# SWE-in-a-team: a coding benchmark for software factories > Our SWE-in-a-team benchmark graded thirteen coding agents and models in an SDLC loop. An Open-Weight builder resolved every ticket at half the price. - URL: https://letsship.ai/blog/swe-in-a-team - Author: Önder Ceylan - Published: 2026-08-31 As an engineer who is used to working in an engineering team, I find the coding benchmarks distant from the engineering teams' tasks, projects and delivery pipeline. They typically grade the destination - functions, patches and environments. Engineering teams ship work through planning, CI/CD pipelines, review, and testing. Those are not the same unit of work. When I decided to build SHIP as a managed cloud based software factory that takes a ticket through a complete SDLC, I knew that I had to see objective results rather than subjective remarks and opinions. I've had my own observations, but numbers speak louder than words. * Can we use a budget or less capable model with smarter redirection and verification gates in a loop to resolve real-world tickets? Can *Haiku* be used for coding in the loop at all? * How much cost can we save just by configuring a builder agent to use a budget model? * How much does a coding agent harness affect cost and cycle time when we pin Open-Weight models? ## SWE-in-a-team benchmark Engineering should not be reduced to generating code - it's also about understanding the business requirements, architecture, decisions, and systems in an environment, communicating with peers, resolving feedback and conflicts, maintaining a code standard and application quality, delivering secure work that is ready for production deployment, and monitoring the application performance post-deployment - among other things. That's why I created a brand new coding benchmark - **SWE-in-a-team**, to grade a coding agent and model's performance while working on 20 tickets of a real-world like full-stack SaaS app in an autonomous SDLC loop - a software factory. Each mission runs the same loop - Plan → Build → CI → Review → Deploy → Test, and goes back to the build stage whenever a gate rejects. One ticket taken through that loop to *Ready for Acceptance* state is called a *mission*. 13 agent teams with variable *Builder* configurations were each assigned 20 tickets, running 260 missions in total during the study: * **claude-code** running `claude-haiku-4-5`, `claude-sonnet-5`, `claude-opus-5` and `claude-fable-5` * **codex** running `gpt-5.6-sol`, `gpt-5.6-terra` and `gpt-5.6-luna` * **pi** running `deepseek/deepseek-v4-flash`, `z-ai/glm-5.2` and `moonshotai/kimi-k3` * **open-code** running `deepseek/deepseek-v4-flash`, `z-ai/glm-5.2` and `moonshotai/kimi-k3` The *Planner*, *Reviewer* and *Tester* agents were assigned the same configuration in every run: `claude-opus-4.8` via `claude-code`, only the *Builder* configuration was dynamic in the team setup. So, the results read as the difference between builders. That's a SWE in a team. ## Summary **Claude Fable** went through 19 of 20 tickets without a single objection from agent peers: green CI, an approving reviewer, a passing tester. **DeepSeek-v4-flash** managed that on 12 tickets, but got correction and direction from peers for the rest. In the end, both completed all of the 20 missions in about the same wall time - medians of 20 minutes, but **Fable cost nearly twice as much**: $122.42 vs $67.11. Hidden tests also showed that one of Fable's completed missions was actually a failure. Below, you can see the insights from the 260 mission runs that caught my attention: * A DeepSeek run that **needed (agent) correction in the loop still cost less** at the median than an Opus run that needed none, $3.44 against $4.58. * **Fable resolved one more ticket than Haiku** on the same harness, 19 of 20 against 18, but paid **52% more per resolved ticket** for it and finished only 9% faster. * Running DeepSeek-v4-flash, GLM-5.2 and Kimi-K3 on **open-code instead of pi cost between 6% less and 26% more** and moved median wall time by as much as 37%, for two more resolved tickets overall. * **The cheapest builder also resolved the fewest**: Luna running on Codex leads on cost at $3.08 resolving 16 / 20, and owns 4 of the 5 runs the loop gave up on. Each of those four couldn't get through the CI gate, which pulled its average cost down. * 4 of the 260 runs **passed every gate and were still not complete**, and the grader's hidden tests caught them. * **Catching late cost money**: the 4 runs the Tester rejected ran to a median of $7.82 against $3.39 for a clean run, while the 27 CI-rejected runs came in at $3.37. * **CI raised the first objection** on 27 of the 46 rejected runs, the Reviewer on 15 and the Tester on 4, so **the cheapest gate did most of the catching**. * **The loop corrected 40 of the 46 runs it objected to** with no human involved, 31 of them on the very next push and all within 4 Builder attempts. * **Correction within the loop cost about a third more** than getting it right first time, for the same builder. ## Methodology SWE-in-a-team uses a private fixture to grade the delivery of backlog tickets by a coding agent working in an SDLC loop. It represents the type of work that engineering teams recognize, as they regularly work on similar tickets. These are the non-negotiables: 1. The coding models must not have seen the application code before. 2. The grade must not come from the models' own opinion of their work. 3. The complete SDLC loop must be measured for key metrics like cycle time and cost. 4. The agent should be working in a team, and be directed and corrected by the peers. 5. The missions must produce evidence for the public to inspect: PRs, reviews and any test artifacts. ### What it measures A ticket is resolved only if it passes two gates: the mission reached *Ready for Acceptance* state *AND* a set of hidden acceptance tests passes. Cost is a computed API-equivalent by multiplying used tokens with the list price per token. Real spend was flat on an Anthropic subscription, so the cost of Anthropic models is a normalization for comparison, not an actual bill. > In SHIP, you use your own provider keys and subscription usage rights as long as AI providers allow it. Read the insights and results as directional, it's not a leaderboard. It is a single trial per cell (k=1), for 20 tickets, in one repository. The CI checks, and pinned `claude-opus-4-8` *Reviewer* and *QA* do course correction during the mission, so the builder model is not solely responsible for the outcome. That is the point of the setup, but it also means these numbers do not isolate raw model's ticket resolution skill, instead evaluates how it performs in a team. ### Building a fixture nobody has trained on Public code has already been read by the models, so a benchmark built on it tests what a model already remembers or was trained on. That is why the fixture in the study is a purposely built booking SaaS called **Studiobook**, kept private, with 20 tickets written in the voice of the people who file them rather than in the voice of a benchmark author. 20 tickets were spread over 10 categories: Bug, Security, Refactor, Dependency Upgrade, Migration, Performance, Feature, Integration, Growth, DevOps. * Two are bugs reported from symptoms rather than stack traces, including a production TypeError on a fully refunded invoice and a member who joined a waitlist twice by double-clicking. * Three are security holes with real blast radius: a cross-tenant leak, a stored XSS in an invoice line, and a vulnerable dependency. * Four are features specified by contract, so the hidden tests grade behavior rather than taste. * Three are integrations covering payment webhooks, error reporting and analytics with third-party services. * The rest is as much familiar: a Zod major upgrade, migrations onto Next.js 16 and Tailwind 4, a move from Supabase to Cloudflare D1, three drifted copies of an invoice total, an N+1 query, an SEO ticket, and a flaky test that broke the suite through an isolation leak. ### Grading with tests the agent never sees A ticket counts as resolved only when it passes two gates: 1. The SDLC pipeline shipped it: the loop reached a PR that passed CI/CD, review and test and was ready for a human. 2. Hidden tests pass on the mission outcome: communicated in the ticket as requirements in the acceptance criteria, kept out of the repository the agent works in, and run against the exact PR the agents produced. ```md Feature (F1) - Export a date-range CSV of bookings for accounting Acceptance Criteria - GET /api/export?type=bookings is supported alongside members and invoices, and requires a signed-in session. - Optional from/to ISO-8601 params. When given, include only bookings whose class session START time falls within [from, to], INCLUSIVE of both ends. An omitted bound is unbounded that side. - CSV columns, in this exact order: Starts, Class, Member, Email, Status. Starts is the session start as its ISO-8601 UTC timestamp. - RFC 4180 escaping: quote fields containing a comma, quote or newline, and double any embedded quotes, exactly as the invoices export does. ``` The second gate runs the hidden tests along with the project's full quality commands: lint, typecheck, test, and build. That is broader than the conventional pass-to-pass definition, which only asks whether previously passing tests still pass. I kept it deliberately, because it also catches a failing test the model wrote itself. And, a PR that fails the repository's own gates is not shippable. ### Measuring the whole loop Most model comparisons hand a model an issue or a task, take one patch, and grade that patch. That measures patch generation, which has driven real progress, but it is not how software reaches production. Real delivery is a loop: somebody plans the change, an assignee codes it, CI gates the integration, a reviewer rejects, the assignee fixes it, the change deploys successfully and is tested - only then is a PR mergeable. A model that writes a rough first draft and takes correction well can beat one that stalls or trips when someone pushes back in such a loop. SWE-in-a-team measures that loop, and grades the PR it produces. Each mission run goes through the software delivery stages with the agent working through CI failures, reviewer and QA feedback the way a person would, so the score covers delivery and correctness together. > The animation above is used across the study to demonstrate a mission of agents working together in a loop on a ticket. > > Click on the image or hit the play button ▶ to see the complete cycle of the actual agent runs, drag the playhead or scrub the image to move backward or forward in time. ## Gates in action The charts, metrics and stats are not enough to demonstrate the complete journey of the agent team. These three are single missions at the durations actually recorded, from plan through to a passing test, so this is where the loop is visible as a mechanism rather than a statistic. Each of the missions was sent back by a different gate on the way: *CI*, *Reviewer*, or the *Tester*. ### See the reviewer send it back On the waitlist bug (B4) the *Reviewer* sent back the most expensive builder in the study, and it corrected course on the next turn. ``` Planner 172s ──▶ Builder 423s ──▶ CI ✓ ──▶ Review 127s ✗ changes requested └─────────────▶ Builder 181s ──▶ CI ✓ ──▶ Review 46s ✓ ──▶ Deploy ✓ ──▶ Test 454s ✓ 30 minutes - $9.18 - 1 review rejection - resolved ``` The second attempt ran in 3m against the first one's 7m, which is a detail I found reassuring. *The feedback narrowed the work instead of restarting it.* ### Watch the Tester catch what Reviewer missed *CI* was green, the *Reviewer* had approved it, and the branch had already deployed to a preview environment. The *Tester* sent it back, and it was the only gate that objected to anything in this mission. ``` Planner 143s ──▶ Builder 281s ──▶ CI ✓ ──▶ Review 38s ✓ ──▶ Deploy ✓ ──▶ Test 387s ✗ failed └─────────────▶ Builder 117s ──▶ CI ✓ ──▶ Review 35s ✓ ──▶ Deploy ✓ ──▶ Test 529s ✓ 36 minutes - $7.23 - 1 test rejection - resolved ``` The Tester cost $5.31 of this run's $7.23, more than the *Planner*, *Builder* and *Reviewer* inference together, because it ran the whole check twice. *The testing stage is the next best candidate to run benchmarks for due to its contribution to the total spend.* ### Follow a hard mission to the pass The SEO ticket (G1) was the hardest mission DeepSeek had, and it still landed. ``` Planner 181s ──▶ Builder 352s ──▶ CI ✗ failed └─────────────▶ Builder 95s ──▶ CI ✗ failed └─────────────▶ Builder 195s ──▶ CI ✓ ──▶ Review 122s ✗ changes requested └─────────────▶ Builder 95s ──▶ CI ✓ ──▶ Review 51s ✓ ──▶ Deploy ✓ ──▶ Test 527s ✓ 38 minutes - $4.26 - 2 CI failures + 1 review rejection - resolved ``` 4 builder attempts and 3 gate rejections produced a resolved ticket for $4.26. For the same ticket: * It's still less than a frontier model's mission cost - claude-opus resolved the same ticket for $6.78. * The same DeepSeek model used with open-code resolved it through a smooth run - no rejections from any of the gates. ## Results After 13 configurations, 20 tickets each, 260 mission runs, and about $1,013 of inference orchestrated by SHIP, let's see what the numbers tell us. ### Compare thirteen builders 7 of the 13 builders resolved all 20 tickets. Ranked by cost per resolved ticket, the cheapest of those was `deepseek/deepseek-v4-flash` running on open-code at $3.16 and the priciest that matched it was `claude-opus-5` at $5.48, which is the same result for 73% more. | Model | Harness | Total spend | Per resolved ticket | vs cheapest | | ---------------------------- | ----------- | ----------- | ------------------- | ----------- | | `deepseek/deepseek-v4-flash` | open-code | $63.17 | $3.16 | baseline | | `deepseek/deepseek-v4-flash` | pi | $67.11 | $3.36 | +6% | | `z-ai/glm-5.2` | open-code | $72.22 | $3.61 | +14% | | `gpt-5.6-sol` | codex | $76.64 | $3.83 | +21% | | `moonshotai/kimi-k3` | open-code | $82.64 | $4.13 | +31% | | `claude-sonnet-5` | claude-code | $86.22 | $4.31 | +36% | | `claude-opus-5` | claude-code | $109.66 | $5.48 | +73% | The most expensive builder in the study `claude-fable-5` running on claude-code has the worst cost per resolved ticket of any configuration at $6.44, and it did not resolve all either: 19 of 20. The cheapest configuration to resolve all 20 did so at $3.16, so Fable cost 104% more for one less ticket resolution. By family, Open-Weight builders resolved 118 of 120, Anthropic 77 of 80, and OpenAI 55 of 60 in total. ### Let the loop do the correcting | Builder | Shipped unopposed | Sent back by a gate | Pushes | Resolved | | ---------------------- | ----------------- | ------------------- | ------ | -------- | | DeepSeek-v4-flash (pi) | 12 of 20 | 13 times | 33 | 20 of 20 | | Claude Opus | 19 of 20 | 1 time | 21 | 20 of 20 | | Claude Fable | 19 of 20 | 1 time | 21 | 19 of 20 | The first column counts tickets where nothing in the loop objected - a single Builder push passes through all the gates. It does not count tickets the model got right, and the gap between those two things is interesting. The mission that Claude Fable lost was lost silently. On the reminders ticket (F5) it wired the live email provider straight into the request path, so the endpoint returns a 500 whenever the API key is missing, even though the ticket only asks it to queue a row. The repository's own test suite stayed green throughout, which is precisely why CI passed, and the Reviewer approved along with the Tester. Fable did not break anything. It built the wrong thing, correctly, and only the hidden tests could tell. DeepSeek-v4-flash via pi is the contrast. It was objected to 13 times and shipped nothing broken. > A model that converges under correction turns out to be worth more inside a loop than one that starts closer to the answer. What makes this practical rather than interesting is the price of those extra pushes. DeepSeek's builder cost about 1% of its own mission bill, because the pinned frontier Planner, Reviewer and Tester dominate the bill, so buying more attempts from a cheap builder costs almost nothing while buying a better first draft costs much more. ### Swap the harness, keep the model Three Open-Weight models each ran on two harnesses - open-code vs pi, which makes this the one place in the study where exactly one variable moves. | Model | Harness | Total spend | Median wall | Resolved | | ---------------------------- | --------- | ----------- | ----------- | -------- | | `deepseek/deepseek-v4-flash` | pi | $67.11 | 20 min | 20 of 20 | | `deepseek/deepseek-v4-flash` | open-code | $63.17 | 22 min | 20 of 20 | | `z-ai/glm-5.2` | pi | $68.07 | 24 min | 19 of 20 | | `z-ai/glm-5.2` | open-code | $72.22 | 21 min | 20 of 20 | | `moonshotai/kimi-k3` | pi | $65.63 | 18 min | 19 of 20 | | `moonshotai/kimi-k3` | open-code | $82.64 | 25 min | 20 of 20 | open-code cost 6% less on DeepSeek and more on the other two, by 6% on GLM and 26% on Kimi, and ran 37% longer on Kimi. It also resolved two more tickets across the 60 runs, so this is a trade-off rather than a ranking. > A result that names the model without naming the harness is describing half of what ran. ### Count what the gates caught Across all 260 runs, 214 landed on the first pass and 46 needed the loop. The gates rejected work 70 times in total: CI 45 times, the Reviewer 21, the Tester 4. Only four rejections reached the Tester, because CI and the Reviewer had caught nearly everything by then. A loop that fails fast early is what makes a cheap builder viable. Inspect any mission below to see how the loop progressed, and where its time and money went. ### Why missions did not resolve A solution is marked as resolved through two gates. 10 of the 260 runs missed one of them, and the two failure modes have almost nothing in common: 1. The mission reaching `Ready for Acceptance` state. 2. The hidden grader tests passing on the PR it opened. Terra and Luna on Codex never passed the first gate. They could not process the CI feedback into a passing build, repeated the same failure across 3 to 5 retries, and hit SHIP's oscillation limit. Fable and Haiku on Claude Code, and GLM and Kimi on Pi, passed the first gate and failed at the second. Below are what the hidden tests caught: * **An edge case named in the acceptance criteria:** An invoice where every line is refunded. An empty line-item list. A date range whose bounds are inclusive at both ends. * **No idempotency**: A second reminder run must queue nothing new. The same webhook delivered twice must pay the invoice once. A repeat booking must be refused without spending another credit. * **Bad input treated as success:** An invalid signature returns 400 and changes nothing. A missing session returns 401 and queues no reminders. Another studio's record is refused, not returned. * **A migration that looked done but had issues:** No Supabase import left anywhere after the move to D1. Next resolved to 16.x in the lockfile, not 15.x. * **Behaviour a unit test never sees:** A bounded number of reads however many bookings there are. A name containing a comma still quoted, so the CSV row keeps its columns. A script payload escaped rather than rendered as markup. None of these failure modes are assertions in repo's own test suite. > Failures were counted equally, not grouped or weighted by severity. ## Takeaways The reflex is to use the best model you can afford, and this study suggests that reflex is wrong. You should choose the agent harness and model your loop can correct. Buying the most expensive model did not buy reliability for the delivery. Fable finished a ticket ahead of Haiku, 19 of 20 against 18, while costing 52% more per resolved ticket. Buying the cheapest one did not save money in all stations either, because although *Build* is often the longest stage, it is just one stage of what a ticket costs once a real loop surrounds it. What the money buys is correction. A budget model that starts worse and takes feedback well resolved every ticket in this study, and together with the 13 rejections it absorbed cost less than a single frontier draft would have. So the question I would ask before picking a model is not which one writes the best code on the first try. It is what plans it, what checks the code afterwards, and whether correction is cheap enough that a rough first draft stops being a problem worth paying to avoid.