← BlogSWE-in-a-team: a coding benchmark for software factories
Our SWE-in-a-team benchmark graded thirteen coding agents and models in an SDLC loop. An Open-Weight builder resolved every ticket at half the price.
As an engineer who is used to working in an engineering team, I find the coding benchmarks distant from the engineering teams' tasks, projects and delivery pipeline. They typically grade the destination - functions, patches and environments. Engineering teams ship work through planning, CI/CD pipelines, review, and testing. Those are not the same unit of work.
When I decided to build SHIP as a managed cloud based software factory that takes a ticket through a complete SDLC, I knew that I had to see objective results rather than subjective remarks and opinions. I've had my own observations, but numbers speak louder than words.
- Can we use a budget or less capable model with smarter redirection and verification gates in a loop to resolve real-world tickets? Can Haiku be used for coding in the loop even?
- How much cost can we save just by configuring a builder agent to use a budget model?
- How much a coding agent harness affects cost and cycle time when we pin Open-Weight models?
SWE-in-a-team benchmark
Engineering should not be reduced to generating code - it's also about understanding the business requirements, architecture, decisions, and systems in an environment, communicating with peers, resolving feedback and conflicts, maintaining a code standard and application quality, delivering a secure work that is ready for production deployment, and monitoring the application performance post-deployment - among other things.
That's why I created a brand new coding benchmark - SWE-in-a-team, to grade a coding agent and model's performance while working on 20 tickets of a real-world like full-stack SaaS app in an autonomous SDLC loop - a software factory.
Each mission runs the same loop - Plan → Build → CI → Review → Deploy → Test, and goes back to the build stage whenever a gate rejects. One ticket taken through that loop to Ready for Acceptance state is called a mission.
13 agent teams with variable Builder configurations were assigned the 20 tickets each, running 260 missions in total during the study:
- claude-code running
claude-haiku-4-5,claude-sonnet-5,claude-opus-5andclaude-fable-5 - codex running
gpt-5.6-sol,gpt-5.6-terraandgpt-5.6-luna - pi running
deepseek/deepseek-v4-flash,z-ai/glm-5.2andmoonshotai/kimi-k3 - open-code running
deepseek/deepseek-v4-flash,z-ai/glm-5.2andmoonshotai/kimi-k3
The Planner, Reviewer and Tester agents were assigned the same configuration in every run: claude-opus-4.8 via claude-code, only the Builder configuration was dynamic in the team setup. So, the results read as the difference between builders. That's a SWE in a team.
Summary
Claude Fable went through 19 of 20 tickets without a single objection from agent peers: green CI, an approving reviewer, a passing tester. DeepSeek-v4-flash managed that on 12 tickets, but got correction and direction from peers for the rest. In the end, both completed all of the 20 missions in about the same wall time - medians of 20 minutes, but Fable cost nearly twice as much: $122.42 vs $66.86. Hidden tests also showed that one of Fable's completed missions was actually a failure.
Below, you can see the insights from the 260 mission runs that caught my attention:
- A DeepSeek run that needed (agent) correction in the loop still cost less at the median than an Opus run that needed none, $3.42 against $4.58.
- Fable resolved one more ticket than Haiku on the same harness, 19 of 20 against 18, but paid 52% more per resolved ticket for it and finished only 9% faster.
- Running DeepSeek-v4-flash, GLM-5.2 and Kimi-K3 on open-code instead of pi cost 22% to 41% more and moved median wall time by as much as 37%, for two more resolved tickets overall.
- The cheapest builder also resolved the fewest: Luna running on Codex leads on cost at $3.08 resolving 16 / 20, and owns 4 of the 5 runs the loop gave up on. Each of those four couldn't get through the CI gate, which pulled its average cost down.
- 4 of the 260 runs passed every gate and were still not complete, and the grader's hidden tests caught them.
- Catching late cost money: the 4 runs the Tester rejected ran to a median of $8.18 against $3.51 for a clean run, while the 27 CI rejected came in at $3.37.
- CI raised the first objection on 27 of the 46 rejected runs, the Reviewer on 15 and the Tester on 4, so the cheapest gate did most of the catching.
- The loop corrected 40 of the 46 runs it objected to with no human involved, 31 of them on the very next push and all within 4 Builder attempts.
- Correction within the loop cost about a third more than getting it right first time, for the same builder.
Cost per resolved task — frontier vs. open
Methodology
SWE-in-a-team uses a fixture that was kept private, grades the delivery of backlog tickets shipped by a coding agent working in SDLC loop. It represents the type of work that engineering teams recognize, as they regularly work on similar tickets.
These are the non-negotiables:
- The coding models must not have seen the application code before.
- The grade must not come from the models' own opinion of their work.
- The complete SDLC loop must be measured for key metrics like cycle time and cost.
- The agent should be working in a team, and be directed and corrected by the peers.
- The missions must produce evidence for the public to inspect: PRs, reviews and any test artifacts.
What it measures
A tickets is resolved only if it passes two gates: the mission reached Ready for Acceptance state AND a set of hidden acceptance tests passes. Cost is a computed API-equivalent by multiplying used tokens with the list price per token. Real spend was flat on an Anthropic subscription, so the cost of Anthropic models are a normalization for comparison, not an actual bill.
In SHIP, you use your own provider keys and subscription usage rights as long as AI providers allow it.
Read the insights and results as directional, it's not a leaderboard. It is a single trial per cell (k=1), for 20 tickets, in one repository. The CI checks, and pinned claude-opus-4-8 Reviewer and QA do course correction during the mission, so the builder model is not solely responsible for the outcome. That is the point of the setup, but it also means these numbers do not isolate raw model's ticket resolution skill, instead evaluates how it performs in a team.
Building a fixture nobody has trained on
Public code has already been read by the models, so a benchmark built on it tests what a model already remembers and trained for. That is why the fixture in the study is a purposely built booking SaaS called Studiobook, kept private, with 20 tickets written in the voice of the people who file them rather than in the voice of a benchmark author.
20 tickets were spread over 10 categories: Bug, Security, Refactor, Dependency Upgrade, Migration, Performance, Feature, Integration, Growth, DevOps.
- Two are bugs reported from symptoms rather than stack traces, including a production TypeError on a fully refunded invoice and a member who joined a waitlist twice by double-clicking.
- Three are security holes with real blast radius: a cross-tenant leak, a stored XSS in an invoice line, and a vulnerable dependency.
- Four are features specified by contract, so the hidden tests grade behavior rather than taste.
- Three are integrations covering payment webhooks, error reporting and analytics with third party services.
- The rest is as much familiar: a Zod major upgrade, migrations onto Next.js 16 and Tailwind 4, a move from Supabase to Cloudflare D1, three drifted copies of an invoice total, an N+1 query, an SEO ticket, and a flaky test that broke the suite through an isolation leak.
Bug (B2) - Production TypeError opening a fully-refunded invoiceSTB-1088
On-call here. Sentry started catching this in production yesterday:
TypeError: Reduce of empty array with no initial value
It fires every time someone opens the invoice detail page for the refunded "Pottery intensive" invoice, the one for Femke. The page dies to the error screen instead of rendering. Every other invoice I tried opens fine.
The thing that's different about that invoice is that every line item on it has been refunded, so there's nothing left to bill. Anything that leaves an invoice with zero billable line items seems to trip the same error, and finance wants to lean on line-level refunds more, so this is only going to happen more often.
Acceptance criteria
- Opening the detail page of an invoice whose line items are all refunded renders normally: subtotal €0.00, tax €0.00, total €0.00, with the refunded lines still listed (and badged as refunded).
- Invoice totals are computed safely for an invoice with zero billable line items (none at all, or all refunded), yielding zero subtotal/tax/total and the correct refunded sum, instead of throwing.
- Invoices with a mix of billable and refunded lines keep today's behaviour (refunded lines tracked but excluded from subtotal and tax).
Bug (B4) - Member got onto a class waitlist twice by clicking 'join' twiceSTB-1089
Front desk flagged this after yesterday's Reformer class filled up.
A member tried to book the full class from the bookings page, the button felt unresponsive, so she clicked it again. Now she shows up twice on that class's waitlist. When a spot opened up this morning, one of her entries got promoted to a confirmed seat, and the second entry is still sitting on the waitlist. It's holding a spot the next person should have, and it's throwing off our occupancy numbers.
I reproduced it in a minute: pick a full class, book the same member, submit again, and the second submit goes through and adds another waitlist row. If the member already holds a confirmed seat, the second submit is correctly rejected ("This member already has a booking for this class"), so it's really only the waitlisted case that slips through.
Acceptance criteria
- Booking a member onto a full class puts them on the waitlist once; any repeat booking attempt for the same member + same class while they are waitlisted is rejected with the same "already has a booking" conflict as a repeat attempt on a confirmed seat.
- After a double submit, the member has exactly one active booking row (waitlisted) for that class.
- Cancelled bookings keep today's behaviour: a member whose booking was cancelled can book the same class again.
Security (S1) - Security advisory: vulnerable semver dependency (CVE-2022-25883)STB-1090
Forwarded from our dependency scanner.
High · Regular Expression Denial of Service (ReDoS) in
semver
- Advisory: GHSA-c2qf-rxjj-qqgw (CVE-2022-25883)
- Package:
semver(npm)- Affected versions:
>=7.0.0 <7.5.2- Patched in:
7.5.2- Dependency path:
apps/web > semverThe
semverrange parser can be driven into super-linear backtracking by a crafted version/range string, allowing a denial of service.apps/webdepends on a version in the affected range.
We pull semver in to gate the minimum supported client version. Please update
the dependency to a patched release so the scanner stops flagging it.
Acceptance criteria
apps/webno longer depends on asemverin the vulnerable range; the version the package manager resolves is>= 7.5.2.- A high-severity dependency audit reports no findings.
- Lint, type-check, tests, and the production build all still pass.
Security (S2) - Any logged-in studio can read another studio's dataSTB-1091
Hi. I run security for one of your pilot studios, and while poking at the app with my own signed-in session I noticed the detail endpoints don't seem to check who owns a record.
Here's what I mean. My session is scoped to my studio, but if I take the id of an
invoice, member, or booking that belongs to a different studio and request it
directly (say GET /api/invoices/<their-id>, GET /api/members/<their-id>, or
DELETE /api/bookings/<their-id>), the app hands it over. In the booking case it
will even cancel their record for me. The id is the only thing guarding it, and
ids are easy to guess. That's an IDOR: one tenant can read and change another
tenant's data.
If a record isn't mine, I'd expect it to look like it simply doesn't exist for me.
Acceptance criteria
- Fetching an invoice or member by id, or cancelling a booking by id, is scoped to the caller's own studio: an id that belongs to a different studio is rejected as not found (the record is neither returned nor modified).
- The same operations continue to work for records that belong to the caller's own studio.
- Lint, type-check, tests, and the production build all still pass.
Security (S3) - Stored XSS: invoice line description renders unescaped HTMLSTB-1092
Hi, reporting a stored cross-site-scripting issue in the invoices area.
Invoice line-item descriptions are free text entered by staff, but the invoice detail page renders that text as raw HTML instead of escaping it. If a description contains markup, it gets injected into the page and runs in the browser of anyone who opens that invoice.
Repro:
- Create an invoice with a line item whose description is
<img src=x onerror="alert(document.cookie)">. - Open the invoice detail page.
- The image tag is parsed and the
onerrorhandler fires, so arbitrary script runs in the viewer's session. One malicious value stored on a single account then executes against every user who later views that invoice.
The description should show as literal text, never interpreted as markup.
Acceptance criteria
- On the invoice detail page, a line-item description containing HTML (e.g.
<img src=x onerror=...>) is rendered as inert, escaped text: the markup appears verbatim and no element or handler is created from it. - Ordinary descriptions still render as readable text.
- Lint, type-check, tests, and the production build all still pass.
Dependency (D1) - Upgrade Zod v3 → v4STB-1093
Automated dependency PR from the dependency bot.
Bumps zod in apps/web from ^3.24 to the latest** v4** release.
Zod 4 is a major version. Most of our schema code is source-compatible, but a
couple of v3-era idioms changed, and the request-error handling that builds the
API validation-error envelope depends on one of them. Move apps/web onto Zod 4,
confirm the request validators still behave exactly as they do today, and keep
the app compiling.
Release notes (upstream)
- Top-level string-format helpers (
z.email(),z.url(), …) are now preferred over the chainedz.string().email()forms. - The
ZodErrorsurface changed: the flaterror.errorsarray accessor was removed in favour oferror.issues;error.flatten()/error.format()are retained (now deprecated). - Default and transform semantics for object schemas are unchanged for the shapes we use.
Acceptance criteria
apps/webdepends on Zod v4: the version the package manager resolves is4.x, not3.x.- The API request validators keep their behaviour: a malformed email is
rejected; a valid email is trimmed and lower-cased; a new member's status
defaults to
active; and a colour that is not a#rrggbbhex value is rejected. - Lint, type-check, tests, and the production build all pass.
Migration (D2) - Migrate the web app to Next.js 16STB-1094
From engineering:
We are a major version behind on Next.js. apps/web is still on Next 15, and
Next 16 is out. Let's upgrade to Next.js 16. Behavior should be unchanged; this
is a framework upgrade, not a feature change.
The main thing Next 16 changes for us is that the dynamic request APIs are now
asynchronous: params and searchParams in pages and route handlers, and
cookies() / headers(), must be awaited rather than read synchronously.
Docs: the Next.js 16 upgrade guide at https://nextjs.org/docs/app/guides/upgrading/version-16.
Acceptance criteria
- The web app runs on Next.js 16:
nextresolves to a16.xversion in the lockfile (not just the declared range). - The dynamic request APIs are accessed the Next 16 way:
params/searchParams(in pages and[id]route handlers) andcookies()/headers()are awaited, not read synchronously. - The app's behavior is unchanged: the full test suite and the production build stay green.
Migration (D4) - Migrate the web app to Tailwind CSS v4STB-1095
From engineering:
We are behind on Tailwind. The web app is still on Tailwind v3 with the old
JavaScript config, and v4 has been out for a while with a faster engine and a
CSS-first setup. Let's upgrade apps/web to Tailwind CSS v4. The app should look
and behave exactly the same afterwards. This is a toolchain modernization, not a
redesign.
Docs: the Tailwind v4 upgrade guide at https://tailwindcss.com/docs/upgrade-guide.
Acceptance criteria
- The web app builds and runs on Tailwind CSS v4:
tailwindcssresolves to v4 in the lockfile;- the PostCSS plugin is
@tailwindcss/postcss(not the oldtailwindcssPostCSS plugin); - the global stylesheet pulls Tailwind in with
@import "tailwindcss", replacing the old@tailwind base; @tailwind components; @tailwind utilities;directives; - the old
tailwind.config.jsis removed (configuration moves into CSS).
- The app's appearance and behavior are unchanged: the full test suite and the production build stay green.
Refactor (R1) - Migrate persistence from Supabase to Cloudflare D1 + DrizzleSTB-1096
Architecture note from the platform team.
We're consolidating Studiobook onto a single Cloudflare stack. The app already
builds and deploys as a Worker (OpenNext), but persistence still runs through
@supabase/supabase-js against a hosted Postgres. That's the last thing keeping
us tied to a second vendor and its connection model. We want the production data
layer on Cloudflare D1 (the app's own SQLite database binding) using** Drizzle ORM**, so everything lives in one account and one deploy.
This is a swap of the adapter, not a change in behaviour. Studiobook already
funnels every read and write through the Repositories interfaces
(lib/db/repos/types.ts). The Supabase client is confined to a single
implementation file behind that seam, and services, routes, and domain code only
ever touch the interfaces. So keep the migration contained to the persistence
adapter: implement the same repository surface with Drizzle over the D1 binding,
then delete the Supabase pieces. Nothing above the repository seam should need to
change, and the in-memory fake repositories (used by local dev and the test
suite) stay exactly as they are.
Keep the change tight and behaviour-preserving. The demo dataset, the booking, invoice, and notification flows, and every existing check should behave identically afterwards.
Acceptance criteria
- The web app no longer depends on
@supabase/supabase-jsor@supabase/ssr; both are removed from the app's dependencies (and the workspace lockfile). - The production repository implementation is written with Drizzle ORM over
the Cloudflare D1 binding, behind the existing
Repositoriesseam. A Drizzle dependency is present in the app. ThecreateD1Repositories(db)factory inlib/db/repos/d1.tstakes the D1 database binding as its argument (so the adapter can be constructed against any D1 instance), andresolveRepositoriespasses the Worker'sDBbinding to it. - Ship the D1 schema as migration SQL under
apps/web/migrations/(the standard wrangler D1 migrations directory) and declare theDBd1_databasesbinding inwrangler.jsonc, so the database can be created from scratch. - No
@supabase/*import remains anywhere under the app'slib. - Behaviour is preserved: the app builds, and the existing unit + integration suite (services and repositories exercised against the in-memory fakes) stays green. Route handlers, services, and domain modules keep depending only on the repository interfaces, untouched by the swap.
Refactor (R3) - Consolidate three drifted copies of invoice total calculationSTB-1097
Eng-lead here, writing this up after finance flagged a discrepancy.
We now compute an invoice's total in three different places and they've drifted:
computeInvoiceTotalsinlib/domain/invoices.ts. This is the original, and the one the invoice page and the stored invoice both rely on. It correctly drops refunded line items out of the taxable subtotal.- The invoice create path (
lib/services/invoices.ts) has grown its own inlined copy of the subtotal/tax math instead of calling the domain function. - The new account statement feature (
lib/services/account-statements.ts) rolled its own copy too, and this one is wrong: it taxes every line, including refunded ones. So a member whose invoice had a line refunded sees a bigger total on their statement than on the invoice itself, and bigger than what we actually charged.
Concretely: an invoice with a €100 billable line and a €50 refunded line, at 9% tax, is €109.00 everywhere it's computed correctly, because tax applies to the €100 only. The statement, though, reports €163.50 because it taxes the full €150.
We need ONE invoice-total implementation. Route every surface through the single domain function so a refunded line is handled identically everywhere, and the statement, the invoice record, and the domain calculation all agree.
Acceptance criteria
- There is a single source of truth for invoice totals; the account-statement and invoice-create paths compute totals through it rather than re-deriving the subtotal/tax math inline.
- For an invoice with a refunded line item, the account statement's total
excludes the refunded line from the taxable subtotal. It equals the
canonical
computeInvoiceTotalsresult and the stored invoice total (for the example above, €109.00, i.e. 10900 cents), never the over-taxed figure. - No behaviour changes for invoices without refunded lines, and invoice creation still stores the same subtotal/tax/total it does today.
Performance (P1) - Bookings list N+1: repository reads grow with the number of bookingsSTB-1098
From engineering, flagged by one of our larger studios.
Since we refactored the bookings list last week, the dashboard has gotten noticeably slow for studios with a full schedule. A few hundred bookings and the page crawls. Our query logs show the list now looking up each booking's member and class session one row at a time, so the number of database reads climbs with the number of bookings. It used to fetch them in a batch.
Please get the list back to a bounded number of reads no matter how many bookings it returns, without changing anything the endpoint returns.
Acceptance criteria
GET /api/bookingsreturns exactly what it returns today: the same rows, the same fields, in the same order.- Listing the bookings issues a number of member and class-session repository reads that does NOT grow with the number of bookings. Fetching a list of N bookings must not read the members repository or the class-sessions repository once per booking; a small, fixed number of reads total is expected, regardless of N.
- The existing test suite stays green.
Feature (F1) - Export a date-range CSV of bookings for accountingSTB-1099
From the founder.
Every quarter I sit down with our bookkeeper and she asks for "the list of what
actually happened": which classes ran, who was in them, over a date range she
picks. Right now the only exports we have are the members list and the invoices
list (/api/export?type=members / ?type=invoices). There's nothing for
bookings, so I end up screenshotting the bookings page, which is a bit
embarrassing.
Can we add a bookings export to the same download endpoint? I want to ask for "everything between 1 June and 30 June" and get a clean CSV I can open in Excel or hand straight over. A couple of our members have commas in how their name is stored ("Rossi, Chiara"), and the current members export handles that fine, so please keep the same quoting.
Acceptance criteria
GET /api/export?type=bookingsis supported, alongside the existingmembersandinvoicesexports, and (like them) requires a signed-in session.- It accepts optional
fromandtoquery parameters (ISO-8601 timestamps). When given, the export includes only bookings whose class session start time falls within[from, to], inclusive of both ends; an omitted bound is unbounded on that side. - The CSV has these columns, in this order:
Starts, Class, Member, Email, Status.Startsis the session start as its ISO-8601 UTC timestamp,Classis the class name,Memberis the member's name,Emailis their email, andStatusis the booking status. - Fields containing a comma, double quote, or newline are quoted with embedded
quotes doubled (RFC 4180, the same escaping the invoices export already
uses), so a member named
Rossi, Chiarastays a single column.
Feature (F6) - Sell class packs (prepaid class credits)STB-1100
From the founder, working on next season's pricing:
Every booking today is a one-off. Our most loyal members would happily pay up front for a bundle instead of paying class by class, the way every studio sells a "10-class pass." I want to sell class packs: a member buys a pack of credits, and each class they book spends one credit until the pack runs out. Then they buy another.
Let's start with two sizes, a 5-credit pack and a 10-credit pack. Once a member has a pack, their bookings should draw from it automatically, so the front desk never has to think about it. When the pack is empty, the booking should stop and tell them to buy another. And if we ever have to refund a pack, its remaining credits should be voided so they can't still be spent.
Acceptance criteria
- Buy a pack.
POST /api/packageswith body{ memberId, credits }, wherecreditsis either5or10. It creates an active pack for that member withcreditsRemainingequal tocredits, and a totalpriceCentsofcredits × 1000— i.e. 1000 cents per credit, so a5-credit pack is5000and a10-credit pack is10000(thepriceCentsfield is the pack's TOTAL price, not the per-credit rate). Responds201with{ id, memberId, creditsTotal, creditsRemaining, priceCents, status, purchasedAt }. - List a member's packs.
GET /api/packages?memberId=<id>responds200with that member's packs (each{ id, creditsTotal, creditsRemaining, priceCents, status, purchasedAt }), newest first. - Bookings draw from the pack automatically. When a member who has an active
pack with
creditsRemaining > 0books a class viaPOST /api/bookings, the booking is confirmed exactly as today AND one credit is spent (the pack'screditsRemainingdrops by one, oldest pack first). A member with no pack books exactly as they do today, unchanged. - A used-up pack stops the booking. Once a member has bought a pack, their
bookings must come from a pack. If they have no credits left to draw (every pack
they own is exhausted or refunded),
POST /api/bookingsis rejected with402and the error codepack_exhausted; they must buy another. A member who never bought a pack is unaffected and books exactly as today. - No double-spend. The existing rule that a member cannot book the same
session twice still holds: a repeated booking is rejected
409and spends no extra credit. - Refund voids the remaining credits.
POST /api/packages/{id}/refundsets the pack'screditsRemainingto0and itsstatustorefunded; a refunded pack is never drawn from again.
Feature (F3) - Per-member calendar subscription behind a private tokenSTB-1101
Let members subscribe to their own class calendar
From the PM:
We already publish a studio-wide calendar feed (/api/ical), but members keep
asking for "just my classes on my phone calendar." A single shared feed doesn't
work for that — it lists everyone's sessions, and it's public.
I'd like each member to have their own private subscription link: a secret per-member token in the URL that returns only the sessions that member is booked into, so they can paste it into Apple/Google Calendar and see their upcoming classes auto-update. Because a calendar app can't log in with our cookie, the secret link itself has to be what authorizes it — so an unknown or made-up token must just 404, never leak someone else's schedule.
Acceptance criteria
- Every member has a private calendar token.
GET /api/ical/[token]returns an iCalendar feed (content-type: text/calendar) containing calendar events for ONLY that token-holder's upcoming booked sessions (future sessions they hold a confirmed seat in). It must not include other members' sessions or the member's past sessions.- An unknown or empty token returns 404.
- No session cookie is required — the secret token is the authorization.
Feature (F5) - Send 24-hour class reminders (idempotent job endpoint)STB-1102
Automatic 24-hour class reminders
From the PM:
Our no-show rate spikes for early classes, and members keep telling us "I just forgot." Every other studio app sends a reminder the day before — we should too. I'd like a reminder to go out to everyone who's booked into a class that starts in the next 24 hours.
I want to drive this from our scheduler (a cron that hits an endpoint), so the important thing is that it's safe to call repeatedly: if the cron runs hourly, a member must still only get one reminder for a given class, not one every hour. And obviously anyone who's turned notifications off shouldn't get one, and people only on the waitlist (no confirmed seat) shouldn't either.
Acceptance criteria
POST /api/reminders/run(requires a signed-in session, like our other write endpoints) finds every class session starting within the next 24 hours and queues abooking_remindernotification — a pending row in the notification outbox — for each member holding a confirmed (booked) seat in one of those sessions.- Members who have opted out of notifications get no reminder. Waitlisted members, and members whose session is outside the 24-hour window, get no reminder.
- The endpoint is idempotent: running it a second time does not queue a duplicate reminder for a booking that already has one queued.
- Returns HTTP 200 on success.
Integration (I1) - Wire up product analytics (PostHog) for the booking funnelSTB-1103
From the founder:
I have no real picture of our funnel. How many bookings get made, how often members join a waitlist, how often they cancel. Right now I'm guessing. Let's wire up PostHog so I can see these moments in one place.
Instrument the events that matter, and keep it clean. Route analytics through a
single helper the way we already route email through the notification adapter,
so it's easy to test and swap. Don't scatter posthog calls through the domain
code.
Docs: PostHog for Next.js (https://posthog.com/docs/libraries/next-js), also
mirrored offline at docs/vendor/posthog-nextjs.md.
Acceptance criteria
- Analytics goes through a single injectable tracker, an interface with a
capturemethod, wired into the booking, waitlist, and cancellation flows, with the real PostHog client constructed only at the composition root (mirroring the notification adapter). Service and domain code depend on the interface, never on aposthogpackage directly. - Expose a
__setTestTracker(tracker)test seam from the tracker composition root — mirroring__setTestRepositorieson the repository seam — so a recording tracker can be injected and the route handlers driven in tests. (How the flows receive the tracker is up to you; the seam is what's graded.) - These events are captured, exactly once, on their flow, each attributed to the
member (the member's id as the analytics distinct id) and carrying the session's
id as a
session_idproperty:booking_createdwhen a booking is confirmed.waitlist_joinedwhen a booking is placed on the waitlist.booking_cancelledwhen a booking is cancelled.
- A waitlisted booking fires
waitlist_joinedand NOTbooking_created; a confirmed booking firesbooking_createdand NOTwaitlist_joined. - No personally-identifying data in the captured events. No email, name, or phone number anywhere in an event's properties.
- The existing test suite stays green.
Integration (I3) - Receive Stripe payment webhooks and mark invoices paidSTB-1104
From the founder:
We're starting to collect payments through Stripe. Right now when a member pays an invoice, someone has to mark it paid by hand, and we forget. Stripe can tell us the moment a payment succeeds via a webhook, so let's receive that and mark the matching invoice paid automatically.
Two things I care about: it must reject anything that isn't genuinely from Stripe (verify the signature), and if Stripe sends the same event twice (it does), we must not double-process.
Docs: Stripe webhooks (https://docs.stripe.com/webhooks) and signature verification (https://docs.stripe.com/webhooks#verify-events).
Acceptance criteria
POST /api/webhooks/stripereceives a Stripe webhook. It verifies the request'sStripe-Signatureheader against our webhook signing secret (STRIPE_WEBHOOK_SECRET); a request with a missing or invalid signature is rejected with400and nothing is changed.- On a verified
invoice.paidevent, the invoice named in the event (its id atevent.data.object.metadata.invoice_id) is marked paid: itsstatusbecomespaidandpaidAtis set. Respond200. - Idempotent: receiving the same event again (same event
id) leaves the invoice paid exactly once and still responds200, with no double-processing and no error. - A verified event that names an unknown invoice responds
200(acknowledged) and changes nothing. A verified event of any other type is acknowledged (200) and changes nothing. - The existing test suite stays green.
Integration (I4) - Report unexpected server errors to SentrySTB-1105
From the founder, after a bad month:
Twice last month a member hit a 500 and we only found out because they emailed us. We need real error monitoring. Wire up Sentry so that whenever one of our API routes hits an unexpected server error, it gets reported to Sentry with the error instead of vanishing into the logs.
The important nuance: our normal, handled responses aren't errors. A 404 for a
missing member, a 400 for bad input, a 409 for a full class, a 402 for an
empty pack: those are expected and must NOT be reported, or Sentry becomes
noise. Only genuine unexpected failures should be captured.
Docs: Sentry for Next.js (https://docs.sentry.io/platforms/javascript/guides/nextjs/).
Acceptance criteria
- When an API route handler throws an UNEXPECTED error (anything that is not one
of our normal handled outcomes: not a validation error, not a deliberate
HttpError), the error is reported to Sentry via the Sentry SDK'scaptureException, and the response is the existing500. - Expected, handled outcomes are NOT reported to Sentry: a
400validation error and anyHttpError(404/409/402/ etc.) are normal and must not be captured. - A successful request reports nothing.
- The existing test suite stays green.
Growth (G1) - Make the existing public studio page rank in search (SEO improvement)STB-1106
From our growth lead:
Every studio already has a public page at /s/<slug> that lists its upcoming
classes, and you can open one without logging in. The problem is that it's
invisible in search. I checked one with Google's Rich Results Test and Lighthouse
and it's a mess. The page is marked noindex so Google won't even index it, the
title is just "Studio", there's no meta description, no social preview when you
paste the link in Slack or WhatsApp, and none of the structured data Google needs
to show class events. We're leaving sign-ups on the table.
Please make the existing public studio page properly SEO-ready so it can rank and
show rich results. Don't build a new page. Improve the one that's already there
(apps/web/app/s/[slug]/page.tsx). Same content, just search-ready.
Docs: Next.js Metadata: https://nextjs.org/docs/app/getting-started/metadata-and-og-images ; sitemap: https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap ; robots: https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots ; schema.org Event: https://schema.org/Event.
Acceptance criteria
- The public page at
/s/[slug]must be indexable: remove thenoindexdirective so search engines can crawl and index it. It must still be public (no login) and still render the studio and its upcoming classes (each class's name, start time, and instructor). A slug that matches no studio still returns404. - Give it real, studio-specific SEO metadata via Next's Metadata API
(
generateMetadata): a<title>and<meta name="description">that name the studio (not a hardcoded "Studio"), Open Graph tags (og:title,og:description,og:type), Twitter card tags, and a canonical URL. - Embed JSON-LD structured data in a
<script type="application/ld+json">using schema.orgEvent, one event per upcoming class, each with at leastname,startDate, andlocation. - Fix the on-page SEO basics: the studio image needs descriptive
alttext, and the call-to-action link needs descriptive text (not "Click here"). - Add
GET /sitemap.xmllisting the public studio page(s) andGET /robots.txtthat allows crawling and references the sitemap (use Next'ssitemap.ts/robots.tsfile conventions). - The page must NOT require authentication, and the existing test suite stays green (the admin app and its auth are unaffected).
The measurable bar: the page's Lighthouse SEO score should go from its current low baseline (~45) up to ~100.
DevEx (O1) - Flaky CI: a test-isolation leak breaks the suiteSTB-1107
From engineering.
CI has gone red and it's blocking everyone. The app itself is fine. The problem is the tests. One test leaves shared state behind (it sets something up and never cleans it up), and that leaks into another test, which then fails when the whole suite runs together. Run that other test on its own and it passes; run the full suite and it fails.
Please fix the test isolation so pnpm test is green and stays deterministic.
Don't just delete or skip the failing tests. Fix the leak so the offending test
cleans up after itself.
Acceptance criteria
pnpm test(the full suite) passes deterministically.- The fix is proper test isolation: the leaking test cleans up the shared state it sets (so it no longer affects any other test), rather than the other test being changed to tolerate the leak.
- No test is deleted, skipped (
.skip/.only/ commented out), or gutted to make the suite pass; every test that existed still exists and still asserts what it did.
Grading with tests the agent never sees
A ticket counts as resolved only when it passes two gates:
- The SDLC pipeline shipped it: the loop reached a PR that passed CI/CD, review and test and was ready for a human.
- Hidden tests pass on the mission outcome: communicated in the ticket as requirements in the acceptance criteria, kept out of the repository the agent works in, and run against the exact PR the agents produced.
Feature (F1) - Export a date-range CSV of bookings for accounting
Acceptance Criteria
- GET /api/export?type=bookings is supported alongside members and
invoices, and requires a signed-in session.
- Optional from/to ISO-8601 params. When given, include only bookings
whose class session START time falls within [from, to], INCLUSIVE of
both ends. An omitted bound is unbounded that side.
- CSV columns, in this exact order: Starts, Class, Member, Email,
Status. Starts is the session start as its ISO-8601 UTC timestamp.
- RFC 4180 escaping: quote fields containing a comma, quote or newline,
and double any embedded quotes, exactly as the invoices export does.The second gate runs the hidden tests along with the project's full quality commands: lint, typecheck, test, and build. That is broader than the conventional pass-to-pass definition, which only asks whether previously passing tests still pass. I kept it deliberately, because it also catches a failing test the model wrote itself. And, a PR that fails the repository's own gates is not shippable.
Measuring the whole loop
Most model comparisons hand a model an issue or a task, take one patch, and grade that patch. That measures patch generation, which has driven real progress, but it is not how software reaches production. Real delivery is a loop: somebody plans the change, an assignee codes it, CI gates the integration, a reviewer rejects, the assignee fixes it, deployment is successful and tested - only then a PR is mergeable. A model that writes a rough first draft and takes correction well can beat one that stalls or trips when someone pushes back in such a loop.
SWE-in-a-team measures that loop, and grades the PR it produces. Each mission run goes through the software delivery stages with the agent working through CI failures, reviewer and QA feedback the way a person would, so the score covers delivery and correctness together.
The animation above is used across the study to demonstrate a mission of agents working together in a loop on a ticket.
Click on the image or hit the play button ▶ to see the complete cycle of the actual agent runs, drag the playhead or scrub the image it to move backward or forward in time.
Gates in action
The charts, metrics and stats are not enough to demonstrate the complete journey of the agent team. These three are single missions at the durations actually recorded, from plan through to a passing test, so this is where the loop is visible as a mechanism rather than a statistic.
Each of the missions was sent back by a different gate on the way: CI, Reviewer, or the Tester.
See the reviewer send it back
On the waitlist bug (B4) the Reviewer sent back the most expensive builder in the study, and it corrected course on the next turn.
Planner 172s ──▶ Builder 423s ──▶ CI ✓ ──▶ Review 127s ✗ changes requested
└─────────────▶ Builder 181s ──▶ CI ✓ ──▶ Review 46s ✓ ──▶ Deploy ✓ ──▶ Test 454s ✓
30 minutes - $9.18 - 1 review rejection - resolvedThe second attempt ran in 3m against the first one's 7m, which is a detail I found reassuring. The feedback narrowed the work instead of restarting it.
Watch the Tester catch what Reviewer missed
CI was green, the Reviewer had approved it, and the branch had already deployed to preview environment. The Tester sent it back, and it was the only gate that objected to anything in this mission.
Planner 143s ──▶ Builder 281s ──▶ CI ✓ ──▶ Review 38s ✓ ──▶ Deploy ✓ ──▶ Test 387s ✗ failed
└─────────────▶ Builder 117s ──▶ CI ✓ ──▶ Review 35s ✓ ──▶ Deploy ✓ ──▶ Test 529s ✓
36 minutes - $7.23 - 1 test rejection - resolvedThe Tester cost $5.31 of this run's $7.23, more than the Planner, Builder and Reviewer inference together, because it ran the whole check twice. Testing stage is the next best candidate to run benchmarks for due to its contribution to the total spend.
Follow a hard mission to the pass
The SEO ticket (G1) was the hardest mission DeepSeek had, and it still landed.
Planner 181s ──▶ Builder 352s ──▶ CI ✗ failed
└─────────────▶ Builder 95s ──▶ CI ✗ failed
└─────────────▶ Builder 195s ──▶ CI ✓ ──▶ Review 122s ✗ changes requested
└─────────────▶ Builder 95s ──▶ CI ✓ ──▶ Review 51s ✓ ──▶ Deploy ✓ ──▶ Test 527s ✓
38 minutes - $4.22 - 2 CI failures + 1 review rejection - resolved4 builder attempts and 3 gate rejections produced a resolved ticket for $4.22.
For the same ticket:
- It's still less than a frontier model's mission cost - claude-opus resolved the same ticket for $6.78.
- The same DeepSeek model used with open-code resolved it through a smooth run - no rejections from any of the gates.
Results
After 13 configurations, 20 tickets each, 260 mission runs, and about $1,030 of inference orchestrated by SHIP, let's see what the numbers tell.
Compare thirteen builders
Cost per resolved task
7 of the 13 builders resolved all 20 tickets. Ranked by cost per resolved ticket, the cheapest of those was deepseek/deepseek-v4-flash running on pi at $3.34 and the priciest that matched it was claude-opus-5 at $5.48, which is the same result for 64% more.
| Model | Harness | Total spend | Per resolved ticket | vs cheapest |
|---|---|---|---|---|
deepseek/deepseek-v4-flash | pi | $66.86 | $3.34 | baseline |
gpt-5.6-sol | codex | $76.64 | $3.83 | +15% |
deepseek/deepseek-v4-flash | open-code | $81.59 | $4.08 | +22% |
z-ai/glm-5.2 | open-code | $82.73 | $4.14 | +24% |
moonshotai/kimi-k3 | open-code | $82.85 | $4.14 | +24% |
claude-sonnet-5 | claude-code | $86.22 | $4.31 | +29% |
claude-opus-5 | claude-code | $109.66 | $5.48 | +64% |
The most expensive builder in the study claude-fable-5 running on claude-code has the worst cost per resolved ticket of any configuration at $6.44, and it did not resolve all either: 19 of 20. The cheapest configuration to resolve all 20 did so at $3.34, so Fable cost 92% more for one less ticket resolution.
By family, Open-Weight builders resolved 118 of 120, Anthropic 77 of 80, and OpenAI 55 of 60 in total.
Let the loop do the correcting
| Builder | Shipped unopposed | Sent back by a gate | Pushes | Resolved |
|---|---|---|---|---|
| DeepSeek-v4-flash (pi) | 12 of 20 | 13 times | 33 | 20 of 20 |
| Claude Opus | 19 of 20 | 1 time | 21 | 20 of 20 |
| Claude Fable | 19 of 20 | 1 time | 21 | 19 of 20 |
The first column counts tickets where nothing in the loop objected - one push from the Builder passes through all the gates. It does not count tickets the model got right, and the gap between those two things is interesting.
The mission that Claude Fable lost was lost silently. On the reminders ticket (F5) it wired the live email provider straight into the request path, so the endpoint returns a 500 whenever the API key is missing, even though the ticket only asks it to queue a row. The repository's own test suite stayed green throughout, which is precisely why CI passed, and the Reviewer approved along with the Tester. Fable did not break anything. It built the wrong thing, correctly, and only the hidden tests could tell.
DeepSeek-v4-flash via pi is the contrast. It was objected to 13 times and shipped nothing broken.
A model that converges under correction turns out to be worth more inside a loop than one that starts closer to the answer.
What makes this practical rather than interesting is the price of those extra pushes. DeepSeek's builder cost about 1% of its own mission bill, because the pinned frontier Planner, Reviewer and Tester dominate the bill, so buying more attempts from a cheap builder costs almost nothing while buying a better first draft costs much more.
Swap the harness, keep the model
Three Open-Weight models each ran on two harnesses - open-code vs pi, which makes this the one place in the study where exactly one variable moves.
| Model | Harness | Total spend | Median wall | Resolved |
|---|---|---|---|---|
deepseek/deepseek-v4-flash | pi | $66.86 | 20 min | 20 of 20 |
deepseek/deepseek-v4-flash | open-code | $81.59 | 22 min | 20 of 20 |
z-ai/glm-5.2 | pi | $63.44 | 24 min | 19 of 20 |
z-ai/glm-5.2 | open-code | $82.73 | 21 min | 20 of 20 |
moonshotai/kimi-k3 | pi | $58.93 | 18 min | 19 of 20 |
moonshotai/kimi-k3 | open-code | $82.85 | 25 min | 20 of 20 |
open-code cost more on all three, by 22%, 30% and 41%, and ran 37% longer on Kimi. It also resolved two more tickets across the 60, so this is a trade-off rather than a ranking.
A result that names the model without naming the harness is describing half of what ran.
Count what the gates caught
Across all 260 runs, 214 landed on the first pass and 46 needed the loop. The gates rejected work 70 times in total: CI 45 times, the Reviewer 21, the Tester 4.
Only four rejections reached the Tester, because CI and the Reviewer had caught nearly everything by then. A loop that fails fast early is what makes a cheap builder viable.
Inspect any mission below to see how the loop progressed, and where its time and money went.
Mission breakdown
Pick any run to see what SHIP records that a one-shot patch benchmark cannot - how many builder attempts it took, where the wall-clock went stage by stage, the reason for every retry, and the proof of delivered work.
Bug: Production TypeError opening a fully-refunded invoice
Takeaways
The reflex is to use the best model you can afford, and this study suggests that reflex is wrong. You should choose the agent harness and model your loop can correct.
Buying the most expensive model did not buy reliability for the delivery. Fable finished a ticket ahead of Haiku, 19 of 20 against 18, while costing 52% more per resolved ticket. Buying the cheapest one did not save money in all stations either, because even though the longest, the Build is just one stage of what a ticket costs once a real loop surrounds it.
What the money buys is correction. A budget model that starts worse and takes feedback well resolved every ticket in this study, and together with the 13 rejections it absorbed cost less than a single frontier draft would have.
So the question I would ask before picking a model is not which one writes the best code on the first try. It is what plans it, what checks the code afterwards, and whether correction is cheap enough that a rough first draft stops being a problem worth paying to avoid.