Our SWE-in-a-team benchmark research is ready.Read the results.
Paper boat sails caught mid-motion against black, some lit white and some vermilion, trailing streaks of light.← Blog
|Önder Ceylan

SWE-in-a-team: a coding benchmark for software factories

Our SWE-in-a-team benchmark graded thirteen coding agents and models in an SDLC loop. An Open-Weight builder resolved every ticket at half the price.

As an engineer who is used to working in an engineering team, I find the coding benchmarks distant from the engineering teams' tasks, projects and delivery pipeline. They typically grade the destination - functions, patches and environments. Engineering teams ship work through planning, CI/CD pipelines, review, and testing. Those are not the same unit of work.

When I decided to build SHIP as a managed cloud based software factory that takes a ticket through a complete SDLC, I knew that I had to see objective results rather than subjective remarks and opinions. I've had my own observations, but numbers speak louder than words.

  • Can we use a budget or less capable model with smarter redirection and verification gates in a loop to resolve real-world tickets? Can Haiku be used for coding in the loop even?
  • How much cost can we save just by configuring a builder agent to use a budget model?
  • How much a coding agent harness affects cost and cycle time when we pin Open-Weight models?

SWE-in-a-team benchmark

Engineering should not be reduced to generating code - it's also about understanding the business requirements, architecture, decisions, and systems in an environment, communicating with peers, resolving feedback and conflicts, maintaining a code standard and application quality, delivering a secure work that is ready for production deployment, and monitoring the application performance post-deployment - among other things.

SWE-in-a-team: Agentic coding benchmark for an agent working in a team environment

That's why I created a brand new coding benchmark - SWE-in-a-team, to grade a coding agent and model's performance while working on 20 tickets of a real-world like full-stack SaaS app in an autonomous SDLC loop - a software factory.

Each mission runs the same loop - Plan → Build → CI → Review → Deploy → Test, and goes back to the build stage whenever a gate rejects. One ticket taken through that loop to Ready for Acceptance state is called a mission.

13 agent teams with variable Builder configurations were assigned the 20 tickets each, running 260 missions in total during the study:

  • claude-code running claude-haiku-4-5, claude-sonnet-5, claude-opus-5 and claude-fable-5
  • codex running gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna
  • pi running deepseek/deepseek-v4-flash, z-ai/glm-5.2 and moonshotai/kimi-k3
  • open-code running deepseek/deepseek-v4-flash, z-ai/glm-5.2 and moonshotai/kimi-k3

The Planner, Reviewer and Tester agents were assigned the same configuration in every run: claude-opus-4.8 via claude-code, only the Builder configuration was dynamic in the team setup. So, the results read as the difference between builders. That's a SWE in a team.

Summary

Claude Fable went through 19 of 20 tickets without a single objection from agent peers: green CI, an approving reviewer, a passing tester. DeepSeek-v4-flash managed that on 12 tickets, but got correction and direction from peers for the rest. In the end, both completed all of the 20 missions in about the same wall time - medians of 20 minutes, but Fable cost nearly twice as much: $122.42 vs $66.86. Hidden tests also showed that one of Fable's completed missions was actually a failure.

Below, you can see the insights from the 260 mission runs that caught my attention:

  • A DeepSeek run that needed (agent) correction in the loop still cost less at the median than an Opus run that needed none, $3.42 against $4.58.
  • Fable resolved one more ticket than Haiku on the same harness, 19 of 20 against 18, but paid 52% more per resolved ticket for it and finished only 9% faster.
  • Running DeepSeek-v4-flash, GLM-5.2 and Kimi-K3 on open-code instead of pi cost 22% to 41% more and moved median wall time by as much as 37%, for two more resolved tickets overall.
  • The cheapest builder also resolved the fewest: Luna running on Codex leads on cost at $3.08 resolving 16 / 20, and owns 4 of the 5 runs the loop gave up on. Each of those four couldn't get through the CI gate, which pulled its average cost down.
  • 4 of the 260 runs passed every gate and were still not complete, and the grader's hidden tests caught them.
  • Catching late cost money: the 4 runs the Tester rejected ran to a median of $8.18 against $3.51 for a clean run, while the 27 CI rejected came in at $3.37.
  • CI raised the first objection on 27 of the 46 rejected runs, the Reviewer on 15 and the Tester on 4, so the cheapest gate did most of the catching.
  • The loop corrected 40 of the 46 runs it objected to with no human involved, 31 of them on the very next push and all within 4 Builder attempts.
  • Correction within the loop cost about a third more than getting it right first time, for the same builder.

Cost per resolved task — frontier vs. open

USD, API-equivalent - every builder, cheapest first — lower is better
Claude (frontier)2/4 resolved every task, $4.23–$6.44GPT (frontier)1/3 resolved every task, $3.08–$3.88Open-weight4/6 resolved every task, $3.10–$4.14
Cost per resolved task = Σ cost of every trial ÷ tasks resolved. 7 of the 13 builders resolved all 20 tasks.

Methodology

SWE-in-a-team uses a fixture that was kept private, grades the delivery of backlog tickets shipped by a coding agent working in SDLC loop. It represents the type of work that engineering teams recognize, as they regularly work on similar tickets.

These are the non-negotiables:

  1. The coding models must not have seen the application code before.
  2. The grade must not come from the models' own opinion of their work.
  3. The complete SDLC loop must be measured for key metrics like cycle time and cost.
  4. The agent should be working in a team, and be directed and corrected by the peers.
  5. The missions must produce evidence for the public to inspect: PRs, reviews and any test artifacts.

What it measures

A tickets is resolved only if it passes two gates: the mission reached Ready for Acceptance state AND a set of hidden acceptance tests passes. Cost is a computed API-equivalent by multiplying used tokens with the list price per token. Real spend was flat on an Anthropic subscription, so the cost of Anthropic models are a normalization for comparison, not an actual bill.

note

In SHIP, you use your own provider keys and subscription usage rights as long as AI providers allow it.

Read the insights and results as directional, it's not a leaderboard. It is a single trial per cell (k=1), for 20 tickets, in one repository. The CI checks, and pinned claude-opus-4-8 Reviewer and QA do course correction during the mission, so the builder model is not solely responsible for the outcome. That is the point of the setup, but it also means these numbers do not isolate raw model's ticket resolution skill, instead evaluates how it performs in a team.

Building a fixture nobody has trained on

Public code has already been read by the models, so a benchmark built on it tests what a model already remembers and trained for. That is why the fixture in the study is a purposely built booking SaaS called Studiobook, kept private, with 20 tickets written in the voice of the people who file them rather than in the voice of a benchmark author.

GitHubGitHub - letsship/ship-bench-app: Studiobook — SHIP's open benchmark fixture: a studio class-booking SaaS (Next.js) with seeded SDLC benchmark tasksStudiobook — SHIP's open benchmark fixture: a studio class-booking SaaS (Next.js) with seeded SDLC benchmark tasks - letsship/ship-bench-app

20 tickets were spread over 10 categories: Bug, Security, Refactor, Dependency Upgrade, Migration, Performance, Feature, Integration, Growth, DevOps.

  • Two are bugs reported from symptoms rather than stack traces, including a production TypeError on a fully refunded invoice and a member who joined a waitlist twice by double-clicking.
  • Three are security holes with real blast radius: a cross-tenant leak, a stored XSS in an invoice line, and a vulnerable dependency.
  • Four are features specified by contract, so the hidden tests grade behavior rather than taste.
  • Three are integrations covering payment webhooks, error reporting and analytics with third party services.
  • The rest is as much familiar: a Zod major upgrade, migrations onto Next.js 16 and Tailwind 4, a move from Supabase to Cloudflare D1, three drifted copies of an invoice total, an N+1 query, an SEO ticket, and a flaky test that broke the suite through an isolation leak.
Bug (B2) - Production TypeError opening a fully-refunded invoiceSTB-1088

On-call here. Sentry started catching this in production yesterday:

TypeError: Reduce of empty array with no initial value

It fires every time someone opens the invoice detail page for the refunded "Pottery intensive" invoice, the one for Femke. The page dies to the error screen instead of rendering. Every other invoice I tried opens fine.

The thing that's different about that invoice is that every line item on it has been refunded, so there's nothing left to bill. Anything that leaves an invoice with zero billable line items seems to trip the same error, and finance wants to lean on line-level refunds more, so this is only going to happen more often.

Acceptance criteria

  • Opening the detail page of an invoice whose line items are all refunded renders normally: subtotal €0.00, tax €0.00, total €0.00, with the refunded lines still listed (and badged as refunded).
  • Invoice totals are computed safely for an invoice with zero billable line items (none at all, or all refunded), yielding zero subtotal/tax/total and the correct refunded sum, instead of throwing.
  • Invoices with a mix of billable and refunded lines keep today's behaviour (refunded lines tracked but excluded from subtotal and tax).
Bug (B4) - Member got onto a class waitlist twice by clicking 'join' twiceSTB-1089

Front desk flagged this after yesterday's Reformer class filled up.

A member tried to book the full class from the bookings page, the button felt unresponsive, so she clicked it again. Now she shows up twice on that class's waitlist. When a spot opened up this morning, one of her entries got promoted to a confirmed seat, and the second entry is still sitting on the waitlist. It's holding a spot the next person should have, and it's throwing off our occupancy numbers.

I reproduced it in a minute: pick a full class, book the same member, submit again, and the second submit goes through and adds another waitlist row. If the member already holds a confirmed seat, the second submit is correctly rejected ("This member already has a booking for this class"), so it's really only the waitlisted case that slips through.

Acceptance criteria

  • Booking a member onto a full class puts them on the waitlist once; any repeat booking attempt for the same member + same class while they are waitlisted is rejected with the same "already has a booking" conflict as a repeat attempt on a confirmed seat.
  • After a double submit, the member has exactly one active booking row (waitlisted) for that class.
  • Cancelled bookings keep today's behaviour: a member whose booking was cancelled can book the same class again.
Security (S1) - Security advisory: vulnerable semver dependency (CVE-2022-25883)STB-1090

Forwarded from our dependency scanner.

High · Regular Expression Denial of Service (ReDoS) in semver

  • Advisory: GHSA-c2qf-rxjj-qqgw (CVE-2022-25883)
  • Package: semver (npm)
  • Affected versions: >=7.0.0 <7.5.2
  • Patched in: 7.5.2
  • Dependency path: apps/web > semver

The semver range parser can be driven into super-linear backtracking by a crafted version/range string, allowing a denial of service. apps/web depends on a version in the affected range.

We pull semver in to gate the minimum supported client version. Please update the dependency to a patched release so the scanner stops flagging it.

Acceptance criteria

  • apps/web no longer depends on a semver in the vulnerable range; the version the package manager resolves is >= 7.5.2.
  • A high-severity dependency audit reports no findings.
  • Lint, type-check, tests, and the production build all still pass.
Security (S2) - Any logged-in studio can read another studio's dataSTB-1091

Hi. I run security for one of your pilot studios, and while poking at the app with my own signed-in session I noticed the detail endpoints don't seem to check who owns a record.

Here's what I mean. My session is scoped to my studio, but if I take the id of an invoice, member, or booking that belongs to a different studio and request it directly (say GET /api/invoices/<their-id>, GET /api/members/<their-id>, or DELETE /api/bookings/<their-id>), the app hands it over. In the booking case it will even cancel their record for me. The id is the only thing guarding it, and ids are easy to guess. That's an IDOR: one tenant can read and change another tenant's data.

If a record isn't mine, I'd expect it to look like it simply doesn't exist for me.

Acceptance criteria

  • Fetching an invoice or member by id, or cancelling a booking by id, is scoped to the caller's own studio: an id that belongs to a different studio is rejected as not found (the record is neither returned nor modified).
  • The same operations continue to work for records that belong to the caller's own studio.
  • Lint, type-check, tests, and the production build all still pass.
Security (S3) - Stored XSS: invoice line description renders unescaped HTMLSTB-1092

Hi, reporting a stored cross-site-scripting issue in the invoices area.

Invoice line-item descriptions are free text entered by staff, but the invoice detail page renders that text as raw HTML instead of escaping it. If a description contains markup, it gets injected into the page and runs in the browser of anyone who opens that invoice.

Repro:

  1. Create an invoice with a line item whose description is <img src=x onerror="alert(document.cookie)">.
  2. Open the invoice detail page.
  3. The image tag is parsed and the onerror handler fires, so arbitrary script runs in the viewer's session. One malicious value stored on a single account then executes against every user who later views that invoice.

The description should show as literal text, never interpreted as markup.

Acceptance criteria

  • On the invoice detail page, a line-item description containing HTML (e.g. <img src=x onerror=...>) is rendered as inert, escaped text: the markup appears verbatim and no element or handler is created from it.
  • Ordinary descriptions still render as readable text.
  • Lint, type-check, tests, and the production build all still pass.
Dependency (D1) - Upgrade Zod v3 → v4STB-1093

Automated dependency PR from the dependency bot.

Bumps zod in apps/web from ^3.24 to the latest** v4** release.

Zod 4 is a major version. Most of our schema code is source-compatible, but a couple of v3-era idioms changed, and the request-error handling that builds the API validation-error envelope depends on one of them. Move apps/web onto Zod 4, confirm the request validators still behave exactly as they do today, and keep the app compiling.

Release notes (upstream)

  • Top-level string-format helpers (z.email(), z.url(), …) are now preferred over the chained z.string().email() forms.
  • The ZodError surface changed: the flat error.errors array accessor was removed in favour of error.issues; error.flatten() / error.format() are retained (now deprecated).
  • Default and transform semantics for object schemas are unchanged for the shapes we use.

Acceptance criteria

  • apps/web depends on Zod v4: the version the package manager resolves is 4.x, not 3.x.
  • The API request validators keep their behaviour: a malformed email is rejected; a valid email is trimmed and lower-cased; a new member's status defaults to active; and a colour that is not a #rrggbb hex value is rejected.
  • Lint, type-check, tests, and the production build all pass.
Migration (D2) - Migrate the web app to Next.js 16STB-1094

From engineering:

We are a major version behind on Next.js. apps/web is still on Next 15, and Next 16 is out. Let's upgrade to Next.js 16. Behavior should be unchanged; this is a framework upgrade, not a feature change.

The main thing Next 16 changes for us is that the dynamic request APIs are now asynchronous: params and searchParams in pages and route handlers, and cookies() / headers(), must be awaited rather than read synchronously.

Docs: the Next.js 16 upgrade guide at https://nextjs.org/docs/app/guides/upgrading/version-16.

Acceptance criteria

  • The web app runs on Next.js 16: next resolves to a 16.x version in the lockfile (not just the declared range).
  • The dynamic request APIs are accessed the Next 16 way: params / searchParams (in pages and [id] route handlers) and cookies() / headers() are awaited, not read synchronously.
  • The app's behavior is unchanged: the full test suite and the production build stay green.
Migration (D4) - Migrate the web app to Tailwind CSS v4STB-1095

From engineering:

We are behind on Tailwind. The web app is still on Tailwind v3 with the old JavaScript config, and v4 has been out for a while with a faster engine and a CSS-first setup. Let's upgrade apps/web to Tailwind CSS v4. The app should look and behave exactly the same afterwards. This is a toolchain modernization, not a redesign.

Docs: the Tailwind v4 upgrade guide at https://tailwindcss.com/docs/upgrade-guide.

Acceptance criteria

  • The web app builds and runs on Tailwind CSS v4:
    • tailwindcss resolves to v4 in the lockfile;
    • the PostCSS plugin is @tailwindcss/postcss (not the old tailwindcss PostCSS plugin);
    • the global stylesheet pulls Tailwind in with @import "tailwindcss", replacing the old @tailwind base; @tailwind components; @tailwind utilities; directives;
    • the old tailwind.config.js is removed (configuration moves into CSS).
  • The app's appearance and behavior are unchanged: the full test suite and the production build stay green.
Refactor (R1) - Migrate persistence from Supabase to Cloudflare D1 + DrizzleSTB-1096

Architecture note from the platform team.

We're consolidating Studiobook onto a single Cloudflare stack. The app already builds and deploys as a Worker (OpenNext), but persistence still runs through @supabase/supabase-js against a hosted Postgres. That's the last thing keeping us tied to a second vendor and its connection model. We want the production data layer on Cloudflare D1 (the app's own SQLite database binding) using** Drizzle ORM**, so everything lives in one account and one deploy.

This is a swap of the adapter, not a change in behaviour. Studiobook already funnels every read and write through the Repositories interfaces (lib/db/repos/types.ts). The Supabase client is confined to a single implementation file behind that seam, and services, routes, and domain code only ever touch the interfaces. So keep the migration contained to the persistence adapter: implement the same repository surface with Drizzle over the D1 binding, then delete the Supabase pieces. Nothing above the repository seam should need to change, and the in-memory fake repositories (used by local dev and the test suite) stay exactly as they are.

Keep the change tight and behaviour-preserving. The demo dataset, the booking, invoice, and notification flows, and every existing check should behave identically afterwards.

Acceptance criteria

  • The web app no longer depends on @supabase/supabase-js or @supabase/ssr; both are removed from the app's dependencies (and the workspace lockfile).
  • The production repository implementation is written with Drizzle ORM over the Cloudflare D1 binding, behind the existing Repositories seam. A Drizzle dependency is present in the app. The createD1Repositories(db) factory in lib/db/repos/d1.ts takes the D1 database binding as its argument (so the adapter can be constructed against any D1 instance), and resolveRepositories passes the Worker's DB binding to it.
  • Ship the D1 schema as migration SQL under apps/web/migrations/ (the standard wrangler D1 migrations directory) and declare the DB d1_databases binding in wrangler.jsonc, so the database can be created from scratch.
  • No @supabase/* import remains anywhere under the app's lib.
  • Behaviour is preserved: the app builds, and the existing unit + integration suite (services and repositories exercised against the in-memory fakes) stays green. Route handlers, services, and domain modules keep depending only on the repository interfaces, untouched by the swap.
Refactor (R3) - Consolidate three drifted copies of invoice total calculationSTB-1097

Eng-lead here, writing this up after finance flagged a discrepancy.

We now compute an invoice's total in three different places and they've drifted:

  1. computeInvoiceTotals in lib/domain/invoices.ts. This is the original, and the one the invoice page and the stored invoice both rely on. It correctly drops refunded line items out of the taxable subtotal.
  2. The invoice create path (lib/services/invoices.ts) has grown its own inlined copy of the subtotal/tax math instead of calling the domain function.
  3. The new account statement feature (lib/services/account-statements.ts) rolled its own copy too, and this one is wrong: it taxes every line, including refunded ones. So a member whose invoice had a line refunded sees a bigger total on their statement than on the invoice itself, and bigger than what we actually charged.

Concretely: an invoice with a €100 billable line and a €50 refunded line, at 9% tax, is €109.00 everywhere it's computed correctly, because tax applies to the €100 only. The statement, though, reports €163.50 because it taxes the full €150.

We need ONE invoice-total implementation. Route every surface through the single domain function so a refunded line is handled identically everywhere, and the statement, the invoice record, and the domain calculation all agree.

Acceptance criteria

  • There is a single source of truth for invoice totals; the account-statement and invoice-create paths compute totals through it rather than re-deriving the subtotal/tax math inline.
  • For an invoice with a refunded line item, the account statement's total excludes the refunded line from the taxable subtotal. It equals the canonical computeInvoiceTotals result and the stored invoice total (for the example above, €109.00, i.e. 10900 cents), never the over-taxed figure.
  • No behaviour changes for invoices without refunded lines, and invoice creation still stores the same subtotal/tax/total it does today.
Performance (P1) - Bookings list N+1: repository reads grow with the number of bookingsSTB-1098

From engineering, flagged by one of our larger studios.

Since we refactored the bookings list last week, the dashboard has gotten noticeably slow for studios with a full schedule. A few hundred bookings and the page crawls. Our query logs show the list now looking up each booking's member and class session one row at a time, so the number of database reads climbs with the number of bookings. It used to fetch them in a batch.

Please get the list back to a bounded number of reads no matter how many bookings it returns, without changing anything the endpoint returns.

Acceptance criteria

  • GET /api/bookings returns exactly what it returns today: the same rows, the same fields, in the same order.
  • Listing the bookings issues a number of member and class-session repository reads that does NOT grow with the number of bookings. Fetching a list of N bookings must not read the members repository or the class-sessions repository once per booking; a small, fixed number of reads total is expected, regardless of N.
  • The existing test suite stays green.
Feature (F1) - Export a date-range CSV of bookings for accountingSTB-1099

From the founder.

Every quarter I sit down with our bookkeeper and she asks for "the list of what actually happened": which classes ran, who was in them, over a date range she picks. Right now the only exports we have are the members list and the invoices list (/api/export?type=members / ?type=invoices). There's nothing for bookings, so I end up screenshotting the bookings page, which is a bit embarrassing.

Can we add a bookings export to the same download endpoint? I want to ask for "everything between 1 June and 30 June" and get a clean CSV I can open in Excel or hand straight over. A couple of our members have commas in how their name is stored ("Rossi, Chiara"), and the current members export handles that fine, so please keep the same quoting.

Acceptance criteria

  • GET /api/export?type=bookings is supported, alongside the existing members and invoices exports, and (like them) requires a signed-in session.
  • It accepts optional from and to query parameters (ISO-8601 timestamps). When given, the export includes only bookings whose class session start time falls within [from, to], inclusive of both ends; an omitted bound is unbounded on that side.
  • The CSV has these columns, in this order: Starts, Class, Member, Email, Status. Starts is the session start as its ISO-8601 UTC timestamp, Class is the class name, Member is the member's name, Email is their email, and Status is the booking status.
  • Fields containing a comma, double quote, or newline are quoted with embedded quotes doubled (RFC 4180, the same escaping the invoices export already uses), so a member named Rossi, Chiara stays a single column.
Feature (F6) - Sell class packs (prepaid class credits)STB-1100

From the founder, working on next season's pricing:

Every booking today is a one-off. Our most loyal members would happily pay up front for a bundle instead of paying class by class, the way every studio sells a "10-class pass." I want to sell class packs: a member buys a pack of credits, and each class they book spends one credit until the pack runs out. Then they buy another.

Let's start with two sizes, a 5-credit pack and a 10-credit pack. Once a member has a pack, their bookings should draw from it automatically, so the front desk never has to think about it. When the pack is empty, the booking should stop and tell them to buy another. And if we ever have to refund a pack, its remaining credits should be voided so they can't still be spent.

Acceptance criteria

  • Buy a pack. POST /api/packages with body { memberId, credits }, where credits is either 5 or 10. It creates an active pack for that member with creditsRemaining equal to credits, and a total priceCents of credits × 1000 — i.e. 1000 cents per credit, so a 5-credit pack is 5000 and a 10-credit pack is 10000 (the priceCents field is the pack's TOTAL price, not the per-credit rate). Responds 201 with { id, memberId, creditsTotal, creditsRemaining, priceCents, status, purchasedAt }.
  • List a member's packs. GET /api/packages?memberId=<id> responds 200 with that member's packs (each { id, creditsTotal, creditsRemaining, priceCents, status, purchasedAt }), newest first.
  • Bookings draw from the pack automatically. When a member who has an active pack with creditsRemaining > 0 books a class via POST /api/bookings, the booking is confirmed exactly as today AND one credit is spent (the pack's creditsRemaining drops by one, oldest pack first). A member with no pack books exactly as they do today, unchanged.
  • A used-up pack stops the booking. Once a member has bought a pack, their bookings must come from a pack. If they have no credits left to draw (every pack they own is exhausted or refunded), POST /api/bookings is rejected with 402 and the error code pack_exhausted; they must buy another. A member who never bought a pack is unaffected and books exactly as today.
  • No double-spend. The existing rule that a member cannot book the same session twice still holds: a repeated booking is rejected 409 and spends no extra credit.
  • Refund voids the remaining credits. POST /api/packages/{id}/refund sets the pack's creditsRemaining to 0 and its status to refunded; a refunded pack is never drawn from again.
Feature (F3) - Per-member calendar subscription behind a private tokenSTB-1101

Let members subscribe to their own class calendar

From the PM:

We already publish a studio-wide calendar feed (/api/ical), but members keep asking for "just my classes on my phone calendar." A single shared feed doesn't work for that — it lists everyone's sessions, and it's public.

I'd like each member to have their own private subscription link: a secret per-member token in the URL that returns only the sessions that member is booked into, so they can paste it into Apple/Google Calendar and see their upcoming classes auto-update. Because a calendar app can't log in with our cookie, the secret link itself has to be what authorizes it — so an unknown or made-up token must just 404, never leak someone else's schedule.

Acceptance criteria

  • Every member has a private calendar token.
  • GET /api/ical/[token] returns an iCalendar feed (content-type: text/calendar) containing calendar events for ONLY that token-holder's upcoming booked sessions (future sessions they hold a confirmed seat in). It must not include other members' sessions or the member's past sessions.
  • An unknown or empty token returns 404.
  • No session cookie is required — the secret token is the authorization.
Feature (F5) - Send 24-hour class reminders (idempotent job endpoint)STB-1102

Automatic 24-hour class reminders

From the PM:

Our no-show rate spikes for early classes, and members keep telling us "I just forgot." Every other studio app sends a reminder the day before — we should too. I'd like a reminder to go out to everyone who's booked into a class that starts in the next 24 hours.

I want to drive this from our scheduler (a cron that hits an endpoint), so the important thing is that it's safe to call repeatedly: if the cron runs hourly, a member must still only get one reminder for a given class, not one every hour. And obviously anyone who's turned notifications off shouldn't get one, and people only on the waitlist (no confirmed seat) shouldn't either.

Acceptance criteria

  • POST /api/reminders/run (requires a signed-in session, like our other write endpoints) finds every class session starting within the next 24 hours and queues a booking_reminder notification — a pending row in the notification outbox — for each member holding a confirmed (booked) seat in one of those sessions.
  • Members who have opted out of notifications get no reminder. Waitlisted members, and members whose session is outside the 24-hour window, get no reminder.
  • The endpoint is idempotent: running it a second time does not queue a duplicate reminder for a booking that already has one queued.
  • Returns HTTP 200 on success.
Integration (I1) - Wire up product analytics (PostHog) for the booking funnelSTB-1103

From the founder:

I have no real picture of our funnel. How many bookings get made, how often members join a waitlist, how often they cancel. Right now I'm guessing. Let's wire up PostHog so I can see these moments in one place.

Instrument the events that matter, and keep it clean. Route analytics through a single helper the way we already route email through the notification adapter, so it's easy to test and swap. Don't scatter posthog calls through the domain code.

Docs: PostHog for Next.js (https://posthog.com/docs/libraries/next-js), also mirrored offline at docs/vendor/posthog-nextjs.md.

Acceptance criteria

  • Analytics goes through a single injectable tracker, an interface with a capture method, wired into the booking, waitlist, and cancellation flows, with the real PostHog client constructed only at the composition root (mirroring the notification adapter). Service and domain code depend on the interface, never on a posthog package directly.
  • Expose a __setTestTracker(tracker) test seam from the tracker composition root — mirroring __setTestRepositories on the repository seam — so a recording tracker can be injected and the route handlers driven in tests. (How the flows receive the tracker is up to you; the seam is what's graded.)
  • These events are captured, exactly once, on their flow, each attributed to the member (the member's id as the analytics distinct id) and carrying the session's id as a session_id property:
    • booking_created when a booking is confirmed.
    • waitlist_joined when a booking is placed on the waitlist.
    • booking_cancelled when a booking is cancelled.
  • A waitlisted booking fires waitlist_joined and NOT booking_created; a confirmed booking fires booking_created and NOT waitlist_joined.
  • No personally-identifying data in the captured events. No email, name, or phone number anywhere in an event's properties.
  • The existing test suite stays green.
Integration (I3) - Receive Stripe payment webhooks and mark invoices paidSTB-1104

From the founder:

We're starting to collect payments through Stripe. Right now when a member pays an invoice, someone has to mark it paid by hand, and we forget. Stripe can tell us the moment a payment succeeds via a webhook, so let's receive that and mark the matching invoice paid automatically.

Two things I care about: it must reject anything that isn't genuinely from Stripe (verify the signature), and if Stripe sends the same event twice (it does), we must not double-process.

Docs: Stripe webhooks (https://docs.stripe.com/webhooks) and signature verification (https://docs.stripe.com/webhooks#verify-events).

Acceptance criteria

  • POST /api/webhooks/stripe receives a Stripe webhook. It verifies the request's Stripe-Signature header against our webhook signing secret (STRIPE_WEBHOOK_SECRET); a request with a missing or invalid signature is rejected with 400 and nothing is changed.
  • On a verified invoice.paid event, the invoice named in the event (its id at event.data.object.metadata.invoice_id) is marked paid: its status becomes paid and paidAt is set. Respond 200.
  • Idempotent: receiving the same event again (same event id) leaves the invoice paid exactly once and still responds 200, with no double-processing and no error.
  • A verified event that names an unknown invoice responds 200 (acknowledged) and changes nothing. A verified event of any other type is acknowledged (200) and changes nothing.
  • The existing test suite stays green.
Integration (I4) - Report unexpected server errors to SentrySTB-1105

From the founder, after a bad month:

Twice last month a member hit a 500 and we only found out because they emailed us. We need real error monitoring. Wire up Sentry so that whenever one of our API routes hits an unexpected server error, it gets reported to Sentry with the error instead of vanishing into the logs.

The important nuance: our normal, handled responses aren't errors. A 404 for a missing member, a 400 for bad input, a 409 for a full class, a 402 for an empty pack: those are expected and must NOT be reported, or Sentry becomes noise. Only genuine unexpected failures should be captured.

Docs: Sentry for Next.js (https://docs.sentry.io/platforms/javascript/guides/nextjs/).

Acceptance criteria

  • When an API route handler throws an UNEXPECTED error (anything that is not one of our normal handled outcomes: not a validation error, not a deliberate HttpError), the error is reported to Sentry via the Sentry SDK's captureException, and the response is the existing 500.
  • Expected, handled outcomes are NOT reported to Sentry: a 400 validation error and any HttpError (404 / 409 / 402 / etc.) are normal and must not be captured.
  • A successful request reports nothing.
  • The existing test suite stays green.
Growth (G1) - Make the existing public studio page rank in search (SEO improvement)STB-1106

From our growth lead:

Every studio already has a public page at /s/<slug> that lists its upcoming classes, and you can open one without logging in. The problem is that it's invisible in search. I checked one with Google's Rich Results Test and Lighthouse and it's a mess. The page is marked noindex so Google won't even index it, the title is just "Studio", there's no meta description, no social preview when you paste the link in Slack or WhatsApp, and none of the structured data Google needs to show class events. We're leaving sign-ups on the table.

Please make the existing public studio page properly SEO-ready so it can rank and show rich results. Don't build a new page. Improve the one that's already there (apps/web/app/s/[slug]/page.tsx). Same content, just search-ready.

Docs: Next.js Metadata: https://nextjs.org/docs/app/getting-started/metadata-and-og-images ; sitemap: https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap ; robots: https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots ; schema.org Event: https://schema.org/Event.

Acceptance criteria

  • The public page at /s/[slug] must be indexable: remove the noindex directive so search engines can crawl and index it. It must still be public (no login) and still render the studio and its upcoming classes (each class's name, start time, and instructor). A slug that matches no studio still returns 404.
  • Give it real, studio-specific SEO metadata via Next's Metadata API (generateMetadata): a <title> and <meta name="description"> that name the studio (not a hardcoded "Studio"), Open Graph tags (og:title, og:description, og:type), Twitter card tags, and a canonical URL.
  • Embed JSON-LD structured data in a <script type="application/ld+json"> using schema.org Event, one event per upcoming class, each with at least name, startDate, and location.
  • Fix the on-page SEO basics: the studio image needs descriptive alt text, and the call-to-action link needs descriptive text (not "Click here").
  • Add GET /sitemap.xml listing the public studio page(s) and GET /robots.txt that allows crawling and references the sitemap (use Next's sitemap.ts / robots.ts file conventions).
  • The page must NOT require authentication, and the existing test suite stays green (the admin app and its auth are unaffected).

The measurable bar: the page's Lighthouse SEO score should go from its current low baseline (~45) up to ~100.

DevEx (O1) - Flaky CI: a test-isolation leak breaks the suiteSTB-1107

From engineering.

CI has gone red and it's blocking everyone. The app itself is fine. The problem is the tests. One test leaves shared state behind (it sets something up and never cleans it up), and that leaks into another test, which then fails when the whole suite runs together. Run that other test on its own and it passes; run the full suite and it fails.

Please fix the test isolation so pnpm test is green and stays deterministic. Don't just delete or skip the failing tests. Fix the leak so the offending test cleans up after itself.

Acceptance criteria

  • pnpm test (the full suite) passes deterministically.
  • The fix is proper test isolation: the leaking test cleans up the shared state it sets (so it no longer affects any other test), rather than the other test being changed to tolerate the leak.
  • No test is deleted, skipped (.skip / .only / commented out), or gutted to make the suite pass; every test that existed still exists and still asserts what it did.

Grading with tests the agent never sees

A ticket counts as resolved only when it passes two gates:

  1. The SDLC pipeline shipped it: the loop reached a PR that passed CI/CD, review and test and was ready for a human.
  2. Hidden tests pass on the mission outcome: communicated in the ticket as requirements in the acceptance criteria, kept out of the repository the agent works in, and run against the exact PR the agents produced.
Feature (F1) - Export a date-range CSV of bookings for accounting

Acceptance Criteria

- GET /api/export?type=bookings is supported alongside members and
  invoices, and requires a signed-in session.
- Optional from/to ISO-8601 params. When given, include only bookings
  whose class session START time falls within [from, to], INCLUSIVE of
  both ends. An omitted bound is unbounded that side.
- CSV columns, in this exact order: Starts, Class, Member, Email,
  Status. Starts is the session start as its ISO-8601 UTC timestamp.
- RFC 4180 escaping: quote fields containing a comma, quote or newline,
  and double any embedded quotes, exactly as the invoices export does.

The second gate runs the hidden tests along with the project's full quality commands: lint, typecheck, test, and build. That is broader than the conventional pass-to-pass definition, which only asks whether previously passing tests still pass. I kept it deliberately, because it also catches a failing test the model wrote itself. And, a PR that fails the repository's own gates is not shippable.

Measuring the whole loop

Most model comparisons hand a model an issue or a task, take one patch, and grade that patch. That measures patch generation, which has driven real progress, but it is not how software reaches production. Real delivery is a loop: somebody plans the change, an assignee codes it, CI gates the integration, a reviewer rejects, the assignee fixes it, deployment is successful and tested - only then a PR is mergeable. A model that writes a rough first draft and takes correction well can beat one that stalls or trips when someone pushes back in such a loop.

SWE-in-a-team measures that loop, and grades the PR it produces. Each mission run goes through the software delivery stages with the agent working through CI failures, reviewer and QA feedback the way a person would, so the score covers delivery and correctness together.

MISSION F1 - opusclaude-code@2.1.211claude-opus-528m 01s$6.95READY FOR ACCEPTANCEPlanner2m 11s - $0.53Builder8m 34s - $2.27CI1m 22sReviewer1m 44s - $0.45Deploy1m 51sTester10m 16s - $3.69
28 MIN COMPRESSED TO 30S - 56X
28m 01s of 28m 01s
F1 - Export a date-range CSV of bookings for accounting - 28m 01s - $6.95 - no rejections - PR #796
tip

The animation above is used across the study to demonstrate a mission of agents working together in a loop on a ticket.

Click on the image or hit the play button ▶ to see the complete cycle of the actual agent runs, drag the playhead or scrub the image it to move backward or forward in time.

Gates in action

The charts, metrics and stats are not enough to demonstrate the complete journey of the agent team. These three are single missions at the durations actually recorded, from plan through to a passing test, so this is where the loop is visible as a mechanism rather than a statistic.

Each of the missions was sent back by a different gate on the way: CI, Reviewer, or the Tester.

See the reviewer send it back

On the waitlist bug (B4) the Reviewer sent back the most expensive builder in the study, and it corrected course on the next turn.

MISSION B4 - fableclaude-code@2.1.211claude-fable-500s$0.001 REJECTION - $9.18 TOTALPlanner2m 52s - $0.00BuilderCIReviewerDeployTester
30 MIN COMPRESSED TO 30S - 59X
00s of 29m 42s
B4 - Member got onto a class waitlist twice - 29m 42s - $9.18 - 1 rejection - PR #836
Planner 172s ──▶ Builder 423s ──▶ CI ✓ ──▶ Review 127s ✗ changes requested
                 └─────────────▶ Builder 181s ──▶ CI ✓ ──▶ Review  46s ✓ ──▶ Deploy ✓ ──▶ Test 454s ✓

30 minutes - $9.18 - 1 review rejection - resolved

The second attempt ran in 3m against the first one's 7m, which is a detail I found reassuring. The feedback narrowed the work instead of restarting it.

Watch the Tester catch what Reviewer missed

CI was green, the Reviewer had approved it, and the branch had already deployed to preview environment. The Tester sent it back, and it was the only gate that objected to anything in this mission.

MISSION F1 - sonnetclaude-code@2.1.211claude-sonnet-500s$0.001 REJECTION - $7.23 TOTALPlanner2m 23s - $0.00BuilderCIReviewerDeployTester
36 MIN COMPRESSED TO 30S - 71X
00s of 35m 37s
F1 - Export a date-range CSV of bookings for accounting - 35m 37s - $7.23 - 1 rejection - PR #584
Planner 143s ──▶ Builder 281s ──▶ CI ✓ ──▶ Review 38s ✓ ──▶ Deploy ✓ ──▶ Test 387s ✗ failed
                 └─────────────▶ Builder 117s ──▶ CI ✓ ──▶ Review 35s ✓ ──▶ Deploy ✓ ──▶ Test 529s ✓

36 minutes - $7.23 - 1 test rejection - resolved

The Tester cost $5.31 of this run's $7.23, more than the Planner, Builder and Reviewer inference together, because it ran the whole check twice. Testing stage is the next best candidate to run benchmarks for due to its contribution to the total spend.

Follow a hard mission to the pass

The SEO ticket (G1) was the hardest mission DeepSeek had, and it still landed.

MISSION G1 - pi-deepseekpi@0.80.10deepseek/deepseek-v4-flash:exacto00s$0.003 REJECTIONS - $4.22 TOTALPlanner3m 01s - $0.00BuilderCIReviewerDeployTester
38 MIN COMPRESSED TO 30S - 76X
00s of 38m 12s
G1 - Make the public studio page rank in search - 38m 12s - $4.22 - 3 rejections - PR #708
Planner 181s ──▶ Builder 352s ──▶ CI ✗ failed
                 └─────────────▶ Builder  95s ──▶ CI ✗ failed
                 └─────────────▶ Builder 195s ──▶ CI ✓ ──▶ Review 122s ✗ changes requested
                 └─────────────▶ Builder  95s ──▶ CI ✓ ──▶ Review  51s ✓ ──▶ Deploy ✓ ──▶ Test 527s ✓

38 minutes - $4.22 - 2 CI failures + 1 review rejection - resolved

4 builder attempts and 3 gate rejections produced a resolved ticket for $4.22.

For the same ticket:

  • It's still less than a frontier model's mission cost - claude-opus resolved the same ticket for $6.78.
  • The same DeepSeek model used with open-code resolved it through a smooth run - no rejections from any of the gates.

Results

After 13 configurations, 20 tickets each, 260 mission runs, and about $1,030 of inference orchestrated by SHIP, let's see what the numbers tell.

Compare thirteen builders

Metric
Group by
Color by
Sort

Cost per resolved task

USD, API-equivalent — lower is better
13/13
Claude (frontier)GPT (frontier)Open-weight
Download per-task data (CSV)All 260 runs - builder, model, ticket, cost, wall-clock and whether the held-out tests passed

7 of the 13 builders resolved all 20 tickets. Ranked by cost per resolved ticket, the cheapest of those was deepseek/deepseek-v4-flash running on pi at $3.34 and the priciest that matched it was claude-opus-5 at $5.48, which is the same result for 64% more.

ModelHarnessTotal spendPer resolved ticketvs cheapest
deepseek/deepseek-v4-flashpi$66.86$3.34baseline
gpt-5.6-solcodex$76.64$3.83+15%
deepseek/deepseek-v4-flashopen-code$81.59$4.08+22%
z-ai/glm-5.2open-code$82.73$4.14+24%
moonshotai/kimi-k3open-code$82.85$4.14+24%
claude-sonnet-5claude-code$86.22$4.31+29%
claude-opus-5claude-code$109.66$5.48+64%

The most expensive builder in the study claude-fable-5 running on claude-code has the worst cost per resolved ticket of any configuration at $6.44, and it did not resolve all either: 19 of 20. The cheapest configuration to resolve all 20 did so at $3.34, so Fable cost 92% more for one less ticket resolution.

By family, Open-Weight builders resolved 118 of 120, Anthropic 77 of 80, and OpenAI 55 of 60 in total.

Let the loop do the correcting

BuilderShipped unopposedSent back by a gatePushesResolved
DeepSeek-v4-flash (pi)12 of 2013 times3320 of 20
Claude Opus19 of 201 time2120 of 20
Claude Fable19 of 201 time2119 of 20

The first column counts tickets where nothing in the loop objected - one push from the Builder passes through all the gates. It does not count tickets the model got right, and the gap between those two things is interesting.

The mission that Claude Fable lost was lost silently. On the reminders ticket (F5) it wired the live email provider straight into the request path, so the endpoint returns a 500 whenever the API key is missing, even though the ticket only asks it to queue a row. The repository's own test suite stayed green throughout, which is precisely why CI passed, and the Reviewer approved along with the Tester. Fable did not break anything. It built the wrong thing, correctly, and only the hidden tests could tell.

DeepSeek-v4-flash via pi is the contrast. It was objected to 13 times and shipped nothing broken.

A model that converges under correction turns out to be worth more inside a loop than one that starts closer to the answer.

What makes this practical rather than interesting is the price of those extra pushes. DeepSeek's builder cost about 1% of its own mission bill, because the pinned frontier Planner, Reviewer and Tester dominate the bill, so buying more attempts from a cheap builder costs almost nothing while buying a better first draft costs much more.

Swap the harness, keep the model

Three Open-Weight models each ran on two harnesses - open-code vs pi, which makes this the one place in the study where exactly one variable moves.

ModelHarnessTotal spendMedian wallResolved
deepseek/deepseek-v4-flashpi$66.8620 min20 of 20
deepseek/deepseek-v4-flashopen-code$81.5922 min20 of 20
z-ai/glm-5.2pi$63.4424 min19 of 20
z-ai/glm-5.2open-code$82.7321 min20 of 20
moonshotai/kimi-k3pi$58.9318 min19 of 20
moonshotai/kimi-k3open-code$82.8525 min20 of 20

open-code cost more on all three, by 22%, 30% and 41%, and ran 37% longer on Kimi. It also resolved two more tickets across the 60, so this is a trade-off rather than a ranking.

A result that names the model without naming the harness is describing half of what ran.

Count what the gates caught

Across all 260 runs, 214 landed on the first pass and 46 needed the loop. The gates rejected work 70 times in total: CI 45 times, the Reviewer 21, the Tester 4.

Only four rejections reached the Tester, because CI and the Reviewer had caught nearly everything by then. A loop that fails fast early is what makes a cheap builder viable.

Inspect any mission below to see how the loop progressed, and where its time and money went.

Mission breakdown

Pick any run to see what SHIP records that a one-shot patch benchmark cannot - how many builder attempts it took, where the wall-clock went stage by stage, the reason for every retry, and the proof of delivered work.

Bug: Production TypeError opening a fully-refunded invoice

Harness
claude-code
Model
claude-opus-5
Wall
21m 24s
Cost
$3.65
Turns
1
Mission
pass
Grader
pass
Sent back
never
Diff
+65 -6 PR #789
MISSION B2 - opusclaude-code@2.1.211claude-opus-521m 24s$3.65READY FOR ACCEPTANCEPlanner4m 14s - $0.87Builder7m 06s - $1.29CI1m 26sReviewer48s - $0.34Deploy1m 28sTester3m 12s - $1.14
21 MIN COMPRESSED TO 30S - 43X
21m 24s of 21m 24s
B2 - Bug: Production TypeError opening a fully-refunded invoice - 21m 24s - $3.65 - no rejections - PR #789

Takeaways

The reflex is to use the best model you can afford, and this study suggests that reflex is wrong. You should choose the agent harness and model your loop can correct.

Buying the most expensive model did not buy reliability for the delivery. Fable finished a ticket ahead of Haiku, 19 of 20 against 18, while costing 52% more per resolved ticket. Buying the cheapest one did not save money in all stations either, because even though the longest, the Build is just one stage of what a ticket costs once a real loop surrounds it.

What the money buys is correction. A budget model that starts worse and takes feedback well resolved every ticket in this study, and together with the 13 rejections it absorbed cost less than a single frontier draft would have.

So the question I would ask before picking a model is not which one writes the best code on the first try. It is what plans it, what checks the code afterwards, and whether correction is cheap enough that a rough first draft stops being a problem worth paying to avoid.

Benchmark your own workflow

See what an agent team can deliver in your repository.