How to build an AI software factory: a roadmap in four stages

To build an AI software factory, move in four stages. Assess your repositories, enable them for agents, run a delivery pilot, and then operate the factory as a standard lane. Each stage has a clear start, a short list of what it needs, and a decision at the end.

I build SHIP with SHIP. Its agents take tickets from SHIP's own backlog and open pull requests on its own repository. In my runs, the model was rarely the reason a mission failed. The ticket text, the CI or the repository usually was. This guide puts those lessons in the order a team meets them on the way to an AI SDLC.

The numbers come from SWE-in-a-team, SHIP's study of 260 missions: 20 tickets for each of 13 builder configurations.

Plan the roadmap in four stages

Take the stages in order. A team that starts a pilot before its CI is stable pays for each flaky test in agent runs. Each extra round is a full run (retry limits).

See what changes for your engineers

In a software factory, the main output of your engineers is no longer code. It is the spec that goes in, the guardrails around the line, and the decision to accept what comes out.

The diagram shows the line SHIP runs. The Planner reads the issue and writes a plan. The Builder writes the change in an isolated sandbox. CI, the Reviewer and the Tester check the work, and a gate that rejects it sends it back to the Builder with the feedback. When every gate passes, the pull request waits for a person.

The work of your engineers moves to the two ends of the line: what goes in, and what they accept.

What is an AI software factory explains each part, and Agentic SDLC, defined follows one change through the loop. This guide covers the order in which you put the parts in place.

Find the constraint before you buy more agents

Most teams that ask me where to start already pay for coding agents. Their engineers write code faster, and the pull requests wait longer. Faros AI measured that "PR review time increases 91%" on teams with high AI adoption. The data is telemetry from more than 10,000 developers across 1,255 teams (The AI Productivity Paradox Report 2025).

A software factory is like 100 new engineers who join your team on the same day. They open pull requests right away. Each pull request needs a CI run, a review, a preview environment and a place to deploy. Your systems and your DevOps team carry all of that load at once. It is the same pressure as 100 new hires who start to ship code together.

Faster agents move the constraint from code to review. A factory puts that check inside the line, so agents do the first rounds of review and test before a person reads anything.

The second reason a pilot stalls is the choice of work. A pilot on tickets that no test can prove done ends in a debate about whether the work is done. In my own runs, a criterion that the preview could not show stopped the mission and sent it to a person.

Assess your repositories for agents

Start with facts about each repository. The question for this agent readiness check is narrow: if an agent took a normal ticket in this repository today, what would stop it?

Check each repository against this list. Each item is a gate the factory depends on, or a failure I saw in a real mission.

• One documented command builds the project and runs its tests from a clean checkout.

  An agent starts from a fresh clone, so a step that exists only on one laptop fails.

• Every install uses a committed lockfile.

  The same versions install on every run, so each environment stays reliable.

• CI runs on every pull request, ends in minutes and has stable required checks.

  A flaky test sends good work back to the Builder, and each round costs a full agent run.

• A failed CI run has logs that the agent can read.

  A bare "CI failed" gives the Builder nothing to fix until the retry limit stops it.

• Tests check behavior, not only that the code compiles.

  The Reviewer and the Tester can confirm only what a test or a preview can show.

• Each pull request gets a preview environment.

  The Tester checks the change the way a user would, on a live copy of the app.

• Tickets have acceptance criteria that a test can check.

  The Planner and the Tester grade against them, and vague text sends the agents in circles.

• Review has a named owner and a time limit.

  The factory opens more pull requests, so the wait for a person grows first.

Record two numbers before you change anything: the lead time from ticket to merge, and the time a pull request waits for its first review. You measure the pilot against this baseline.

The milestone at the end of the assessment is a clear list of what your environment needs for agentic engineering. Use it to pick the pilot repository: the one that passes the most checks, not the one with the most urgent backlog.

Enable your codebase and your engineers

Enablement fixes the blockers from the assessment, in the order of the list. Most of the work is ordinary engineering discipline. Agents only make its absence more expensive.

Start with the instruction file. AGENTS.md at the repository root holds the conventions that every agent needs. Every agent picks it up by default (agent prompts). Keep it short and specific, because every line costs context on every run.

# Conventions

## Commands
- Install: `pnpm install --frozen-lockfile`
- Check before a push: `pnpm lint && pnpm typecheck && pnpm test`

## Code
- TypeScript strict mode. No `any`: use `unknown` and a type guard.
- One module does one job. Split a file before it passes 500 lines.
- Every new function with logic gets a unit test next to it.

## Boundaries
- Do not edit `.github/workflows/`. Do not skip or delete a test to make CI pass.
- Database changes go in a new migration file. Never edit an applied one.

To add rules for one role, give that role its own prompt entry in ship.yml. SHIP adds the file to what the role already gets from AGENTS.md:

agents:
  builder:
    prompt: .ship/agents/builder.md

Then change how your team writes tickets. A ticket the factory can finish names the outcome and the way to check it, not the steps to get there.

---
name: Agent task
about: A change the factory can build and verify
---

## Outcome
What a user can do after this change.

## Acceptance criteria
- [ ] A result that a test or the preview can prove
- [ ] An edge case, with the exact expected result

## Out of scope
What the change must not touch.

Write criteria about results, not methods. The ticket is the source of truth for the whole SDLC. The Builder builds to it, the Reviewer reviews against it, and the Tester tests to confirm it. So an instruction in a ticket becomes a criterion for every agent. In one of my missions, a note about how to do the work became a check that the change had to pass. A correct change then failed on a detail no user cared about.

Last, harden CI from the assessment list: required checks, readable logs on a failed run, and no flaky test in the required set. The milestone is one ticket that agents take from issue to a pull request, with your conventions.

Orchestrate a delivery pilot on one backlog slice

The pilot is where the chain of agents runs on real work. Tickets go in, and reviewed and tested pull requests come out, each with its proof and its cost.

Take the slice from work a test can prove: bug fixes with a reproduction, dependency upgrades, migrations, and small features with exact criteria. The use cases list ready-made slices with their acceptance criteria already written.

To keep a human in the loop after a specific stage, set a checkpoint on that role. The mission holds when the role finishes, and a person reads its result before the next stage starts. You can set a checkpoint on the Planner, the Builder (after CI) and the Reviewer (human in the loop). This example holds the mission after the plan and after the review:

agents:
  planner:
    checkpoint: true
  reviewer:
    checkpoint: true

Keep the retry limits at their defaults. A mission that fails the same way twice stops and goes to a person. The docs say that this usually costs less than five failed rounds (retry limits).

Expect the loop to do most of the correction. In SWE-in-a-team, it fixed 40 of the 46 runs its gates rejected, with no person involved. Among the configurations that resolved all 20 tickets, the cost per resolved ticket was $3.16 to $5.48.

Expect it to miss some things too. 4 of the 260 runs passed every gate and were still not complete, and only the hidden acceptance tests caught them. So read each pull request in the pilot, and keep a list of every place the agents stopped, and why. That list is the backlog for the next round of enablement.

Measure the pilot against the baseline from the assessment. Track the cost per accepted change, the cycle time from ticket to merge, and the share of pull requests accepted as they are. Divide the cost by the changes you accepted, not by the ones the agents opened. The milestone is a software factory floor where you orchestrate agents for a division of work.

Operate the factory as a service for your team

After the pilot, the factory is a lane in how your team delivers. Tickets of the proven types go to it by default, and one person who owns the configuration decides when a new type joins.

SHIP records the cost, the duration and the result of each stage. It attributes them to the harness, model and prompt that ran (capabilities). That makes the quarterly review concrete. You compare configurations on your own tickets, not on a public benchmark.

Widen in one direction at a time: one new ticket type, or one checkpoint fewer, never both at once. When a number gets worse, you then know which change caused it.

The last decision is how much the factory merges with no human in the loop. By default, a person accepts every pull request. Auto-merge, for a project or for one mission, lets a change that passes review and QA merge itself. It stays off until you turn it on (human in the loop). Before you turn it on, read what a dark software factory misses.

The milestone is software delivery on several projects, with or without a human in the loop.

Choose to build the line or run it on SHIP

You can build the line yourself from open parts: tracker webhooks, your own CI, a container for each run and a headless coding agent. What is an AI software factory shows the Claude Code version step by step. The part you write yourself is the orchestration, with the retry limits, the gates and the cost ledger.

SHIP runs that line as a managed service on your repository, with the coding agents and model keys you choose. Stages 1 and 2 are work in your own repositories either way. SHIP matters from stage 3, when the pilot needs the gates and a cost record for each change.

You now have the roadmap and an agent readiness checklist for each repository. Run the checklist on one repository first, and write down the two baseline numbers before you change anything.