# How to build an AI software factory: a roadmap in four stages > A four-stage roadmap from a first readiness check to a factory that runs every day. See what each stage needs, when it starts and what it delivers. - URL: https://letsship.ai/blog/how-to-build-an-ai-software-factory - Author: Önder Ceylan - Published: 2026-10-19 To build an AI software factory, move in four stages. Assess your repositories, enable them for agents, run a delivery pilot, and then operate the factory as a standard lane. Each stage has a clear start, a short list of what it needs, and a decision at the end. I build SHIP with SHIP. Its agents take tickets from SHIP's own backlog and open pull requests on its own repository. In my runs, the model was rarely the reason a mission failed. The ticket text, the CI or the repository usually was. This guide puts those lessons in the order a team meets them on the way to an AI SDLC. The numbers come from [SWE-in-a-team](https://letsship.ai/blog/swe-in-a-team), SHIP's study of 260 missions: 20 tickets for each of 13 builder configurations. ## Plan the roadmap in four stages Take the stages in order. A team that starts a pilot before its CI is stable pays for each flaky test in agent runs. Each extra round is a full run ([retry limits](https://letsship.ai/docs/configuration/retry-limits)). ## See what changes for your engineers In a software factory, the main output of your engineers is no longer code. It is the spec that goes in, the guardrails around the line, and the decision to accept what comes out. The diagram shows the line SHIP runs. The Planner reads the issue and writes a plan. The Builder writes the change in an isolated sandbox. CI, the Reviewer and the Tester check the work, and a gate that rejects it sends it back to the Builder with the feedback. When every gate passes, the pull request waits for a person. > The work of your engineers moves to the two ends of the line: what goes in, and what they accept. [What is an AI software factory](https://letsship.ai/blog/what-is-an-ai-software-factory) explains each part, and [Agentic SDLC, defined](https://letsship.ai/blog/agentic-sdlc) follows one change through the loop. This guide covers the order in which you put the parts in place. ## Find the constraint before you buy more agents Most teams that ask me where to start already pay for coding agents. Their engineers write code faster, and the pull requests wait longer. Faros AI measured that "PR review time increases 91%" on teams with high AI adoption. The data is telemetry from more than 10,000 developers across 1,255 teams ([The AI Productivity Paradox Report 2025](https://www.faros.ai/blog/ai-software-engineering)). A software factory is like 100 new engineers who join your team on the same day. They open pull requests right away. Each pull request needs a CI run, a review, a preview environment and a place to deploy. Your systems and your DevOps team carry all of that load at once. It is the same pressure as 100 new hires who start to ship code together. Faster agents move the constraint from code to review. A factory puts that check inside the line, so agents do the first rounds of review and test before a person reads anything. The second reason a pilot stalls is the choice of work. A pilot on tickets that no test can prove done ends in a debate about whether the work is done. In my own runs, a criterion that the preview could not show stopped the mission and sent it to a person. ## Assess your repositories for agents Start with facts about each repository. The question for this agent readiness check is narrow: if an agent took a normal ticket in this repository today, what would stop it? Check each repository against this list. Each item is a gate the factory depends on, or a failure I saw in a real mission. An agent starts from a fresh clone, so a step that exists only on one laptop fails. The same versions install on every run, so each environment stays reliable. A flaky test sends good work back to the Builder, and each round costs a full agent run. A bare "CI failed" gives the Builder nothing to fix until the retry limit stops it. The Reviewer and the Tester can confirm only what a test or a preview can show. The Tester checks the change the way a user would, on a live copy of the app. The Planner and the Tester grade against them, and vague text sends the agents in circles. The factory opens more pull requests, so the wait for a person grows first. Record two numbers before you change anything: the lead time from ticket to merge, and the time a pull request waits for its first review. You measure the pilot against this baseline. The milestone at the end of the assessment is a clear list of what your environment needs for agentic engineering. Use it to pick the pilot repository: the one that passes the most checks, not the one with the most urgent backlog. ## Enable your codebase and your engineers Enablement fixes the blockers from the assessment, in the order of the list. Most of the work is ordinary engineering discipline. Agents only make its absence more expensive. Start with the instruction file. `AGENTS.md` at the repository root holds the conventions that every agent needs. Every agent picks it up by default ([agent prompts](https://letsship.ai/docs/configuration/agent-prompts)). Keep it short and specific, because every line costs context on every run. ```markdown title="AGENTS.md" # Conventions ## Commands - Install: `pnpm install --frozen-lockfile` - Check before a push: `pnpm lint && pnpm typecheck && pnpm test` ## Code - TypeScript strict mode. No `any`: use `unknown` and a type guard. - One module does one job. Split a file before it passes 500 lines. - Every new function with logic gets a unit test next to it. ## Boundaries - Do not edit `.github/workflows/`. Do not skip or delete a test to make CI pass. - Database changes go in a new migration file. Never edit an applied one. ``` To add rules for one role, give that role its own prompt entry in `ship.yml`. SHIP adds the file to what the role already gets from `AGENTS.md`: ```yaml title="ship.yml" agents: builder: prompt: .ship/agents/builder.md ``` Then change how your team writes tickets. A ticket the factory can finish names the outcome and the way to check it, not the steps to get there. ```markdown title=".github/ISSUE_TEMPLATE/agent-task.md" --- name: Agent task about: A change the factory can build and verify --- ## Outcome What a user can do after this change. ## Acceptance criteria - [ ] A result that a test or the preview can prove - [ ] An edge case, with the exact expected result ## Out of scope What the change must not touch. ``` Write criteria about results, not methods. The ticket is the source of truth for the whole SDLC. The Builder builds to it, the Reviewer reviews against it, and the Tester tests to confirm it. So an instruction in a ticket becomes a criterion for every agent. In one of my missions, a note about how to do the work became a check that the change had to pass. A correct change then failed on a detail no user cared about. Last, harden CI from the assessment list: required checks, readable logs on a failed run, and no flaky test in the required set. The milestone is one ticket that agents take from issue to a pull request, with your conventions. > A factory must work on a repository that has never heard of it, and SHIP does. Enablement is not a condition for access. It raises the share of tickets that pass every gate without help. ## Orchestrate a delivery pilot on one backlog slice The pilot is where the chain of agents runs on real work. Tickets go in, and reviewed and tested pull requests come out, each with its proof and its cost. Take the slice from work a test can prove: bug fixes with a reproduction, dependency upgrades, migrations, and small features with exact criteria. The [use cases](https://letsship.ai/use-cases) list ready-made slices with their acceptance criteria already written. To keep a human in the loop after a specific stage, set a checkpoint on that role. The mission holds when the role finishes, and a person reads its result before the next stage starts. You can set a checkpoint on the Planner, the Builder (after CI) and the Reviewer ([human in the loop](https://letsship.ai/docs/configuration/human-in-the-loop)). This example holds the mission after the plan and after the review: ```yaml title="ship.yml" agents: planner: checkpoint: true reviewer: checkpoint: true ``` Keep the retry limits at their defaults. A mission that fails the same way twice stops and goes to a person. The docs say that this usually costs less than five failed rounds ([retry limits](https://letsship.ai/docs/configuration/retry-limits)). Expect the loop to do most of the correction. In SWE-in-a-team, it fixed 40 of the 46 runs its gates rejected, with no person involved. Among the configurations that resolved all 20 tickets, the cost per resolved ticket was $3.16 to $5.48. Expect it to miss some things too. 4 of the 260 runs passed every gate and were still not complete, and only the hidden acceptance tests caught them. So read each pull request in the pilot, and keep a list of every place the agents stopped, and why. That list is the backlog for the next round of enablement. Measure the pilot against the baseline from the assessment. Track the cost per accepted change, the cycle time from ticket to merge, and the share of pull requests accepted as they are. Divide the cost by the changes you accepted, not by the ones the agents opened. The milestone is a software factory floor where you orchestrate agents for a division of work. ## Operate the factory as a service for your team After the pilot, the factory is a lane in how your team delivers. Tickets of the proven types go to it by default, and one person who owns the configuration decides when a new type joins. SHIP records the cost, the duration and the result of each stage. It attributes them to the harness, model and prompt that ran ([capabilities](https://letsship.ai/docs/cli/capabilities)). That makes the quarterly review concrete. You compare configurations on your own tickets, not on a public benchmark. Widen in one direction at a time: one new ticket type, or one checkpoint fewer, never both at once. When a number gets worse, you then know which change caused it. The last decision is how much the factory merges with no human in the loop. By default, a person accepts every pull request. Auto-merge, for a project or for one mission, lets a change that passes review and QA merge itself. It stays off until you turn it on ([human in the loop](https://letsship.ai/docs/configuration/human-in-the-loop)). Before you turn it on, read [what a dark software factory misses](https://letsship.ai/blog/what-is-a-dark-software-factory). The milestone is software delivery on several projects, with or without a human in the loop. ## Choose to build the line or run it on SHIP You can build the line yourself from open parts: tracker webhooks, your own CI, a container for each run and a headless coding agent. [What is an AI software factory](https://letsship.ai/blog/what-is-an-ai-software-factory) shows the Claude Code version step by step. The part you write yourself is the orchestration, with the retry limits, the gates and the cost ledger. SHIP runs that line as a managed service on your repository, with the coding agents and model keys you choose. Stages 1 and 2 are work in your own repositories either way. SHIP matters from stage 3, when the pilot needs the gates and a cost record for each change. You now have the roadmap and an agent readiness checklist for each repository. Run the checklist on one repository first, and write down the two baseline numbers before you change anything. ## Questions ### How do you build an AI software factory? Build it in four stages. First assess your repositories for agents. Then enable them with an instruction file, fast CI and testable tickets. Run a delivery pilot on one part of your backlog, and then operate the factory as a standard delivery lane. Each stage ends with a decision, so you widen the scope only when the numbers support it. ### Where should a company start with an AI SDLC? Start with an assessment of one or two repositories, not with a tool purchase. Check that each repository builds from a clean checkout, that CI is fast and required, and that tickets have acceptance criteria a test can check. Those three facts decide how much of the work agents can finish, in an agentic or an autonomous SDLC alike. ### What does a repository need before AI agents can work on it? A repository needs a build that works from a clean checkout, and CI that is fast, stable and required. It also needs tests that check behavior and an AGENTS.md file with its conventions. Each ticket needs acceptance criteria that a test or a preview can prove. A preview environment for each pull request lets a test agent check the change as a user would. ### What is a chain of agents in software delivery? A chain of agents gives each delivery stage to its own AI agent, with CI between them. In SHIP, the agents are a Planner, a Builder, a Reviewer and a Tester. A stage that rejects the work sends it back to the Builder with the feedback. In SHIP's 260-mission SWE-in-a-team study, that loop fixed 40 of the 46 runs its gates rejected. ### What is a software factory model? A software factory model describes how work moves from intent to a shipped change, and who does each part. In an AI software factory, people write the specs, set the guardrails and accept each change. AI agents plan, build, review and test the work between those points. ### Can you get an AI software factory as a service? Yes. SHIP runs the agent line as a managed service on your own repository, with the coding agents and model keys you choose. Your team keeps the specs, the guardrails and the decision to merge, and SHIP records the cost and the result of each change.