Software Factories

Software production as a system

A software factory is an organized system for turning authorized software requests into accepted, delivered changes using people, tools, and automation. Its boundary extends beyond implementation: it must connect incoming work to requirements, checks, delivery decisions, and feedback from operation. Unattended delivery is one possible ambition, not a defining requirement. Eno Reyes's account of the approach at the company Factory emphasizes the organizational work needed to connect an incoming request to deployment, even when an individual change proceeds without human intervention. [3]

A candidate is a proposed change awaiting assessment. Creating one establishes something different from accepting it, delivering it, or benefiting from it. The complete-process perspective introduced in Workflow Automation applies directly: completing an activity is not necessarily completing the requested work.

These distinctions determine the factory's design. If its responsibility ends at a reviewed pull request, it should say so. If it promises working production changes, it must also own the path through integration, release, and confirmation. Increasing implementation capacity without extending those responsibilities merely produces more candidates. [2]

Turning points in repeatable production

Software factories have a longer history than generated code. The name has described organizational arrangements, systematic reuse, model-driven development, and automated delivery. These traditions address different sources of repeated effort; they are not successive versions of one architecture. [7][8]

The unit of repetition differs across these approaches: a component family, a supported product configuration, a model transformation, or an identified release candidate. A factory must choose which of these it can reproduce and assess reliably.

Generated implementation broadens what can be proposed without a prewritten transformation for every change. It does not settle which requests deserve execution, which variations remain compatible, or which results are acceptable. Those decisions explain why earlier production disciplines remain relevant. [9]

Receive requests, admit bounded work

Work intake receives, clarifies, prioritizes, and assigns responsibility for proposed work. Admission is the separate decision to start execution. A request can be understandable but not yet worth starting, authorized but blocked by a dependency, or important but too ambiguous for implementation. Devin's scoping discussion identifies understanding the task, selecting the repository, and seeking clarification as prerequisites to autonomous backlog work. [10]

For an intake decision, establish the intended benefit, scope, permitted repository access, dependencies, consequences of error, and how completion will be assessed. Assign an owner who can resolve uncertainty about the request. Then consider capacity: eligibility does not mean immediate selection. The Kanban Guide uses capacity signals to govern new starts, while the Scrum Guide distinguishes backlog refinement from selecting work. [10][1][11][12]

For example, adding a specified API response format may be ready for implementation. A complaint that the API is slow may instead justify an investigation identifying where time is spent. Investigation is legitimate production work; its deliverable is an answer that supports a later decision, not an unspecified optimization.

Production fit also has historical weight. Michael Cusumano's study of System Development Corporation describes a standardized factory organization begun in 1976 that lapsed in 1978. Uneven incoming work, incompatible customer hardware, management resistance, and insufficient continuing tool investment complicated the arrangement. Standard procedures could not make every incoming project suitable for the same production system. [13]

Specify behavior and preserved guarantees

A specification states the agreed behavior and constraints. An acceptance criterion is an observable condition used to assess the requested result. Describe inputs, outputs, required relationships, operating constraints, and interactions with other systems. Record the reason for consequential requirements so later contributors can distinguish their purpose from an incidental implementation choice. NASA's requirements guidance makes this distinction between what must happen and how well it must happen. [14]

The agreement should cover both the addition and what must remain unchanged, along with exclusions, dependencies, and unresolved choices. Shared completion obligations are different from feature-specific criteria. A Definition of Done states common quality conditions for completed work; one feature's behavior does not replace those conditions. The Scrum Guide uses this distinction without requiring every factory to adopt Scrum. [11][15]

Specification by example expresses a rule through concrete circumstances, an event, and an expected outcome. Cucumber's Gherkin reference connects such examples to executable steps while keeping implementation details out of the requested behavior. Examples clarify an agreement; they do not establish that it is complete. [16]

Keep the agreement inspectable and owned. Kiro's original workflow separates requirements and acceptance criteria from design and dependency-ordered tasks. That separation allows a reviewer to challenge the intended behavior before examining the proposed implementation. When a coding conversation resolves an ambiguity, incorporate the approved decision into the specification instead of leaving it only in the conversation. Generating a test from an implementation does not grant permission to change the requirement that test should assess. [17][18][19]

Separate execution from acceptance

A production stage has inputs, a required result, permitted actions, retained outputs, and a condition for advancement. An artifact is a retained output, such as source changes, a test report, or a deployable package. Investigation, implementation, checking, integration, and delivery are distinct responsibilities, but they need not be separate services, separate agents, or an inflexible sequence that forbids feedback. [20][21]

A coding agent inspects repository information, modifies files, executes tools, and uses observed results to choose further work. It therefore supplies an implementation capability rather than merely generating source text. Coding Agents develops that loop; Agent Runtimes and Harness Engineering explains the environment that keeps it running and preserves its work. A factory can surround one such agent with existing build, test, and release automation. [22]

Required sequencing belongs in executable control flow. If a candidate must pass a check before advancing, the workflow should enforce that condition rather than asking the agent to remember it. The agent can still choose local implementation and repair steps. Deterministic ordering controls which operation may run next; it does not guarantee that the operation succeeds or that its output is correct. [20]

Separate permission to propose from permission to accept. A pull request presents source changes for review and possible integration. GitHub's documented cloud-agent controls allow its agent to work on a branch but not approve or merge pull requests. This is one concrete authority boundary. A factory can also require acceptance of the candidate and separate release authorization before allowing deployment. Those decisions sit outside the implementation agent's authority: an accountable owner can approve, redirect, or stop work even when machines execute every intervening operation. [23][21]

Bound work for recombination

A work package is an assignment bounded around a coherent deliverable. Before dispatch, settle its dependencies, interface obligations, acceptance conditions, and how its output will join the rest of the system. Small assignments are useful when they remain independently assessable. Dividing implementation into tiny tasks and then regrouping everything into one large testing or release batch postpones the feedback that small batches were meant to provide. [24][25]

Reuse depends on similarly deliberate boundaries. Feature-Oriented Domain Analysis, or FODA, was described by Kyo Kang and colleagues at Carnegie Mellon's Software Engineering Institute in November 1990. It studies related applications before constructing reusable assets. Mandatory, optional, and alternative user-visible features describe what is common and what may vary. This makes variation something to design for rather than discover after assembling incompatible pieces. [26]

A software product line carries that reasoning into production: shared architecture and assets, including tests and integration plans, support a bounded family of products. Excessive variation strains those assets; overly narrow scope limits their usefulness. Greenfield and Short's model-driven factories further proposed domain-specific descriptions and transformations supported by prepared frameworks. A model-generated implementation can propose changes outside a prepared transformation, but those proposals still need an explicit agreement about acceptable behavior. [27][8]

The same boundary reasoning governs parallel work. Two independent documentation corrections may proceed separately. A producer and consumer changing a shared API must first agree on its representation and deployment order, even if they edit different files. Git worktrees provide separate checked-out working directories, each with its own index and HEAD; they do not isolate credentials, shared services, or behavioral assumptions. Git's documentation describes workspace separation, not a guarantee of safe recombination. [28][29]

Different workloads justify different arrangements. Factory Missions' architecture uses serial feature implementation with parallel read-only investigation and review, responding to conflicting edits and architectural choices. OpenHands' refactoring workflow decomposes separable contributions and reviews them through a shared integration branch. Neither establishes a universal worker count. Parallel tests need no extra reasoning agents, and multiple agents must justify their coordination cost against simpler complete designs. [30][24]

Carry identity through handoffs

Provenance records an artifact's origin and production history. A receiving stage needs to identify what it received, what produced it, and which observations describe it. The Supply-chain Levels for Software Artifacts (SLSA) build-provenance specification connects identified outputs to build definitions, inputs, and execution details. That is a useful part of a production handoff, but it does not establish behavioral correctness or release authorization. [31]

A summary helps someone navigate this record; it cannot replace it. The claim that tests passed should lead to their observed results for the candidate under consideration. Uber's autonomous pull-request presentation attaches completed checks and screenshots so reviewers can inspect work beyond initial generation. That improves visibility without making the table itself proof of correctness. [33]

The factory therefore needs both candidate-specific verification and artifact retention beyond the worker. The next stage should be able to inspect the result after the original agent session has ended. Git history identifies changes; the work record also identifies what remains unfinished. [32]

Make checks govern advancement

An acceptance gate is an enforced decision about advancing a particular candidate under agreed criteria. Verification assesses conformity to specified requirements; validation assesses suitability for intended use in context. Tests, inspection, analysis, and demonstrations can contribute different evidence. Select them in proportion to the change and its risk rather than treating every available check as equally informative. [4]

A test oracle is the basis for deciding whether observed behavior is correct. It might be an expected value, an invariant, or an informed assessment. Oracles are often partial: a formatting test does not establish access control, and passing selected examples does not settle every operating condition. Evals and Benchmarks explains how to match checks to claims. Here the factory's responsibility is to collect the required evidence and prevent advancement when it is insufficient. [19][34]

Protect the acceptance machinery from the work it assesses. Changes to tests, thresholds, or workflow rules should be reviewed against the specification, not accepted merely because they turn an indicator green. GitHub branch protections can require designated reviewers, dismiss stale approvals, and require another person's approval of the latest push. Their defaults need inspection: administrator bypass may remain available, and required status checks can accept skipped or neutral results. A factory policy requiring an executed behavioral check must enforce more than that status alone. [35]

Human judgment belongs where the criteria are ambiguous, coverage is inadequate, or a tradeoff needs an accountable decision. An agent can gather evidence and propose a disposition; the owner must understand enough to defend accepting the residual risk. Adding a reviewer persona without defining its scope and authority does not create this responsibility. [21][36]

Accept the combined candidate

Integration combines contributions and checks their interactions. It creates a new object of assessment: the combined candidate. A semantic conflict is incompatible behavior despite a clean textual merge. The detailed combined-result treatment applies regardless of whether the contributions came from people, successive sessions of one agent, or concurrent workers. [29]

For example, one contribution changes a producer's output field from total to amount; another builds a consumer against a mock that still supplies total. Each isolated test suite can pass, and the files can merge without conflict. The combination fails because the consumer's assumption no longer matches the producer. An interaction check must exercise their actual exchange. Similar mismatches can occur through configuration or deployment order rather than source text. [29]

Continuous integration, or CI, is the practice of frequently combining contributors' changes and checking them with automated builds and tests. In their September 2000 account, Martin Fowler and Matthew Foemmel described a ThoughtWorks project doing this many times daily. Reproducible build inputs included scripts, configuration, database definitions, and installation files—not just application code. Frequent integration exposed interactions while changes were recent, making them easier to investigate. A failed build still required repair; repeatedly attempting it did not establish successful integration. [37]

A merge queue is one way to manage current combinations. GitHub's queue checks a candidate containing the pull request, the latest target branch, and preceding queued changes. Its merge-group commit differs from the original pull-request commit; Actions workflows must handle merge_group to report required checks. Failed checks or conflicts can remove work from the queue. Exact acceptance depends on configuration, including whether a passing combined group may contain an individually failing contribution. [38]

Assign integration responsibility even if assembly is automated. That owner must resolve incompatible assumptions and accept the resulting combination, rather than add up earlier approvals. OpenHands' refactoring workflow makes this visible by reviewing contributions into an integration branch before proposing the accumulated change to main. [24]

Revalidate changed inputs

Evidence invalidation means an old result no longer supports current acceptance; its historical record remains. Requirements traceability links requirements, implementation, and checks. Revalidate affected work and approvals; broaden checking when impact is uncertain. [39]

Changed source, dependencies, configuration, or requirements can change what a result supports. A package tested with one dependency version has not thereby been tested with another. An approval of one diff may not apply after another push. GitHub's option to dismiss stale reviews is a narrower concrete instance of this general rule. [31][35]

For example, approved changes to report columns require renewed report acceptance. A documentation result remains applicable only if its artifact, inputs, and obligations are unchanged. [39]

Requirement revisions need their own approval. They must reach affected workers and checks before acceptance resumes. Otherwise an agent can appear to succeed by changing the stated goal to fit what it produced. Living specification reviews, as described in Kiro's engineering practice, make changed intent and its design consequences visible. [40]

Authorize release and confirm its outcome

Release responsibility covers authorizing exposure, checking its outcome, and directing recovery. Deployment installs a version into an environment; release makes functionality available to intended users. Feature flags can separate them. A deployment acknowledgment therefore does not establish user exposure, correct behavior, or the benefit that motivated the change. [25][21]

Promotion advances an identified candidate through environments or approval stages. Farley's pipeline retained binaries rather than recompiling them at every stage, avoiding accidental compiler or dependency changes between checks. Its release-candidate identifier connected source, binaries, configuration, and deployment scripts. The same discipline connects today's accepted candidate to the actual release; Software Engineering Fundamentals develops that identity boundary. [41]

Authorization must control the deployment path. GitHub environment protection rules can require approval, restrict branches or tags, and withhold environment secrets until approval. Coverage depends on jobs referencing the environment and on the actual credential paths. Administrator bypass is enabled by default unless disabled, and preventing self-review is a separate option. The existence of an environment name is not sufficient enforcement. [42]

Once deployment starts, observe the candidate separately. A canary release exposes a bounded portion of production traffic while the existing version provides a control. Google's canary guidance connects version-specific errors, latency, and resource use to advance-or-stop decisions. Define indicators, sufficient observation, and recovery authority before exposure. Small or unrepresentative traffic samples can miss defects, and shared dependencies can confound the comparison. [43]

Expand ⤢

Operational readiness concerns the ability to run and support the service: dependencies, monitoring, capacity, emergency response, documentation, and prepared operators. Google's production-readiness review is a prerequisite to a site reliability engineering (SRE) team accepting production responsibility. This is distinct from initial release approval: a service may already be running when another team agrees to operate it. A smaller organization can assign these duties without a separate SRE department. Broader staffing and accountability choices belong in AI Engineering Leadership; the factory still needs a named owner for this release and its recovery. [44]

Route exceptions to the right owner

An exception is work requiring treatment outside its expected path. The appropriate response depends on what remains unresolved. Repeating implementation will not clarify a disputed requirement or restore an unavailable test runner. An audit record should preserve the affected artifacts, failed obligation, observations, known completed effects, and next decision so the recipient can act without reconstructing the entire run. [45][46]

Notification is not accepted responsibility. PagerDuty's incident lifecycle separates triggering, acknowledgment, and resolution: acknowledgment records that a responder has claimed the still-unresolved incident, while lack of acknowledgment permits escalation to continue. A factory needs the same distinction when handing work to a person or another team. Track the exception until someone has accepted it and the blocker has a recorded disposition. [48]

Clarification changes understanding; repair changes a defective artifact; retry repeats an operation under a justified policy; rejection ends the proposed path; takeover transfers execution responsibility. Set retry bounds rather than permitting indefinite repair. For uncertain effects, reconcile first. Compensation performs new corrective actions and may preserve concurrent changes rather than restore an exact past state. It can also fail and require an operator. [47][49]

Exceptions remain part of the workload while they wait. Their owners need time and usable diagnostics, not just notifications. The broader treatment of exception capacity explains why a low exception count can still consume substantial human effort. [50]

Improve the production arrangement

Repairing an item changes the software under production. Improving the factory changes how future work is admitted, supplied, checked, or delivered. A recurring misunderstanding may need a clearer specification. Repeated integration failures may need a shared interface check. A task class that routinely exceeds available tools may need narrower admission rather than more retries. [51][52]

Treat operational signals as proposals for investigation, not permission for arbitrary edits. In Agents Building Agents, failure clusters and suggested root causes are reviewed with subject-matter experts before being fixed, deferred, or discarded. A negative user rating can reflect intended behavior or an incorrect diagnosis. Confirmed traces then become regression cases, followed by observation of the change in production. [53][54]

Preventive work needs an owner and a completion condition. Google's postmortem guidance recommends concrete, prioritized, tracked actions rather than vague instructions to improve. The factory can apply that discipline to a shared check, reusable component, task policy, or deployment safeguard. Resolving the immediate defect and completing the preventive action are separate outcomes. [55]

Apply the same versioning and acceptance boundaries to changes in the factory itself. An agent proposing a weaker gate is proposing a production-policy change, not simply repairing its task. Keep the previous configuration recoverable, review the intended effect, and test the revised arrangement before expanding its use. These improvements can change instructions, tools, or workflow rules without retraining a model. [51][56][57]

Regression cases check whether known failures recur; independent assessment asks whether the revised system also improves on other work. Once failures or scores have guided revisions, that evaluation has participated in development—even if individual cases remained hidden. Retain useful failures as regression tests. For a straightforward independent assessment, use fresh cases sampled from the intended workload that have not guided the revisions. Protecting independent assessment explains this separation, and failure-driven change decisions connects it to adoption. [58]

Control unfinished work

Throughput counts completed items per unit time at a declared boundary. Work in progress counts started but unfinished items; work-item age measures time since an unfinished item started. Here, cycle time means elapsed time from the chosen start to finish, including waiting. Define whether work starts at admission or implementation and whether it finishes at review, deployment, or accepted delivery. Those choices change what the measurements mean. [1]

The limiting stage may be implementation, testing, review, integration, or release. Tasks differ in size, blocked work waits, and rejected candidates revisit stages that have already spent effort on them. Backpressure limits upstream starts when downstream capacity cannot handle the arriving work. Uber's software-factory account identifies CI capacity, feasible experiment volume, and product decisions as continuing constraints after implementation becomes easier. [59][2][1]

A queue with repeated work

Consider a deliberately fixed review model. Reviewers can perform six equally sized review attempts per workday. Each day they first accept two repaired items returned from the preceding day. They then review four previously unreviewed items: two pass and two need repair, returning ready the next morning. The initial unfinished population is two repaired items. Compare admitting six new items daily with admitting four; every other assumption stays fixed.

Both policies produce four review acceptances per day: two first-pass acceptances and two after repair. With six new arrivals, unfinished work grows by two items daily. With four, it remains at two. After five days, both have accepted twenty items, but their unfinished populations are twelve and two respectively. These are review outcomes, not claims of production delivery. The extra arrivals create waiting without increasing completion.

Matching average arrivals to capacity preserves this model's backlog; draining it requires spare capacity. Real work also varies, so the example's fixed rates are not a staffing target. Track age, waiting by stage, review effort, and first-pass acceptance—the fraction of assessed items accepted without return. Small, coherent changes and fewer unnecessary returns can improve the complete path. The underlying capacity and backlog-recovery principle applies to people as well as machines. [25][50]

Consistent accounting

Measure the complete arrangement

Compare the factory with a credible existing workflow on comparable work. Keep task mix, change size, and completion rules visible, and retain unsuccessful, abandoned, and unfinished items in the accounting. Evals and Benchmarks explains why output volume, labor, elapsed time, quality, and usefulness are separate outcomes. A factory can improve one while worsening another. [61][62]

The DevOps Research and Assessment (DORA) delivery metrics help separate delivery speed from instability. Change lead time measures elapsed time from commit to production, and deployment frequency measures how often deployments occur. Failed-deployment recovery time measures how long recovery takes after a failed deployment. Change fail rate measures the proportion of deployments needing immediate intervention; deployment rework rate measures the proportion of unplanned deployments caused by production incidents. Interpret these for an application or service, not as interchangeable team rankings. Commit-to-production time excludes intake, deployment rework excludes much review effort, and none directly measures product value. [63]

METR's February 2026 experiment-design update illustrates the measurement difficulty. Selective participation and withheld tasks changed the observed workload; some tasks remained incomplete; concurrent agent use complicated labor accounting. METR consequently described its central estimate as an unreliable proxy for the real productivity effect. Timing an agent while a developer does other work is not the same as measuring the developer's active labor. [61]

Hao He and colleagues' Cursor longitudinal study, in its January 2026 version, compared 806 repositories with observable adoption against 1,380 matched controls. It estimated transient increases in code-output measures alongside persistent increases in static-analysis warnings and complexity. Adoption was inferred from configuration files, usage intensity was unknown, and causal interpretation depends on observational assumptions. Static warnings are not production incidents. The useful lesson is to measure output and downstream quality separately over time. [62]

Include effort transferred into specification, scenario maintenance, and exception handling before declaring a saving. Detailed cost per accepted outcome belongs in workload economics. Here the decision is whether the complete production arrangement delivers acceptable work more effectively—not whether its fastest stage became faster. [64]

Choose the boundary of unattended work

The justified automation boundary depends on several conditions together: clear intent, bounded changes, compatible contributions, adequate checks, controlled release authority, feasible recovery, and capacity for exceptions. Strong offline checks cannot resolve ambiguous requirements. More reviewers cannot undo an irreversible action. More implementation agents cannot clear a review queue whose effective capacity is already exhausted. [10][30][47][50]

Factory Missions describes clarification and plan approval followed by orchestrated execution that users can monitor and interrupt. Its documentation identifies a scriptable application and dependencies as prerequisites for reliable user-facing verification, and recommends ordinary sessions for straightforward work. This design retains an explicit planning boundary without requiring continuous supervision of every implementation action. [65]

StrongDM's reported design instead removes human code review and emphasizes end-to-end scenarios, often outside the codebase, with model-assisted assessment. Its behavioral replicas of services such as Okta, Jira, and Slack permit controlled testing and failure injection. These replicas emulate service behavior; they are not necessarily learned world models. Their fidelity and the validity of the judges constrain what passing scenarios establish. The account does not demonstrate that this allocation preserves quality or improves complete delivery across other organizations. [66]

The comparison is about where scrutiny occurs, not which design has the stronger autonomy label. Planning, scenario maintenance, acceptance, exceptions, and operating ownership still require work. Someone must be able to explain why the evidence is sufficient and own what reaches production, even if no person reads every line. [21]

Expand unattended execution one supported class of work at a time. Start with a real bottleneck, explicit completion conditions, and an operable exception path. Compare accepted delivery, human effort, waiting, and later quality before expanding the boundary. Treat the factory as a product for its engineering users: retain what helps, revise what transfers work elsewhere, and keep investment and staffing decisions connected to leadership responsibilities. [57][67]

Open questions

1. 01

   Selective revalidation needs trustworthy dependency information across requirements, source, configuration, and checks. Missing relationships can preserve stale permission; excessive invalidation consumes scarce capacity. Progress would make these relationships inspectable and test whether targeted checking misses consequential changes. [39]

2. 02

   Replacing code inspection with behavioral assessment shifts work into scenario design, judge validation, and environment maintenance. Establishing the useful boundary requires comparable delivery outcomes, including that labor and later defects—not merely successful autonomous runs. [65][21]

3. 03

   Productivity measurement must handle selective task admission, incomplete work, and people supervising concurrent executions. Progress would compare credible alternatives while preserving task populations, active labor, elapsed delivery time, and follow-up quality as separate observations. [61]

4. 04

   Factory improvement uses the same checks whose limitations it is trying to overcome. Repeated optimization can fit known failures while weakening independent assessment. Progress would combine protected acceptance rules, useful regression cases, and fresh evaluations of revised production policies. [58][51]