Purpose and foundations
From suggestions to repository work
A coding agent uses a model to choose code-related actions, observes their results, and selects further actions toward a task. It might search for a function, inspect its callers, edit an implementation, and run a test before deciding what to do next. What distinguishes this from a predefined workflow is who selects the subsequent operation—not whether execution happens in a loop or in the background. Agent Engineering develops that distinction.
A repository contains a project's version-controlled files and history. Working on a repository means changing an existing system of relationships, not merely producing a self-contained answer. The model proposes operations; surrounding software executes permitted requests and returns their results so the model can choose its next action. This surrounding software is the harness, which supplies instructions, tools, and the execution loop. Agent Runtimes and Harness Engineering explains its execution responsibilities.
| Responsibility | Code suggestion | Delegated repository task |
|---|---|---|
| Locate relevant code | The programmer usually supplies the immediate editing context. | The agent can investigate files and dependencies. |
| Modify and check | The programmer incorporates the suggestion and directs checks. | The agent can edit, execute checks, and investigate failures. |
| Accept the result | The programmer judges the resulting change. | Acceptance remains a separate, accountable decision. |
GitHub's delegated maintenance workflow illustrates this separation: an issue initiates work, the agent prepares a draft pull request and runs tests, and review can request revisions. The documented agent cannot merge its own pull request. This is one concrete allocation of authority, not a requirement that every coding agent use the same interface.
Define behavior and authority
Acceptance criteria describe testable conditions for the requested behavior. The definition of done adds the completion obligations surrounding that behavior, such as required checks and review. State both what should change and what must remain unchanged. A feature adds behavior; a bug fix restores intended behavior; a refactor changes internal structure while preserving observable behavior. That preservation requirement is central to Fowler's definition, not an optional quality goal.
Behavioral requirements and delegated authority answer different questions. Specify whether the agent may inspect files, edit them, execute code, publish a proposal, merge, or deploy. Permission at one boundary need not extend to the next. Likewise, distinguish an implementation detail the repository can answer from a product decision it cannot: locating the authorization check calls for investigation; deciding whether another role should gain access calls for an authorized decision.
| Starting condition | Action | Expected outcome |
|---|---|---|
| The user owns the draft. | Attempt to publish it. | The published article becomes available. |
| The draft belongs to another user. | Attempt to publish it. | The request is refused; the draft stays unchanged. |
Use close collaboration when intent is unsettled or a wrong direction would be expensive to reverse. Bounded background delegation fits clearer tasks with executable checks and a reviewable stopping point. Backlog.md's workflow places reviews at specification, plan, and code; these are useful decision locations, not mandatory ceremony for a tiny edit. Stop when the agreed boundary is reached, a necessary decision is missing, or the attempt budget is exhausted. See requesting the right human decision.
Turning points in automated change
Program synthesis constructs programs from specifications or examples. Automated program repair instead searches for changes to an existing program. Coding agents connect learned generation to repository tools, but inherit an older problem: constructing a candidate and establishing that it meets the intended specification are different achievements.
| Contribution | Date | What changed |
|---|---|---|
| Imperative deductive synthesis — Zohar Manna and Richard Waldinger | 1987 | Extended proof-directed program construction to mutable data by representing computational states explicitly. |
| GenProg — Westley Weimer, ThanhVu Nguyen, Claire Le Goues, and Stephanie Forrest | 2009 | Searched for repairs to existing C programs, using compilation and tests to evaluate candidate edits. |
| Codex — Mark Chen and colleagues | July 2021 | Studied learned Python function generation from docstrings using a model specialized on public GitHub code. |
| SWE-agent — John Yang, Carlos E. Jimenez, and Princeton collaborators | May 2024 | Studied how repository navigation, editing, and execution interfaces affect an agent's ability to repair software. |
In deductive synthesis, a program is constructed from a proof that the specified result exists. Manna and Waldinger's 1987 contribution extended this approach to mutation: changing data can invalidate statements that were previously true. Their hand-derived list reversal required finite lists and restrictions on shared pointers; guaranteeing storage efficiency also required specifying it. Test-directed repair can work without a complete formal specification, but inherits the limits of its tests. Interactive agents add observations from development tools, while still needing a clear account of acceptable behavior and checks that distinguish success from failure.
Repository knowledge
Establish the working baseline
Before editing, identify the state being changed. A commit records a project version; a branch is a movable reference to a line of development. The working tree contains checked-out files, and HEAD identifies the current branch or a detached commit. HEAD identifies the committed baseline; the working tree can also contain uncommitted modifications and untracked files. The index, or staging area, holds content selected for the next commit.
Git status distinguishes staged changes, unstaged modifications, and untracked files. Record these before work begins; their presence does not tell you who owns them. Default output also omits ignored files. Preserve unrelated work rather than treating every difference as disposable agent output. When an operation would overwrite or remove existing content, inspect its scope and resolve ownership first.
A linked worktree is another checked-out directory for the same repository, with its own HEAD and index. Git worktrees separate editable files while sharing repository history and some configuration. They do not isolate processes, credentials, ports, or databases. Two checkouts pointed at one test database can still interfere. A sandbox restricts execution and resource access; its filesystem controls solve a different problem from checkout separation.
The baseline also includes the toolchain, dependency versions, configuration, and required services. A lockfile records dependency resolution; it is not the installed dependency tree. For example, pnpm can update only a lockfile without populating installed packages. Its frozen-lockfile option rejects missing or inconsistent dependency records. Neither an unchanged lockfile nor successful installation establishes that the application can run.
Locate the behavior
Repository investigation should answer the questions that control the change: where a request enters, which implementation handles it, who calls that implementation, what configuration changes its behavior, and which tests exercise it. Model context is the information supplied for the current decision. Selecting for that decision means gathering the relationships needed to justify the next operation, rather than loading as much source as possible.
| Method | Useful question | What the result does not establish |
|---|---|---|
| Exact-text search | Where does this spelling or error message occur? | Symbol identity or runtime reachability. Matches can be comments or unrelated names; filters can exclude files. |
| Definition and reference navigation | Which declaration is this symbol, and where is it referenced? | Complete dynamic behavior, reflective uses, or external consumers. |
| Repository map | Which files and declarations should I inspect next? | Complete source or exhaustive dependency coverage. |
| Semantic search | Where is code related to this behavior when I do not know its spelling? | That a similar-looking candidate uses the intended API or has the required semantics. |
Aider's repository map supplies selected paths and declarations, using dependency relationships to prioritize a bounded overview. The model can then request complete files. This is progressive disclosure: reveal enough structure to find relevant detail before loading it. Context Engineering explains the general method; Search and Retrieval covers meaning-based candidate retrieval.
AGENTS.md provides a designated place for build commands, testing procedures, and project conventions, including nested guidance for subprojects. It is guidance, not enforcement or proof of compliance. A thin index can point toward detailed runbooks, while comments near relevant code can help agents discover them during search—a practice described in Amazon AGI. Keep arbitrary repository text and tool output distinct from authorized instructions.
Follow candidate locations into actual code, inspect callers and tests, and consult documentation matching the installed dependency version. Preserve file references and unresolved questions so another inspection can close the remaining gaps. After an edit or another person's change, revisit affected observations: a previously correct excerpt may no longer describe the workspace. This is the local form of context freshness.
Planning and modification
Plan around dependencies
A useful plan connects the desired behavior to the work needed to establish it. Investigation resolves uncertainty; implementation changes the system; verification checks the resulting obligations. Keep those purposes distinct even when the plan is only a few sentences. For a small, obvious edit, a formal planning phase may add little. For an unfamiliar or cross-cutting change, investigating before committing to an approach can prevent extensive work on a mistaken premise.
File boundaries do not determine task independence. Changing a producer's return contract can require its implementation, consumers, and tests to move together. Dividing those files among workers does not remove their shared decision. Prefer increments whose behavioral obligation is coherent and checkable. Excessively fine decomposition can replace one ambiguous task with many coordination decisions; revise the plan when repository observations contradict its assumptions.
For larger migrations, the OpenHands refactor workflow groups related files into reviewable contributions and orders work around dependencies. Directory grouping is a practical heuristic, not proof that batches are independent. Its accompanying engineering account describes workers starting from an integration branch, submitting reviewed changes there, and incorporating later integration changes. Broader communication and coordination designs belong in Multi-Agent Systems.
Dynamic action selection is not always necessary. Agentless, introduced by Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang in July 2024, used a fixed localization-and-repair workflow. It narrowed a repository overview to declarations and then edit locations, generated patches, filtered invalid candidates, and ranked survivors. Its contribution is a useful architectural alternative: models perform bounded subtasks while ordinary software controls their sequence. Choose autonomy where adapting the next action has value, not merely because a model can call tools.
Make bounded edits
A diff compares file states; a patch describes edits that can be applied. A patch hunk groups a local change with surrounding context. Always identify the comparison: Git diff can compare working files, staged content, or commits, and its three-dot form uses a merge base. These comparisons answer different questions. A diff also does not automatically inventory untracked files.
Choose the editing interface for the transformation. Exact replacement needs current text and a unique match. A context-based patch uses surrounding lines to locate edits. A codemod is a programmatic source transformation; tools such as jscodeshift operate on an abstract syntax tree, a structured representation of source constructs. Codemods suit repeated structural changes, but parser support and target selection still matter. None of these interfaces establishes behavioral correctness by successfully writing files.
Git apply normally rejects a patch without modifying the working tree if some hunks cannot apply. Its explicit --reject option instead permits partial application. An applicability check is not a lock: files can change afterward. Read the selected tool's failure semantics, and inspect the resulting state rather than assuming all-or-nothing behavior.
For example, suppose an internal API changes from returning a complete list to returning one page at a time. Each page contains items and a continuation cursor—a value the caller supplies to request the next page. A consumer that still needs the complete collection must now follow those cursors, not just read the new return shape.
Traverse pages, preserve the complete result
| Consumer | Exchange | Producer |
|---|---|---|
| Complete result: A, B, C | Producer to consumer: Return A, B, C | Complete collection |
| Consumer | Exchange | Producer |
|---|---|---|
| Start with no accumulated items | Consumer to producer: Request first page | Read first page |
| Accumulate A, B; retain c1 | Producer to consumer: Return A, B; cursor c1 | First page |
| Use the returned cursor c1 | Consumer to producer: Request next page with c1 | Read next page |
| Append C once; stop at null | Producer to consumer: Return C; cursor null | Final page |
Complete consumer result in both contracts: ABC
The fewest changed lines would alter only the producer; the smallest complete change also updates its obligations. Stay within the existing implementation boundary, reuse established dependencies, and keep unrelated cleanup separate. Inspect additions, deletions, generated files, manifests, and lockfiles as part of the actual candidate. This applies controlled change to agent-produced edits.
Editing interfaces can materially affect performance. SWE-agent paired bounded edits with immediate file display and rejection of selected lint errors. In its historical GPT-4 Turbo comparison on 300 SWE-bench Lite tasks, the custom interface resolved 18%, versus 11% for a shell-only baseline supplied with a demonstration. The model and task setting were held fixed; the interface changed. Lint acceptance still did not establish that the repaired behavior was correct.
Execution and diagnosis
Interpret execution results
Running code turns a proposal into an observation, but the observation depends on how execution was launched. Record the command and arguments, working directory, relevant configuration, and completion state. Standard output and standard error are separate output streams. An exit status reports process termination according to the command's contract; a terminating signal is another possible outcome. Neither a launch acknowledgment nor an intermediate log line is a completed result.
| Observation | Supported claim | Still unresolved |
|---|---|---|
| Dependency installation completed. | The selected installation operation succeeded. | Required services and application behavior. |
| A build completed. | That build accepted its inputs. | Behavior of an exercised application path. |
| A development server reports readiness. | The server reached its readiness condition. | Whether the requested interaction works. |
| A test runner reports passing assertions. | Those assertions passed in that execution. | Requirements they did not exercise. |
For background work, retain the operation reference and observe its eventual result. If output is truncated, recover the relevant diagnostics or narrow the command; missing output is not evidence that nothing failed. Keep references to produced artifacts so later steps inspect what was actually generated. Operational records should identify relevant settings without copying credentials or unnecessary sensitive content.
Execution also has consequences. Repository scripts and tests can modify files or contact services using available credentials. Keep side effects isolated and prefer test environments without production secrets. A passing test is not worth an accidental live mutation. Sandboxes and Execution Isolation owns those controls; controlled dispatch explains the boundary between a model's requested call and application execution.
Check the intended behavior
A test suite is a collection of automated checks. A regression occurs when a change breaks previously working behavior. Select checks from the requested outcomes and affected dependencies: reproduce the target failure, run relevant existing tests, add coverage for a real gap, and exercise collaborating components where an isolated check cannot establish the result. See observable obligations and integration boundaries.
| Obligation | Useful check | Remaining gap |
|---|---|---|
| Return the selected items. | Check known items and the continuation cursor. | A correct first page does not prove complete traversal. |
| Preserve the consumer's complete output. | Exercise the consumer across multiple pages. | Only tested input conditions are covered. |
| Handle the end of the collection. | Check empty results and the agreed terminal cursor. | The expectation must match the intended API contract. |
The test oracle is the rule deciding whether observed behavior is acceptable. Execution can be independent while expectations are not. If an agent assumes a cursor always exists and writes both implementation and tests around that assumption, the tests may faithfully confirm the wrong contract. More cases do not repair an incorrect expectation. In a VS Code demonstration, generated tests passed, yet the developer still found inadequate route error handling and requested changes to both code and tests.
Plausible patches and correct repairs
Automated repair exposed this distinction before language-model agents. GenProg generated candidate edits, compiled and tested them, and minimized a passing patch. Its motivating greatest-common-divisor program printed the correct value for one input but failed to terminate: output alone was insufficient. The method made existing tests useful search feedback, but passing them remained a bounded acceptance condition.
Qi, Long, Achour, and Rinard's 2015 audit of later repair collections distinguished two failures. Some validation scripts accepted a successful exit without checking output. Other patches really passed the intended tests but failed new counterexamples, often because they removed needed functionality. Their patch-plausibility study explains why a plausible patch, meaning one that passes validation tests, is not necessarily a correct repair. These audited collections were distinct from GenProg's original 2009 experiments.
Review generated tests for durable behavioral expectations, not volume. Requirements can legitimately change, and incidental assertions may deserve removal or revision. But a failing assertion must be explained before weakening it. Curating tests means preserving useful obligations and removing unjustified constraints—not discarding failures until the suite becomes green.
Diagnose before editing again
A red result is a symptom, not a diagnosis. Narrow the failure before editing again: isolate the interaction that fails and inspect what actually happened. Software Engineering Fundamentals explains reproduction and hypothesis-driven debugging.
Compare identical checks on baseline and candidate under matching conditions. Baseline passes: investigate the change. Same failure: investigate shared causes. Baseline untestable: leave the comparison unresolved.
A flaky test produces intermittent outcomes. Shared state, incomplete cleanup, test order, timing assumptions, and concurrency can all contribute. One successful rerun does not identify the cause of the preceding failure. pytest's guidance is useful for separating repeatability problems from straightforward regressions.
When building the coding agent itself, narrow failures similarly. Zed's testing account moves from an end-to-end failure to a focused, repeated model interaction and finally to a conventional regression test when the defect lies in deterministic tooling. That progression turns a vague agent failure into a repairable software boundary.
After a justified fix, rerun the focused check and the broader affected checks. Avoid accumulating speculative changes. The same diagnosis applies in continuous integration (CI), which runs automated checks on repository versions. A CI timeout may need a bounded retry rather than a code edit; a compatibility failure may need a further code change that preserves the original security fix. The PatchPilot account uses this distinction and escalates after a retry limit. Stop earlier when authority, intent, or required execution conditions are unavailable.
Collaboration and review
Steer and resume work
Human intervention is most useful when it changes a consequential decision: the intended behavior is ambiguous, a dependency broadens scope, the proposed design conflicts with an accepted constraint, or the next action exceeds authority. Routine, reversible edits within scope need not each become an approval interruption. Active steering keeps the programmer engaged with implementation direction, while independent execution controls enforce permissions.
A decision-bearing update explains discovery, consequence, and the needed decision. For example: “The requested response change also affects an external client. Changing it now would break that client's parser. I can add a versioned response or coordinate a client update; the choice changes the compatibility promise. I am continuing with the unaffected validation tests.” This is more useful than listing searches because it exposes the decision the agent cannot settle from code alone.
An interruption requires a different kind of handoff: preserve the goal, accepted decisions, changed-file references, observed checks, pending commands, and unresolved work. Context compaction replaces a longer active history with a shorter representation; it can lose details and preserve observations that later become stale. On resumption, read the handoff, inspect current files and changes, and establish which operations remain active. Accepted intent and historical observations have different lifetimes.
Anthropic's long-running coding account uses progress notes, Git history, unfinished feature records, and a basic application check to orient a new session. Apply the same principle after human edits: refresh affected files and revise only the work their changes disturb. Context reconstruction owns the input-recovery method; runtime steering owns delivering accepted direction across long waits.
Review the actual change
A pull request proposes integrating changes through a review interface. The review object is the actual change, not its generated summary. Follow the behavior through producer, consumers, configuration, and tests rather than treating alphabetical file order as an explanation. Chris Kelly's review discussion emphasizes this gap between a list of modified files and an understanding of how the software changes.
Inspect design, required behavior, preserved contracts, and maintainability in surrounding code. Pay particular attention to altered test expectations, dependency or configuration changes, unsafe data handling, and incidental artifacts. Would the tests fail for a broken implementation? Does a new abstraction serve the current requirement? Google's review guidance provides a concrete reference for this examination. Keep the change focused as described in controlled change.
The handoff should connect the symptom or requested behavior, diagnosis, implementation, and supporting checks. State what changed and why, identify the candidate tested, name the checks that actually ran, and retain material unresolved limitations. A long list of commands is less useful than an explanation of which obligations their results cover. The human owner must be able to stand behind that explanation, whether or not an agent helped draft it.
Author self-review, automated review, and accountable acceptance have different roles. Automated review can surface focused findings, but its usefulness depends on both detection capability and whether its comments help the recipient act. Another model's agreement does not make shared assumptions independent. In OpenHands' reported workflow, bot-owned pull requests obscured responsibility and interacted badly with review rules. Assigning the initiating human ownership made responsibility for completion and later breakage explicit.
Verification and integration
Verify the final candidate
Verification provenance connects a result to the source, dependencies, configuration, command, and environment it exercised. A check does not belong permanently to a branch name. If relevant inputs change, the earlier result remains a statement about the earlier candidate. Artifact identity explains this foundation; the practical task is deciding which results still support acceptance now.
| Checked candidate | Relevant inputs | Observation | Claim supported |
|---|---|---|---|
| A | Source A; identified dependencies D and configuration C. | Affected check passed on A. | Checked property supported on A; retain this historical result. |
| B, before rerun | Relevant implementation edit creates source B; D and C held fixed. | Affected check not yet run on B. | B remains unsupported for this property; not-run is neither passed nor failed. |
| B, after rerun | Same source B, D and C; new execution of the affected check. | Affected check passed on B. | Checked property now supported on B; no claim about untested behavior. |
Before packaging, check that the proposed candidate includes every file needed to reproduce the tested behavior. A test that depended on an omitted new file does not verify the delivered patch. Preserve passed, failed, skipped, and unavailable outcomes separately. GitHub's branch protections are configurable: approvals can be dismissed after changes, and accepted status checks can include skipped or neutral outcomes. An accepted badge therefore does not necessarily mean the intended tests executed.
Readiness is a claim about the actual candidate and the agreed completion boundary. When a required check cannot run, report the unsupported claim and its cause instead of replacing it with confidence. The harness's current-outcome check should examine that candidate, not merely whether the agent has finished speaking.
Check the combined result
Integration incorporates an accepted change into the intended shared development version. A merge combines histories; a rebase instead replays changes onto another base. Git can combine files without unresolved textual conflicts, but it does not execute the resulting application. A semantic conflict is an incompatibility in combined behavior, even when the text merges cleanly.
Consider two changes from the same base. One removes a returned display_name field after updating all existing consumers. The other adds an export that reads that field. Each can work independently; their combination can fail even if the new export file produces no textual conflict. Integration must reconcile the shared contract and check the resulting candidate, not merely collect each branch's earlier passing results.
A clean merge can break a contract
ExampleTwo changes can each preserve their own tested behavior while introducing incompatible assumptions when combined.
Read the diagram as text
- Common base. The producer returns display_name.
- A: remove display_name. Updates existing consumers to use other fields.
- B: add export. A new file reads display_name from the producer.
- Combined candidate. The new export is present, but the field it needs is absent.
- Export behavior fails. A combined behavioral check exposes the incompatible assumptions.
- Common base → A: remove display_name: branch from base.
- Common base → B: add export: branch from base.
- A: remove display_name → Combined candidate: combine changed producer.
- B: add export → Combined candidate: combine new consumer.
- Combined candidate → Export behavior fails: exercise export.
CI must check the version that will actually be integrated. GitHub's merge queue tests a proposed change together with the current target and preceding queued changes. Its temporary combined candidate has a different commit identifier from the pull request's head commit. GitHub Actions checks must handle the merge_group event to report on that candidate. This checks the combined source, but only for behavior covered by the configured checks.
Keep the outcomes distinct: applied means files changed; checked means specified observations were obtained; accepted means the responsible reviewer approved; integrated means the intended shared version includes the change; released means the authorized delivery step occurred. Do not infer deployment from merge or permission to deploy from permission to merge. Organization-wide production stages and ownership belong in Software Factories.
Evaluation and usefulness
Build a software-task evaluation
An evaluation systematically assesses behavior against an intended purpose. A case specifies a task and its conditions; a trial is one attempt; a grader assesses designated observations. For coding agents, the system under assessment includes the model, harness, tools, environment, budgets, and human assistance. Case design provides the general framework; here the unit is a software task rather than a final chat response.
A software-task case needs enough information to reconstruct both the opportunity to act and the basis for assessment.
- Starting conditions — Request, repository and base revision, toolchain, installed environment, fixtures, and required services.
- Permitted work — Available information, tools, permissions, human assistance, attempt and time budgets, retry rules, and stopping conditions.
- Delivered result — Final patch or artifact, command observations, assistance received, and remaining incomplete work.
- Assessment — Required behavior, preserved behavior, grader version, and separate task, infrastructure, and integration outcomes.
Separate what the solver may inspect from what assesses its result. The SWE-bench dataset structure distinguishes the issue and base commit from a reference solution and test material; its guide warns against inspecting the reference solution while solving. In your own evaluation, protect assessment checks from silent agent modification. Solver-visible tests provide feedback during work; protected checks assess the delivered candidate under the declared protocol.
Separate solving from assessment
ExampleThe solver's development feedback and the grader's protected checks have different access boundaries.
Read the diagram as text
- Declared task inputs. Issue, base revision, environment, permitted tools, and solver-visible tests.
- Solver workspace. Model and harness work within declared permissions and budgets.
- Delivered candidate. Identified patch or artifact plus retained execution observations.
- Protected assessment material. Outside solver access and mutation: required-behavior checks, preservation checks, and assessment references.
- External grader. Assesses the delivered candidate under the specified protocol.
- Outcome report. Separates task results, execution problems, assistance, and unfinished integration.
- Declared task inputs → Solver workspace: data: permitted inputs.
- Solver workspace → Delivered candidate: data: final output.
- Delivered candidate → External grader: data: assess this candidate.
- Protected assessment material → External grader: data: protected criteria.
- External grader → Outcome report: data: grades and dispositions.
Choose a task mix that resembles the proposed delegation: repairs, features, refactors, and integration work have different burdens. Sampling the intended work explains representativeness. Start repeated trials in clean environments and inspect trajectories alongside grades. A failed setup, incorrect repair, and completed patch awaiting integration are different outcomes, even if none satisfies the full delivery objective.
Keep development separate from final assessment. Once case results have guided prompt, tool, policy, or grader changes, those cases are useful development and regression material. Freeze the configuration before evaluating untouched tasks for a generalization claim. That does not prevent the agent from adapting its actions within a trial according to its declared policy. Clean environments address state contamination; untouched cases address a different form of independence. See protecting assessment.
Read benchmark claims precisely
A benchmark standardizes a task and an assessment protocol, not every property of useful software work. HumanEval, introduced in the 2021 Codex study, evaluates generated functions through executable tests. SWE-bench instead supplies a repository and an issue to resolve, making localization, coordinated edits, and execution part of the task. These are different assessment units, not interchangeable measures of coding ability.
| Benchmark | Task and input | Assessment boundary |
|---|---|---|
| HumanEval | Generate a Python function from its supplied specification. | Tests assess sampled programs; repository integration is not the task. |
| SWE-bench | Edit an identified repository version to address an issue. | Associated repair and preservation checks assess the patch; maintainer acceptance and deployment remain outside the score. |
Fail-to-pass checks fail before the intended repair and must pass afterward. Pass-to-pass checks protect behavior already working. To interpret a resolved-task score, also check how that grader version handles skipped tests and missing results: an accepted outcome does not always mean every test ran. A separate measure, pass@k, asks whether at least one of k sampled candidates passes. It measures candidate coverage, not whether the system selects the passing candidate for delivery or succeeds consistently. The attempt-policy treatment develops the estimator and its assumptions.
Graders can reject valid alternatives as well as accept inadequate repairs. OpenAI's July 2026 SWE-Bench Pro audit found an example whose prompt showed one leading space while hidden tests required two. The report retracted an earlier recommendation to adopt that benchmark. Its detailed review followed automated selection of potentially problematic tasks, so it does not establish a universal defect rate. The general lesson is to inspect whether an expectation is an actual requirement or an accidental property of one reference implementation.
Contamination occurs when evaluation information reaches training or a solving path that violates the intended assessment. Finding a public solution through repository history is different from legitimately consulting API documentation when that is allowed. Tools, network access, retries, and assistance therefore belong in the protocol. SWE-rebench's practical account also highlights environment and configuration failures that can masquerade as capability changes.
Even resource settings change the assessment. Anthropic's infrastructure experiments varied allocations while holding model, harness, and tasks fixed. Additional headroom could prevent transient failures, while larger allocations could enable different solution strategies. Record guarantees, hard limits, and time budgets instead of attributing every score change to the model. Finally, evaluate bug discovery directly if that is the intended work: repairing a supplied issue does not test the same capability as finding an unknown defect.
Measure useful delivery
The final adoption question concerns the complete programmer-agent workflow. Measure task completion, human effort, elapsed time, review, repair, integration, and quality against a credible alternative. Generated lines and pull-request counts can increase without reducing delivery work. Faster implementation may move effort into someone else's review queue, so an individual's apparent gain can differ from the team's outcome.
| Study | Population and task | Finding and scope |
|---|---|---|
| Peng and colleagues, reported February 2023 | A May–June 2022 randomized study assigned 95 professional programmers a JavaScript HTTP-server task, with or without Copilot. | Among 35 completers per arm, mean completion time was 71.17 versus 160.89 minutes: a 55.8% reduction, with a reported 95% confidence interval of 21–89%. Success-rate differences were not statistically significant. Completion required passing 12 visible, unmodifiable tests. |
| METR, early-2025 tools | Sixteen experienced developers worked on 246 predefined tasks in mature repositories they knew well, with AI availability randomized. | Allowing the studied AI tools increased completion time by 19%, despite participants believing afterward that they had saved time. This does not estimate the effect of current tools or every development setting. |
The first study evaluates assistance on a standardized construction task and conditions its time estimate on completion. The second examines experienced contributors working in familiar systems. Neither establishes the long-term effect of autonomous agents across implementation, integration, and later maintenance. Their differences show why local task type, developer familiarity, tool version, and completion criteria matter.
Measurement becomes harder when people select which tasks enter the comparison or work elsewhere while agents run. METR's February 2026 update described participation and task-selection problems, unequal completion, and ambiguous concurrent-time accounting; it treated the later data as an unreliable estimate of the current effect. Agent elapsed time is not automatically human effort, and overlapping work should not be counted twice.
For a local comparison, define completion and quality first, retain unfinished tasks, and account for steering, review, and rework as well as implementation. Use matched-work comparisons where appropriate and choose a defensible live assignment and follow-up design. Track later corrections and regressions rather than ending observation at the first passing patch.
Start bounded delegation with clear, verifiable tasks; use closer collaboration when design or intent needs continuing judgment. Prefer manual work when it is the simpler reliable path for the task at hand. Expand delegation when observed delivery outcomes justify it—not merely when the agent can generate more code. The operating goal remains an understandable change that satisfies its obligations in the version where it will be used.
Open questions
Long-term delivery value remains difficult to establish because early implementation savings can be offset by review, integration, and later maintenance. Progress would mean representative studies that retain unfinished work, account for concurrent human effort, and follow accepted changes for regressions and repair costs.
Behavioral assessment must become stronger without forcing every valid implementation to resemble one reference patch. Incomplete requirements and implementation-specific tests create errors in opposite directions. Progress would combine clearer contracts, counterexample testing, and independent review of both accepted and rejected repairs.
Safe parallelism depends on recognizing semantic dependencies that file separation and import graphs do not fully capture. Progress would identify shared behavioral contracts early and validate combined candidates without making coordination and review cost exceed the benefit of parallel work.
The right amount of autonomy changes with the task, model, and available feedback. Fixed workflows can simplify control, while dynamic agents can respond to discoveries. Progress would compare these policies on matched repository tasks, including intervention effort and unsuccessful outcomes rather than only successful patches.
