Execution responsibilities
What the runtime promises
An agent selects actions from observations and retained information. Application code executes those actions and returns observations for the next choice, forming the loop explained in Agent Engineering. The harness supplies the software and configuration around that loop: tools, retained state, execution controls, and checks. Within it, the runtime manages when work runs and how its lifetime is controlled; it need not be a separate service. Here, a worker is a process that carries out assigned work. An evaluation harness serves a different purpose: it runs cases and assesses results rather than managing the agent's ongoing work.
Durable execution preserves enough accepted progress to continue after an interruption. Its value grows when restarting would discard substantial computation or repeat a long wait. Durability also introduces obligations: retained work must remain discoverable, resumption must respect earlier effects and limits, and completion must have a defined meaning. Samuel Colvin's durable-agent demonstration motivates this distinction through iterative tool use rather than a single request and response.
| Responsibility | What it establishes |
|---|---|
| Decision-making | Selects a proposed next action; the choice can still be wrong. |
| Execution management | Retains progress and controls when work runs, waits, resumes, or stops. |
| Authorization | Determines whether the particular actor may perform the requested operation. |
| Isolation | Restricts the resources and interfaces executed code can reach. |
| Outcome checking | Establishes the properties actually tested or observed. |
A generated tool call is an action request, not an executed effect; Structured Outputs and Tool Calling develops that boundary. A sandbox restricts execution but does not supply the task's permission policy or recovery logic. Anthropic's Managed Agents architecture illustrates a useful separation: a session event log survives outside the harness process, while the execution sandbox can be replaced independently. Reliable execution preserves and controls decisions; it does not establish that they were good decisions.
Identify the work that continues
Recovery needs identities that outlast a process. This chapter uses task for requested work, task attempt for one pursuit of it, operation for one logical action, and execution for a physical worker activation. These are application distinctions, not universal product field names. For example, Temporal's Workflow ID and Run ID distinguish a stable business identity from a particular run. A retry can create a new run, as can Continue-As-New, which continues the workflow in a fresh run. A Temporal Run ID therefore does not identify the worker activation defined here.
| Identity | Before interruption | After resumption | Meaning |
|---|---|---|---|
| Task | T1 | T1 | The requested work is unchanged. |
| Task attempt | A1 | A1 | The same pursuit continues. |
| Logical operation | O1 | O1 | Recovery concerns the same intended action. |
| Worker execution | E1 | E2 | A different physical activation performs recovery. |
A new pursuit or experimental branch should be distinguishable from recovery. Likewise, a fresh network invocation can retain the same logical-operation identifier: identical parameters alone cannot establish whether a caller intended one action twice or two separate actions. Keep provider request IDs and queue delivery IDs as linked transport records. Conversation IDs group interactions; trace IDs correlate telemetry. They do not replace the identities used to authorize or accept work. Observability explains how to connect these records.
Execution condition and outcome also answer different questions. Runnable work is eligible to start; running work is active; waiting work depends on an event; terminal work will not continue under that attempt's contract. Its outcome can separately be succeeded, failed, cancelled, or unresolved. A terminal attempt can therefore retain an unresolved external operation rather than falsely recording that operation as failed.
Turning points in recoverable execution
Recoverable execution combines several systems traditions. Database recovery preserves committed records. Process supervision assigns another process responsibility for detecting failure and managing replacement. A workflow coordinates steps and their dependencies; persistent workflow systems retain that coordination when execution stops. Environment restoration preserves selected machine state. These different scopes explain why restarting a process, restoring files, and continuing an agent task are not the same operation.
| Development | Date | Contribution |
|---|---|---|
| ARIES | March 1992 | Mohan and colleagues combined recovery logging and checkpoints for database transactions with concurrent updates and partial rollback. |
| Erlang/OTP | Project begun in 1996 | Ericsson consolidated a reliable process-based software platform for the AXD301 project; supervision assigned responsibility for handling process failures. |
| Mentor-lite | February/March 2000 | Weissenfels and colleagues separated workflow interpretation, reliable communication, and database-backed recovery logging in a lightweight prototype. |
| Amazon Simple Workflow | February 21, 2012 | AWS introduced managed coordination between activity workers and history-driven deciders, including retained state, task queues, signals, and timeouts. |
| Durable Functions | May 7, 2018 | Microsoft announced general availability of stateful serverless orchestration built on Durable Task Framework, including long waits and human interaction. |
| Trigger.dev v3 | September 17, 2024 | General availability brought a hosted task runtime with checkpoint-and-restore waits, queues, scheduling, and concurrency controls. |
Persistent workflows addressed coordination that could not depend on one process remaining alive. Serverless orchestration carried that concern into short-lived function environments. Agents add variable action selection and valuable intermediate work, but still need the earlier distinctions. Mentor-lite even separated recovery logging from operator-facing history: information useful for inspection need not be sufficient to restart execution. These approaches coexist; their chronological order does not establish direct influence or replacement.
Retaining and reconstructing execution
Make accepted state durable
Authoritative execution state is the application-controlled record used to decide what may happen next. An execution journal retains the events or outcomes from which that state can be understood or reconstructed. The model's current input is a selected representation of retained information, not the complete record. Context Engineering owns that selection. A transcript saying that a file was produced does not establish that another worker can retrieve it.
A useful runtime record preserves the obligations needed for continuation, not just past conversation:
- Accepted direction — Task and attempt identity, accepted inputs and decisions, and state version.
- Unfinished work — Logical operations, pending dependencies, received results, and their dispositions.
- Retained outputs — Resolvable artifact identities and producing-operation references.
- Continuing controls — Retry use, remaining allowances, deadlines, approvals, cancellation, and terminal reason.
Acceptance is a state transition, not merely a successful computation. Related records in one database—for example, an accepted result, its operation status, and newly pending work—can commit in one transaction. That prevents observers from seeing only part of the transition. It does not include an unrelated remote API effect. State invariants and acceptance explain the underlying rule.
Acknowledge durable acceptance only after the promised storage boundary. PostgreSQL's write-ahead-log settings make this concrete: asynchronous commit can return success before recent records are safe against a server crash. Local flushing and standby confirmation provide different boundaries, with additional storage assumptions. Thus “transaction committed” must be interpreted using the configured acknowledgment contract, not as an unlimited promise against every failure.
Checkpoint what resumption needs
A checkpoint is a durably recorded recovery boundary: it determines which saved progress a replacement worker can rely on. A continuation identifies unfinished computation and what should execute or be awaited next. For an approval-gated tool call, conversation history alone does not identify the pending call or whether approval arrived. SnapLogic's Agent Continuations prototype retains messages, a resume request identifying pending work, and approval-processing information. That record describes how to continue. The application must separately store it durably, authenticate the approval, and make any repeated external action safe.
Checkpoint coverage must match the program's continuation needs. LangGraph's checkpoint records, for example, include graph values, next nodes, identifiers, metadata, and pending tasks. Application allowances, output references, and control decisions are recoverable only if recorded. Nor does graph state automatically contain an arbitrary process's memory or the current state of an external service.
Suppose a worker receives an operation result. If it stops before committing that result, the replacement has only the earlier recovery boundary: the operation remains unconfirmed. If the result and next-work transition committed, the replacement can reuse them. If commitment happened but its acknowledgment was lost, recovery must inspect storage rather than infer absence. External actions taken during the unconfirmed interval need their own recovery rules.
Coverage and granularity
Recovery granularity and write timing are separate choices. Smaller recovery units reduce repeated work but create more persistence boundaries. Asynchronous writes permit execution to advance before recording finishes; synchronous writes wait. LangGraph's node-granularity guidance describes these tradeoffs without implying that an asynchronous checkpoint is already committed. ARIES supplies an older, complementary lesson: its checkpoints summarize recovery information while a log retains changes; the checkpoint does not replace the log.
Name what each saved representation covers. A conversation handoff reconstructs model input. A sandbox snapshot preserves specified environment state. A training checkpoint supports continued parameter optimization; Accelerate, for example, saves model, optimizer, random-generator, and gradient-scaler state. None substitutes for an application record of pending operations and accepted outcomes.
Resume state or replay history
Two common reconstruction methods differ in what they load. Explicit-state resumption reads accepted state and dispatches the next recorded work. History replay reruns workflow control code while substituting recorded outcomes for completed operations. In Colvin's Temporal demonstration, earlier model calls received stored results instead of contacting the model again. Recovering a response is not generating an identical replacement response.
Consider a retained response R1 from completed model operation M1, followed by tool operation O1 whose outcome is unconfirmed. Both reconstruction paths must preserve R1 and reach O1’s recovery policy.
Deterministic replay requires the orchestration code to produce commands compatible with its recorded history. A command might schedule a timer or invoke an activity, Temporal's unit of external or otherwise nondeterministic work whose outcome is recorded. Changing the order of those commands can break replay. Model calls and external queries belong inside activities so recovery can reuse their recorded results; the model itself need not generate deterministically. LangGraph's Functional API similarly places model calls, clock reads, randomness, and side effects inside persisted tasks. Ordinary entrypoint code runs again, while completed task results are restored.
Recover R1, then reach O1’s recovery boundary
Code changes during execution
Persisted execution becomes an interface to future code. Temporal's Worker Versioning can pin workflows to a deployment version while replacing individual workers; operators must retain suitable versions for unfinished work. Auto-Upgrade instead requires replay compatibility. LangGraph documents a different default: existing threads resume on the latest graph. Removed node names or incompatible state schemas can prevent recovery. Even a technically loadable checkpoint can encounter changed business rules. Choose a supported compatibility strategy, explicitly transform state where appropriate, or refuse an unsafe resume.
Scheduling and accepting work
Wake work without a live worker
Scheduling selects eligible work for execution. Suspension retains a continuation while releasing active worker resources. Eligibility can depend on a persisted timer becoming due, a callback arriving, or another operation producing a result. The durable object is the waiting obligation, not a process-local sleep. Temporal's activity queues illustrate how apparently sequential orchestration can dispatch work to replaceable workers.
When state and pending work share one transactional database, workers can discover committed work there. A separate queue introduces a dual-write gap: committing state and then publishing a message can lose dispatch if the publisher crashes between them. A transactional outbox stores outgoing intent alongside the state change in one transaction. A relay later publishes committed entries. Recovery can repeat delivery, so consumers must handle duplicates; the outbox does not create exactly-once remote execution.
Commit state and dispatch intent together
Wait registration also races with event arrival. Azure's durable external-event handling buffers events that arrive before an existing instance starts listening, but discards events addressed to a nonexistent instance. It recommends event identifiers for duplicate handling and a durable timer to bound waiting. A custom design needs an equally explicit rule for early, duplicate, late, and unmatched events; saving a continuation alone does not resolve those races.
Eligibility is not capacity. Concurrency slots limit active work; queue capacity limits waiting work; backpressure slows or rejects admission when downstream capacity is insufficient. Google SRE's queue guidance explains why buffering cannot absorb sustained overload indefinitely. Check remaining task deadlines before dispatch rather than starting work that no longer has a useful completion window.
Reject stale worker results
A lease grants temporary permission to work; expiry does not prove the worker stopped. Amazon SQS likewise permits duplicate delivery even during its visibility timeout.
A fencing token identifies an ownership generation. Mike Burrows's 2006 Chubby paper describes recipients checking sequencers to reject obsolete lock holders. A harness can enforce this at acceptance; protecting external effects requires the receiving service to participate too.
Combine ownership with optimistic acceptance. In a single-row design, atomically accept only when the operation is pending, the ownership generation matches, and the state version is still the one used by the worker. PostgreSQL rechecks an UPDATE predicate after a concurrent update. Zero affected rows means rejection, not permission to overwrite the newer record. Retain rejected observations separately when useful for diagnosis.
For example, A begins O1 under generation 7. Reassignment gives B generation 8. After B's completion commits, A's late result cannot replace it.
Acceptance checks current authority
ExampleReassignment makes A stale before B commits.
Read the diagram as text
- O1.
- A: generation 7.
- B: generation 8.
- Receiving-side acceptance gate. Atomically check current ownership generation, expected state version, and pending operation status before committing.
- B accepted.
- A rejected.
- O1 → A: generation 7: Initial assignment.
- O1 → B: generation 8: Reassignment: current generation becomes 8.
- A: generation 7 → Receiving-side acceptance gate: Submit A's result.
- B: generation 8 → Receiving-side acceptance gate: Submit B's result.
- Receiving-side acceptance gate → B accepted: Generation and version match; O1 pending.
- Receiving-side acceptance gate → A rejected: Generation 7 does not match 8.
Recovery without repeated harm
Bound retries and restarts
A retry invokes the same logical operation again. It is appropriate only when repetition is permitted and another invocation might help. An agent issuing slightly different invalid requests is doing something else: attempting strategy repair, possibly without progress. Nishant Gupta's runaway-retry account illustrates how repeated tool rejection can consume resources unless surrounding software imposes a stopping boundary.
| Known condition | Runtime response |
|---|---|
| Transient failure; repetition is safe | Retry within retained attempt and time limits. |
| Invalid input or denied authority | Reject unchanged repetition; return the actionable failure. |
| External outcome unknown | Use the receiver's recovery contract or reconcile. |
| Action completed but did not solve the task | Return observations for a new decision, not a transport retry. |
Exponential backoff increases delays between retries; jitter randomizes delays to avoid synchronized retry waves. Bound attempts and total elapsed time, and choose the layer that owns repetition. Nested policies multiply: three total attempts at each of three layers can produce 27 deepest calls. Backoff reduces pressure but does not make duplicate mutations safe. Persist used attempts and the original deadline so replacing a worker cannot replenish the allowance.
Process supervision supplies a related boundary. Erlang/OTP's supervision tree assigns processes responsibility for their children; restart-intensity limits escalate repeated failure instead of restarting indefinitely. This bounds restart responsibility, not task semantics. A restarted process still needs retained state and safe effect recovery. Keep decisions about changing strategy with Agent Engineering, and let the runtime enforce the limits around those decisions.
Recover uncertain external effects
The most consequential recovery gap lies between an external effect and its recorded acknowledgment. An activity can change a remote system and lose its worker before reporting completion. A workflow journal then lacks the successful result even though the effect occurred. Temporal explicitly documents this effect/reporting gap: recorded completions can be reused, but unrecorded completion can lead to another invocation.
Idempotency means that repeated execution of one logical operation adds no further effect; safe repetition requires receiver enforcement. Before dispatch, retain intent, parameters, and an operation key. Append invocation attempts and receipts to that record. The receiver must coordinate duplicate recognition with its mutation and reject the same key paired with different intent. A separate check-then-write is insufficient under concurrency.
The receiver defines safe repetition
Receiver contracts differ. Stripe stores the first executing request's status and body, including failures. Parameter mismatches are rejected; validation failures and concurrent conflicts before execution do not create a saved result. Keys may be pruned once at least 24 hours old, after which reuse starts a new request. Thus an in-progress conflict, a cached failure, and an expired key are different recovery conditions. Never change the key merely to bypass an ambiguous result.
Reconciliation checks authoritative external state to resolve what happened. Retain provider handles and local operation references so later responses or events can be matched correctly. Stripe's low-level error guidance distinguishes an API response from the eventual outcome: an HTTP 500 can remain indeterminate, with partial mutations later reconciled and reported through events. If neither safe repetition nor authoritative lookup is available, preserve an unresolved outcome and assign recovery responsibility instead of guessing.
Recovery can create new work
Compensation is a new operation addressing a known completed effect. Hector Garcia-Molina and Kenneth Salem's Sagas, 1987, decomposed long transactions into smaller commits with application-defined compensations. Recovery could continue forward or compensate earlier work. Compensation does not erase history: other work may have observed the changes, some actions are irreversible, and compensation itself can fail. Give reconciliation and compensation durable identities, bounded attempts, authorization, and explicit unresolved dispositions just like other work.
Limits and long-running control
Preserve budgets and deadlines
A budget limits an aggregate quantity; a deadline names the latest permitted time for a specified boundary. A task attempt can have limits on model requests, tool invocations, elapsed duration, concurrency, and metered usage. Per-call ceilings do not bound the whole attempt. Model tokens are vocabulary units used to represent text, not necessarily words; Tokenization explains request accounting.
Admission control decides whether new work may start. Retrospective accounting records what it consumed. Pydantic AI's spend-limit documentation illustrates the gap: concurrent requests can pass admission before either records usage, and a request crossing the threshold can finish. A reservation holds part of the shared allowance for work before it starts, so other calls cannot also claim that allowance. Recovery must preserve both outstanding reservations and recorded consumption, with duplicate accounting prevented, rather than let a replacement worker start from zero.
| Admission design | First call | Second call | Consequence |
|---|---|---|---|
| Independent balance reads | Reads 10; starts | Also reads 10; starts | Twelve units are admitted against ten. |
| Atomic shared reservation | Reserves 6; leaves 4 | Cannot reserve 6 | Outstanding commitments participate in admission. |
After reserved work finishes, settlement replaces its hold with observed consumption and releases any justified remainder. TigerBeetle's two-phase transfers implement the accounting distinction between pending and posted amounts; applying it to an agent budget requires the harness to track actual usage. If a response is lost, consumption may remain unknown. Keep that possible cost distinct from confirmed usage and other outstanding reservations: expiry of a hold does not prove the dispatched request was free. A ledger cannot impose a hard external spending cap unless downstream execution has enforceable bounds.
Preserve the attempt's allowance across workers and allocate child work from it rather than granting each child an independent fresh limit. Attempts, backoff, and local overhead must fit the remaining deadline. Propagate the job's deadline downstream; a durable job may intentionally outlive its initiating HTTP request. Keep bounded capacity for checks and cleanup, so ordinary generation cannot consume everything needed to finish responsibly. Choosing how much reasoning deserves that allowance belongs in Reasoning and Test-Time Compute.
Stop owned work
Cancellation requests that remaining work stop. An acknowledgment confirms acceptance of that request. Quiescence means no work covered by the stop contract remains active. These boundaries underpin honest interface semantics. Persist stop intent, enforce it before admitting new covered work, and continue shutdown after worker replacement. A timeout or accepted cancel request alone is not confirmation that execution ended.
Cancellation is often cooperative: running code must observe it. Temporal activities can receive cancellation through heartbeats and must respond through supported mechanisms such as an abort signal; throttled heartbeats can delay delivery. Configure both an individual-attempt timeout and a total duration covering retries. Then distinguish dispatch stopped, cancellation delivered, and worker exit instead of compressing them into one Boolean.
Children and cleanup
Structured concurrency ties child lifetimes to an owning scope. Trio's nurseries wait for child tasks to exit and cancel siblings after an unhandled child failure; bounded shielded cleanup can complete despite outer cancellation. This is an in-process ownership mechanism, not durable recovery. Intentionally detached work needs a separate durable owner and control handle, otherwise it becomes unaccounted-for background work.
For unresponsive local work, an execution service can escalate from cooperative shutdown to forced termination. Erlang supervisors illustrate a finite shutdown interval followed by killing the child, with care required for descendant lifetimes. Check the actual engine's scope: Azure explicitly documents that terminating an orchestration does not terminate its activities or sub-orchestrations. Sandbox lifecycle management covers environment teardown.
Completion can win a race with cancellation, and remote work may be impossible to stop. gRPC's cancellation contract explicitly does not roll back changes already made. Retain those effects and uncertain outcomes while stopping further task work. Reconciliation or an authorized compensation may remain necessary even after local execution is quiescent.
Stop the covered execution; retain the remote effect
Direct work across long waits
A signal is an external event addressed to retained execution. Pause requests a supported stopping boundary while preserving continuation; resume makes that continuation eligible again. Steering accepts changed direction into task state rather than editing a live call stack. Commands need identities and retained dispositions, with acceptance coordinated against results, cancellation, and terminal state.
Distinguish receipt from execution of a command. In Temporal's message-passing interface, a Signal returns after server acceptance, before workflow delivery. Updates distinguish validated acceptance from handler completion and expose a result handle. Async handlers can interleave at awaits; the main workflow can also finish before handlers do. Coordination and an explicit handler-completion boundary matter even when messaging is durable.
Approval releases a particular action
An approval binding associates a human decision with one retained action: actor, represented requester, scope, operation identity, parameters or proposal version, and validity period. Vinoth Govindarajan's scoped-approval account emphasizes preserving that association through callbacks and retries. The continuation makes waiting possible; it does not make an incoming approval trustworthy.
At release, match the decision to the pending action, reject denial or expiry, and recheck current authorization. Changed parameters must not inherit approval for an earlier proposal. OWASP's transaction-authorization guidance supports operation-specific authorization, limited validity, and a final execution gate. The runtime implements these boundaries; Agent Engineering owns deciding which human intervention is needed.
Long waits also need an administrative ending. Retain the pending action and control history when the client disconnects. Bound the wait with a durable timeout. If an approval expires or new code cannot interpret the continuation, expose a blocked or expired disposition for an operator to renew, transform through a supported path, or terminate. Do not turn an unusable continuation into an endless wake-and-retry loop.
Artifacts, completion, and status
Retain outputs beyond the worker
An artifact is a produced item the task retains or delivers: a file, patch, report, or structured result. A manifest lists outputs and their identities. Record immutable content identity, storage location, and the producing operation so another worker can retrieve the same item. A content digest is derived from content; provenance identifies its production history. SLSA's provenance specification supplies useful fields for output subjects, dependencies, builder identity, and invocation identity. These establish lineage, not correctness or delivery.
Separate scratch output, partial upload, complete stored content, and accepted output reference. S3 multipart uploads illustrate the first boundary: stored parts are not yet a completed object and continue consuming storage until completed or aborted. A complete object can then exist without a corresponding accepted manifest entry if registration fails. Conversely, recording a local pathname does not make the referenced bytes available outside the worker.
A practical publication design stages content under a stable identity, verifies successful storage, then conditionally accepts its reference for the expected task state. S3's conditional writes can prevent overwriting an existing key or require a matching current ETag. They guard that storage operation, not an unrelated task database, and an ETag is not a universal content digest. Recover upload-to-registration failures by reconciling staged objects with accepted references; delete true orphans only under a retention policy that excludes still-owned work.
Collect required outputs before destroying their environment, and keep durable identities separate from temporary download access. Filesystem exports cover the secure transfer boundary. Cleanup also has a postcondition: S3 warns that in-flight part uploads can finish after an abort, so repeated aborts and an empty ListParts result may be needed to confirm removal. Accepted cleanup requests are not the same as removed resources.
Check the current outcome
A completion check evaluates a required property of an identified result or authoritative external state. That property is a postcondition, developed in Establish the achieved outcome. A finished model turn, empty to-do list, exited process, and available file each establish something narrower. The runtime must arrange the checks promised by the task; Evals and Benchmarks explains how to choose their methods.
Make checking recoverable: retain artifact identity, requirement, checker identity and version, result, and failure disposition. Guard success against the current task version so artifact replacement cannot race with acceptance.
GitHub's required-status-check rules require checks on the latest applicable commit: the test-merge commit when it has a status, otherwise the head. Protection can require a particular GitHub App. Skipped or neutral conclusions can satisfy the gate, so inspect which tests actually ran.
Failure must block the intended transition
The checker's failure contract is equally important. Claude Code's TaskCompleted hook can block completion with exit code 2. Ordinary exit code 1 without a valid blocking decision is nonblocking for most hook events; a hook that cannot start can also leave execution proceeding. Registering a test command therefore does not establish an enforced gate. Exercise failed, unavailable, and timed-out checks, and preserve their distinct outcomes.
Require only the boundaries the task promises. Preparing a report can end with a retained, checked artifact; delivering it adds a delivery obligation. Govindarajan recounts a message tool reporting success while the interface displayed nothing, showing why internal acceptance and user-visible availability need separate receipts. Preserve usable partial artifacts when execution ends unsuccessfully, but label their verification and delivery status explicitly.
Report authoritative progress
Expose a stable handle that clients can query independently of the original connection. Report accepted phase, pending dependency, retained progress, last worker contact, resource use, and terminal reason. Google's long-running Operations interface separates progress metadata, completion, and result or error. Its cancellation is asynchronous, and deleting the operation expresses loss of interest rather than stopping execution.
| Observation | What it establishes | What it does not establish |
|---|---|---|
| Heartbeat | Recent worker contact | Useful progress or task success |
| Committed checkpoint | Retained progress within its coverage | Every later effect or local file |
| Tool receipt | The receiving component's stated outcome | Every downstream or user-visible outcome |
| Current passing check | The tested property of its identified subject | Untested correctness or promised delivery |
An event cursor identifies a position from which progress delivery can resume. The HTML Server-sent events standard sends Last-Event-ID when reconnecting after identified events. That supplies a transport mechanism, not event retention or durable execution. Workflow DevKit's streaming walkthrough similarly associates a stream with a run ID and position rather than starting the agent again on reconnection.
Include state versions or observation times so clients can recognize stale snapshots, and handle repeated notifications without repeating accepted transitions. Missing heartbeats indicate missing contact, not proven failure. When the remaining work is unknown, show completed stages and the current dependency instead of an invented percentage. Link task, attempt, operation, and execution identities into traces, but keep sampled telemetry separate from the records that authorize continuation.
Verification and architecture choice
Test interruption boundaries
A recovery test should establish what survives an interruption, not merely show that a process restarted. Safety properties forbid outcomes such as accepting a stale result. Liveness properties require eventual progress under stated conditions, such as restored storage and available workers. FoundationDB's simulation-testing account provides a useful method: control network, disk, time, and randomness, inject faults, and assert system invariants and recovery under recoverable conditions.
| Controlled interruption or race | Required observation |
|---|---|
| Worker stops around acceptance | Acknowledged durable work remains discoverable; uncommitted proposals do not become accepted. |
| External effect succeeds; response is lost | Recovery retains the operation identity and follows the receiver's contract. |
| Duplicate delivery; old worker returns late | No second accepted transition and no stale overwrite. |
| Callback arrives before the wait begins | The documented buffering or rejection behavior occurs without unexplained loss. |
| Worker restarts after usage is recorded | Spent allowance is neither reset nor counted twice. |
| Cancellation races with dispatch | Accepted stop state prevents subsequent covered admission; in-flight work is accounted for. |
| New code interprets saved execution | Incompatibility is detected rather than silently changing continuation. |
| Artifact changes or checker cannot run | Stale or unavailable checks cannot authorize success. |
Use controlled dependencies for model responses, clocks, queue delivery, and external outcomes. Then exercise real storage and service boundaries in integration tests, especially immediately before and after commitment. Explicit transition coverage helps locate untested paths, but it cannot establish that the transition model or tool effects are correct.
Temporal's testing tools can skip timer waits and replay representative histories from open and closed executions. Time skipping pauses while activities run; successful history replay checks compatibility, not new external effects. For architecture comparisons, measure recovery delay, repeated invocations, and retained work separately. Colvin's retry demonstration also encountered an unexplained stall, a reminder that a successful recovery path does not explain every failed path.
Choose sufficient execution machinery
Choose execution machinery from the required lifetime and failure contract. Short, safely repeatable work may need only a bounded in-process harness. Background work needs an addressable owner independent of the client. Work that must survive worker loss needs persisted progress and recoverable wakeups. A long-lived process or graph-shaped API does not establish those properties: LangGraph's persistence documentation explicitly distinguishes an in-memory saver from a persistent backend.
| Implementation | What it can simplify | What remains to establish |
|---|---|---|
| Bounded in-process harness | Direct control flow and local cancellation | Acceptable restart loss, safe effects, limits, and completion |
| Long-lived worker | Background execution independent of a request handler | Persistence, recovery, durable waits, and ownership after replacement |
| Persisted state machine with workers | Explicit phases, transactional acceptance, inspectable pending work | Dispatch recovery, duplicate handling, timers, ownership, and state evolution |
| History-based workflow engine | Recorded outcomes, durable waits, configured retries, and worker replacement | Replay-compatible code, version operations, and application-specific controls |
Across all four, assign responsibility for uncertain effects, shared allowances, cancellation scope, artifact retention, and completion checks. An engine may supply useful primitives without supplying the finished application contract. Persistence writes, history growth, supported code versions, and recovery operations are real maintenance costs. Prefer existing machinery when it removes more correctness-critical code than it adds operational complexity.
Environment restoration is another dimension. Trigger.dev's January 2024 v3 proposal contrasted replaying cached chunks with provider-hosted execution that saved containers at explicit waits. This can preserve valuable machine state, but does not replace task identity or external-effect recovery. Eric Allam's replay-versus-snapshot account makes the distinction concrete: a repository, installed packages, in-memory data, and a running development server are not merely conversation history.
Start by stating what may be lost, what must remain discoverable, which effects may repeat, and which controls must survive. Map each obligation to an implementation and an operating owner. Build the smallest complete path through acceptance, execution, recovery, and checked completion, then test its interruption boundaries. Add machinery when an uncovered obligation requires it—not because orchestration complexity looks like reliability.
Open questions
Portable recovery across changing harnesses remains difficult because saved state can be technically readable while tool meanings, business rules, or external permissions have changed. Progress would mean explicit compatibility contracts and representative-history tests that distinguish safe continuation from a new pursuit.
Strict aggregate spending control remains difficult when concurrent providers report usage late or incompletely. Local reservations help, but cannot constrain an unbounded external request. Progress requires enforceable downstream ceilings and reconciliation of outstanding liability without resetting allowances after recovery.
Cross-service stopping remains incomplete when an orchestrator can terminate while external work continues. Progress would provide inspectable ownership and shutdown contracts spanning descendants and remote operations, while preserving unresolved effects instead of reporting premature quiescence.
