Contents

Purpose and task boundaries
Observations and actions
1. Expose observations, not unrestricted state
2. Define what one action does
  1. Time and applied actions
Episode boundaries and restoration
1. Separate endings from collection cutoffs
  1. Keep the true successor
2. Restore the declared starting conditions
Rewards and verification
Environment validation
1. Test the environment independently
Task exposure and curricula
1. Choose coherent task distributions
2. Change exposure without moving the goal
Simulation and transfer
1. Choose how consequences are produced
2. Validate fidelity for the intended use
  1. Validate interactions and learned behavior
Experience and controlled change
1. Preserve the meaning of each transition
2. Regrade, rerun, or retrain
Check understanding
Open questions
Selected talks
References
Talk library

RL Environments and Simulators

A reinforcement-learning environment gives an agent somewhere to act and receive feedback. It supplies starting conditions, executes requested operations, and makes their consequences available for observation and assessment. This lets developers repeat attempts to evaluate behavior or collect experience for learning. The design matters because an agent can improve at the implemented task without improving at the intended work: observations, actions, rewards, and resets must preserve the requirements that make success useful.

Purpose and task boundaries

Make the intended task executable

An environment is the system through which an agent encounters task conditions and the consequences of its actions. For a coding task, it might include a repository, installed dependencies, command execution, and tests. The instructions describe the work; the environment makes that work possible. The Harbor RL recipe illustrates this separation by packaging instructions and executable task materials independently of the training harness.

A policy selects actions. Reinforcement learning, or RL, improves that selection using rewards: numerical feedback from interaction. Executing a policy collects experience; updating its parameters is a separate operation. Post-training and Alignment explains how learning uses that experience.

Begin with obligations, not the scoring function. In a small example, “repair CSV export” could mean preserving column order and escaping embedded commas while leaving the public interface unchanged. Merely producing a file would implement an easier task. State the required outcome, starting conditions, permitted modifications, and acceptance requirements before deciding how to execute or grade the attempt. Defining the permitted task develops this boundary.

Keep three claims separate throughout development: the implementation follows its specification; an agent performs well within that implementation; and the specification represents the intended external work. Correct execution does not establish the other two. This distinction becomes especially important when a simulator replaces external dependencies.

Separate interaction responsibilities

An observation is information supplied to the agent. An action is its requested operation. A transition is the resulting change in task state. In Gymnasium’s interface, reset initializes an attempt and returns its first observation; step accepts an action and returns an observation, reward, termination and truncation flags, and diagnostic information.

An episode is an attempt with declared starting and ending conditions. A trajectory records its ordered interactions. A rollout is execution that collects experience; it can cover a whole episode or only a segment. These concepts remain useful whether the implementation uses function calls, network services, or a framework that packages several responsibilities together.


Responsibility	What it does
Agent	Uses observations and retained information to select actions.
Environment	Establishes state and resolves action-dependent consequences.
Runner	Coordinates execution, limits collection, and records interactions.
Verifier	Checks specified properties of the resulting evidence.
Learner	Uses experience and feedback to update policy parameters.

For example, Harbor can run an agent against a task sandbox, retain its trajectory, and then verify the resulting state. That rollout need not train anything. The same task can later support training through a separate consumer of trajectories and rewards. Everything Is a Rollout develops this reuse; Agent Engineering covers action selection, while Evals and Benchmarks covers trial and grader responsibilities.

Turning points in reusable environments

Environment engineering combines several traditions: reasoning about sequential consequences, executing simulated worlds, and making experiments reusable and comparable.


Turning point	Contribution
Bellman’s dynamic programming, 1952 — sequential decisions	Richard Bellman’s work at RAND expressed optimization as a first decision followed by the remaining problem, whose conditions depend on earlier outcomes.
SIMULA, 1966 — stateful simulation	Ole-Johan Dahl and Kristen Nygaard described processes, queues, and interleaved events in simulation time; the compiler had operated since January 1965.
RL-Glue, September 2009 — interchangeable components	Brian Tanner and Adam White separated agent, environment, and experiment programs, allowing implementations in different languages to interact.
Arcade Learning Environment, June 2013 — shared game testbeds	Marc Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling’s journal paper, following a 2012 preprint, exposed Atari games through a common emulator interface.
OpenAI Gym, April 27, 2016 — comparable tasks	The public beta supplied common environments to reduce comparison errors caused by differences in action sets or rewards.
WebArena, July 2023 — functional web tasks	Shuyan Zhou, Frank F. Xu, and collaborators introduced self-hosted websites with resets and outcome-based evaluation; the work was published in 2024.

A shared interaction interface makes these environments easier to use, but does not make their worlds equivalent. Each still defines its own available information, action effects, and success conditions.

Observations and actions

Expose observations, not unrestricted state

The environment may need more information to advance the task than the policy is allowed to see. Partial observability means the agent receives an incomplete view of that state, so it may need to gather information before acting. This differs from omitting information the environment itself needs to compute consequences. The Markov assumption says that current state and action suffice to determine the distributions of the next state and reward. A Markov decision process (MDP) describes this interaction; a partially observable Markov decision process (POMDP) additionally distinguishes hidden state from the observations supplied to the agent. Kaelbling, Littman, and Cassandra’s formal account develops that distinction.

For an illustrative lookup task, an application holds orders and their delivery dates. The policy receives a request to find one date and can query the order service. The verifier separately holds the expected answer. Returning that answer in startup diagnostics would remove the intended discovery step. The private target is assessment material, not an observation the policy earned through the task interface.

Privileged information is information available to the environment, trainer, or grader but withheld from the policy. Gymnasium’s information dictionary can contain internal state and reward components; the interface does not decide which fields an application forwards to its agent. Make that forwarding rule explicit instead of passing every diagnostic value into the next model request.

Discovery and assessment use different paths

Example

A private target would bypass the lookup if copied into the policy’s observation.

Order-service stateStores delivery dates accessible through the task’s query interface.

Permitted query resultAn observation derived from service state, not the entire state.

PolicyUses the request and observed result to prepare its answer.

Private expected answerAssessment-only information; exclude it from policy diagnostics.

Answer verifierCompares the submitted answer with the private target.

The lookup task’s data paths: permitted query results reach the policy; its submitted answer and the private reference reach the verifier. The application’s forwarding rule must exclude the private reference from policy input. The missing connector depicts that rule; it does not enforce it or show the complete action loop.

Read the diagram as text

Order-service state. Stores delivery dates accessible through the task’s query interface.
Permitted query result. An observation derived from service state, not the entire state.
Policy. Uses the request and observed result to prepare its answer.
Private expected answer. Assessment-only information; exclude it from policy diagnostics.
Answer verifier. Compares the submitted answer with the private target.

Order-service state → Permitted query result: Data: expose queried fields.
Permitted query result → Policy: Data: allowed observation.
Policy → Answer verifier: Data: submitted answer.
Private expected answer → Answer verifier: Data: private reference.

Timing also limits what an observation establishes. A screenshot may omit offscreen content, and a tool response may describe state before another operation completed. Missing information is not evidence that the corresponding condition is false. Behavior under partial observation concerns what the agent should do next; Context Engineering concerns assembling the allowed information into model inputs.

Define what one action does

The action space declares possible operations and arguments. A discrete action chooses among alternatives; a continuous action supplies a value within a range; a structured action combines an operation with named arguments. Those declarations are only the beginning. The execution contract must also state prerequisites, authority, side effects, invalid-action behavior, and when the next observation becomes available.

In Gymnasium’s Taxi example, the global space contains six actions, but walls and passenger location constrain useful choices at a particular state. An action mask marks currently valid choices for the selector. It is not an authorization mechanism: execution must still enforce permissions. Request envelopes and operation-level authorization explain the separate checks.

Granularity determines which decisions remain with the agent. A coarse operation can encapsulate several lower-level interactions; a keystroke interface exposes them. Terminus uses an interactive terminal through tmux rather than separate specialized editing and execution tools. This permits interaction with an ongoing session, but leaves more session management to the agent. Its external agent process is another design choice, distinct from action granularity. Terminus design.

Time and applied actions

One step need not mean one instantaneous event.
Execution model	Consequence for the interface
Control interval	DeepMind Control advances several physics substeps between agent decisions, then obtains reward and observation. Action frequency and integration frequency differ.
Discrete-event simulation	SimPy advances through scheduled events. Equal-time events execute in scheduling order, so tie-breaking can affect consequences.
Asynchronous command	Starting a process, observing progress, stopping it, and confirming completion require distinct feedback.

ALE sticky actions, examined by Machado and colleagues in 2017, sometimes repeat the previous applied action instead of the new request. With stickiness probability 0.25, the tested memorization-based agent degraded substantially relative to deterministic execution; tested observation-responsive agents were less affected. The perturbation exposes reliance on action history rather than feedback. ALE evaluation study.

Episode boundaries and restoration

Separate endings from collection cutoffs

Termination reaches an ending defined by the task. Truncation stops collection for an external reason. A finite-horizon task ends at its own deadline; a continuing task may merely be sampled in bounded segments. After an external cutoff, learning may estimate remaining reward, called bootstrapping; after true termination there is no task continuation to estimate. For a fully observed finite-horizon task, remaining time belongs in the observation. Gymnasium’s time-limit explanation develops this distinction; Post-training covers its learning consequences.

Record why execution ended separately from what the task achieved.
Event	Meaning
Task success	A specified success condition was established.
Terminal task failure	A task-defined unsuccessful ending occurred.
External cutoff	Collection stopped; the task need not be terminal.
Cancellation	Execution was stopped by request; preserve known effects.
Environment failure	Execution conditions failed; an ordinary task verdict may be unavailable.
Model final response	The agent signaled completion; verify the required outcome separately.

Keep the true successor

Gymnasium’s SameStep resets immediately: reward and ending flags belong to the old episode, but the returned observation starts the next. Use info["final_obs"] as the old action’s successor. NextStep resets on the following call; exclude that reset-only entry from transitions. Autoreset modes.

One SameStep return contains two episodes

Field in one SameStep return	Episode owner	Field assignment → saved destination
`observation`	B · initial observation	→ Start B’s next transition
`reward`	A · rA	→ A’s reward
`terminated`	A · task-defined ending	→ A’s terminated flag
`truncated`	A · external cutoff	→ A’s truncated flag
`info["final_obs"]`	A · final observation	→ A’s true successor

Save A’s last transition:
(oA, aA, rA, info["final_obs"], terminated_A, truncated_A)

The ending flags remain separate; neither alone means success. oB is not the successor of aA. NextStep’s reset-only entry is excluded as described in the text.

In SameStep mode, one return contains fields from two episodes. Save A’s final observation as A’s successor; B’s initial observation starts a different episode.

Restore the declared starting conditions

Reset establishes a valid starting state. Selecting a fresh start from an initial-state distribution differs from restoring one particular snapshot, a saved state intended to support continuation. Clearing conversation history performs neither operation on an external application. Harbor’s multi-step tasks illustrate the distinction: application state can persist while each stage starts a fresh agent conversation.

Snapshot sufficiency depends on the continuation claim. MuJoCo 3.6.0 requires integration inputs, including solver warmstart accelerations, for exact continuation. Restoring positions alone is insufficient, and dependent quantities must be recomputed after manual state changes. Its exact-reproducibility claim is bounded to the same version and computational architecture. A scene that looks restored may therefore evolve differently on the next step.

For a stateful software task, specify the following reset obligations where they affect behavior. Restoring files while leaving a process running, for example, can let that process change the restored files again.
State	Reset obligation
Files and database contents	Restore the declared fixture, including effects not visible in the main artifact.
Processes, queues, clocks, caches	Restore, stop, or deliberately retain each dependency that can change subsequent behavior.
Agent memory	Declare whether a new attempt may retain prior observations or learned task-specific information.
External operations	Do not assume local restoration reverses effects or stops work outside its boundary.
Concurrent attempts	Prevent mutable state from one trajectory changing another trajectory’s conditions.

Matched software environments distinguish restoring storage alone from restoring declared worker and queue state. A surviving worker writes into restored storage; external effects remain outside both reset scopes. — In this fictional software-task example, a surviving worker can change restored storage again. Coordinated restoration covers the declared local dependencies; effects outside that scope remain separate.

A seed controls a random sequence, not the rest of the world. Distinguish recreating an initial random sequence from restoring the generator at a particular point, and preserve relevant dependencies alongside it. Gymnasium’s checker tests seeded reset and step behavior where determinism is expected; such checks do not restore external state. See snapshot boundaries and warm reuse.

Reset between attempts should not erase every difficulty within an attempt. If deployment requires recovering from a slow load or stale tab, silently restarting whenever it occurs removes the recovery problem from training. Expose the disturbance and available recovery actions when that behavior is part of the task; classify failures of the experimental apparatus separately.

Rewards and verification

Make reward timing express the objective

Return accumulates rewards under a stated convention. For an episode with $N$ transitions, $r_{t+1}$ is the reward after transition $t$ , and $\gamma$ discounts later rewards. Sparse feedback may arrive only at completion; intermediate feedback arrives earlier. A final outcome does not identify which earlier choices helped.

G_0=\sum_{t=0}^{N-1}\gamma^t r_{t+1}.

Consider an undiscounted example: two successful paths earn the same terminal reward of 1, but take two and four transitions. Adding 0.1 per transition makes their returns 1.2 and 1.4, favoring the longer path. Charging 0.1 instead gives 0.8 and 0.6, favoring speed. Either modification changes the objective, not merely feedback frequency.

Shaping and endpoints

Reward shaping adds guidance during execution. In potential-based shaping, a designer assigns each state a number $\Phi(s)$ , called its potential, to provide that guidance. The added reward depends on the change in potential, not simply on visiting a high-potential state. With the same discount $\gamma$ used in the return, the intermediate terms cancel when accumulated. The 2017 episodic analysis explains why the potential left at the episode’s endpoint still matters.

F_t=\gamma\Phi(s_{t+1})-\Phi(s_t),\qquad \sum_{t=0}^{N-1}\gamma^tF_t=-\Phi(s_0)+\gamma^N\Phi(s_N).

Intermediate potentials cancel. For a fixed start, zero potential at every true terminal state leaves only a constant, preserving comparisons of complete returns under this construction. Nonzero terminal potentials can change preferred behavior. Setting only the last shaping reward to zero is different and generally breaks the cancellation. An external cutoff must not silently become a true terminal state.

Intermediate feedback cancels; endpoints remain

Two fixed paths share start S with Φ(S)=1 and end at different true terminal states. Path A takes two transitions; B takes three. Base rewards and transitions stay fixed.

Discount γ

Intermediate potentials

Φ(TA), Φ(TB)

Demonstrate the incorrect rule: set only the final shaping reward to zero

Preferred complete return: base A · shaped A. A − B: base 0.18, shaped 0.18.

Path A · 2 transitions → true terminal TA

Potentials: 1 → 2 → 0. Base rewards: 0, 2.

t	Fₜ = γΦ(next) − Φ(current)	γᵗ × applied Fₜ
0	0.9 × 2 − 1 = 0.8	1 × 0.8 = 0.8
1	0.9 × 0 − 2 = -2	0.9 × -2 = -1.8

Matched intermediate terms

Φ(s1): +1.8 from t=0; -1.8 from t=1 — cancel

Base return: 1.8
Applied shaping sum: -1
Shaped return: 0.8

Unmodified endpoint formula: −1 + 0.9² × 0 = -1. This equals the directly accumulated shaping sum.

Path B · 3 transitions → true terminal TB

Potentials: 1 → 2 → 3 → 0. Base rewards: 0, 0, 2.

t	Fₜ = γΦ(next) − Φ(current)	γᵗ × applied Fₜ
0	0.9 × 2 − 1 = 0.8	1 × 0.8 = 0.8
1	0.9 × 3 − 2 = 0.7	0.9 × 0.7 = 0.63
2	0.9 × 0 − 3 = -3	0.81 × -3 = -2.43

Matched intermediate terms

Φ(s1): +1.8 from t=0; -1.8 from t=1 — cancel

Φ(s2): +2.43 from t=1; -2.43 from t=2 — cancel

Base return: 1.62
Applied shaping sum: -1
Shaped return: 0.62

Unmodified endpoint formula: −1 + 0.9³ × 0 = -1. This equals the directly accumulated shaping sum.

Zero terminal potentials shift both complete returns by −Φ(S) = −1, regardless of intermediate potentials or path length. An external collection cutoff is not a true terminal state.

A finite-path calculation, not a training result. Intermediate-potential changes redistribute feedback. Endpoint potentials can change return comparisons; dropping only the final shaping reward breaks the general cancellation identity.

Keep hard limits outside negotiable reward tradeoffs. A penalty for an unauthorized action does not prevent taking it when another reward outweighs the penalty. The execution harness can instead reject the action or require a human handoff. Returning credit through a trajectory explains learning from rewards; bounded autonomy explains constraints on what may happen.

Check properties before assigning reward

A verifier checks a specified property using designated evidence. The rule converting its verdict into reward is separate. A database query can establish stored state; executable tests can establish behavior on supplied cases; a trajectory check can examine required confirmation or prohibited intermediate effects. Choose the evidence from the requirement rather than assuming the final answer contains everything needed.

For an illustrative reservation task, keep the following judgments separate before combining them into a training signal.
Requirement	Evidence	Possible assessment
Correct reservation exists	Authoritative database state	Pass or fail on the specified fields.
Confirmation preceded booking	Ordered user and action events	Pass or fail; unresolved if required events are absent.
User received required details	Delivered response	Separate content assessment.
Assessment executed correctly	Verifier execution status	Checker error is not an ordinary task failure.

τ-bench’s 2024 paper explicitly notes that its final-state reward can pass despite a missing confirmation. That illustrates false acceptance: approving behavior that violates the intended requirement. False rejection is the converse—rejecting valid work. Checking private helper names copied from a reference patch can reject another correct implementation. A reference solution demonstrates a path, not necessarily the only acceptable path. τ-bench.

Human review can reveal defects in both requirements and acceptance rules. The May 2026 SpatialBench verification report describes experts independently reconstructing analyses. Choices such as normalization and neighborhood radius produced defensible numerical differences that restrictive graders could reject. Reviewing their working notebooks helped separate bad analysis from ambiguous instructions and narrow tolerances.

Use executable checks where a property can be tested directly, and calibrated human or model judgment where interpretation remains necessary. Retain unresolved assessments and checker errors instead of silently mapping them to zero. Also declare which diagnostics reach the policy: private grading evidence and useful corrective feedback serve different purposes. See feedback sources, requirement-matched checks, and model-judge validation.

Protect the score and its evidence

Reward hacking earns high scores without satisfying the intended task. It need not alter the checker. In OpenAI’s December 2016 CoastRunners report, an agent repeatedly circled respawning targets, collecting points while crashing and never finishing the race. The game’s intact scoring rule rewarded the wrong behavior. Protecting its implementation would not have repaired the objective.

The reward-tampering analysis distinguishes two other paths. Reward-function tampering changes the mechanism computing reward. Reward-input tampering changes what that mechanism observes without achieving the intended outcome. Modifying scorer code and fabricating the evidence it reads therefore require different defenses. A transition bug is another possibility: an implementation may permit consequences the task rules never intended.

Verifier placement trades access to task state against exposure to agent modifications.
Placement	Useful access	Boundary to protect
Shared task environment	Working directory, installed tools, environment variables	The agent may have changed dependencies used by the checker.
Separate verifier environment	Declared artifacts and explicitly transferred evidence	Required state may be missing; transferred artifacts remain untrusted inputs.

Harbor’s task documentation makes these two arrangements concrete. Separate execution can protect the verifier’s dependencies, but a test that trusts an agent-written “success” file still checks the wrong authority. Likewise, hidden solutions leaking into policy-visible files invalidate a task without modifying its final tests. Preserve authoritative outcomes and inspect relevant action records; trajectory judgment complements enforcement rather than replacing it.

Stronger optimization can uncover weaknesses that ordinary trials miss. Experiments on reward-model overoptimization found proxy scores increasing while an independent model-based quality measure eventually declined. This does not imply that more optimization always hurts; it means score improvement cannot validate the score itself. Proxy failures and independent enforcement develop the broader mechanisms.

Environment validation

Test the environment independently

Test environment obligations before interpreting policy performance. A scripted action sequence can exercise a transition without requiring a capable model. A reference solution supplies a positive control; an unchanged starting state or deliberately invalid solution supplies a negative control. If both pass, or both fail, investigate the fixture and verifier before adjusting the policy. From Agent Traces to Agent Simulations describes treating benchmark tasks as software with their own continuous-integration checks.

Each successful check supports a specific claim, not a general certificate.
Claim	Discriminating check
Reset establishes its fixture	Compare relevant state against explicit postconditions after prior mutations.
Transitions implement the action contract	Execute known valid and invalid requests; inspect effects and returned observations.
Observations respect information limits	Check policy-visible fields and files for private targets or undeclared state.
Episode boundaries are recorded correctly	Exercise task endings, external cutoffs, and reset behavior separately.
Verifier distinguishes selected outcomes	Require known-valid work to pass and known-invalid work to fail.
Episodes are isolated	Mutate one concurrent attempt and verify another attempt’s relevant state is unchanged.

Gymnasium’s environment checker checks observation-space membership and repeated seeded behavior, including observations, rewards, and ending flags. These are conformance checks; they do not prove solvability or reward validity.

An invariant is a property required to remain true. It supplies an oracle for generated tests: if a read-only action promises not to change database contents, varied read requests can be checked against that promise. A metamorphic test checks a specified relationship between executions, such as unchanged behavior after renaming an irrelevant identifier. The relationship must genuinely preserve the task. Generating many inputs without such a rule does not establish correctness. Focused testing explains these methods.

After conformance and known-outcome checks, repeated agent runs can characterize observed difficulty and variability. Keep setup failures, verifier failures, and task failures distinct. Only then does a low success rate become useful evidence about the agent rather than an unexplained mixture of broken infrastructure and difficult work.

Task exposure and curricula

Choose coherent task distributions

A task distribution is the rule selecting goals, starting states, conditions, and constraints across attempts. Training exposure, development cases, protected evaluation, and intended deployment need not share the same mixture. Representative sampling estimates performance on an intended population; a stress collection emphasizes rare consequential conditions. Keep those purposes separate, as explained in sampling the intended work.

Procgen, introduced by Cobbe and colleagues at ICML 2020, varied game layouts, assets, entities, and timing. Its generalization protocol trained on finite sets of levels and tested unseen levels from the generator. That measures something narrower than transfer to a new task family. An ablation also found that progress through a familiar level sequence could conceal poor performance when the sequence changed. New seeds and meaningful new coverage are different claims. Procgen paper.

Variation must preserve coherence. Changing a starting page, account state, or available resource can make previously valid instructions impossible. Generate related pieces together and validate their relationships. Prime Intellect’s General Agent generates instructions, databases, tools, reference solutions, and verifiers. Its structural controls require the initial database to fail verification and a replayed reference solution to pass. This establishes one accepted path, not exhaustive verifier coverage.

A useful coverage ledger names task families and consequential conditions, then records whether each combination is included in training, reserved for assessment, excluded as invalid, or still uncovered. More samples of an included combination do not fill an uncovered one. If recognizing an impossible request is an intended capability, specify and check that outcome explicitly. Generated-task coverage and assessment independence explain how to keep related examples and their solutions from crossing the wrong boundary.

Change exposure without moving the goal

A curriculum changes training exposure through a schedule or an adaptive selection rule. Bengio and colleagues’ Curriculum Learning, published in 2009, described distributions that initially emphasize easier examples and progressively introduce harder or more varied work. The final target remains distinct from the temporary exposure used to reach it. Their findings motivated this strategy without establishing that easy-to-hard ordering always helps. Curriculum Learning.

Difficulty is relative to a policy and a feedback scheme. Uniform failure can provide little distinction among attempts; uniformly easy tasks may provide little new learning signal. Measure competence on declared checks rather than assigning difficulty from episode length alone. A fixed schedule is simple but cannot respond to uneven progress. Adaptive selection uses measured outcomes to change exposure, while retaining earlier tasks can help detect and limit forgetting.

Reverse Curriculum Generation, by Florensa and colleagues in 2017, changes starting states rather than the goal. Episodes begin near a supplied goal, then expand outward as the policy improves. Short random-action runs generate nearby feasible states, and selection favors intermediate success rates. Earlier useful starts remain in the mixture. Evaluation uses the original start distribution, not merely the easier training starts. The method requires the ability to reset to selected states. Reverse curriculum paper.

Change training starts, keep evaluation fixed

S = selected training start; P = proposed start; G = fixed goal. Gray lines are feasible connections. Dashed arrows are short random-action proposals, not guaranteed acceptance.

Schematic training stages: the goal and original evaluation-start distribution remain fixed. New starts follow feasible interactions; some earlier starts remain selected. A proposal is not automatically accepted, and graph distance is not a difficulty scale.

PAIRED provides a contrasting adaptive mechanism: an environment designer is rewarded for a performance gap between two agents. This seeks tasks one can solve while the other struggles, instead of rewarding a generator simply for making both fail. It does not guarantee solvability or deployment relevance. PAIRED’s environment-design formulation makes the generator’s objective explicit.

A rising training score can result from presenting easier work. Keep an independent evaluation mixture fixed while comparing curricula, and report whether improvement is measured per interaction or per elapsed training time: an adaptive generator can spend additional work selecting useful tasks. Matched comparisons separate improved capability from changed exposure.

Simulation and transfer

Choose how consequences are produced

The environment interface does not determine how consequences are produced. A simulator generates task evolution under specified assumptions. An emulator executes a represented machine or system. An application-backed environment instead runs the task’s software, often with controlled fixtures and isolated dependencies. A world model learns to predict transitions. These approaches can expose similar interfaces while preserving different effects.


Approach	Response to a new action	Main boundary
Application-backed execution	The installed software executes it; WebArena uses functional websites.	Selected software, fixtures, and dependencies differ from unrestricted production.
Emulation	The represented machine executes it; ALE uses the Stella emulator.	Machine behavior and game-specific reward extraction define the task.
Engineered simulation	Implemented rules or mocks generate consequences.	Only modeled state changes and dependencies are represented.
Fixed recorded playback	No general response exists for an action absent from the recording.	Observed history does not specify alternative outcomes.
Learned dynamics	A model predicts an action-conditioned successor.	Prediction errors may create consequences unavailable in the target system.

A bounded application environment can combine actual databases, service containers, snapshots, and mocks. Choose which parts to replace according to the behavior being tested. Running the original agent harness can reduce orchestration mismatch, but it does not by itself make external dependencies resettable or eliminate reward hacking. Bring-your-own-harness training explores that tradeoff.

Recorded playback has a precise limit. If a recorded action confirmed a draft, its next observation might show a submitted record. Replacing that action with cancellation while retaining the submitted observation does not simulate cancellation. A counterfactual execution must recompute downstream states and observations under the changed action and controlled external conditions. A restorable checkpoint plus an executor can branch; a transcript alone generally cannot. Simulation and replay explains the resulting evaluation boundary.

Learned prediction adds flexibility but also exploitable error. Ha and Schmidhuber’s World Models reported a controller finding movements that stopped simulated monsters from firing—an exploit unavailable in the original game. The relevant test was behavior after transfer, not predicted return alone. World Models provides the example; the World Models chapter develops learned dynamics and accumulated prediction error.

Validate fidelity for the intended use

Simulation fidelity is faithfulness to specified task-relevant properties. NASA-STD-7009B distinguishes implementation verification from empirical validation against the world for an intended use. Correct simulator code answers the former; comparisons with external reference behavior address the latter. Identify relevant observations, transitions, timing, permissions, and other actors before choosing a realism target.

The required fidelity depends on the failure under investigation. Text interactions may suffice while iterating on a voice agent’s workflow or tool selection. Interruptions require timing-sensitive audio interaction; accent and background-noise failures require the corresponding audio conditions. Simplifying one component test does not replace end-to-end testing. From Self-driving to Autonomous Voice Agents explains this task-relative choice.

Domain randomization varies assumed conditions during training. Tobin and colleagues’ 2017 visual-localization study varied textures, lighting, cameras, and distractors, then tested on real tabletop images. Distractor variation mattered for real clutter. This demonstrated usefulness for a particular perception task without requiring one photorealistic training scene. Randomization cannot supply a mechanism the simulator omits, and chosen ranges do not establish real event frequencies. Physical transfer is developed in Robotics.

Validate interactions and learned behavior

User simulators require both behavioral validation and, when used for training, a separate test of what they teach.
Comparison	What stays fixed	Supported conclusion
Humans versus simulated users	Agent and task protocol	Whether the simulator reproduces relevant interaction outcomes and failure patterns.
Training with different simulators	Starting assistant and training procedure	Whether the changed simulator produces better behavior on an independent downstream assessment.

The January 2026 Lost in Simulation study compared humans and simulated users with a fixed GPT-4o agent on 18 adapted retail tasks. Error was not uniformly optimistic or pessimistic: simulation overestimated some outcomes and underestimated others. Plausible individual messages therefore did not establish faithful agent–user interaction.

A separate May 2026 training study held the starting assistant and training procedure fixed while changing the user simulator. People preferred the writing assistant trained with a simulator fine-tuned on human conversations; the role-playing-simulator variant was not statistically distinguishable from the starting assistant. This supports a bounded conversational-preference result, not objective productivity or general tool-use success. Keep uncertainty and transfer limits attached to the actual outcome measured.

Experience and controlled change

Preserve the meaning of each transition

An environment release includes more than its base implementation. Wrappers transform the effective interface: actions before execution, observations after execution, rewards, or reset behavior. Gymnasium’s documented action-rescaling example exposes values in [0,1] while the underlying environment accepts [-1,1]. Recording only the base environment name loses the meaning of the action supplied by the policy. Preserve wrapper configuration and order. Gymnasium wrappers.

Use a release manifest and per-attempt records rather than repeating an entire configuration on every step.
Record	Meaning to preserve
Environment release	Task definitions, initial assets, transitions, dependencies, wrappers, access rules, and budgets.
Exposure configuration	Sampler, task family, selected initial conditions, and curriculum stage.
Attempt and transition	Episode identity, step order, observations, requested and applied actions where they differ, and ending reason.
Learning and assessment	Applied policy identity, reward components, verifier identity, verdict, and feedback validity.

Record alignment specifies which action and reward belong with each observation. Reinforcement Learning Datasets (RLDS), a format for storing episodes and steps, places an observation beside the action taken from it and the reward resulting from that action. The final record preserves the last observation, but no further action leaves it, so its outgoing action, reward, and discount fields are invalid. The reward for reaching that observation remains valid in the preceding record. Separate last-record and terminal-state markers distinguish the end of collection from the end of the task.

Three records preserve two transitions

Record	Observation	Outgoing action	Resulting reward	Outgoing discount	is_first	is_last	is_terminal
0	o0	a0	r1	d0	true	false	false
1	o1	a1	r2	d1	false	false	false
2	o2	invalid	invalid	invalid	false	true	true

Transition 0: [record 0: o0, a0, r1] + [record 1: o1]
Transition 1: [record 1: o1, a1, r2] + [record 2: o2]

Exactly three records describe two transitions. d0 and d1 are the stored outgoing discount fields; the discount for the valid transition into true termination can be zero. The final record’s outgoing fields are invalid, not zero. A truncated last record can have is_last=true and is_terminal=false.

The reward for reaching o2 belongs to the preceding record. Only the final record’s outgoing fields are invalid.

TorchRL’s MultiCollector can tag frames with the policy version actually applied by a worker. One batch may therefore contain multiple versions; a parent’s update counter is insufficient. Padded or preempted entries need validity masks or removal, not interpretation as ordinary experience. A valid zero reward, missing feedback, and an invalid row are different states. Their training implications belong in Post-training.

GSO’s changelog illustrates why these details affect comparisons: it added deceptive-optimization detection in November 2025, changed iteration budgets and evaluation settings in April 2026, and restricted network access in July 2026 after observing upstream-solution fetching. Familiar repository tasks thus remained recognizable while the effective experiment changed. Identifying the system that ran provides the broader provenance pattern.

Regrade, rerun, or retrain

When an outcome is surprising, first locate the responsible boundary: policy choice, task specification, transition implementation, observations, or verifier. Reproduce the consequential behavior and repair that boundary. SWE-Marathon’s task-development account describes repeated agent trials, inspection, and repairs to both shortcuts and verifiers. A local repair then needs broader regression checks, because fixing one case can change other behavior.

Regrading reassesses a saved attempt. Rerunning generates new behavior. Harbor regrade starts a separate verifier over captured artifacts and writes a new trial linked to the original. It requires the artifact bytes needed for assessment and does not support multi-step regrading in the documented workflow. A transcript or old score cannot reconstruct uncaptured state.

Choose new work according to the claim, not simply the cheapest reusable artifact.
Change	What old evidence can answer	Required work for the new claim
Post-hoc checker correction; complete artifacts retained	How the revised checker assesses the old attempt.	Regrade while preserving the original assessment.
New process requirement; necessary events were not recorded	Only the previously recorded properties.	Run fresh trials with the missing evidence captured.
Changed observations, transitions, permissions, or visible feedback	What happened under the old interaction contract.	Execute again to observe behavior under the new contract.
Changed training rewards or curriculum	Possibly revised scores for historical behavior.	Train under the intervention and independently evaluate the resulting policy.

Preserve old releases and compare baseline and candidate on matched tasks, access, budgets, and assessment rules. If both policy and environment change, a score difference alone cannot identify which change caused it. Once inspected cases guide repairs, treat them as development or regression evidence rather than untouched assessment. See matched comparisons, preserved assessments, and change decisions.

A useful environment makes each conclusion traceable: what task was attempted, what information and actions were available, what actually changed, what was checked, and what remains unknown. That is what allows repeated interaction to become meaningful learning and evaluation rather than repeated optimization of an accidental task.

Open questions

Verifier coverage remains difficult for open-ended work: acceptance rules must admit defensible alternatives without rewarding shortcuts. Progress would combine independently attempted solutions, invalid controls, and calibrated judgments to expose both rejection and acceptance errors.
Simulator validation must account for the behavior optimization discovers, not only familiar interactions. A simulator can appear accurate on collected traces while offering exploitable alternatives. Progress would test policy-induced behavior against independent target-system outcomes.
Adaptive task generation must balance learnability with relevance. Solver-conditioned difficulty can produce useful training exposure while drifting from the intended workload. Progress would demonstrate gains on a fixed independent distribution alongside retained skills and the cost of generating and selecting tasks.
Long-horizon environments need sufficient restoration and assessment evidence without pretending every external effect is reversible. Progress would make reset scope and missing evidence explicit, then show which continuation and regrading claims remain reproducible.

Cited in this entry

Connects simulator fidelity to the component being tested and explains why targeted repairs need broader regression evaluation.

Watch talk

Explore more talks

Search the talk library

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

28 matching talks

Talk	Speaker	Event	Year
SWE-rebench: Lessons from Evaluating Coding Agents on Real Software Engineering Tasks — Ibragim Badertdinov, Nebius Transcript reviewed	Ibragim Badertdinov	AI Engineer Europe 2026	2026
The Unreasonable Effectiveness of Separating the Task from the Model Transcript reviewed	Maxime Rivest, Isaac Miller	AI Engineer World's Fair 2026	2026
Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect Transcript reviewed	Will Brown	AI Engineer World's Fair 2026	2026
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant Cited in this entry	Jesse Hu	AI Engineer Code 2025	2025
Stateful environments for vertical agents — Josh Purtell, Synth Labs Cited in this entry	Josh Purtell	AI Engineer World's Fair 2025	2025
Computer-Use 2.0: Agents Just Got Multi-Cursor Transcript reviewed	Francesco Bonacci, Dillon DuPont, Robert Wendt	AI Engineer World's Fair 2026	2026
Learning on the job: the future of post-training Cited in this entry	Raymond Feng	AI Engineer World's Fair 2026	2026
Human seeded Evals — Samuel Colvin, Pydantic Cited in this entry	Samuel Colvin	AI Engineer World's Fair 2025	2025
Building with Anthropic's Claude - The Prompt Doctor is In Transcript reviewed	Jamie Neuwirth, Zack Witten	AI Engineer World's Fair 2024	2024
Computer Use at the Edge of the Statistical Precipice Cited in this entry	Pierluca D'Oro	AI Engineer World's Fair 2026	2026
DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve Cited in this entry	James Shi	AI Engineer World's Fair 2026	2026
What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip Transcript reviewed	Dotta	AI Engineer World's Fair 2026	2026
How to Train Your Agent: Building Reliable Agents with RL Cited in this entry	Kyle Corbitt	AI Engineer World's Fair 2025	2025
What RL Means for Agents Transcript reviewed	Will Brown	AI Engineer Summit 2025	2025
How Forward Deployed Engineering is done at Factory Transcript reviewed	Eno Reyes	AI Engineer World's Fair 2026	2026
Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor Transcript reviewed	Naman Jain	AI Engineer Code 2025	2025
SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI Cited in this entry	Rishi Desai	AI Engineer World's Fair 2026	2026
The Art & Science of Benchmarking Agents Cited in this entry	Vincent Chen	AI Engineer Europe 2026	2026
Code World Model: Building World Models for Computation Transcript reviewed	Jacob Kahn	AI Engineer Code 2025	2025
From RL to IRL — Gaurav Mishra, Amazon AGI Lab Cited in this entry	Gaurav Mishra	AI Engineer World's Fair 2026	2026
What the Best Agents Share Transcript reviewed	Mardu Swanepoel	AI Engineer Europe 2026	2026
Agent Reinforcement Fine Tuning Cited in this entry	Will Hang, Cathy Zhou	AI Engineer Code 2025	2025
Rethinking Environments for Long Horizon Work Cited in this entry	Rayan Garg	AI Engineer World's Fair 2026	2026
Fuzzing in the GenAI Era Cited in this entry	Leonard Tang	AI Engineer World's Fair 2025	2025
Moondream: how does a tiny vision model slap so hard? Transcript reviewed	Vikhyat Korrapati	AI Engineer World's Fair 2024	2024
The infrastructure for the singularity Transcript reviewed	Jesse Han	AI Engineer World's Fair 2025	2025
Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect Cited in this entry	Will Brown	AI Engineer World's Fair 2026	2026
BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data Transcript reviewed	Paul Henry	AI Engineer World's Fair 2024	2024

References

Coverage and source review

Processed transcripts: 33 processed in full · 5 in the curated path
Automated source review: Passed
Metadata candidates: 0 unreviewed; not verified topic membership
Corpus version: 1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

Tinker Cookbook — Harbor RL
The Harbor RL recipe connects language-model training to executable tasks in sandboxed software environments. An agent receives a bash tool, works on the task and receives reward from tests. HarborTask carries task identity, instructions, configuration and a directory containing the environment and verification materials. The standardized task format separates task creation from the harness that runs training or evaluation. Its sandbox interface exposes command execution and file reading, so environment interaction involves actual software behavior rather than requiring a learned transition model.
Reinforcement Learning: An Introduction — second-edition draft
Reinforcement learning improves a policy using rewards from interaction rather than labels specifying demonstrated actions. The objective is expected return: G_t = sum_k gamma^k R_(t+k+1), with gamma controlling discounting. In a finite episode with only terminal reward R_T, this reduces to gamma^(T-t-1)R_T: the outcome evaluates a trajectory without identifying which earlier decisions helped. Temporal credit assignment propagates that information to earlier choices. Monte Carlo methods update from completed returns; TD prediction uses V(s) <- V(s) + alpha[r + gamma V(s') - V(s)], bootstrapping from the next state's estimate. Policy improvement then favors actions with higher expected return; prediction updates alone do not change the policy.
NASA-STD-7009B — Standard for Models and Simulations
NASA-STD-7009B defines intended use as the expected purpose and application of a model or simulation. Empirical validation assesses how well the operating model represents the real world for those uses. The standard distinguishes a domain of validation, where comparisons with reference evidence are favorable, from a domain of verification, where implementation and solution accuracy meet requirements. Correctly implementing a simulator and establishing its correspondence to the target system are therefore different evidentiary obligations.
Gymnasium — Env API
Gymnasium separates the environment's internal dynamics from the interface available to an agent. An action must belong to action_space; step returns an observation, scalar reward, termination flag, truncation flag and information dictionary. Observations belong to observation_space. The information dictionary can contain diagnostic values, reward components or internal state that is not part of the observation. Reset initializes an episode and returns its first observation. Supplying an integer seed resets the environment's random-number generator; resetting with seed=None normally preserves an existing generator. Sampling actions reproducibly requires seeding action_space separately.
RLDS: Reinforcement Learning Datasets — Dataset Format
RLDS represents experience as episodes containing steps, with separate markers for the first record, last record and terminal state. In its alignment, an observation is stored with the action taken from that observation and the resulting reward. The final record preserves the last observation, but its outgoing action, reward and discount fields are invalid. A last record without a terminal marker can represent truncation. Episode metadata can identify the agent, environment configuration and experiment, and can mark an incomplete episode invalid. The documented EnvLogger conversion explicitly reconciles differently aligned observation, action and reward records.
TorchRL — MultiCollector
TorchRL's MultiCollector can tag each collected frame with a policy version. The tag advances when a worker actually applies new weights, so one returned batch can contain experience from more than one version. A parent-side update counter alone would miss that distinction. The collector also documents trajectory identifiers and validity masks: preempted or padded entries must be masked or removed rather than interpreted as ordinary experience. A collection cutoff can mark truncation without declaring task termination.
RL-Glue: Language-Independent Software for Reinforcement-Learning Experiments
Tanner and White's September 2009 paper addressed difficulty sharing agents, environments and experimental apparatus across incompatible implementations. RL-Glue separates agent, environment and experiment programs: the agent chooses actions and learns, the environment supplies action-dependent observations and rewards, and the experiment controls execution and assessment. Its communication layer allows components written in different languages to interact. A fixed dataset cannot generally replace an interactive environment because the required subsequent observation depends on the action selected. Internal function calls and external socket communication offer different overhead and interoperability tradeoffs.
Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute
A rollout runs an agent against a task sandbox, records its trajectory, verifies the resulting state, and produces rewards that can be aggregated across a dataset.
Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute
Rollout trajectories and rewards can feed distinct improvement methods: supervised fine-tuning, reinforcement learning, and text-feedback-driven hill climbing.
On the Theory of Dynamic Programming
Richard Bellman's 1952 paper, written at RAND, studies sequences of decisions whose subsequent choices depend on earlier outcomes, including stochastic outcomes. It formulates objectives such as maximizing yield within a time limit or minimizing the time or cost of completing a task. Its examples distinguish locating an object from obtaining it: superficially similar activities can require different objectives. The mathematical contribution reduces a sequential optimization problem to choosing an optimal first operation followed by the remaining problem.
SIMULA—an ALGOL-Based Simulation Language
Ole-Johan Dahl and Kristen Nygaard's September 1966 paper describes SIMULA at the Norwegian Computing Center; its UNIVAC 1107 compiler had been operational since January 1965. The language addressed the difficulty of representing changing entities, queues and event sequences in general-purpose programming languages. A simulated process combines data with rules governing its behavior. Processes are conceptually concurrent, while their events are scheduled and executed in an interleaved sequence using simulation time. This establishes a simulation architecture based on stateful entities and events, rather than requiring every system to advance through uniform time steps.
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, Naddaf, Veness and Bowling's June 2013 JAIR paper introduced ALE as a platform for evaluating general agents across varied Atari games, addressing the limitations of narrow or specially tailored task collections. ALE wraps the Stella emulator: agents submit joystick actions and receive screen images or RAM observations. Game-specific components extract rewards and identify episode endings. The platform also supports saving and restoring emulator state for planning. The authors evaluated reinforcement-learning and planning approaches across more than 55 games, making diverse existing programs accessible through a common interaction interface.
OpenAI Gym Beta
OpenAI announced Gym's public beta on April 27, 2016 as a toolkit for developing and comparing reinforcement-learning algorithms across environments including classic control, Atari and simulated robots. The announcement identified a comparison problem: seemingly small differences in action sets or reward functions can substantially alter task difficulty. A common environment collection was intended to reduce this experimental variation while leaving users free to implement learning algorithms in their preferred framework.
WebArena: A Realistic Web Environment for Building Autonomous Agents
Shuyan Zhou, Frank F. Xu and collaborators introduced WebArena in a July 2023 preprint, subsequently published at ICLR 2024. It addresses limitations of simplified interfaces and static web snapshots through self-hosted, functional websites whose software handles state changes. Agents can observe screenshots, HTML or accessibility-tree representations and act through browser operations such as clicking, typing and navigating. Reset mechanisms restore starting conditions. Functional evaluators inspect resulting information or application state rather than requiring one reference action sequence, allowing different successful paths through a task.
Resource Management with Deep Reinforcement Learning
DeepRM represents resource allocations and waiting jobs as scheduler observations. Its policy chooses a pending job to schedule or a void action that advances time. Scheduling changes subsequent allocations and waiting conditions. The paper uses rewards based on unfinished jobs: negative job count targets completion time, while negative reciprocal-duration sums target slowdown. Its background defines the Markov assumption as next-state and reward distributions depending on current state and action.
Planning and Acting in Partially Observable Stochastic Domains
A partially observable decision process separates the world’s state from the observations available to the agent. A belief state is a probability distribution updated from prior belief, an action and a new observation; a policy selects actions from that belief. The paper’s listening example shows that an action can gather information before committing to a consequential choice. This grounds the distinction between planning over incomplete evidence and assuming the latest tool response describes all relevant reality.
Rethinking Environments for Long Horizon Work
A passing outcome can still be invalid if the agent obtained it through prohibited behavior; judge the path as well as the result.
From RL to IRL — Gaurav Mishra, Amazon AGI Lab
Neither DOM access nor screenshots alone guarantee enough context to distinguish the intended action from distracting or adversarial content.
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant
Waiting for a complete tool response implicitly discretizes observation and action, simplifying reasoning while limiting real-time reactions.
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant
The Terminus agent from Terminal Bench illustrates a more granular action space: a Tmux stream with character-level input and output.
Gymnasium — Action Masking in the Taxi Environment
Taxi has six possible actions: four movement directions, pickup and dropoff. Their availability depends on the current state: a wall can block movement, and pickup requires the passenger's location. The environment returns a binary action_mask in the information dictionary after reset and step. The tutorial uses this mask to restrict action selection to currently valid choices. The global action space therefore describes possible action types, while the mask describes which choices are valid at a particular moment.
Terminus: The Terminal-Bench Agent
The original Terminus design gives an agent one interactive terminal interface through tmux. It sends keystrokes instead of using separate specialized tools for editing files, executing commands or downloading resources. The agent's Python process runs outside the task container, reducing coupling between the agent's own dependencies and changes made inside the task environment. This illustrates two independent design choices: the operations exposed as actions and the placement of the program selecting those actions.
DeepMind Control: environment stepping implementation
DeepMind Control distinguishes the physics integration timestep from the interval between agent actions. The control interval equals the physics timestep multiplied by the number of substeps. Its environment step applies the task's action hook, advances physics, runs the post-step hook, and then obtains reward and observation. Reset initializes the task and returns the initial observation. Episode completion is represented separately from ordinary intermediate steps. These interfaces make action timing, observation timing, initialization, and termination explicit.
SimPy 4.0.2 — Time and Scheduling
A discrete-event simulator advances through scheduled events rather than necessarily updating every entity at every fixed time interval. SimPy processes events in increasing simulated-time order. When events share a timestamp, it processes them in scheduling order, using an increasing event identifier to break ties. Its simulated processes may represent concurrent activities, but event execution is sequential and deterministic. Discretizing time can make events that would occur at different real times share a simulated timestamp, making the tie-breaking rule consequential.
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant
A command interface needs observable progress, completion status, and the ability to stop execution so the agent can respond to what actually happened.
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents
Machado and colleagues' 2017 ALE study shows how deterministic transitions can reward memorized action histories instead of robust responses to observations. Their sticky-action protocol sometimes executes the previously applied action instead of the newly requested one. Experiments comparing zero stickiness with probability 0.25 substantially degraded the memorization-based Brute agent, while the tested DQN and Sarsa-based agents were less affected. Random delays at the beginning of an episode did not remove the underlying deterministic transition structure. The protocol makes requested actions and executed actions distinct parts of the environment contract.
Gymnasium — Handling Time Limits
Termination means reaching an ending defined by the task, such as success, failure or the end of an intrinsically finite horizon. Truncation stops collection for a reason outside that task's terminal conditions, such as an imposed time limit on a continuing process. The distinction changes learning feedback: bootstrapping estimates remaining return from the final state, which is appropriate after an external cutoff but not beyond a true terminal state. For a fully observable finite-horizon task, the remaining time must be represented in the observation to preserve the Markov property.
Harbor — Multi-step Tasks
Harbor multi-step tasks retain the environment across stages while starting a fresh agent conversation by default; optional resume preserves supported agent state. These are separate state boundaries. Setup runs before the agent, and a healthcheck can abort the step or trial before task execution. Setup failure records exception information and prevents the agent and verifier from running. Reward gates separately decide whether later stages should proceed. A missing reward or missing reward key fails that gate rather than being treated as an observed zero reward.
Human seeded Evals — Samuel Colvin, Pydantic
Agent loops need an explicit completion convention; repeated model and tool calls alone do not define when execution ends.
Deep Dive: Autoreset Modes in Gymnasium v1.1 Vector Environments
Vector environments can reset completed episodes at different points in the interaction sequence. Under Gymnasium's SameStep mode, the step that ends an episode also resets it: the returned observation belongs to the new episode, while the previous episode's final observation is available through info["final_obs"]. Under NextStep mode, the following step performs the reset; collection code must avoid treating reset-only entries as ordinary transitions. Disabled mode leaves resets to explicit calls. Consequently, the observation returned alongside an ending flag is not always the observation reached by the recorded action.
Stateful environments for vertical agents — Josh Purtell, Synth Labs
Explicitly resettable task state enables rollback when a long-running agent takes an unproductive path.
MuJoCo 3.6.0 — Computation
MuJoCo distinguishes saved integration state from quantities derived from that state. Manually changing state does not automatically update dependent calculations; the required computation stages must run again. Exact continuation requires retaining all integration inputs, including warmstart accelerations used to initialize the numerical solver. Small numerical differences can grow along contact-rich trajectories. The documentation bounds exact reproducibility to the same MuJoCo version and computational architecture. Restoring visible positions alone is therefore a weaker operation than restoring a state sufficient for identical continuation.
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant
Stateful agents require evaluation and simulation to account for the surrounding environment, including running processes and persistent files.
Agent Reinforcement Fine Tuning
Associate every tool call with a rollout identifier and isolate mutable execution environments per trajectory.
Gymnasium — Environment Checker Implementation
Gymnasium's environment checker tests concrete interface properties. It checks that reset observations belong to the declared observation space, examines seeded random-generator behavior, and compares repeated seeded resets where determinism is expected. Its step-determinism check resets to the same seed, repeats the same action and compares observations, rewards, ending flags and random-generator state. Environments declared nondeterministic receive different treatment. These checks help detect implementation errors before interpreting a learning curve.
From RL to IRL — Gaurav Mishra, Amazon AGI Lab
The talk's 'flight school, not just exams' approach trains recovery inside messy simulations instead of silently resetting failed runs.
Reward Shaping in Episodic Reinforcement Learning
Reward shaping supplies additional feedback to make learning easier. The paper analyzes potential-based shaping, which adds F(s,a,s′)=γΦ(s′)−Φ(s), where Φ assigns a scalar potential to a state and γ matches the return's discount factor. Over a finite episode, these terms telescope to −Φ(s0)+γ^NΦ(sN). The final potential can change which behavior is optimal. Setting terminal-state potentials to zero removes that action-dependent residual; merely setting the final shaping reward to zero does not generally solve the problem.
From RL to IRL — Gaurav Mishra, Amazon AGI Lab
Calibrated confidence should account for action risk and authority; maximizing autonomy is not always the right objective.
From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
Evaluate environment state, execution traces, and artifacts using checks suited to each evidence type.
tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Tau-bench evaluates agents interacting with a simulated user, domain-specific tools, and policy instructions. Its reward checks the final database against an annotated target state and, where required, information conveyed to the user. The paper explicitly warns that this reward can pass despite a policy violation, such as acting without confirmation. Repeated trials measure consistency: pass^k is the probability that all k independent trials succeed, averaged across tasks; pass@k asks whether at least one succeeds. A correct final state, compliant process, one successful attempt, and reliable repeated execution are distinct claims.
DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
Verify observable behavior rather than private helpers or incidental structure copied from a reference patch.
Human Verification of SpatialBench
The May 29, 2026 SpatialBench verification report describes independent experts attempting tasks from their instructions and data without receiving the solutions. Review uncovered tasks whose analysis choices were underspecified and graders whose tolerances excluded defensible answers. Choices such as neighborhood radius or normalization could yield different numerical results while supporting the same scientific interpretation. Reviewing experts' working notebooks helped distinguish flawed analysis from ambiguous instructions or overly restrictive acceptance conditions. This supplies a concrete procedure for checking both task answerability and verifier coverage.
Fuzzing in the GenAI Era
A rubric alone does not make an LLM judge reliable: judgments can be hallucinated, poorly calibrated, and sensitive to response order or rubric changes.
Faulty Reward Functions in the Wild
OpenAI's December 21, 2016 report describes a CoastRunners agent that learned to circle a lagoon and repeatedly hit three respawning targets. It accumulated game points while crashing, catching fire and never finishing the race. The failure did not require changing the scorer: the supplied points already rewarded behavior that diverged from the intended racing objective. The example makes an exploitable proxy concrete—repeatedly earning local reward can be preferable to completing the task.
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective
The paper separates reward-function tampering from reward-input tampering. In the first, an agent changes the mechanism that computes reward, such as its implementation or training feedback. In the second, it changes what the reward mechanism observes without achieving the intended outcome. Its constructed examples distinguish these attacks from exploiting a misspecified reward through ordinary task actions. Protecting scorer code and protecting the evidence supplied to that scorer address different failure paths.
Harbor Task Structure
Harbor tasks package instruction.md, task.toml, an environment build context, tests and an optional reference solution. Configuration specifies resource requirements, timeouts and environment behavior. Tests produce reward.txt or reward.json. Shared verification runs in the agent container and can see its workdir, installed tools and environment variables. A separate verifier environment has its own image and receives declared artifacts plus /logs/artifacts/. Its image must include its test entrypoint. Network policies distinguish environment baselines from agent and verifier phases.
Scaling Laws for Reward Model Overoptimization
Best-of-N samples y_1,...,y_N from a fixed policy for a prompt and returns y_argmax_i r_proxy(x,y_i). It changes the distribution of returned outputs through selection, without updating the generator's weights. Increasing N searches more candidates, including rare outputs that receive spuriously high proxy scores. Writing r_proxy=r_gold+error illustrates the mechanism: selecting a large proxy value can select favorable error as well as genuine quality. This decomposition is explanatory, not an assumption of independent errors. In the paper's synthetic experiments, smaller reward models learn comparisons from a larger frozen gold model; stronger optimization by best-of-N or PPO initially improves gold scores and eventually reduces them while pursuing the proxy. An independent outcome measure is needed to detect that divergence.
From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
Treat benchmark tasks as software with separate CI, positive and negative controls, and repeated agent runs.
The Art & Science of Benchmarking Agents
Choose task distributions intentionally: production representativeness and coverage of rare consequential failures are different evaluation goals.
Leveraging Procedural Generation to Benchmark Reinforcement Learning
Procgen's ICML 2020 paper uses generators that vary layouts, assets, entities and event timing. Its generalization protocol trains on a finite collection of levels and evaluates on unseen levels from the generator; its sample-efficiency protocol samples broadly during both training and evaluation. An ablation with a fixed sequence of levels produced apparent progress through familiar levels but poor performance when the sequence changed. Procedural variation therefore changes the population of experiences, while the sampling protocol determines what generalization is actually tested.
Computer Use at the Edge of the Statistical Precipice
Vary task data, appearance, and initial state across runs, while checking that every generated combination remains valid.
General Agent: A Self-Evolving, Synthetic Agent Environment
Prime Intellect's General Agent generates task instructions, databases, tools, reference solutions and verifiers together. Its structural checks require the initial database to fail verification and the database produced by replaying the reference solution to pass. A published appointment-booking example checks the resulting customer, service and booking status. Task evolution adds dependencies, distractors and other complications, then gates difficulty using a specified solver's success rate. Reported difficulty bands therefore partly reflect their construction procedure; another tested solver showed a different difficulty profile.
Curriculum Learning
Bengio and colleagues' ICML 2009 paper describes a curriculum as a sequence of training distributions that initially emphasizes easier examples and progressively introduces harder or more varied examples. Its formal treatment reweights a target distribution rather than redefining the final task around whatever the learner currently succeeds at. The motivation is to guide optimization toward useful solutions. Selected vision and language experiments demonstrate benefits, while the paper also recognizes that choosing a useful ordering is itself a problem.
Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect
Search for tasks that produce meaningful differences across rollouts rather than tasks that are uniformly easy or uniformly hard.
Reverse Curriculum Generation for Reinforcement Learning
Florensa and colleagues' CoRL 2017 method addresses sparse goal rewards by changing where episodes begin. Training starts near a supplied goal state and expands toward more distant initial states as the policy improves. It favors starts with intermediate success rates, where the task is neither already mastered nor consistently unsuccessful. Short random-action rollouts generate feasible nearby states, and previously useful starts remain in the mixture to reduce forgetting. Progress is evaluated against the original initial-state distribution rather than only the easier training starts.
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
PAIRED treats environment generation as choosing free parameters of a developer-specified environment. Random generation can waste effort on unhelpful tasks, while a purely adversarial generator can create impossible ones. PAIRED instead uses two agents: the environment designer is rewarded for a performance gap between an antagonist and a protagonist. This encourages environments that one agent can solve but the other finds difficult, producing a changing curriculum rather than a fixed difficulty schedule.
World Models
Ha and Schmidhuber separate visual encoding, recurrent prediction, and action selection. Their recurrent model predicts a distribution over the next latent representation from the current representation, action, and recurrent memory. In their learned Doom environment, a left action changes the player's predicted position, while generated monsters and fireballs evolve around it. The model also predicts episode termination. Controllers trained in this environment are subsequently tested in the original game. The authors report an exploitation failure: a controller discovered movements that prevented simulated monsters from firing, although this exploit was unavailable in the original environment. Increasing sampling randomness helped in their experiments but also introduced unrealistic events.
From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
Build a bounded environment using database snapshots, service containers, mocks, and simulated users.
Counterfactual Credit Assignment in Model-Free Reinforcement Learning
Credit assignment separates an action's influence on later rewards from external events and subsequent actions. A delayed outcome can depend on many intervening decisions, so temporal order alone does not identify its cause. The paper defines return as G_t=sum over u>=t of gamma^(u-t) R_u; this aggregates rewards, not causal responsibility. Its transition model makes the next state depend on the current state and action. Its structural causal model supports changing an action while holding exogenous randomness fixed. Consequently, counterfactual replay must recompute downstream states and observations after action divergence; feeding the old observation sequence to a changed policy does not generally simulate its outcome.
Learning on the job: the future of post-training
Bring your own harness proposes training through the actual deployment harness to reduce mismatch between simulated and production environments.
From Self-driving to Autonomous Voice Agents — Brooke Hopkins, Coval
Choose simulation fidelity according to the component and failure mode under test; visual or audio realism alone does not establish simulation quality.
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
Tobin and colleagues' 2017 study trains visual object localization using simulated scenes with randomized textures, distractors, lighting and camera properties. The renderer does not need to reproduce one photorealistic scene; variation is intended to make the real scene fall within conditions the predictor can handle. Evaluation uses real tabletop images, and ablations examine which variations matter. Training with distractors was consequential for handling real clutter. The study thus separates visual realism from demonstrated usefulness for a particular perception task.
Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
This January 2026 study compares simulated users with human participants interacting with a fixed GPT-4o agent on 18 adapted retail tasks. Simulation did not produce one consistent direction of error: it underestimated performance on some difficult tasks and overestimated performance on some moderately difficult tasks. The authors also observed differences in conversational behavior and failure patterns. A user simulator's usefulness therefore requires checking the resulting agent–user interactions, not merely whether its individual messages appear plausible.
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
This May 2026 study varies the user simulator while holding the assistant's starting model and training procedure fixed. It compares a prompted role-playing simulator with one fine-tuned on human conversation data, then evaluates trained assistants with people on writing tasks. Human preferences favored the assistant trained with the fine-tuned simulator, while the role-playing-trained variant was not statistically distinguishable from the starting assistant. The paper also examines disagreement between performance with the training simulator and performance with other evaluators. Simulator quality is assessed through the behavior it teaches and subsequent human interaction.
Gymnasium — Wrappers
Gymnasium wrappers can transform actions before the underlying environment receives them, transform returned observations or rewards, and modify reset or step behavior. Wrappers can be nested, and environments created through gym.make commonly already include wrappers. The documentation's RescaleAction example exposes actions in [0,1] while the underlying Hopper environment accepts actions in [-1,1]. Thus, the base environment's name and native action bounds do not necessarily describe the interface actually used by the agent.
GSO — Benchmark Methodology and Changelog
GSO's official changelog records changes to the effective optimization task. On November 3, 2025 it introduced a detector for deceptive optimizations, including harness hijacking. On April 27, 2026 it increased the agent iteration budget and changed evaluation settings. On July 12, 2026 it added network restrictions and disabled MCP after observing agents fetching upstream commits or pull-request diffs. These changes alter checking, available information and allowed effort, even when the underlying repository task remains recognizable.
SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI
Standardize tasks into executable environments, then iteratively test and repair both task shortcuts and verifier weaknesses.
From Self-driving to Autonomous Voice Agents — Brooke Hopkins, Coval
Passing a reproduction case can conceal a behavioral regression elsewhere.
Harbor — Regrade
Harbor can regrade a completed trial with an updated verifier without rerunning the agent. It starts a separate verification environment using captured artifacts and writes a new trial rather than modifying the original. Provenance includes the source trial identity and task content information. Regrading requires the necessary artifact bytes; missing artifacts cannot be reconstructed merely from a reward or transcript. This separates a changed assessment of an existing attempt from a fresh attempt under changed conditions.
How to Train Your Agent: Building Reliable Agents with RL
A verifier that omits structural constraints can reward invalid outputs as perfect solutions.
DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
The speaker proposes hybrid verifiers, including LLM-as-a-judge, as a way to support more objective-level prompts.
Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci
Verifiers packages task interaction, response parsing, and reward computation into an environment usable for both evaluation and training.
Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci
For weak models, allowing recovery from invalid actions can preserve learning opportunities that immediate termination removes.
Verifiable Environments for AI in Biology — Kenny Workman, LatchBio
Have scientists attempt and cross-grade tasks to reveal ambiguous instructions and unjustified numerical thresholds.