World Models: How Agents Learn Inside Imagined Worlds

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. System Identification Overview

   System identification estimates dynamic-system models from measured input and output signals, a selected model structure, and fitted parameters, followed by application-specific validation. The documentation's spring-mass-damper example represents state using displacement and velocity while observing displacement alone. Its sampled model explicitly includes a sampling interval, and fitted discrete-time parameters depend on that interval as well as physical system constants. This provides a concrete explanation of why measured position alone can omit information needed to predict motion and why changing the prediction clock changes the modeled transition.

2. Code World Model: Building World Models for Computation

   A learned world model could generate imagined execution traces to evaluate actions before interacting with the actual environment.

3. Reinforcement Learning: An Introduction — second edition in-progress manuscript

   Dyna uses real experience both to improve a model and to update action values directly. The learned model generates simulated experience for additional updates using the same learning machinery. Search control selects the starting states and actions for those simulated experiences. In the presented Dyna-Q algorithm, the model stores a next state and reward for previously encountered state-action pairs, and additional simulated updates occur between real interactions. Thus planning in Dyna includes learning from generated experience, rather than only searching futures immediately before choosing an action.

4. Underactuated Robotics: Trajectory Optimization

   Trajectory optimization chooses states and controls over a time horizon to minimize a cost while satisfying dynamics and additional constraints. Direct transcription makes intermediate states variables and imposes state-transition equations; shooting derives states by simulation. Obstacle avoidance often makes the optimization nonconvex. Model predictive control repeatedly measures state, solves a finite-horizon trajectory problem, applies its first control action, advances the physical system, and solves again. The prediction model, objective, constraints and repeated feedback update distinguish MPC from merely replaying a geometric path. They also distinguish it from a learned policy that samples observation-conditioned action chunks: both can replan over a horizon, but only the stated optimization formulation supplies explicit dynamics and constraints.

5. GraphCast: Learning skillful medium-range global weather forecasting

   Remi Lam and colleagues' GraphCast learns global weather evolution from historical weather-state estimates. Two states six hours apart initialize a prediction of the next six-hour state; repeated predictions produce a ten-day forecast. A graph neural network transfers information between a latitude–longitude grid and a multiscale mesh. The reported implementation produces a ten-day forecast in under one minute on one TPU v4 and evaluates forecast variables and severe-weather events against held-out observations or reference analyses.

6. CWM: An Open-Weights LLM for Research on Code Generation with World Models

   The FAIR CodeGen team's September 30, 2025 CWM report extends world modeling to computational environments. Training includes observation–action trajectories from Python execution and agentic Docker sessions. In the Python representation, an observation records local variables and call-stack information around execution; a code line acts as the transition-producing operation. The model learns to predict subsequent execution information. Published examples demonstrate stepwise predicted execution, providing a nonvisual example of learned state transitions.

7. Planning and Acting in Partially Observable Stochastic Domains

   A POMDP separates actual hidden state s from observation o generated after action a. Belief b(s) is a probability distribution over possible states conditioned on experience, not the observation itself or merely the most likely state. With transition probabilities T(s,a,s') and observation likelihood O(s',a,o), predict p(s') = sum_s T(s,a,s')b(s), then update b'(s') = O(s',a,o)p(s')/Z, where Z = sum_s' O(s',a,o)p(s'). Prediction accounts for action effects; likelihood weighting incorporates new evidence; normalization makes probabilities sum to one. The policy selects actions from the updated belief, including actions useful for gaining information.

8. Learning Latent Dynamics for Planning from Pixels

   PlaNet infers an approximate hidden-state distribution from observation and action history. Its recurrent state-space model combines deterministic temporal memory with stochastic latent variables. The transition advances memory using the previous state and action; an observation-conditioned encoder revises the stochastic state when measurements arrive. Observation and reward readouts train the representation, but planning predicts latent states and rewards without rendering images. Standard training matches one-step predictive distributions to observation-informed posteriors. Latent overshooting additionally matches multi-step predictions to their corresponding posteriors, avoiding repeated image decoding. With restricted model capacity, the best one-step predictor need not be the best multi-step predictor.

9. Predictive Representations of State

   Littman, Sutton and Singh's 2001 paper represents state through predictions about selected future action–observation sequences. Each test asks how likely specified observations are if specified actions are taken. A sufficient predictive state retains enough information to determine predictions for all such tests. This differs both from inventing hidden physical-state labels and from retaining only a fixed window of observations. The paper establishes a linear predictive representation requiring no more predictions than the number of states in a minimal POMDP representation.

10. Forward Models: Supervised Learning with a Distal Teacher

    Michael Jordan and David Rumelhart's 1992 paper addresses learning actions when training specifies desired outcomes but not the actions that produce them. A forward model predicts an action's consequences; an inverse model chooses actions for desired consequences. Their method propagates an outcome error backward through a learned forward model to update the controller. Crucially, actual performance error and model prediction error are different signals. Simulated arm experiments show improved trajectory tracking as learned feedforward control takes over more of the work from an auxiliary feedback controller.

11. Code World Model: Building World Models for Computation

    Code World Model (CWM) represents program execution as state transitions that an autoregressive LLM can learn to predict.

12. DeepMind Control: environment stepping implementation

    DeepMind Control distinguishes the physics integration timestep from the interval between agent actions. The control interval equals the physics timestep multiplied by the number of substeps. Its environment step applies the task's action hook, advances physics, runs the post-step hook, and then obtains reward and observation. Reset initializes the task and returns the initial observation. Episode completion is represented separately from ordinary intermediate steps. These interfaces make action timing, observation timing, initialization, and termination explicit.

13. How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position

    The authors propose evaluating alternative action sequences from matched starting histories, with actually executed branches supporting intervention claims. Their protocol separates teacher-forced diagnostics from policies acting on model-generated histories. It recommends measuring task outcomes, policy-ranking agreement, success-probability calibration, and the gap between optimized model return and environment return. Downstream optimization comparisons fix data, compute, and interaction budgets. The paper explicitly distinguishes externally setting actions from stronger counterfactual claims that hold a particular realization of external randomness fixed.

14. Genie: Generative Interactive Environments

    The 2024 Genie paper from Jake Bruce and colleagues combines a video tokenizer, a latent-action model and an action-conditioned dynamics model. During training, consecutive frames supply information for learning discrete action codes without recorded controller labels. At interaction time, choosing a code conditions the predicted continuation. The CoinRun imitation experiment separately uses a small action-labeled dataset to map learned codes to executable game controls. Thus inferred action categories and an environment's actual action interface remain distinct.

15. When observational action records identify causal effects

    For action A, pre-action covariates L and potential outcome Y^a, standard adjustment identifies E[Y^a]=sum_l E[Y|A=a,L=l]P(L=l) under consistency, conditional exchangeability and positivity. Consistency links observed outcomes to a sufficiently specified intervention. Exchangeability requires Y^a independent of A given L: measured covariates adequately remove action-outcome confounding. Positivity requires P(A=a|L=l)>0 wherever the target population has support. Unmeasured common causes can invalidate exchangeability; actions never taken in relevant states leave their effects unidentified by this adjustment. Predictive accuracy on observed actions does not repair either failure.

16. A New Approach to Linear Filtering and Prediction Problems

    R. E. Kalman's 1960 paper reformulated linear filtering and prediction using state transitions and recursive error-covariance calculations. Its motivation included making calculations suitable for digital computers and handling changing signal statistics without repeatedly deriving new filters. The formulation connects prediction, measurement-based correction, and regulation through an explicit dynamical model.

17. Model Predictive Heuristic Control: Applications to Industrial Processes

    Richalet, Rault, Testud and Papon's 1978 industrial-control paper describes identifying an input–output model from plant measurements and using it to calculate future controls. The approach addressed multivariable processes, delays and operational constraints. Predicted outputs follow a reference trajectory starting from the measured output; actual measurements correct subsequent predictions. Identification experiments must excite the process enough to reveal its response while limiting disruption to normal operation. The report describes application to industrial processes rather than only a simulated benchmark.

18. Integrated Modeling and Control Based on Reinforcement Learning and Dynamic Programming

    Richard Sutton's NIPS 1990 report presents Dyna-AHC and Dyna-Q as systems that alternate interaction with the environment and computation using a learned forward model. In its navigation experiment, the agent learns routes while learning the model itself. Increasing model-based updates between real interactions reduces the real experience required to obtain effective routes. This supplies an early concrete demonstration of exchanging additional computation for fewer environment interactions.

19. World Models

    Ha and Schmidhuber separate visual encoding, recurrent prediction, and action selection. Their recurrent model predicts a distribution over the next latent representation from the current representation, action, and recurrent memory. In their learned Doom environment, a left action changes the player's predicted position, while generated monsters and fireballs evolve around it. The model also predicts episode termination. Controllers trained in this environment are subsequently tested in the original game. The authors report an exploitation failure: a controller discovered movements that prevented simulated monsters from firing, although this exploit was unavailable in the original environment. Increasing sampling randomness helped in their experiments but also introduced unrealistic events.

20. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

    Julian Schrittwieser and the DeepMind team introduced MuZero in a November 2019 preprint. A representation network encodes observed history; a dynamics network advances hidden state under a candidate action and predicts reward; a prediction network supplies policy and value estimates for tree search. Training targets rewards, search policies and returns rather than reconstructed observations. The reported board-game and Atari results demonstrate that useful planning can operate through task-specific hidden dynamics without predicting every visible detail.

21. The Value Equivalence Principle for Model-Based Reinforcement Learning

    Grimm, Barreto, Singh and Silver's 2020 paper formalizes when different dynamics models can be interchangeable for particular planning calculations. Reward scores a transition; value estimates expected accumulated, discounted reward. A Bellman update combines immediate reward with discounted next-state value. Models are value equivalent for specified policies and value functions when these updates agree. Requiring agreement for more policies and functions narrows the acceptable model class. Experiments in four rooms, catch and cart-pole show that task-directed model fitting can produce better downstream policies than maximum-likelihood fitting under the tested capacity restrictions.

22. Temporal Difference Learning for Model Predictive Control

    TD-MPC learns an encoder, action-conditioned latent transition, reward predictor, value predictor, and policy without reconstructing observations. Training encodes the initial observation and recursively advances predicted latent states; subsequent observations supply latent targets through a slowly updated target encoder. Reward, value, and latent-consistency losses jointly shape the representation. Planning combines predicted short-horizon rewards with a terminal value estimating return beyond that horizon. This supplies a concrete alternative to preserving every visual detail: the representation is trained for the task's rewards and decisions.

23. Action-Conditional Video Prediction using Deep Networks in Atari Games

    The Atari predictors encode frame history, transform features using executable game actions, and decode future frames. Multi-step training feeds predicted frames back as inputs. Actions are selected every four emulator frames, producing prediction steps at 15 frames per second. Figure 4 supplies a concrete compounding-error example: a small position error leaves predicted Ms Pacman inside a corridor when the actual character reaches a junction; an upward action then produces different futures. Small bullets contribute little to whole-image squared error, and an action-blind baseline can score relatively well because the controlled object occupies few pixels. The experiments separately measure prediction error across horizons and game scores when a fixed controller receives predicted frames.

24. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

    Matching learned embeddings admits an uninformative solution when every input receives the same vector. VICReg counters this collapse by penalizing embedding dimensions whose batch standard deviation falls below a threshold, alongside agreement and covariance penalties. This is a concrete mechanism for requiring representations to retain variation rather than merely agree.

25. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

    Mahmoud Assran and the FAIR at Meta team distinguish action-free video pretraining from action-conditioned prediction in their June 2025 V-JEPA 2 report. After learning to predict representations of missing visual content, they freeze the encoder and train an action-conditioned predictor using robot trajectories. These actions describe changes in end-effector position, orientation and gripper state. Planning searches for actions whose predicted visual representations approach a goal image, then executes an action and observes again. The paper reports reaching, grasping and placement experiments in two laboratories absent from the action-training dataset.

26. Simulate and Predict Identified Model Output

    The documentation distinguishes simulation using inputs and initial conditions from prediction that additionally uses measured input-output history. Multi-step prediction propagates internally predicted quantities; one-step prediction can continually benefit from actual measurements. It recommends validating against measurements excluded from model estimation and choosing prediction horizons relevant to the intended application. Consequently, a model's response with refreshed measured history and its free-running response answer different validation questions.

27. Dream to Control: Learning Behaviors by Latent Imagination

    Dreamer separates dynamics learning, behavior learning, and environment interaction. Recorded sequences initialize latent states; imagined trajectories then alternate policy-selected actions and predicted transitions without receiving the corresponding real observations. Predicted rewards and values train an action model, which subsequently chooses actions during actual interaction without online trajectory search. Value estimates account for consequences beyond the imagination horizon. The experiments compare behavior-learning alternatives and representation objectives using downstream task performance. Pixel reconstruction outperformed the tested contrastive alternative on most tasks, while reward prediction alone was insufficient in those experiments.

28. Transformers T5 documentation: Training

    Teacher forcing supplies the recorded target sequence as decoder context, shifted so that each position predicts the following target. The documented T5 procedure prepends its start token to decoder inputs and uses target tokens followed by EOS as labels. This establishes the distinction between conditioning training predictions on recorded predecessors and feeding generated choices back during generation. Padding tokens are artificial entries used when batching sequences of different lengths.

29. PETS: Learning Dynamics and Planning with Uncertainty

    A learned dynamics model predicts consequences rather than directly selecting actions. PETS fits pθ(s′|s,a) to recorded state-action-next-state triples by minimizing negative log likelihood, −Σ log pθ(s′|s,a). Networks predict Gaussian means and diagonal covariances; separately bootstrapped datasets produce an ensemble. Within-model variance represents aleatoric noise; disagreement between models approximates epistemic uncertainty from limited data. For each candidate action sequence, particles recursively sample predicted transitions. The planner scores sequences by particle-averaged cumulative reward; cross-entropy search repeatedly refits its sampling distribution toward better candidates. It executes only the first action, observes the actual next state, and replans. Outcomes enter the dataset for subsequent model training. Recursive prediction feeds predictions back as inputs, allowing small biases to compound. The paper’s horizon experiments show that longer planning is not automatically better. Engineering implication: replanning limits open-loop commitment but cannot undo an already unsafe action or make an inaccurate model trustworthy.

30. Counterfactual Credit Assignment in Model-Free Reinforcement Learning

    Credit assignment separates an action's influence on later rewards from external events and subsequent actions. A delayed outcome can depend on many intervening decisions, so temporal order alone does not identify its cause. The paper defines return as G_t=sum over u>=t of gamma^(u-t) R_u; this aggregates rewards, not causal responsibility. Its transition model makes the next state depend on the current state and action. Its structural causal model supports changing an action while holding exogenous randomness fixed. Consequently, counterfactual replay must recompute downstream states and observations after action divergence; feeding the old observation sequence to a changed policy does not generally simulate its outcome.

31. Model Predictive Control: Theory, Computation, and Design, second edition

    Model predictive control plans N steps by minimizing Σk=0…N−1 ℓ(xk,uk)+Vf(xN), subject to dynamics xk+1=f(xk,uk), initial state, and state/input constraints. Here ℓ is step cost and Vf is terminal cost. It executes only the first optimized input, obtains a new state measurement or estimate, and solves again over a shifted horizon. Thus execution uses feedback even though each nominal planning problem considers an open-loop sequence. This requires a predictive model, an actionable state estimate, explicit objectives and constraints, and a feasible optimization solved in time. Continuity and suitable compactness/coercivity conditions support existence of a minimizer; stability and continuing feasibility need additional design conditions.

32. When to Trust Your Model: Model-Based Policy Optimization

    MBPO replaces a few long imagined episodes with many short model rollouts initialized from states in real-experience replay. This separates model rollout length from the task's episode length and limits exposure to recursively accumulated model error. The paper distinguishes error on the data-collecting policy's distribution from error after policy changes. Its evaluation compares against model-free SAC and includes an ablation giving SAC similar numbers of updates without model data, helping isolate the contribution of generated transitions. Short rollout results are assessed through performance in the benchmark environment, not solely through predicted model return.

33. Model uncertainty versus observation uncertainty

    Aleatoric uncertainty describes observation noise conditional on the available inputs; epistemic uncertainty describes uncertainty about the learned model. The paper represents epistemic uncertainty through approximate distributions over network weights and learns input-dependent aleatoric variance. In its regression formulation, predictive variance combines variation in predicted means across weight samples with the mean predicted noise variance. More relevant data can reduce parameter uncertainty without eliminating outcome noise. A normalized class score alone does not represent both components. Engineering limitation: uncertainty within one chosen model family does not exhaust uncertainty about whether that specification is appropriate.

34. Distribution-aware Evaluation of Multimodal Trajectory Predictions with Energy Score

    The authors represent a forecast as samples of a joint distribution over future positions across time, while each observed case supplies one realized trajectory. Their synthetic experiments show that best-sample displacement metrics improve as more samples are offered, without changing the underlying predictive distribution. They propose energy-score evaluation combining forecast-to-observation distances with distances between forecast samples. A proper scoring rule rewards reporting the true distribution in expectation; this differs from selecting whichever sampled future happened to match the observation best.

35. MOPO: Model-based Offline Policy Optimization

    MOPO addresses policy learning from a fixed transition dataset, where a new policy can visit states and actions poorly represented by the data-collecting behavior. It subtracts a dynamics-uncertainty penalty from predicted reward before policy optimization. The intended tradeoff is between improvement outside recorded behavior and exploiting inaccurate predictions there. Its theoretical lower bound assumes an admissible estimator that upper-bounds model error. The implementation instead uses a heuristic derived from predicted ensemble variances and explicitly states that this estimator lacks the theoretical guarantee.

36. Planning and Acting in Partially Observable Stochastic Domains

    A partially observable decision process separates the world’s state from the observations available to the agent. A belief state is a probability distribution updated from prior belief, an action and a new observation; a policy selects actions from that belief. The paper’s listening example shows that an action can gather information before committing to a consequential choice. This grounds the distinction between planning over incomplete evidence and assuming the latest tool response describes all relevant reality.

37. Neural general circulation models for weather and climate

    Kochkov, Yuval and colleagues' 2024 NeuralGCM combines a numerical atmospheric dynamical core with learned representations of processes unresolved by its grid. Training through model trajectories accounts for interactions between these components. The study evaluates weather forecasts and longer climate statistics separately. In its forty-year experiments, 22 of 37 initializations remained stable throughout, and analysis focused on those runs. Experiments with warmer prescribed sea-surface temperatures exposed climate drift and responses departing from expectations under the largest tested warming.

38. From RL to IRL — Gaurav Mishra, Amazon AGI Lab

    Neither DOM access nor screenshots alone guarantee enough context to distinguish the intended action from distracting or adversarial content.

39. Underactuated Robotics: Planning and Control through Contact

    Hybrid contact models combine continuous dynamics within a mode with discrete events at mode transitions. A guard identifies when an event occurs; a reset changes the state, for example the velocity change during impact. A foot can transition between flight, heel contact, full-foot contact and toe contact. These modes impose different motion constraints, so a smooth free-space controller cannot simply assume its previous dynamics still apply. For a bouncing ball, height reaching zero triggers an impact reset that reverses and scales vertical velocity. Planning with a fixed contact-mode sequence can connect continuous trajectory segments using guard/reset constraints; discovering the sequence is a separate harder problem.

40. Physion: Evaluating Physical Prediction from Vision in Humans and Machines

    Physion evaluates whether two marked objects will contact as a scene evolves. Its simulated scenarios include collisions, support, containment, falling, linked objects, rolling, and cloth. Event prediction permits comparison across models that predict pixels, object representations, or physical states without requiring every model to render images. This supplies a concrete task-relevant alternative to judging future frames only by visual similarity.

41. Towards Accurate Generative Models of Video: A New Metric & Challenges

    Unterthiner, van Steenkiste and colleagues introduced Fréchet Video Distance in a December 2018 preprint to assess video quality, temporal coherence and diversity together. FVD embeds complete videos using a pretrained video network and compares Gaussian approximations fitted to the feature distributions of reference and generated samples. The paper reports agreement with human judgments in its experiments. It explicitly distinguishes distribution-level assessment from measuring one predicted continuation against its corresponding reference.

42. On the Content Bias in Fréchet Video Distance

    Songwei Ge and colleagues test temporal sensitivity by separating per-frame distortion from changes in distortion across frames. Their experiments show that substantial temporal corruption can produce relatively small changes in conventional FVD. One published example receives a better FVD despite visibly worse temporal consistency. Replacing supervised I3D features with the tested self-supervised video features reduces this content bias.

43. Project Genie: Experimenting with infinite, interactive worlds

    Google's January 29, 2026 launch report describes Project Genie as an experimental web prototype for creating, exploring and remixing interactive worlds. During exploration, generated continuations respond to the user's movement and camera actions. The report explicitly lists failures to follow real-world physics, imperfect character controllability, control latency and a sixty-second generation limit. It also distinguishes the prototype from the earlier Genie 3 model preview: some announced model capabilities were not included.

44. Training Agents Inside of Scalable World Models

    Danijar Hafner, Wilson Yan and Timothy Lillicrap's September 2025 Dreamer 4 report trains a transformer world model on recorded Minecraft video and mouse–keyboard interactions, then improves behavior through imagined experience. Its shortcut-based dynamics design targets the cost of repeatedly generating training trajectories. Evaluation subsequently executes the learned agent in Minecraft rather than scoring only imagined outcomes. Across 1,000 hour-long evaluation episodes, the reported diamond acquisition rate is 0.7%, demonstrating successful transfer on some runs while making its limited reliability explicit.

45. Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo

    The presented benchmark results use replay-based open-loop evaluation; the speaker treats simulation and road testing as additional requirements for validating the prototype.

46. Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

    Simulation should represent realistic starting conditions and support counterfactual trajectories as agent behavior changes.

47. SFT Trainer: objective, shifting, and masking

    For prompt x and target response y, supervised fine-tuning minimizes negative conditional log likelihood: L = -sum_t m_t log pi_theta(y_t | x,y_<t), optionally normalized by the count of included tokens. The mask m_t excludes padding and can exclude prompt or non-assistant tokens. Each prediction conditions on the demonstrated prefix, rather than a prefix sampled from the model; this is teacher forcing. The loss rewards probability assigned to the demonstrated next token, not execution success or truth directly. Current TRL supports completion-only and assistant-only loss with dataset/template requirements. Changing masking changes which behavior receives direct supervision, even when the visible example text is identical.

48. Scaling to Long Horizons

    Value-model bootstrapping provides a training signal before the final outcome, at the cost of introducing value-model bias.

49. Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

    Build an offline improvement loop that grounds simulation in deployment logs and uses categorized failure triage to decide what to improve.

50. Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo

    The research uses generated video with controllable weather and time of day to evaluate EMMA, reporting worse camera-only planning under rain and at night.

51. Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

    Build simulators grounded in production traces to gain backend control and construct tasks with known reachable outcomes.