Chapter 1 — Why an Agent Needs to “Imagine the Future”

The big picture

A tiny story

A robot sees the wall, doorway, and goal perfectly. It still drives into the wall because it always chooses the move that looks closest to the goal right now. The useful route first moves away, reaches the doorway, crosses, and turns back.

Seeing nouns is not enough. Acting needs a purpose, a model of consequences, and time. A direct policy can still be powerful and use history; the distinction is that a world-model planner explicitly rolls out candidate futures and searches them at decision time.

The real rule

In this course, “imagination” means a conditional computation, not a human mental movie:

current latent history + proposed action -> predicted next latent

LeWM’s encoder describes the current situation in latent space. Its predictor asks, “What follows if I apply this action?” A useful decision simulator needs three things:

• Action sensitivity: consequentially different actions can change the prediction.
• Rollout: a predicted state can be advanced again.
• A decision interface: a cost can compare imagined outcomes with a goal.

The core model predicts latent states, not future videos. The paper’s auxiliary decoder is for inspection, not the training target or planner. One deterministic prediction can also hide several plausible futures; the baseline interface supplies no calibrated uncertainty distribution.

MPC makes a plan provisional: propose sequences, imagine endpoints, score them, execute a configured prefix, then observe again. In the reported LeWM v3 setup, the horizon is five model steps and all five are executed before replanning. With five environment actions per model step, feedback arrives after 25 environment steps. A one-action loop is valid MPC, but it is not that reported timing.

The baseline endpoint score is squared Euclidean distance to the encoded goal. That is a convenient latent ruler, not reachability, path length, or travel time.

The trick that fools us

Remove look-ahead and a greedy agent hugs the wall because every immediate move toward the goal seems good. Remove feedback and an initially sensible route can become stale after an observable block appears. Add MPC and the agent may reroute—but only if the observation, representation, model, cost, and search are useful.

The same collision can begin in different places:

1. Perception: the latent omits the wall or motion.
2. Imagination: the predictor passes through the wall or drifts.
3. Selection: the cost prefers the wrong endpoint or CEM misses the door.

Replanning cannot repair a systematic model that redraws the same fake shortcut every cycle. A final failure does not identify its cause.

Evidence receipt

LeWorldModel v3 and the frozen official repository support the pixel encoder, action-conditioned latent predictor, autoregressive rollout, terminal latent cost, CEM search, and configurable MPC prefix. The repository delegates parts of planning and evaluation to stable-worldmodel; the exact solver convention must be recorded rather than guessed.

The TwoRoom detour, blocked-door story, and three-part diagnosis are teaching constructions. They show why look-ahead and feedback can matter; they do not prove that every direct policy fails or every model-based controller succeeds.

Quick check

1. Why can moving away from the goal be the correct first action?
2. What action-swap result would warn that the predictor ignores control?
3. Which error can MPC keep repeating even after every new observation?