Chapter 15 — Goal-Conditioned Planning: From “Where Am I?” to “Where Do I Want to Go?”

The big picture

A tiny story

The world model has finished learning an atlas from reward-free observation–action journeys. A traveler now arrives with a current photograph and a destination postcard. The atlas can predict consequences, but prediction alone has no preference. The postcard supplies one.

The latent atlas may still distort walls, direction, time, and data support. A goal image is not a success detector, route, reward used in training, or proof that the destination is reachable.

The technical backpack

Current and goal observations use the same frozen visual encoder, so their coordinates can be compared. Their roles remain different:

current history -> initial condition for dynamics
candidate actions -> controls for dynamics
goal image -> terminal comparison only

Swapping the goal while keeping current history and candidate actions fixed must not change baseline rollouts. It should change only their ranking. Goal preprocessing, camera, environment, and support still need to match the checkpoint’s input contract. A shared encoder cannot repair mismatched crops or normalization.

CEM samples complete action sequences over horizon H. Every candidate begins from the same current latent history and receives its own actions. The frozen predictor rolls each sequence autoregressively. The baseline emits latent tokens, not videos, and supplies one deterministic path without calibrated confidence.

For every candidate, the frozen criterion takes the final predicted latent and computes:

J(a0:H-1) = sum_d (z_hat_H[d] - z_goal[d])^2

Only the terminal token is scored. There is no explicit wall, collision, intermediate safety, control effort, route length, or support penalty. Squared Euclidean distance is cheap and parameter-free after representation learning; it is not reachability by definition.

CEM uses low-cost samples as elites and refits its action proposal. “Best” always means under this frozen model, terminal cost, finite samples, and recorded solver convention. Appendix B prose describes returning the final proposal mean; Algorithm 2 also permits the best sample or the first action of that mean. The LeWM repository’s external solver revision is unpinned, so exact reproduction must record it.

Reward-free training and goal-directed action are therefore compatible: dynamics are learned without a task reward, then an inference-time objective tells search which imagined future to prefer.

The trick that fools us

Roll one fixed candidate population once. Then swap among three supported goal images. The terminal latents must stay identical while costs reorder. If rollouts change, the goal leaked into dynamics or randomness was not held fixed.

Now use an impossible or support-poor goal. Every candidate still receives a number, and CEM still returns a plan unless an explicit abstention rule is added. Numerical ranking is not feasibility.

Protocol details can dominate results. LeWM v1/v2 used TwoRoom goal offset/budget 100/150; current v3 revised these to 25/50, matching the repository on those fields. One independent audit reported 14% versus 84% on the same released weights under the two protocol readings. In a paired comparison, changing to each episode’s recorded target with a 150-step budget moved 84% to 8%; because both goal construction and budget changed, that is not a one-variable causal result.

Evidence receipt

LeWorldModel v3, frozen jepa.py, and eval.py support frozen goal-image planning and terminal latent cost.

The same-system Research Frontier follow-up arXiv:2608.12959v1 reported that terminal latent L2 saturated and then decreased with true separation at long range in one TwoRoom evidence chain, and that alternative objectives changed behavior. It does not add a reachability head to original LeWM or establish a universal repair.

Quick check

1. Which two inputs enter dynamics, and which input enters only the cost?
2. Why can an unreachable goal still produce a CEM plan?
3. What should remain unchanged when only the goal postcard is swapped?