Chapter 2 — Why Not Predict the Next Image Directly?

The big picture

A tiny story

Forecast A gets the floor texture and shadows wrong, but keeps the agent, wall, and doorway in the right places. Forecast B looks almost perfect, except for a thin strip that closes the doorway. A pixel score may prefer B. A controller should not.

Pixels are not useless. A small red light, cable, wet patch, or contact edge can matter enormously. The lesson is narrower: the place where we measure error decides which differences training pressures the model to preserve.

The real rule

LeWM does not generate the next image for its core objective. It encodes the recorded next observation, predicts that observation’s code from earlier codes and actions, and compares the two codes.

observations: [B,T,C,H,W] -> codes: [B,T,D]
used actions: [B,T-1,A]
prediction and target: [B,T-1,D] <-> [B,T-1,D]

This creates room for selective abstraction. Different lighting can share a useful code if it does not change consequences. But the word “latent” guarantees nothing. A learned code may keep wallpaper and discard the doorway. The core rollout is a chain of vectors, not a hidden video; rendering requires a separate diagnostic decoder.

A control-useful representation should preserve situation differences, action effects, temporal clues, and a comparison useful to the planner. It should also be treated cautiously outside the state–action support of the offline data. Those are desiderata to test, not properties promised by compression.

The frozen path uses one shared trainable encoder. The encoded next observation is connected to it—no EMA teacher and no stop-gradient. SIGReg later opposes the easiest constant-code shortcut, but SIGReg does not label physics or reachability.

The trick that fools us

Turn a “compression dial.” At one end is the full image with every nuisance. At the other is one constant vector for every image. The constant is tiny and perfectly predictable: the predictor can always return it. It is also useless.

A good diagnostic changes one factor at a time. Keep geometry fixed while changing texture and lighting; then keep appearance similar while opening or closing the doorway. The desired pattern is invariance to the declared nuisance and sensitivity to the route-changing cue. Reverse the task—for example, make floor texture indicate friction—to prove that “nuisance” was conditional, not an intrinsic property of pixels.

A pretty cluster plot is not enough. Pair it with held-out prediction, action swaps, and planning outcomes.

Evidence receipt

LeWorldModel v3 and frozen train.py support compact next-embedding prediction, connected targets, SIGReg, and an auxiliary decoder outside the core objective.

A narrow comparison in I-JEPA v3 reported 66.9 top-1 with representation targets versus 40.7 with pixel targets for ViT-L/16 on ImageNet-1K with a 1%-label linear evaluation. The pretraining schedules differed—500 versus 800 epochs. That is a named static-image result, not proof that latent prediction always beats pixel prediction in control.

Permitted claim: latent prediction allows, but does not guarantee, selective abstraction. Do not claim that pixel prediction is inherently inferior, the smallest code is best, or latent error equals task error.

Quick check

1. How can many wrong pixels matter less than one wrong door edge?
2. Why is a constant code easy to predict but useless to control?
3. What controlled pair would test whether the code kept doorway geometry?