Chapter 5 — Trajectory Data: To the Model, the World Is Not an Image Collection

The big picture

A tiny story

Make two photo albums from one TwoRoom journey. The first keeps every frame in order and writes the controls between them. The second shuffles the same pictures and throws the controls away. Both show what the rooms look like. Only the first can teach what followed after a particular action.

An episode boundary matters too. Joining the final frame of one run to the first frame of the next teaches a fake teleport. A world model needs transitions, not merely a visual inventory.

The real rule

LeWM’s world-model training is offline and reward-free. It replays recorded episodes and uses no task reward or goal in the base objective. That does not mean action-free, goal-free deployment, or unlimited extrapolation.

In the frozen data configuration, frameskip=5. One model transition groups five consecutive lower-level environment actions into a dense block; it does not keep only the last action or repeat one action five times. train.py z-scores non-pixel columns including action, and the action encoder width is set programmatically to:

frameskip × environment action dimension

The frozen defaults are history_size=3 and num_preds=1. A sampled window therefore contains four observation positions. The first three latents form causal context; the sequence shifted by one supplies three one-step targets:

z0 -> z1
z1 -> z2
z2 -> z3

num_preds=1 is the offset, not “use only the last pair.” The v3 paper reports history one for TwoRoom and three for PushT/OGBench-Cube, whereas the frozen global configuration uses three. Record which source the run follows.

Coverage is more than an x-y heatmap. A transition depends on what is seen, recent motion or contact, and the proposed action. Visiting every position slowly does not cover fast doorway motion. Seeing a PushT object at one angle while still does not cover the same image during a collision. A fluent prediction outside this combined support is still unsupported.

The trick that fools us

Collector A records millions of beautiful frames while circling one room. Collector B records fewer frames but approaches both doors, collides, pauses, crosses in both directions, and varies speed. A has more images; B may contain much better cross-room dynamics evidence.

Now shift B’s action log by one time index. Its visitation map still looks excellent, but every consequence is paired with the wrong control. Or let a training window cross a reset: the model learns teleportation. Shape checks alone may miss both errors.

Before training, replay complete episodes, inspect timestamps and action blocks, count full doorway crossings, condition action histograms on region and motion, and list missing combinations.

Evidence receipt

LeWorldModel v3, frozen train.py, and the frozen TwoRoom data configuration support the trajectory interface and configuration facts. The paper reports 10,000 TwoRoom episodes averaging 92 environment steps, collected by a noisy doorway-travel heuristic. That is a dataset description, not a universal minimum or proof of complete coverage.

The independent TwoRoom reproduction identified dense action gathering, runtime action width, ImageNet pixel normalization, and action z-scoring as reproduction-critical conventions not all visible in YAML alone. It remains one single-seed environment.

Quick check

1. Why can shuffled copies of the same frames not teach action-conditioned dynamics?
2. What false event appears when a window crosses an episode reset?
3. Why does full x-y visitation still fail to prove transition coverage?