Chapter 0 — Before You Begin

The big picture

This course has one spine: observe → compress → imagine → search → act → diagnose. Every later box belongs somewhere on that line.

A tiny story

Put three tools on a robot’s workbench. A camera records one journey. A direct policy chooses an action, possibly with rich memory, without explicitly searching many futures at decision time. A world-model planner rehearses proposed actions inside a learned model before choosing.

LeWorldModel, or LeWM, is the third tool. It learns from ordered visual–action trajectories. An encoder turns an observation into a compact latent “passport.” An action-conditioned predictor advances those passports. After training, CEM searches action sequences through the frozen model, and MPC executes a configured prefix before looking again.

The passport is not a literal map. Two nearby latent points may sit on opposite sides of a real wall. A compact state can preserve wallpaper and forget the doorway.

The real rule

Keep four names separate:

• A world model is a job: represent enough change to reason about consequences.
• JEPA is a broad idea: predict a learned target representation from a context representation.
• LeJEPA studies predictive representation learning with SIGReg and non-collapse.
• LeWM adds time, actions, latent dynamics, and a planning loop.

Our baseline is LeWorldModel v3, revised 2026-06-03, plus the official code frozen at 8edfeb336732b5f3ce7b8b210d0ba370a09e2cac, dated 2026-05-22. LeJEPA v3 and the JEPA position paper v0.9.2 explain related ideas, not interchangeable implementations.

Original LeWM v3 uses one shared trainable visual encoder. Its next-observation targets stay connected to that encoder. It uses no training-time stop-gradient, no EMA teacher, and no pretrained visual encoder. Baseline world-model training needs no task reward or goal, but it absolutely needs ordered actions. The goal arrives later, during planning.

LEARN:     observations + actions -> latent dynamics + non-collapse pressure
FREEZE:    encoder and predictor
ACT:       current + goal -> CEM plans -> execute a prefix -> observe again
DIAGNOSE:  data -> representation -> dynamics -> cost -> search -> execution

The trick that fools us

Imagine a predictor that cannot hear actions. Most recorded TwoRoom trips follow the same doorway route, so history alone may predict the common continuation and produce a respectable loss. During planning, however, “up,” “right,” and “stay” all create nearly the same imagined future. CEM is holding an audition for a judge with earplugs.

That counterexample proves only this: predicting common chronology is not enough for an action-conditioned planning model. A different prediction after an action swap is necessary evidence, but it still does not prove the direction, size, or long-term effect is correct.

Evidence receipt

The course’s evidence cutoff is 2026-08-20. arXiv:2608.10145v1 independently reimplemented one single-seed TwoRoom system and reproduced a position probe and repository-protocol planning; it did not reproduce PushT, OGBench-Cube, or multi-seed robustness. arXiv:2608.12959v1 reused that system, so it is a follow-up in the same evidence chain, not a second independent replication.

TwoRoom makes walls, detours, and bad costs easy to see. PushT adds contact, rotation, and sliding. Neither benchmark proves general physics or safe real-robot control. Tutorial stories and diagrams explain mechanisms; they are not paper results.

Quick check

1. Why is LeWM a rehearsal room rather than a camera or policy?
2. Which original-v3 facts rule out an EMA-teacher diagram?
3. Why can a reward-free world model still need a goal when it acts?