Chapter 13 — The Original LeWM’s End-to-End Training Mechanism

The big picture

A tiny story

An easy but wrong sketch shows a context encoder, a pale target tower, a stop sign, and an EMA arrow. It looks familiar because BYOL, I-JEPA, and V-JEPA use related target mechanisms in named versions. It is not original LeWM v3.

LeWM is more like a mapmaker and route-rule learner editing one live map together. SIGReg is a zoning rule that prevents every address from becoming one lot. Zoning can spread a city without making its roads correct.

The technical backpack

The v3 training topology is:

observations -> one shared trainable encoder -> all latents
context latents + recorded actions -> predicted future latent
predicted future <-> connected future latent
observation latents -> SIGReg
one total objective -> joint update

The word target means “what the prediction is compared with.” It does not specify who owns the weights or how they update. Frozen train.py slices future latents from the same encoded sequence and does not detach them before the prediction loss. Thus prediction pressure reaches the shared encoder through both context and future uses, as well as the action encoder and predictor.

The Gaussian reference is analytic, not a learned teacher. The recorded next observation is data, not a teacher network. Original v3 also sets visual pretraining off; “from scratch” means the visual encoder exists but does not begin from an external pretrained checkpoint.

Prediction alone permits every place to receive the same address. At the ideal SIGReg target, a repeated point cannot match a non-degenerate isotropic Gaussian. The joint objective therefore opposes the easiest exact constant solution. It does not make collapse impossible in finite optimization or ensure that a spread representation preserves rare task variables.

“End to end” stops at the training boundary. Dataset construction is not learned. During later CEM planning, encoder and predictor weights are frozen and no planning gradient returns to retrain them.

Named graphs remain different: BYOL v3, I-JEPA v3, and V-JEPA v1 use stop-gradient/EMA target mechanisms in their cited recipes; LeWorldModel v3 uses a shared connected encoder plus SIGReg. This is a mechanism comparison, not a performance ranking across unequal data and tasks.

The trick that fools us

Three .detach() calls can look identical in a keyword search while meaning different things:

• future target detached before training loss: changes the method;
• metric copy detached after loss: logging bookkeeping;
• goal detached during frozen planning: no model update was possible anyway.

Audit value, timing, and phase. Then run a tiny topology probe:

A: future target connected
B: future target detached before comparison
C: target connected + SIGReg

Hooks should show B losing only the target-side prediction route, while C adds an encoder-side population route. One batch reveals connectivity, not which variant will train or plan better.

Evidence receipt

The graph comes from LeWorldModel v3, frozen train.py, and frozen model/lewm.yaml. Comparisons are tied to BYOL v3, I-JEPA v3, and V-JEPA v1.

Allowed claim: LeWM v3 is fully end to end and structurally differs from those named stopped-target/EMA recipes. Do not claim EMA is obsolete, SIGReg universally replaces teachers, from-scratch training is inherently better, or connectivity proves learned physics.

Quick check

1. Why does “target” not imply “teacher”?
2. Which learned paths receive prediction pressure, and which receive SIGReg directly?
3. Which detach changes v3: logging, frozen goal, or future target before loss?