Chapter 28 — How LeVJEPA might feed a future world model

Add one rung, test it, then climb

Suppose you have built a good pair of eyes and want to add a tabletop imagination. Lock the eyes first. Teach the new box only this: “Given what I see now and this push, what should I see next?” If the experiment fails, you know which box to open.

Training everything end to end on day one makes a failure much harder to locate. The encoder may drift, the dynamics may ignore actions, or the planner may simply search badly—and one success rate cannot tell those stories apart.

A conservative first build

Start with the released encoder frozen. Feed it an observation history shaped [B,3,T,224,224]. Use patch states associated with the current temporal slot, z_t:[B,N,D], rather than the [CLS] token that can see the entire supplied clip. Add recorded actions a:[B,H,A] and a causal, action-conditioned dynamics model:

(z_t, a_t) -> z_hat_t+1 -> z_hat_t+2 -> ... -> z_hat_t+H

Encode the real future frames with the same frozen encoder to make targets. Train both one-step and multi-step latent losses. Everything after the frozen encoder—the action input, predictor, rollout loss, and control dataset—is a proposed addition, not part of LeVJEPA v1.

Token alignment needs its own contract. Token i can be compared across time only when camera, crop, frame rate, and patch grid give it a stable meaning. With camera motion and occlusion, state whether the model predicts every patch, only visible patches, aligned object features, or a pooled state. A mean error over a mostly static background can look excellent while every moving object is wrong, so report those errors separately.

The first baseline is almost embarrassingly simple: copy the current state forward. Adjacent video frames are similar. A dynamics model that cannot beat copy-last has not earned its extra machinery. Add no-action and shuffled-action models next; they reveal whether the predictor actually listens to a_t.

Close the loop only after prediction works

For image-goal control, encode the goal as z_goal. A search method such as the cross-entropy method (CEM) samples K action sequences, the learned dynamics rolls each sequence through latent space, and a cost ranks the futures. Model-predictive control executes only the first action, observes the world again, and repeats:

# Proposed pseudocode, not an upstream repository function.
while not done:
    z0 = encode_causal(observation_history)
    candidates = cem.sample(K, horizon=H)
    futures = dynamics.rollout(z0, candidates)  # [K, H, N, D]
    scores = cost(futures, z_goal, constraints)
    env.step(candidates[scores.argmin(), 0])

That final observation matters. It lets the controller correct model error and unexpected changes instead of trusting a long imagined path.

Six gates before a robot demo

1. Interface: shapes are asserted and future-leakage tests pass.
2. One step: the predictor beats copy-last, no-action, and shuffled-action baselines.
3. Rollout: error versus horizon and uncertainty calibration are reported.
4. Counterfactual: different plausible actions produce distinguishable futures.
5. Simulation: data, compute, CEM samples, horizon, and baselines are fixed before comparing success.
6. Closed loop: only then test distribution shifts, stopping rules, human takeover, and constraint violations.

Stop at the first failed gate and diagnose it. A larger robot run will not repair an ignored action input.

A deterministic MSE predictor can blur several possible futures into one. Compare an ensemble, stochastic latent, or energy-ranked candidates, then test coverage and calibration rather than admiring a sample. Longer time scales may warrant a slow latent layer and subgoals, but each addition must beat the single-scale, short-horizon baseline. If the encoder is later unfrozen, track its original frozen probes and embedding spectrum so a narrow control dataset does not erase general visual features or collapse the representation.

The paper notes that block-causal structure permits reuse of past computation. The pinned repository and released Transformers interface do not ship a complete incremental KV-cache API. A streaming implementation is therefore new engineering and should be checked token by token against recomputing the full prefix. “Cacheable in principle” and “a verified cache is available” are different statements.

Sources for the build, not claims of completion

• LeCun’s 2022 autonomous-machine blueprint
• LeVJEPA v1: block-causal design and future directions
• V-JEPA 2-AC and LeWorldModel v3 as primary design comparisons
• Pinned LeVJEPA implementation

Three build rules

1. A world-model extension must add action-conditioned prediction, a cost, search, and MPC explicitly.
2. Freeze the encoder and try to falsify each layer before end-to-end tuning or long-horizon control.
3. A cache that the architecture permits is not a streaming interface the public code already supplies.

Before adding the next rung

1. What are the two indispensable inputs to the smallest dynamics predictor?
2. Why compare it with copy-last?
3. Why does MPC execute only the first action in its chosen sequence?