Chapter 5 — Keep the meaning; do not repaint every pixel

Three kinds of homework

A teacher covers half a picture of a dog chasing a ball. One pupil must redraw every blade of grass and shadow. Another guesses a high-level feature for the hidden region. Mina gets a different task: compare a wide shot with a crop showing only the dog’s head, then write two nearby story cards.

All three exercises can be made without human labels. They are still different objectives.

Follow the answer tensor

Pixel reconstruction makes color values the answer. Methods such as VideoMAE hide most space-time patches and use a decoder to rebuild them. Neighboring video frames make low-level copying unusually easy, which is one reason structured tube masks make sense for that assignment.

Feature prediction changes the answer from pixels to a learned representation. In I-JEPA and V-JEPA, a context encoder and predictor estimate the encoded target regions. V-JEPA gets those targets from a stopped-gradient EMA target encoder. It avoids repainting every texture, but it most certainly has a “predict the hidden target” graph.

LeVJEPA view invariance asks something else. One global and several local views go through the same encoder and projector. The loss reads only their [CLS] embeddings and brings the local summaries toward the global summary. No predictor. No target encoder. No stop-gradient. No decoder. The global summary changes under gradient descent; it is not a fixed label.

This is why “token dropping” should not be casually renamed “masked prediction.” Randomly discarded LeVJEPA patch tokens never enter the encoder, and no term asks the model to recover them. The surviving sparse tokens are all the model sees on that pass. Dropping changes the compute bill and the difficulty of observation.

“Keep the meaning” also needs a footnote. The objective rewards information shared across crops and appearance perturbations; it does not announce which factors are semantic or which must be ignored. If a downstream task depends on a tiny motion and sparse observation regularly deletes its clue, the representation can suffer. Frozen probes, retrieval, and dense tasks must settle that question empirically.

A reliable diagnostic

Whenever you meet a self-supervised loss, ask: Where does the answer tensor come from? Original color values imply reconstruction. Hidden-region output from another encoder branch implies feature prediction. A distance between global and local [CLS] cards is the LeVJEPA-style view-invariance case.

Source shelf

• VideoMAE: pixel reconstruction with tube masking
• I-JEPA: block prediction in image representation space
• V-JEPA: video representation prediction
• LeVJEPA v1: the view-level [CLS] invariance objective

Take these with you

1. Reconstructing pixels, predicting hidden features, and matching view summaries are three separate training questions.
2. LeVJEPA neither predicts discarded tokens nor rebuilds pixels.
3. It brings same-clip [CLS] embeddings together; downstream evidence must tell us whether their meaning is useful.

Spot the graph

1. Why does LeVJEPA need no decoder?
2. Is there a reconstruction loss on randomly dropped tokens?
3. What is the clearest graph-level difference between V-JEPA and LeVJEPA?