Chapter 4 — Video can set its own homework

Homework made from the video itself

Nobody at Mina’s school has time to label every frame in ten thousand videos. Her teacher tries something cheaper. He cuts one recording into a wide view and a close-up, then asks, “Do these tell the same story?” The video itself reveals the pairing. No one has to type pouring water or opening a door.

Mina could still cheat by memorizing the wallpaper. Free homework does not guarantee causal understanding.

Where the learning signal comes from

A supervision signal is simply a computable rule telling the parameters how to change. In conventional supervised learning, a person supplies a category. In self-supervised learning, the structure of the data supplies the rule.

LeVJEPA draws global and local views from one 16-frame interval and treats their common clip identity as a positive pair. One shared encoder makes [CLS] embeddings for them. No class words, hand-drawn boxes, or action labels enter pretraining. The objective does not require designated negative videos either.

Video offers a rich source of regularity. Objects move while remaining the same objects. Hands meet tools. Events have an order. If both views preserve the whole time interval, some of those clues can survive their different crops. Still, the loss does not explicitly calculate optical flow, track instances, or judge causation. It asks same-clip summaries to agree and SIGReg to preserve a rich batch distribution. What actually emerges depends on the footage, crop policy, sparse tokens, capacity, and optimization.

Keep training homework separate from the final exam. Pretraining never sees the category answers from ImageNet, Kinetics-400, or Something-Something-v2. Afterward, researchers freeze the encoder and train a light probe on its features. The resulting score shows that a particular protocol could read some appearance or motion information. It does not demonstrate human common sense or planning.

Block-causal attention gives another limited promise: a frame’s tokens use only their own and earlier frames. That supports streaming. Passive viewing still never says, “If I execute action A, this will happen.” Such action-conditioned data and predictors belong to later world-model work.

Make one pair by hand

Take a two-second clip. Do not name its action. Duplicate it. Keep a broad view in one copy and crop the hand in the other, while retaining exactly the same 16 instants. You have made a self-supervised positive pair. Replace the second copy with another clip—even another clip containing a hand—and it is no longer the pair LeVJEPA defines.

Where the pairing comes from

• LeVJEPA v1: unlabeled video, multi-view objective, and frozen evaluation
• Pinned repository: data and multi-crop implementation
• V-JEPA paper: a self-supervised video feature-prediction baseline

Three takeaways

1. In self-supervision, answers come from data structure rather than human class labels.
2. LeVJEPA pairs multiple views of the same 16 frames; it does not directly supervise optical flow or causality.
3. A frozen-probe score is evidence for one task and protocol—not proof of general understanding or planning.

Quick check

1. Does LeVJEPA pretraining require action-class labels?
2. Does same-clip agreement guarantee causal knowledge?
3. Why must we name pretraining and frozen evaluation separately?