Chapter 14 — Same-frame teamwork, no peeking into tomorrow

Classrooms that cannot phone tomorrow

Each video frame is a classroom. Children in one room may talk freely and read notes from earlier rooms, but nobody may call a future classroom. At the end of the hall, [CLS] is a master mailbox that receives notes from the whole building. Classrooms cannot read from it. If they could, the last room could mail tomorrow’s answer in, and an early room could retrieve it on the next layer.

The exact visibility rule

For a patch query in frame t and patch key in frame s, attention is allowed when s ≤ t. Thus all spatial patches at s=t see one another bidirectionally. Earlier frames are visible; later frames are not.

“Causal” therefore describes the frame-level patch representations. Training [CLS] reads the entire clip and is not an online state at time t. After random dropping, the implementation recovers each retained patch’s original frame ID from token_ids; it does not infer time from the shortened sequence order.

In one controlled setting—ViT-B/16, 20% K710, tubelet=1, ρ=0.95, V=4, random dropping—the authors report frozen attentive-probe ImageNet top-1 of 50.7 with fully bidirectional attention and 51.2 with block-causal attention. That means no measurable penalty appeared under this protocol. It is not a theorem about all datasets and tasks.

The topology also permits a streaming implementation to cache past keys and values instead of re-encoding history. That is an architectural opportunity. The paper does not train action-conditioned dynamics, a cost, or a planner, so “streaming-friendly foundation” must not become “already plans.”

Color the mask

Draw a 3×3 frame-level attention matrix, query rows and key columns. Color the diagonal and lower triangle green, the upper triangle red. Add [CLS]: its query row is green; the column that would let patches read it is red. If you color that column green, trace the two-layer leak future → CLS → past.

Mask records

• LeVJEPA v1: block-causal definition and ablation
• Official project page: attention pattern and 50.7/51.2
• Pinned anti-leakage mask for [CLS]

Three rules of the hallway

1. Patches attend both ways within a frame and only backward or level across time.
2. [CLS] sees the clip, but patches cannot read it; otherwise future information leaks across layers.
3. 51.2 versus 50.7 is an author-reported probe comparison, not a universal no-cost theorem or planning evidence.

Leakage check

1. Can a frame-3 patch read other patches in frame 3?
2. Why may patches not read [CLS]?
3. Is training [CLS] an online state containing only the first three frames?