Chapter 13 — Why throwing away 95% can help

Read a moving puzzle through a handful of pieces

Close your eyes and grab a few tiles from an animated picture book. This time you find a wheel and lamp; next time, a driver and patch of sky. Because the visible places keep changing, you have to piece together the event from shifting clues. A tube-shaped mask, by contrast, might hide the hero’s position in every frame.

Do not stretch the story into reconstruction. LeVJEPA has no mask token, decoder, or fill-in-the-pixels objective.

What the code removes

Sampling happens after patch embedding. For each sample, the code assigns random values across the full space-time grid and keeps the required subset. [CLS] is prepended afterward, so it always survives. Dropping runs only in training mode; evaluation and inference use the full sequence.

In a controlled sweep, the authors report ImageNet-1K rising from 33.9 with no dropping to 47.6 at 95%; 90% and 95% give 47.4 and 47.6. Their explanation has two parts: fewer tokens reduce work per step, while changing sparse observations act as augmentation. The paper’s 1 / (1 − ρ) = 20 is a theoretical reduction factor for feed-forward work at ρ=0.95, not a guarantee that every operator—or wall-clock training—becomes 20× faster.

The pattern matters too. Random keeping versus a same-location-through-time tube gives author-reported ImageNet 50.7 versus 39.6. In the stated tubelet=2 comparison, SSv2 is 28.8 versus 26.4. Yet extreme sparsity is not a free victory: in short training, SSv2 declines once dropping exceeds 0.3. Longer training helps in the paper, and motion-preserving sparse schemes remain an open question.

There is also a caption wrinkle. Section 4 broadly says “frozen attentive probe,” while the arXiv v1 Figure 4 caption says “linear probing”; the project page says attentive probing. Treat sweep numbers as reliable within-figure trends, not as free-standing values to compare across tables with different protocols.

Draw random versus tube

Sketch four frames with four cells each. Circle different cells in every frame for random keeping. Circle the same spatial cell in all four for a tube. Which scheme can hide an object sitting at one location for the entire clip? The tube can.

Evidence trail

• LeVJEPA v1: token-dropping ablation
• Official project page: random-dropping plot and values
• Pinned training keep-set implementation
• Pinned config: 95% during training

What survives the drop

1. Dropping is real removal during training; discarded tokens are neither encoded nor reconstructed.
2. At 95%, global/local views keep 157/29 patches; [CLS] is separate and always present.
3. The authors report better ImageNet behavior and efficiency, but short-run motion performance can fall.

Sparse-reading test

1. Are removed positions replaced by mask tokens?
2. How long is the global sequence including [CLS] after 95% dropping?
3. Does the ImageNet trend prove 95% is best for every motion task?