Chapter 6 — Match the cards, but do not leave every card blank

The perfect score that teaches nothing

Suppose the teacher checks only whether the wide and narrow cards agree for each video. A flawless cheat is to write the number 0 for the cat, the train, and the pouring cup. Every pair matches. The loss is zero. The cards distinguish nothing.

The optimizer has not malfunctioned. The assignment allowed this answer. Once we say “pull paired views together,” we must also say how the entire class avoids crowding onto one point.

Two kinds of collapse

Invariance means a representation should stay stable under changes we consider irrelevant. LeVJEPA measures squared distance between a global embedding z_0 and each local z_v. Crops and color can differ; if the views still cover the same 16 frames, their summaries should be near. Gradients flow through both ends, so the global and local paths move together rather than acting as teacher and student.

With that term alone, the constant function z(x) = c is an exact solution: every difference is zero. This is complete collapse.

There is a quieter failure too. A vector may have hundreds of coordinates while only two independent directions vary. Other coordinates stay fixed or copy one another. This dimensional collapse leaves a representation that looks large on paper but carries little usable variation.

Other self-supervised systems fight these shortcuts with negatives, variance/covariance penalties, or asymmetric EMA-teacher and predictor designs. LeVJEPA chooses SIGReg. It looks at the cloud formed by a batch of embeddings and pushes that cloud toward a zero-mean, equal-scale, uncorrelated isotropic Gaussian. A dot or a flattened line cannot satisfy that shape.

The division of labor is clean. Invariance says how different views of one sample should relate. SIGReg asks what many samples together look like. Invariance alone permits one answer. SIGReg alone could make a beautiful cloud without matching the views of any video. The fixed weighted sum supplies both pressures.

Non-collapse is not understanding. A healthy cloud may encode wallpaper, camera model, or another shortcut. Dataset audits and downstream evaluations are still required.

Four points on paper

Write (0,0), (0,0), (0,0), (0,0). Any paired-view distance can be zero, yet the points vary in neither direction. Now write (−1,0), (1,0), (0,−1), (0,1). The cloud at least spreads out. Its shape alone still cannot tell you whether the points mean “running,” “pouring,” or “which camera filmed this.”

Evidence

• LeJEPA v3: isotropic Gaussian target and anti-collapse theory
• LeVJEPA v1: bidirectional invariance and SIGReg
• Pinned LeVJEPA implementation

The compact version

1. Invariance alone gives a constant encoder a zero-loss escape route.
2. SIGReg constrains the batch distribution to resist complete and dimensional collapse.
3. “Did not collapse” means “did not degenerate,” not “understood the world.”

Check the shortcut

1. Why can a constant encoder fool the invariance loss?
2. Is gradient stopped on the global path?
3. Does a well-spread cloud alone prove the model learned actions?