Chapter 7 — SIGReg checks a cloud by looking at its shadows

Inspect the cloud by its shadows

There is a tangled pile of blocks on a table, but its full shape is hard to see. Shine a lamp from different sides. A shadow that shrinks to a dot reveals a flattened direction. Many smooth, centered, bell-shaped shadows give you more confidence that the pile has not been squeezed into a crack.

An afternoon permits only finitely many lamp positions. One small pile cannot check every direction in the universe.

From the lamp to SIGReg

Each view produces a K-dimensional embedding. A batch of them forms the cloud. SIGReg stands for Sketched Isotropic Gaussian Regularization. “Sketched” means checking lower-dimensional summaries. The intended shape is the standard isotropic Gaussian N(0, I): centered at zero, unit scale in every unit direction, with no direction crushed flat.

The Cramér–Wold idea supplies the bridge. If a high-dimensional random variable has the right one-dimensional distribution along every direction, then its joint distribution is determined; in this case it is the standard isotropic Gaussian.

An implementation cannot test every direction. It samples unit vectors a_m and compresses embedding z_i to the scalar inner product <z_i, a_m>. It then compares an empirical characteristic function—a collection of sine-and-cosine fingerprints—with the analytic fingerprint of a standard normal. The paper uses an Epps–Pulley-style statistic, approximates an integral numerically, and aggregates across directions and views.

Why does this notice collapse? If every embedding is equal, its shadows have almost no width, unlike a unit-variance Gaussian. If the cloud lives in a low-dimensional sheet, directions perpendicular to that sheet expose near-zero variance. The penalty sends gradients through the embeddings into both projector and encoder. It needs neither negative examples nor a teacher branch.

Now the boundary. The theorem concerns an ideal distribution under its assumptions. A training step sees a finite batch, finitely many random directions, and finitely many quadrature points; optimization may not reach its target either. A low observed SIGReg loss is not proof of an exactly Gaussian population in every direction. It certainly is not proof that coordinates correspond to true physical state. It is a theoretically motivated pressure against collapse.

Try a deceptive cloud

Draw ten dots on one horizontal line. A horizontal lamp sees a broad shadow; a vertical lamp sees one point. One direction can miss dimensional collapse. More random directions improve the chance of finding it, but finite inspection remains an approximation.

Paper trail

• LeJEPA v3: SIGReg, Cramér–Wold, and the risk argument
• Pinned LeJEPA repository: finite-projection implementation
• LeVJEPA v1: view-wise SIGReg with quadrature
• Pinned LeVJEPA module.py

Three shadows to remember

1. SIGReg uses random one-dimensional projections to test whether high-dimensional embeddings approach an isotropic Gaussian.
2. Constant points and flattened clouds leave abnormal shadows in some directions.
3. Finite batches, directions, and quadrature approximate the theory; a low loss is not a full distributional proof.

Shine the light

1. What does “sketched” mean here?
2. How could a horizontal-only check miss collapse?
3. Does low SIGReg establish that the encoder learned physics?