Chapter 27 — Give Latent Space a “Health Check”

The big picture

A tiny story

A patient can have perfect blood pressure and still be unable to climb stairs. The measurement is real; it is simply not the whole body.

A latent cloud can have broad variance while losing contact, velocity, or wall side. It can draw a beautiful smooth path by remembering only a slow background. Freeze the specimen first: checkpoint, preprocessing, history, attachment point, episodes, and sampling rule.

The real rule

Use several instruments and keep their questions separate:

• Mean and coordinate variance: catches constants, dead axes, scale drift, and reload bugs.
• Covariance spectrum and effective dimension: shows where population spread lives. A low value can be collapse or a legitimately simple task.
• Pairwise distance and duplicates: catches pile-ups, but not reachability.
• Nearest neighbors: show original observations, temporal distance, actions, state differences, and next outcomes; exclude self and adjacent duplicates unless duplication is the test.
• Trajectory continuity: place visual change, latent change, and task-variable change on the same time axis. Separate real encoded states from self-fed predictions.

Slice every reading by doorway, wall, free motion, approach, first contact, and maintained contact. Broad nuisance variation can hide a missing task distinction.

LeWM v3 reports that PushT latent trajectories become straighter during training under its chosen post-hoc measure and are straighter than PLDM in that analysis. Baseline LeWM has no explicit straightness loss. Pair straightness with representation spread, action sensitivity, and matched trajectory regions: a collapsed line is perfectly straight.

The trick that can fool us

t-SNE and UMAP are mapmakers that flatten a learned high-dimensional world. Repeat seeds, neighborhood settings, and reasonable metrics; keep every map, audit original-space neighbors, and do not name axes “position” or “time.” Islands, bridges, empty gaps, areas, and far-cluster distances can be projection artifacts.

LeWM v3 uses t-SNE on a structured PushT state grid and reports a qualitative sampled neighborhood pattern. UMAP is a tutorial extension. Neither reveals true topology, reachability, causality, or understanding.

An auxiliary decoder is another viewer. The paper describes a lightweight transformer decoder from the 192-dimensional final [CLS] representation; a 224-pixel, patch-16 example uses 196 learned patch queries. Its chronology wording is inconsistent across sections. Use one explicit tutorial protocol: freeze a named checkpoint, train the decoder afterward on an episode split, and keep reconstruction outside baseline LeWM training and planning.

A sharp image shows decoder-dependent reconstructability, not complete state or model use. The paper’s PushT reconstruction-loss ablation has lower planning success than baseline under its tested setup; it does not prove reconstruction is always harmful.

Experiment receipt

For every neighbor gallery or projection, save sample identities, original latent distances, labels added after fitting, algorithm settings, seed, and the original observations. For every trajectory, mark door crossing, contact, camera change, and action; use the same reference projection across episodes.

Do not confuse baseline’s emergent post-hoc straightening with the separate Temporal Straightening for Latent Planning, which uses an explicit curvature regularizer, a stop-gradient target, and gradient-based planning. Those ingredients do not belong to original LeWM v3.

The source boundary is LeWorldModel v3 and frozen core commit 8edfeb3. The frozen repository does not include the complete t-SNE, straightening, decoder, or VoE scripts and settings, so exact reproduction of those analyses is not established here.

Quick check

1. Why can low effective dimension be healthy in one task and harmful in another?
2. What turns a nearest-neighbor gallery into a dynamics audit?
3. What does a sharp post-hoc decoder establish—and what does it not?