Chapter 31 — Global Non-Collapse Does Not Guarantee Preservation of Task-Relevant Dynamics

The big picture

A tiny story

A passport office gives everyone a different card. The archive never collapses. But each card records shirt color and wallpaper while omitting direction of motion and which gate can be crossed.

Latents can do that too. Background texture can fill many dimensions while a small velocity or contact variable disappears. Global variance, rank, and a pretty projection rule out some failures; they do not prove task-relevant dynamics survived.

The real rule

Use matched interventions.

• Hold simulator state and transition fixed; change only an appearance factor the environment guarantees is dynamically irrelevant.
• Hold appearance nearly fixed; change one physical variable known to alter the next outcome.
• Hold the complete observable history fixed; change only the proposed action, reset the simulator, and compare every predicted branch with its matching real transition.

“Appearance” is not automatically nuisance. Traffic-light color, shadow, or texture can signal real control state. The experiment—not intuition—must guarantee invariance.

Observation sufficiency comes first. One frame cannot reveal two opposite velocities that render identically. Give enough history to make motion observable before diagnosing representation failure. If history contains the clue but latents and predicted futures still collide, the test is stronger.

Action sensitivity has three levels: does output change, does it change in the right direction, and does the response depend on state–action interaction? Shuffled, zero, opposing, and withheld actions help, but can create out-of-support pairs. Nonzero response alone is not correct dynamics.

The trick that can fool us

Make the same PushT pose appear under three harmless floor textures. Then make a nearly identical view with opposite motion or different contact. If the harmless texture changes prediction more than the physical intervention, the model prioritizes the wrong distinction in this controlled environment.

PhyLatent names three operational failures:

1. physical invariance collapse: appearance-only movement dominates a selected physical transition;
2. physical identifiability collapse: states far apart under selected physical variables map close;
3. counterfactual dynamics collapse: materially different action consequences compress together.

These are definitions inside one recent paper, not a universal taxonomy or theorem-level identification. Thresholds, simulator variables, scale, observability, and pair selection matter. PhyLatent’s described baseline uses a stopped future target, unlike original connected-target LeWM v3, and its proposed training uses privileged physical targets.

Experiment receipt

Later LeWM-family proposals make different supervision purchases:

• PSG-JEPA keeps LeWM forward loss and SIGReg, then adds training-only heads for proprioceptive state and multi-horizon joint-angle change (q_{t+k}-q_t). Privileged labels are absent at test time.
• SCALE adds (1-\mathrm{corr}(d_z,d_q)), aligning squared latent pair distances with squared distances in a standardized selected simulator state. Its regression control uses the same labels without shaping pairwise geometry. Authors report improvement in all 15 task–solver averages; each training configuration has one seed, while evaluation sets measure episode sampling.
• AC-MTM removes SIGReg and keeps connected forward MSE plus a training-only Action-NCE inverse head that identifies the recorded continuous-action block among in-batch actions. It has no target network, stop-gradient, pretrained encoder, or reconstruction loss; the inverse head is discarded at test time. In a matched three-training-seed study it matches SIGReg on average over four standard tasks but trails on PushT, and reports 80% versus 58% on its OGBench Visual Scene trajectory-goal stress test. The 52% random baseline is one 50-episode run.

PSG-JEPA and SCALE use privileged simulator labels. AC-MTM instead requires informative action variation and observable consequences. None is an ingredient of original v3, and none is independently reproduced here.

For every new diagnostic, save simulator reset state, history, intervention, real outcomes, appearance control, support, checkpoint, thresholds, and seeds. Cross any representation improvement with fixed-planner ranking and success; a better probe can remain unread by the cost.

Quick check

1. How can a globally non-collapsed representation lose a control variable?
2. What must be true before an appearance change is called irrelevant?
3. Why must action branches be compared with real matching outcomes?