Chapter 24 — Run a smoke test that cannot flatter you

Five gates before the expensive run

A new toy train takes two laps before a cross-country journey. Count the carriages. See whether the wheels turn. Verify the red signal stops it. Two laps do not prove national reliability; they catch crossed wires before the costly trip.

Gate 1: shapes

The default loader emits global [B,16,3,224,224] and local [B,10,16,3,96,96]. main.py converts them to encoder order [B,C,T,H,W]. With a 5% training keep-set, 3136 global patches become about 157 plus [CLS] = 158 tokens. Each local’s 576 becomes about 29 plus [CLS] = 30.

Gates 2 and 3: numbers and gradients

pred_loss, sigreg_loss, and total loss must be finite. After backward(), both encoder and projector should have at least one finite, nonzero gradient. Do not demand monotonically falling loss over two batches; random crops, projections, and small-batch noise will wobble.

For a tiny-batch overfit test, freeze the batch and seed, and record feature variance too. Falling MSE alone can mean collapse.

Check dtype boundaries. With normalize_on_gpu=true, workers send uint8 to save roughly fourfold host/IPC memory; main.to_float_normalized() divides by 255 and applies ImageNet normalization. Normalizing a float batch twice quietly ruins it. Log dtype, range, shape, and finiteness before the encoder: uint8 should lie in [0,255]; the model input should be finite float. Global receives spatial crop and normalization; locals add color jitter, grayscale, and flip. Their time indices must match exactly.

Gate 4: future leakage

This test changes only the final two frames. Patch tokens from the first two should remain equal. [CLS] is deliberately excluded because it reads the full clip.

uv run python - <<'PY'
import torch
from module import vit_tiny
torch.manual_seed(0)
m = vit_tiny(img_size=32, patch_size=16, num_frames=4, tubelet_size=1,
             use_rope=True, token_drop_rate=0, attn_mode="block_causal").eval()
x = torch.randn(1, 3, 4, 32, 32)
y = x.clone(); y[:, :, 2:] = torch.randn_like(y[:, :, 2:])
with torch.inference_mode():
    a, b = m(x), m(y)
d = (a[:, 1:9] - b[:, 1:9]).abs().max().item()  # 2 frames × 4 patches
print(f"causal prefix max diff: {d:.3g}")
assert d < 1e-5
PY

Gate 5: resources—and useful failure modes

Record peak device memory, step time, and actual token counts. For OOM diagnosis, reduce batch size, local crops, or switch to vit_tiny, while labeling the changed experiment. For stuck workers, set loader.num_workers=0. For NaNs, check dtype/normalization first, then learning rate, mixed precision, SIGReg batch size, and distributed synchronization. The official workshop notebook runs views, model, both losses, and distribution plots on CUDA, MPS, or CPU.

Turn success criteria into assertions: exact shapes; finite losses and gradients; over 50–100 fixed-batch steps, a lower moving-average loss without all feature standard deviations tending to zero; causal-prefix difference below tolerance; reproducible first-step loss under the same seed. SIGReg resamples directions, so a single value need not match a reference digit for digit. Persistent explosion, NaN, or a near-zero embedding spectrum is the real alarm.

Debugging sources

• Pinned training workshop
• Pinned tensor rearrangement and losses
• Pinned token dropping and causal mask

The smoke-test contract

1. The loader uses [B,T,C,H,W]; the encoder uses [B,C,T,H,W].
2. A causality test compares prefix patches; whole-clip [CLS] may change.
3. Two successful batches prove plumbing, not accuracy or independent reproduction.

Diagnose it

1. Roughly how many tokens does the global training view output by default?
2. Why watch representation variance when MSE falls?
3. Must [CLS] stay fixed after future frames change?