Chapter 26 — Freeze the encoder and test your own videos

Split families before cutting clips

Randomly cut one long recording into train and test windows and the score may look wonderful. The test question could simply be the second after a training question, with the same person, wall, and watermark. Group by source video, scene, subject, or date before extracting windows.

Four questions need four readouts

Fix preprocessing, model revision, and frame rate first. For [B,3,16,224,224], freeze every encoder parameter and assert requires_grad=False.

A minimal linear probe maps [CLS] features [B,1024] through one linear layer. Alongside accuracy, report balanced accuracy or macro-F1 when classes are uneven.

The paper’s attentive probe gives a learned query cross-attention access to all 3,137 frozen tokens, adds a query residual, then applies a two-layer MLP, GELU, LayerNorm, and classifier. This is higher capacity and must not be called a linear probe.

For retrieval, L2-normalize [CLS], rank cosine similarity, and report Recall@1/5 and mAP. For dense study, remove [CLS] and reshape out.last_hidden_state[:,1:] to [B,16,14,14,1024]. Query-patch maps, small segmentation or tracking heads, and metrics such as mIoU, J&F, or point error answer different questions. Remember: patch tokens have the per-frame causal guarantee; whole-clip [CLS] does not.

@torch.inference_mode()
def features(model, raw_bcthw):
    mean = raw_bcthw.new_tensor([.485,.456,.406])[None,:,None,None,None]
    std = raw_bcthw.new_tensor([.229,.224,.225])[None,:,None,None,None]
    out = model(pixel_values=(raw_bcthw - mean) / std)
    cls = torch.nn.functional.normalize(out.pooler_output, dim=-1)
    patches = out.last_hidden_state[:, 1:].reshape(-1, 16, 14, 14, 1024)
    return cls.cpu(), patches.cpu()

This assumes [0,1] input already cropped to 224. Make conversion explicit otherwise.

Split source groups into train/validation/test, then sample windows inside each. Fit normalization and class weights on train only; choose hyperparameters with train/validation; keep test sealed. Include at least three negative controls: shuffled training labels should fall to chance; repeated single frames reveal reliance on appearance; reversed or shuffled frames test sensitivity to time. Compare with a randomly initialized same-shape encoder or simple color/motion baseline.

For a linear probe, caching train features and optimizing only torch.nn.Linear(1024, n_classes) is enough. Verify an encoder parameter digest remains unchanged each epoch. Select with validation, touch test once, run at least three seeds, and report mean, standard deviation, counts per class, and a confusion matrix. Small tests benefit from bootstrap confidence intervals. A stratified split cannot fix identity leakage; use group splitting when subjects recur.

Time tasks deserve an appearance-matched control: use exactly the same frames in forward and reverse order. Failure to beat chance suggests the probe is reading objects rather than sequence. For retrieval, remove same-source neighbors from the gallery. Score dense tasks on a fixed annotated set, not hand-picked pretty PCA clips. Record fps, seconds per window, crop, probe parameters, and revision.

The paper’s 69.5 and 55.0 belong to its own frozen attentive-probe protocol. A new dataset, split, or head is a new experiment—not an automatic reproduction or victory over those values.

Evaluation references

• Pinned model revision: intended use, shapes, and preprocessing
• Paper: frozen attentive-probe protocols and task scope
• Official dense-feature notebook

A clean evaluation

1. Split source videos, people, or scenes before making clips to block near-neighbor leakage.
2. Linear, attentive, retrieval, and dense probes answer different questions.
3. [CLS] summarizes a full clip; use time-indexed patch tokens for causal per-frame state.

Leakage test

1. Why not randomize adjacent two-second windows across train and test?
2. What does far-above-chance accuracy with shuffled labels suggest?
3. How do you reshape [B,3137,1024] after removing [CLS]?