Chapter 25 — Skip training: extract features from the public checkpoint

Inspect a borrowed machine before plugging it in

The download includes more than numbers. It includes custom assembly code. If time and color axes are swapped, the machine may still run and return plausible-shaped nonsense. Safe use begins with code review, a pinned version, and explicit tensor checks.

Pin and audit the release

LeVJEPA-VideoMix-Large is a 303.1M-parameter ViT-L/16 trained on 1,806,869 VideoMix clips. It uses 16 frames, tubelet 1, RoPE, and block_causal attention.

The repository requires trust_remote_code=True, meaning downloaded Python executes locally. Inspect config.json, configuration_levjepa.py, and modeling_levjepa.py; use an isolated environment without sensitive credentials; pin the audited model commit e831a03 instead of floating HEAD.

At minimum ask: Which files does auto_map import? Does code initiate network calls, read environment variables, or dynamically execute strings? Are weights in safetensors? Does the license permit your use? After a reviewed online download, test the same cache offline with local_files_only=True. revision pins both config and custom code.

import torch
from transformers import AutoModel

repo = "galilai-group/LeVJEPA-VideoMix-Large"
revision = "e831a0347737fcaa660b39c57d41c109de399845"
model = AutoModel.from_pretrained(
    repo, revision=revision, trust_remote_code=True
).eval()

# raw is already in [0,1]. Real video also needs matching resize/center crop to 224.
raw = torch.rand(1, 3, 16, 224, 224)  # [B,C,T,H,W]
mean = torch.tensor([0.485, 0.456, 0.406])[None, :, None, None, None]
std = torch.tensor([0.229, 0.224, 0.225])[None, :, None, None, None]
video = (raw - mean) / std
with torch.inference_mode():
    out = model(pixel_values=video)
print(out.last_hidden_state.shape, out.pooler_output.shape)

Expected shapes are torch.Size([1, 3137, 1024]) and torch.Size([1, 1024]): one [CLS] plus 16×14×14 patch tokens. Eval mode sets token dropping to zero, so all patches return. There is no classification head; pooler_output is the [CLS] representation, not class probabilities.

Full-token block-causal ViT-L attention is memory-hungry. Begin at batch 1. Do not quietly switch to full attention to make it fit. Add assertions for (1,3137,1024) and torch.isfinite. On CUDA or MPS, model, video, mean, and standard deviation need one device. Log peak memory before increasing batch. eval() disables training-only dropping and dropout; encoder LayerNorm does not keep BatchNorm-style running statistics. Repeated inference should agree within the hardware’s numerical tolerance.

Training samples roughly 7.5 fps, so 16 frames span about two seconds. Repeating an image with image.unsqueeze(2).repeat(1,1,16,1,1) can produce image features, but contains no true motion. Preserve model.config.attn_mode == "block_causal". Released tensors are the encoder’s EMA evaluation weights; they are not an EMA teacher and do not include the training projector.

The model card uses CC BY-NC 4.0. Review licensing before commercial use.

Checkpoint records

• Release model card and pinned revision
• Pinned custom model code
• Pinned official weight instructions

Before inference

1. trust_remote_code=True executes downloaded code; audit it and pin a revision.
2. Input is ImageNet-normalized [B,C,T,H,W], normally 16×224×224.
3. Output is a frozen representation, not a classification answer; the release uses evaluation EMA weights.

Plug-in test

1. Why are there 3137 output tokens?
2. Can you switch attn_mode to full and still call it protocol-matched inference?
3. Is pooler_output an ImageNet probability vector?