Chapter 10 — Cut a video into space-time tiles

Turn a flipbook into numbered coins

Lay out all 16 pages of a flipbook and draw the same grid on each. Every square becomes a numerical coin stamped with its page, row, and column. A special [CLS] coin sits at the mouth of the bag to gather a summary. The Transformer can now work with addressed tokens rather than raw pixels.

Count before thinking

Ignore batch and RGB channels for a moment. LeVJEPA’s default spatial patch is 16 × 16; temporal tubelet size is τ = 1. The 3D patch embedding spans only one frame at a time. It does not blend two neighboring frames into one initial token.

For a 16-frame 224 × 224 global view:

grid per frame = (224 / 16) × (224 / 16) = 14 × 14 = 196
patch tokens    = 16 × 196 = 3136

For a 16-frame 96 × 96 local view:

grid per frame = (96 / 16) × (96 / 16) = 6 × 6 = 36
patch tokens    = 16 × 36 = 576

Patch embedding maps each tile to width d, producing a sequence shaped like [T × Hpatch × Wpatch, d]. The published large-model recipe uses factorized 3D RoPE to encode relative time, vertical position, and horizontal position. This lets one encoder accept both grid sizes without interpolating a learned absolute position table. RoPE describes positions; the attention mask, not RoPE, controls future visibility.

During training, 95% of patch tokens are sampled away after embedding. With the pinned code’s rounding, the model keeps about 157 of 3136 global patches and 29 of 576 local patches. Only then is the always-kept learned [CLS] token prepended. The loss directly reads projected [CLS]; no decoder restores the missing patches. Evaluation and inference turn dropping off and use the full token set.

τ = 1 is the paper and released large model’s choice, not a law for every video Transformer. With τ = 2, 16 frames make eight temporal slots and half as many initial patch tokens, but adjacent frames have already been fused. A comparison must account for keep rate and evaluation length too.

Do one count yourself

Take 8 frames at 128 × 128, patch size 16, τ = 1. Each frame has 8 × 8 = 64 patches; the clip has 8 × 64 = 512. Add [CLS] to reach 513. Never include [CLS] in the 95% patch-dropping pool.

Tokenization records

• LeVJEPA v1: per-frame tokenization and architecture
• Pinned module.py: PatchEmbed3D, random keep-set, and [CLS]
• Pinned config: patch 16, tubelet 1, 16 frames, and 95% dropping

Three numbers to retain

1. τ = 1: one initial patch token spans one 16 × 16 area in one frame.
2. A global clip has 3136 patch tokens and a local clip 576, before [CLS].
3. Training discards 95% without reconstruction; evaluation uses all patches.

Token count test

1. What does τ = 1 mean?
2. Why are there 3136 global patch tokens?
3. Can random dropping remove [CLS]?