Chapter 22 — Ten long walks become a training set

Ten long walks become short training clips

Ten thick travel albums are awkward to lift onto a desk for every question. A better archive keeps photographs in order, stamps every 128 with one journey ID, then fetches 16 at a time without crossing into the next journey. That is roughly the job of the Walking Tours pipeline.

From URLs to Lance rows

The Walking Tours dataset publishes YouTube URLs, not video binaries. The pinned script downloads ten first-person videos, each roughly one to seven hours, preferring 720p60 without audio. The stated download is about 25 GB. Web sources may disappear or be region-blocked, so check terms and research use before downloading. Leave at least 70 GB for downloads plus the roughly 33 GB Lance store.

uv sync --extra data
bash scripts/download_walking_tours.sh
uv run python scripts/build_lance_walking_tours.py --workers 16

The builder derives a sampling step from each source’s true frame rate: nine are about 60 fps, while Wildlife is about 30 fps. All become roughly 15 fps. Frames are resized to short side 384 and saved at JPEG quality 90. Consecutive groups of 128 frames form about 8.5-second episodes.

Each Lance row stores episode_idx:int32, step_idx:int32, frame:binary, h/w:int16, and label:int16; because the set is unlabeled, label=-1. Rows must remain ordered by episode or the loader will infer bad boundaries.

Only complete 128-frame episodes are written. A short tail at the end of a source is omitted. A decoding failure increments failed and skips that episode; successful episodes are then numbered continuously. A zero process exit is therefore not enough. Save source URLs, downloaded sizes, observed fps, per-video episode counts, and failed. If a web video later changes, this log becomes the data fingerprint.

The training default draws num_frames=16 at frame_stride=2 from the 15 fps store: effective 7.5 fps over about 2.1 seconds. A start must remain inside one episode and only selected rows are read. The config comment records 6,067 episodes in the authors’ build; web sources drift, so your builder output—not that number—is authoritative.

Indices are start + [0,2,4,…,30]. The implementation uses span=16×2=32 to test length. With pad_short=false, short episodes are excluded; if padding is enabled, the final frame repeats. clips_per_video=200 samples each valid episode 200 times per epoch. These are repeated draws, not 200 new independent videos. Neighboring clips remain highly correlated, so downstream supervised splits must group by source.

Point to external data without editing code:

LEVJEPA_DATA_ROOT=/mnt/levjepa-data uv run python main.py --cfg job

The loader looks for $LEVJEPA_DATA_ROOT/walking_tours/train.lance. The builder refuses to overwrite an existing output. Keeping the old archive and selecting a new --out path is safer than deleting blindly.

Inspect the store before training

uv run python - <<'PY'
import lance
ds = lance.dataset("data/walking_tours/train.lance")
print(ds.count_rows())
print(ds.schema)
PY

Expect a positive row count and all six fields. Also require failed == 0 in the build log and multiple episode_idx values. One enormous episode signals broken indexing or ordering.

Data provenance

• Pinned download script
• Pinned Lance builder
• Pinned loader and Walking Tours dataset page

Three data facts

1. Walking Tours supplies URLs; the script retrieves about 25 GB of video.
2. The builder stores 15 fps, 128-frame Lance episodes; training draws 16 frames at stride 2.
3. Roughly 33 GB and 6,067 episodes describe the authors’ snapshot, not an eternal guarantee.

Archive check

1. Why not use one fixed frame step for Wildlife and the 60 fps videos?
2. Roughly how much time does the default 16-frame clip cover?
3. Why must Lance rows not be shuffled before writing?