Chapter 17 — CEM: Searching for Actions Through an Elimination Tournament

The big picture

A tiny story

A director cannot watch every possible dance. Instead, 300 dancers perform complete routines. The 30 routines with the best model-predicted score influence the next audition. After several rounds, the audition concentrates near promising routines.

That is the Cross-Entropy Method (CEM) used for LeWM planning. Here “cross-entropy” names a sampling optimizer, not the classification loss. The trained encoder and predictor stay frozen. Only the probability distribution used to sample action sequences changes.

The real rule

A candidate is a whole sequence, not one greedy action. A useful conceptual shape is [B, N, H, A]: environment batch, candidate, planning horizon, and action-block width.

initialize a proposal over complete action sequences
repeat for the declared number of rounds:
  sample candidates
  roll them through the frozen world model
  compute one terminal goal cost per candidate
  keep low-cost elites
  refit the proposal from those elites
return the plan using the declared solver convention

The named LeWM setup uses 300 candidates and 30 elites. The generic paper description and frozen shared YAML contain 30 refinement rounds, while Appendix D says PushT uses at most 30 and the other environments use 10. Keep those source cards separate.

The paper describes a multivariate Gaussian proposal for its continuous-control tasks. Exact clipping, covariance, smoothing, and bound handling belong to the resolved external solver revision; the LeWM core did not pin that dependency. A categorical discrete-action solver is possible in general, but is not established for this frozen baseline.

The trick that can fool us

Imagine two equally good corridors around an obstacle: one goes left, one goes right. The elites split into two groups. Their Gaussian average points straight into the obstacle.

So record the best sampled plan and the final proposal mean separately. LeWorldModel Appendix B uses the final mean; Algorithm 2 allows the best found sequence or the first action of the final mean. Both inspected stable-worldmodel revisions—tag 0.0.5 and snapshot addbab4—return the final mean. Because LeWM did not pin its historical dependency, this does not prove the authors ran either exact revision. Executing best_seen is a useful labelled variant, not a silent replacement.

More search is not always safer. CEM can miss a narrow good corridor, collapse too early, see flat costs, average modes, saturate action bounds, or discover a loophole in the learned model more efficiently. Elite means “best in this sampled round under this model and cost,” not reachable or successful in the real environment.

Experiment receipt

Log horizon, action width and transformation, proposal seed and scale, population, elites, rounds, candidate and elite cost distributions, proposal spread by horizon position, boundary fraction, support warnings, model calls, hardware, and time. Count real action budget separately from imagined predictor work.

The source boundary is LeWorldModel v3, frozen config 8edfeb3, and the separately inspected current solver snapshot. The independent TwoRoom reimplementation compares 10 and 30 rounds in one environment; it does not recover the historical dependency or validate every benchmark.

The defensible claim is small: CEM concentrates finite samples around low-model-cost action sequences. It has no global-optimum or real-success guarantee.

Quick check

1. What changes during CEM, and what remains frozen?
2. Why can two good elite families produce a bad proposal mean?
3. Which paper and code cards disagree about non-PushT refinement rounds?