A taxonomy of world models
Two comprehensive surveys give ready-made organizations of the field:
Ding et al. (arXiv:2411.14499, ACM Computing Surveys) split the field by function:
- Implicit/internal representation for understanding world dynamics — LeCun’s JEPA line, and the Dreamer/MuZero decision-making tradition;
- Future-state prediction via generative simulation — Sora-style video generation, embodied and spatial prediction.
The survey is explicitly motivated by the tension between those two camps (see the debates).
Zidan et al. (arXiv:2606.00133, May 2026 preprint) instead use a four-axis scheme: architecture; methodological family (state-space/recurrent, transformer-based, diffusion-based, physics-informed, language-augmented multimodal); reasoning strategy (imagination-based planning, latent policy learning, counterfactual reasoning, planning under uncertainty); and application domain.
The map this site uses
For a practitioner, the most useful cut is what the model predicts, and for whom. We organize the survey into six branches:
| Branch | Predicts | Exemplars |
|---|---|---|
| Latent prediction | abstract representations, not pixels | I-JEPA, V-JEPA, V-JEPA 2 |
| Video-generative | (interactive) video | Sora, Veo, Genie 1–3 |
| 3D / spatial | persistent 3D space | World Labs Marble/RTFM, 3DGS generation |
| Agents & robotics | consequences of actions, for control | Dreamer, MuZero, Cosmos, V-JEPA 2-AC |
| Game / simulation | playable environments | Oasis, GameNGen, Muse/WHAM |
| Driving | traffic futures, for autonomy | GAIA-1/2, Vista |
The branches are converging. NVIDIA’s Cosmos 3 (May 2026) is deliberately omnimodal in both input and output — text, images, video, audio, and action trajectories in; text, images, video, audio, and actions out — making it simultaneously a video-generative world model and an action-conditioned one, collapsing two branches into a single system. That is exactly the “unified multimodal world models” future the surveys forecast.
A framing to avoid: an “explicit vs. implicit world models” two-way taxonomy sometimes attributed to Oh’s ACM Computing Surveys tutorial (arXiv:2606.12783) does not survive verification against the tutorial itself — we do not use it. The tutorial remains a good source on V-JEPA 2 and Genie mechanics.
← What is a world model?Latent prediction: the JEPA family →