Evaluating world models
How do you score “understands how the world works”? The surveys name fragmented evaluation as one of the field’s three principal open problems. Three benchmarks now anchor the answer.
Physics-IQ — filmed reality as ground truth
Physics-IQ (Google DeepMind + INSAIT, arXiv:2501.09038, WACV 2026) tests whether video models predict physical outcomes: 396 real filmed videos (66 scenarios × 3 camera angles × 2 takes; Sony Alpha a6400, 4K/30fps — filmed, not rendered) spanning solid mechanics, fluids, optics, thermodynamics, magnetism. The model sees up to 3 s of conditioning (a single frame for image-to-video models) and must generate the 5-second continuation; the score aggregates four metrics (Spatial IoU, Spatiotemporal IoU, Weighted Spatial IoU, MSE), normalized so that two real takes of the same scene score 100% — physical randomness defines the ceiling.
The headline result (2024-era models, finalized Feb 2025): the best model, VideoPoet-multiframe, scored 29.5 / 100. And the dissociation: Sora was the most visually realistic model tested (Gemini-judged 2AFC 55.6%, near-perfect deception) yet scored the worst on physics — 10.0%. Across models, visual realism was statistically uncorrelated with physical understanding (r = −0.46, p = .249 — a small-sample null, but the direction is damning). The paper’s conclusion: “physical understanding is severely limited, and unrelated to visual realism.”
Mid-2026 standings (Original leaderboard, checked live 2026-07-18; entries are author-submitted, not independently re-run): reward-guided Magi-1 + GeoPhys (BoN) leads video-to-video at 64.5%, with NVIDIA Cosmos3-Super + WMReward (BoN) at 63.4%; plain Sora 2 scores 42.3% on image-to-video. DeepMind now recommends the audited Physics-IQ Verified variant (arXiv:2606.18943), which refined 57.6% of samples and requires 4-run averages for SOTA claims — absolute scores drop further there (best v2v 58.2; Sora 2 i2v 26.5). Two years of progress moved the frontier from ~30% to ~60% of the physical-variance ceiling — real progress, and still nowhere near closed.
WorldScore — one yardstick across 3D, 4D, and video
WorldScore (Stanford — Duan, Yu, Chen, Fei-Fei Li, Jiajun Wu; arXiv:2504.00983, ICCV 2025) is the first unified benchmark across 3D scene generation, 4D generation, and video models: world generation decomposed into next-scene generation tasks with explicit camera-trajectory layouts. 3,000 test examples; ten sub-metrics across controllability (camera, object, content), quality (3D consistency, photometric, style, subjective), and dynamics (accuracy, magnitude, smoothness); ~20 models evaluated.
The instructive result from the paper: 3D scene-generation methods beat video models at static world-building — WonderWorld scored 72.69 WorldScore-Static vs. 62.15 for the best video model (CogVideoX-I2V) — because video models fail at camera controllability. But 3D methods score 0.00 on dynamics: they build dioramas, not worlds that move. Nobody wins both. On the live leaderboard (read 2026-07-18): Voyager, a 3D scene-generation method, leads Static at 77.62, while CogVideoX-I2V still leads Dynamic at 59.12 — the static/dynamic split persists a year after publication.
The physical-law study — why scaling won’t fix it
The ByteDance study How Far is Video Generation from World Model: A Physical Law Perspective (arXiv:2411.02385, ICML 2025; phyworld) is the controlled experiment behind the benchmark numbers. DiT video models (DiT-S→XL) trained on synthetic 2D physics — uniform motion, elastic collision, parabolic motion, up to 3M videos:
- In-distribution error 0.012 vs. out-of-distribution 0.427 (velocity, DiT-L at 3M examples) — OOD errors an order of magnitude higher in every setting, and “scaling up the training data and model size has little or negative impact” on them.
- Combinatorial generalization is different: abnormal-generation rate fell from 67% to 10% when template diversity rose from 6 to 60 — coverage helps where volume doesn’t.
- The failure mechanism: case-based imitation — models mimic the nearest training example rather than abstracting a law, with a measurable attribute priority of color > size > velocity > shape.
The rest of the toolbox
- VideoPhy (arXiv:2406.03520, ICLR 2025) and VideoPhy-2 (arXiv:2503.06800, ICLR 2026) — text-to-video physical commonsense, human-rated semantic adherence + physical commonsense. VideoPhy-2’s hard subset is brutal: best model (Wan2.1-14B) ~22% joint; original Sora 5.3%. Models particularly fail conservation laws (mass, momentum).
- WorldModelBench (arXiv:2502.20694, CVPR 2025 workshop) — video models judged as world models across 7 domains with 67K human labels; top model (KLING) completes the instructed task in only 61% of videos, with ~12% violating mass conservation.
- EWMBench (arXiv:2505.09694, AgiBot) — embodied world models on scene consistency, motion correctness, and semantics; domain-tuned models beat general video generators (Kling-1.6: 3.87, Cosmos-7B: 3.29 vs. fine-tuned EnerVerse at 4.70).
- ⚠️ PhysBench (arXiv:2501.16411, ICLR 2025) is often miscited in this company: it benchmarks VLM physical understanding (GPT-4o: 49.5%), not video generation — don’t put it in the same table.
WorldArena — does the world model help the robot?
WorldArena (arXiv:2602.08971, Feb 2026; world-arena.ai) targets embodied world models and makes the argument the other benchmarks miss: perceptual fidelity is not utility. It scores 16 perceptual metrics plus functional roles — data engine, policy evaluator, action planner — aggregated into an EWMScore across 14 models. WorldArena 2.0 (arXiv:2605.17912, May 2026) extends the coverage.
What we take from this
- Never trust demo reels — the Physics-IQ dissociation is the quantitative proof that impressive video ≠ physical understanding.
- Leaderboards are living and partly self-reported — top entries arrive by vendor pull request; the Verified variant exists precisely because the original needed an audit.
- The 3D-vs-video split is measurable — WorldScore quantifies the trade this whole field is trying to escape: controllable-but-static versus dynamic-but-drifting.
- Scaling is not the answer to OOD physics — the ByteDance result says the fix is diversity and mechanism, not more of the same data. That is an open research door, and it is not compute-gated.