Game & simulation world models
Games are the natural proving ground for interactive world models: bounded physics, plentiful data, and an unforgiving test — a human with a keyboard notices immediately when the world stops making sense. This branch also produced the field’s most useful public numbers on what world models actually cost to train.
GameNGen — DOOM in a diffusion model
GameNGen (Diffusion Models Are Real-Time Game Engines, arXiv:2408.14837, ICLR 2025) simulates DOOM by fine-tuning Stable Diffusion v1.4 (all U-Net parameters unfrozen), running at 20 FPS on a single TPU-v5 — 50 FPS with a distilled 1-step variant — and staying stable over multi-minute autoregressive play.
The honest numbers: next-frame PSNR 29.4 (the authors’ own comparison: lossy JPEG at quality 20–30). Human raters picked the real game over the simulation only 58–60% of the time (chance = 50%) — but on 1.6–3.2 s passive clips with 10 raters, which does not establish indistinguishability during interactive play. Context is ~3 s of frames, so long-term map/state memory is limited. Training: 128 TPU-v5e, 700k steps over 70M frames from RL-agent trajectories (excluding the ~900M-frame agent data collection).
Muse / WHAM — the Nature paper with a research-only catch
Microsoft’s WHAM (World and Human Action Models towards gameplay ideation, Nature, Feb 2025) trained on Ninja Theory’s Bleeding Edge: ~500,000 player trajectories from 60,986 matches (27.89 TiB; ~1.4B frames at 10 Hz; >7 years of continuous play).
The flagship is a 1.6B causal transformer over a 300M ViT-VQGAN tokenizer (300×180 frames → 540 tokens, vocab 16,384), trained 200k updates within ~1×10²² FLOPS on 98 H100s for 5 days (~490 H100-days). Checkpoints at microsoft/wham: 200M (3.7 GiB) and 1.6B (18.9 GiB), plus a Windows “Demonstrator” app.
⚠️ License: Microsoft Research License — research-only, non-commercial. WHAM is publicly downloadable but not open-source, contrary to how it is often described.
DIAMOND — the single-GPU existence proof
DIAMOND (Diffusion for World Modeling, arXiv:2405.12399, NeurIPS 2024 Spotlight; MIT license, eloialonso/diamond) matters twice:
- Result: mean human-normalized score 1.46 on Atari 100k — best at publication among agents trained entirely inside a world model (vs. STORM ~1.27, DreamerV3 ~1.10, IRIS ~1.05).
- Feasibility: its playable CS:GO world model (381M params) trained on 95 h of Dust2 gameplay in 12 days on a single RTX 4090; inference at ~10 fps on an RTX 3090 with 1.5 GB of weights. Atari runs take ~2.9 days per game on one 4090. (Numbers live on the csgo branch README.)
Matrix-Game — open weights at two scales
Skywork’s Matrix-Game 1.0 (May 2025, arXiv:2506.18701) is a 17B diffusion image-to-world model for controllable Minecraft generation — MIT weights on HF, but inference needs ≥80 GB GPU memory. Matrix-Game 2.0 (Aug 2025, arXiv:2508.13009) drops to 1.8B params with real-time 25 FPS streaming — three MIT distilled checkpoints (universal, GTA driving, TempleRun) that run on a single 24 GB GPU. A 3.0 exists (~Mar 2026) but its circulated spec sheet failed verification — we don’t publish it.
Oasis (Decart + Etched) completes the branch — see its entry above: DiT + Diffusion Forcing on Minecraft, open 500M checkpoint, full model hosted-only.
The compute ladder
The verified training budgets, assembled:
| Tier | System | Compute |
|---|---|---|
| Single consumer GPU | DIAMOND CS:GO | 12 days × 1 RTX 4090 |
| Single consumer GPU | DIAMOND Atari | ~2.9 days per game × 1 RTX 4090 |
| Mid-tier cluster | WHAM 1.6B | 98 H100 × 5 days (~1×10²² FLOPS) |
| Pod scale | GameNGen | 128 TPU-v5e × 700k steps |
Inference is uniformly cheap — DIAMOND at 10 fps on a 3090, GameNGen at 20 FPS on one TPU, Matrix-Game 2.0 on a 24 GB card. The implication for small labs: you can train DIAMOND-class world models from scratch, and run or probe everything else. See open problems.
← Agents & robotics: world models for controlDriving world models →