Open problems
Per the May-2026 survey (Zidan et al.), the principal persistent challenges in world modeling:
-
Compounding prediction errors. Every autoregressive world model accumulates error over long rollouts — the world drifts, mutates, forgets. This is the single failure mode most visible in interactive/game world models (persistent-memory failures) and the reason imagination horizons in model-based RL stay short.
-
Sim-to-real transfer. A world model trained on video or simulation must still close the gap to physical deployment — the robotics branch’s core bottleneck.
-
Fragmented evaluation. No unified benchmark; physics consistency, coherence, action fidelity, and control success are measured by disjoint suites that don’t rank models consistently (see evaluation).
The forecast future directions: unified multimodal world models (Cosmos 3 is the first big data point), foundation-scale interactive simulators, and safe deployment in safety-critical domains.
Where a small lab can contribute
Our read (and our research agenda): the open problems are not equally compute-hungry, and the verified compute numbers from the game/sim branch prove it.
The compute ladder (all figures from primary sources — see game & simulation):
| Tier | Evidence | What it buys |
|---|---|---|
| 1 consumer GPU | DIAMOND’s CS:GO model: 12 days on one RTX 4090; Atari ~2.9 days/game | training real world models from scratch |
| ~500 H100-days | WHAM 1.6B: 98 H100 × 5 days | a Nature-paper-scale foundation model |
| Pod scale | GameNGen: 128 TPU-v5e | industrial results |
And inference is uniformly cheap: DIAMOND plays at 10 fps on an RTX 3090 with 1.5 GB of weights; Matrix-Game 2.0 streams 25 FPS from a 24 GB card.
So the tractable agenda for a single researcher with RunPod-scale compute:
- Train at DIAMOND scale. A 381M diffusion world model on a domain nobody has done (not Minecraft, not CS:GO) is one consumer-GPU-month.
- Probe, don’t train, at the top of the ladder. V-JEPA 2’s MIT checkpoints (0.3B–1B) and Matrix-Game 2.0 (1.8B) run on single GPUs — physical-understanding probes and consistency benchmarks against them are compute-light.
- Evaluation is the underbuilt layer. The field’s own surveys call evaluation fragmented; benchmarks need task design more than FLOPs (see evaluation).
- 3D/spatial generation at toy scale is tractable: our own prompt→panorama→depth→3DGS pipeline runs on a single consumer GPU or a Mac.