Latent prediction: the JEPA family
The Joint-Embedding Predictive Architecture (JEPA) line is Yann LeCun’s answer to generative world modeling: rather than predicting raw sensory inputs in pixel space, “JEPA learns to predict abstract representations in a latent embedding space, making the learning process more efficient and robust” (Ding et al.). The bet is that pixel-level generation wastes model capacity on unpredictable, task-irrelevant detail (every leaf on a tree), while latent prediction keeps only what is predictable and useful.
V-JEPA 2 — the flagship
V-JEPA 2 (Meta FAIR, arXiv:2506.09985, June 2025) is the strongest current instantiation:
- Pretraining: 1M+ hours of internet video with a joint-embedding predictive objective — no pixel reconstruction, no action labels.
- Robotics adaptation (V-JEPA 2-AC): an action-conditioned head trained with only ~62 hours of unlabeled DROID robot video (~23k trajectories). At inference it controls real robot arms via goal-conditioned planning in representation space — a form of model-predictive control, sampling candidate action sequences with the Cross-Entropy Method in a receding-horizon loop.
- Open weights: eight MIT-licensed checkpoints (~0.3B–1B params; ViT-L/H/g backbones
at 256/384 resolution, plus Something-Something-v2 and Diving48 fine-tuned variants) in
the official facebook/v-jepa-2
collection. The action-conditioned
vjepa2-accheckpoint ships separately. A V-JEPA 2.1 follow-up exists.
Honest limitations
Meta’s own authors state the current constraints: planning relies on goal images rather than language or abstract task specifications, and a base JEPA without action-conditioning “cannot directly evaluate the consequences of candidate actions or support planning through internal simulation” — the action-conditioned stage is what turns a representation learner into a usable world model.
Why this branch matters for evaluation
Because JEPA models never render pixels, they cannot be judged by video benchmarks — you evaluate them by probing their representations (do they encode object permanence, contact, occlusion?) and by downstream control performance. This makes the branch a natural home for compute-light research: probing suites run on single-GPU budgets against freely downloadable 0.3B–1B checkpoints.