DUET-DINO Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation

Nisarga Nilavadi1, Ralf Römer2, Moritz Reuss3,4, Michael Krawez1, Tobias Jülg1,
Angela P. Schoellig2,5, Rudolf Lioutikov3,5, Wolfram Burgard1,5

1 University of Technology Nuremberg2 Technical University of Munich3 Karlsruhe Institute of Technology4 NVIDIA5 Robotics Institute Germany

arXiv Code CheckpointsTBD
Reach Coffee Pot
Angled-reach Drill
Real-world Angled-reach Bottle
Angled-lift Ketchup

DUET-DINO enables fine-grained robot control through full 7-DoF end-effector action planning.

Abstract

Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control. However, their predictions for fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. To address this gap, we introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations through cross-view conditioning. By exploiting complementary global scene and gripper-centric information, DUET-DINO enables latent planning over the full 7-DoF action space. Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. DUET-DINO is trained from scratch on DROID and RoboArena datasets and generalizes robustly under visual distribution shifts. We further show that while V-JEPA 2 wrist-view predictions underestimate visual dynamics induced by fine-grained actions, DINOv3 predictions better capture action-conditioned scene changes, leading to stronger downstream planning.

Architecture and Training of DUET-DINO

DUET-DINO architecture with side and wrist encoders, cross-attention, and action-conditioned predictors

The pretrained encoder maps side- and wrist-view observations into latent space. Cross-attention blocks condition each view on complementary information from the other view in the latent space. The view-specific predictor heads then predict future latents conditioned on the action and end-effector state. The dual-view prediction loss combines the teacher-forcing loss computed over all one-step predictions in the clip with an autoregressive loss over the K-step rollout predictions.

Latent Planning in Robot Manipulation

Zero-shot latent planning pipeline using side and wrist goal images

Overview of the proposed simultaneous cross-view world modeling method, showing predicted dual-view latent rollouts, CEM action planning using predicted and goal latents, and the normalized dual-view planning cost.

Results

DUET-DINOSuccessful episode in source log
Orange Juice Carton
Side-view V-JEPA 2Outcome not labeled in source log
Orange Juice Carton
VJEPA 2 - AC*Outcome not labeled in source log
Orange Juice Carton
DUET-DINOSuccessful episode in source log
Bagel
Side-view V-JEPA 2Outcome not labeled in source log
Bagel
VJEPA 2 - AC*Successful episode in source log
Bagel
DUET-DINOSuccessful episode in source log
Coffee Can
Side-view V-JEPA 2Outcome not labeled in source log
Coffee Can
VJEPA 2 - AC*Outcome not labeled in source log
Coffee Can

Embeddings Analysis

Side- and wrist-view latent changes comparing encoders, DINOv3 predictors, and V-JEPA 2 predictors across episode transitions
Mean latent changes between successive frames captured by frozen encoders, DINOv3 action-conditioned predictors, and V-JEPA 2 action-conditioned predictors over 30 runs of an angled-reach task, showing weaker modeling of fine-grained action-conditioned dynamics by the V-JEPA 2 wrist predictor.
DINOv3 and V-JEPA 2 qualitative patch correspondences across observed frames, encoder features, and predicted features
Patch-wise latent distances for DINOv3 and V-JEPA 2 feature maps. Lower l1 distances indicate greater similarity to the selected reference patch. Frames at t and t + 1 are shown after resizing to the respective encoder inputs. Feature maps are visualized at 14 × 14 for DINOv3 and 16 × 16 for V-JEPA 2. DINOv3 produces 196 patches of 1280 dimensions, whereas V-JEPA 2 produces 256 patches of 1408 dimensions. DINOv3 embeddings show coherent correspondences over semantically related object regions, and its AC-predictor closely preserves the encoded t+1 pattern. In contrast, V-JEPA 2 embedding exhibits noisier correspondences, with predicted low-distance regions also appearing in semantically unrelated areas.

BibTeX

@article{nilavadi2026duetdino,
    title   = {DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation},
    author  = {Nilavadi, Nisarga and R{\"o}mer, Ralf and Reuss, Moritz and Krawez, Michael and J{\"u}lg, Tobias and Schoellig, Angela P and Lioutikov, Rudolf and Burgard, Wolfram},
    journal = {arXiv preprint arXiv:2609.10506},
    year    = {2026}
    }