FlowDreamer: A RGB-D World Model with Flow-Based Motion Representations for Robot Manipulation
Jun Guo, Xiaojian Ma, Yikai Wang, Min Yang, Huaping Liu, Qing Li
AI summary
Problem
Current RGB-D world models merge dynamics prediction and visual rendering into a single network, which obscures physical motion understanding and degrades prediction performance by prioritizing visual fidelity over accurate dynamics.
Approach
FlowDreamer decouples the task into two stages: a U-Net explicitly predicts 3D scene flow to capture motion dynamics, which then conditions a diffusion model to generate future RGB-D frames, all trained end-to-end.
Key results
- 7% improvement in semantic similarity on video prediction benchmarks
- 11% gain in pixel quality metrics across RT-1 and Language Table datasets
- 6% increase in robot manipulation success rates on visual planning tasks
- Demonstrates superior dynamics modeling over single-stage and separately trained baselines
Why it matters
Enables more reliable and interpretable robot simulators for planning and control, reducing reliance on precise physical modeling in real-world manipulation.
Abstract
This paper investigates training better visual world models for robot manipulation, i.e., models that can predict future visual observations by conditioning on past frames and robot actions. Specifically, we consider world models that operate on RGB-D frames (RGB-D world models). As opposed to canonical approaches that handle dynamics prediction mostly implicitly and reconcile it with visual rendering in a single model, we introduce FlowDreamer, which adopts 3D scene flow as explicit motion representations. FlowDreamer first predicts 3D scene flow from the past frame and action conditions with a U-Net, and then a diffusion model will predict the future frame utilizing the scene flow. FlowDreamer is trained end-to-end despite its modularized nature. We conduct experiments on 4 different benchmarks, covering both video prediction and visual planning tasks. The results demonstrate that FlowDreamer achieves better performance compared to other baseline RGB-D world models by 7% on semantic similarity, 11% on pixel quality, and 6% on success rate in various robot manipulation domains.