GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control
Anthony Chen, Wenzhao Zheng, Yida Wang, Xueyang Zhang, Kun Zhan, Peng Jia, Kurt Keutzer, Shanghang Zhang
AI summary
Problem
Current driving world models lack robust 3D geometric consistency and precise action controllability, often producing structurally incoherent views or relying on dense annotations that fail to capture true vehicle dynamics.
Approach
The framework extracts a 3D point cloud from a single input frame, renders dynamic sequences along user-specified trajectories using a novel dynamic editing module, and conditions a latent video diffusion model on these geometric cues to generate trajectory-faithful future scenes.
Key results
- Reduces trajectory-following errors by 42% compared to Vista
- Outperforms baselines on all video quality and action fidelity metrics
- Enables zero-shot novel-view synthesis with superior 3D consistency
- Provides interactive object editing and non-ego vehicle trajectory control
Why it matters
It delivers a data-efficient, geometrically reliable simulation tool critical for safe autonomous vehicle planning, scenario generation, and dynamic environment understanding.
Abstract
Recent advancements in world models have revo- lutionized dynamic environment simulation, allowing systems to foresee future states and assess potential actions. In autonomous driving, these capabilities help vehicles anticipate the behavior of other road users, perform risk-aware planning, accelerate training in simulation, and adapt to novel scenarios, thereby enhancing safety and reliability. Current approaches exhibit deficiencies in maintaining robust 3D geometric consistency or accumulating artifacts during occlusion handling, both critical for reliable safety assessment in autonomous navigation tasks. To address this, we introduce GeoDrive, which explicitly integrates robust 3D geometry conditions into driving world models to enhance spatial understanding and action controllability. Specifically, we first extract a 3D representation from the input frame and then obtain its 2D rendering based on the user-specified ego-car trajectory. To enable dynamic modeling, we propose a dynamic editing module during training to enhance the renderings by editing the positions of the vehicles. Extensive experiments demonstrate that our method significantly outperforms existing models in both action accuracy and 3D spatial awareness, leading to more realistic, adaptable, and reliable scene modeling for safer autonomous driving. Additionally, our model can generalize to novel trajectories and offers interactive scene editing capabilities, such as object editing and object trajectory control.