Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecasting
Guangxun Zhu, Xuan Liu, Nicolas Pugeault, Chongfeng Wei, Edmond S. L. Ho
AI summary
Problem
Prior pedestrian prediction methods rely on coarse 2D representations or lack realistic 3D multi-agent datasets, limiting their ability to capture fine-grained motion dynamics and the influence of surrounding vehicles.
Approach
The authors enhance the Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes and adapt the TBIFormer architecture with a dedicated vehicle encoder and cross-attention module to fuse historical pedestrian motion with surrounding vehicle context.
Key results
- Enhanced Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes and a scene-level sampling scheme
- Proposed a vehicle-conditioned 3D pose forecasting network with a dedicated vehicle encoder and Pedestrian-Vehicle Interaction Cross-Attention module
- Demonstrated substantial forecasting accuracy improvements over the TBIFormer baseline by explicitly modeling pedestrian-vehicle interactions
- Validated interaction modeling approaches and highlighted the necessity of vehicle-aware 3D pose prediction for autonomous driving
Why it matters
Provides a realistic framework and benchmark for modeling complex multi-agent interactions, directly advancing the safety and reliability of autonomous driving systems.
Abstract
Accurately predicting pedestrian motion is crucial for safe and reliable autonomous driving in complex urban environments. In this work, we present a 3D vehicle-conditioned pedestrian pose forecasting framework that explicitly incor- porates surrounding vehicle information. To support this, we enhance the Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes, enabling realistic modeling of multi- agent pedestrian–vehicle interactions. We introduce a sampling scheme to categorize scenes by pedestrian and vehicle count, facilitating training across varying interaction complexities. Our proposed network adapts the TBIFormer architecture with a dedicated vehicle encoder and pedestrian–vehicle in- teraction cross-attention module to fuse pedestrian and vehicle features, allowing predictions to be conditioned on both his- torical pedestrian motion and surrounding vehicles. Extensive experiments demonstrate substantial improvements in forecast- ing accuracy and validate different approaches for modeling pedestrian–vehicle interactions, highlighting the importance of vehicle-aware 3D pose prediction for autonomous driving.