Benchmarking Multi-View BEV Object Detection with Mixed Pinhole and Fisheye Cameras
Xiangzhong Liu, Hao Shen
AI summary
Problem
Existing BEV 3D object detection models assume uniform pinhole cameras, causing severe performance degradation when applied to real-world mixed pinhole and fisheye camera configurations due to radial distortion and non-linear projections.
Approach
The authors convert the KITTI-360 dataset to a standardized nuScenes format to benchmark mixed-camera detection, adapt three BEV architectures using a unified MEI camera model for distortion-aware view transformation, and introduce polar coordinate representations to align with fisheye geometry.
Key results
- Established the first real-data multi-view BEV detection benchmark on KITTI-360
- Demonstrated projection-free architectures are inherently more robust to fisheye distortion
- Developed distortion-aware view transformation modules using the MEI camera model
- Introduced polar coordinate transformations to align BEV representations with fisheye geometry
Why it matters
Provides a standardized benchmark and practical adaptation guidelines for building robust, cost-effective 3D perception systems in real-world autonomous vehicles using mixed camera setups.
Abstract
Modern autonomous driving systems increasingly rely on mixed camera configurations with pinhole and fisheye cameras for full view perception. However, Bird’s-Eye View (BEV) 3D object detection models are predominantly designed for pinhole cameras, leading to performance degradation under fisheye distortion. To bridge this gap, we introduce a multi-view BEV detection benchmark with mixed cameras by converting KITTI-360 into nuScenes format. Our study encompasses three adaptations: rectification for zero-shot evaluation and fine-tuning of nuScenes-trained models, distortion-aware view transformation modules (VTMs) via the MEI camera model, and polar coordinate representations to better align with radial distortion. We systematically evaluate three representative BEV architectures, BEVFormer, BEVDet and PETR, across these strategies. We demonstrate that projection-free architectures are inherently more robust and effective against fisheye dis- tortion than other VTMs. This work establishes the first real- data 3D detection benchmark with fisheye and pinhole images and provides systematic adaptation and practical guidelines for designing robust and cost-effective 3D perception systems. The code is available at https://github.com/CesarLiu/ FishBEVOD.git.