DiffPlace: Street View Generation Via Place-Controllable Diffusion Model Enhancing Place Recognition
Ji Li, Zhiwei Li, ShiHao Li, ZhenJiang Yu, Boyang Wang, Haiou Liu
AI summary
Problem
Existing multi-view diffusion models for street view generation lack precise control over background and place information, limiting their ability to produce realistic, place-aware scenes needed for training visual place recognition models.
Approach
DiffPlace integrates a place-ID controller that maps visual place recognition embeddings into a fixed CLIP space using linear projection, a perceiver transformer, and contrastive learning, enabling controllable generation of multi-view street images with consistent backgrounds but flexible foregrounds and weather.
Key results
- First place-controllable diffusion model for place recognition augmentation
- High-fidelity street view generation with consistent backgrounds across varying weather and objects
- State-of-the-art generation quality (FID 13.4) and place controllability (AR@1: 57.6, AR@5: 75.4)
- Significant performance gains in downstream place recognition and 3D object detection via augmented training
Why it matters
Provides a scalable data augmentation pipeline that improves the robustness and accuracy of visual place recognition systems for autonomous driving and robotics.
Abstract
Generative models have advanced significantly in realistic image synthesis, with diffusion models excelling in quality and stability. Recent multi-view diffusion models improve 3D-aware street view generation, but they strug- gle to produce place-aware and background-consistent urban scenes from text, BEV maps, and object bounding boxes. This limits their effectiveness in generating realistic samples for place recognition tasks. To address these challenges, we propose DiffPlace, a novel framework that introduces a place- ID controller to enable place-controllable multi-view image generation. The place-ID controller employs linear projection, perceiver transformer, and contrastive learning to map place- ID embeddings into a fixed CLIP space, allowing the model to synthesize images with consistent background buildings while flexibly modifying foreground objects and weather conditions. Extensive experiments, including quantitative comparisons and augmented training evaluations, demonstrate that DiffPlace outperforms existing methods in both generation quality and training support for visual place recognition. Our results highlight the potential of generative models in enhancing scene- level and place-aware synthesis, providing a valuable approach for improving place recognition in autonomous driving.