Sim-To-Real Grasp Detection with Global-To-Local RGB-D Adaptation

This paper focuses on the sim-to-real issue of RGB-D grasp detection and formulates it as a domain adapta- tion problem. In this case, we present a global-to-local method to address hybrid domain gaps in RGB and depth data and insufficient multi-modal feature alignment. First, a self- supervised rotation pre-training strategy is adopted to deliver robust initialization for RGB and depth networks. We then propose a global-to-local alignment pipeline with individual global domain classifiers for scene features of RGB and depth images as well as a local one specifically working for grasp features in the two modalities. In particular, we propose a grasp prototype adaptation module, which aims to facilitate fine-grained local feature alignment by dynamically updating and matching the grasp prototypes from the simulation and real-world scenarios throughout the training process. Due to such designs, the proposed method substantially reduces the domain shift and thus leads to consistent performance improve- ments. Extensive experiments are conducted on the GraspNet- Planar benchmark and physical environment, and superior results are achieved which demonstrate the effectiveness of our method. Code is available at https://github.com/ mahaoxiang822/GL-MSDA.