Open-Vocabulary Mobile Manipulation Based on Double Relaxed Contrastive Learning with Dense Labeling
Daichi Yashima, Ryosuke Korekata, Komei Sugiura
AI summary
Problem
Domestic service robots struggle to retrieve specific target objects and receptacles from thousands of similar images using complex, open-vocabulary instructions. Standard contrastive learning methods fail in this setting because they incorrectly treat unlabeled similar images as negatives, degrading retrieval accuracy.
Approach
The authors propose RelaX-Former, which uses a Dense Labeler to identify unlabeled positive samples and a Double Relaxed Contrastive loss to properly balance positive, unlabeled positive, and negative samples during training. This is combined with a Spatial Overlay Grounding module that extracts fine-grained visual features using segmentation masks and multimodal large language models.
Key results
- Outperforms baseline models on standard image retrieval metrics across unseen environments
- Achieves 75% overall success rate in zero-shot physical fetch-and-carry experiments
- Introduces a Spatial Overlay Grounding module for fine-grained, linguistically grounded visual features
- Proposes a Dense Representation Learning module with a novel DRC loss to effectively leverage unlabeled positives
Why it matters
Advances the reliability of open-vocabulary instruction following for domestic service robots, enabling them to assist humans in complex, real-world indoor environments.
Abstract
Growing labor shortages are increasing the demand for domestic service robots (DSRs) to assist in various settings. In this study, we develop a DSR that transports everyday objects to specified pieces of furniture based on open-vocabulary instructions. Our approach focuses on retrieving images of target objects and receptacles from pre-collected images of indoor environments. For example, given an instruction “Please get the right red towel hanging on the metal towel rack and put it in the white washing machine on the left,” the DSR is expected to carry the red towel to the washing machine based on the retrieved images. This is challenging because the correct images should be retrieved from thousands of collected images, which may include many images of similar towels and appliances. To address this, we propose RelaX-Former, which learns diverse and robust repre- sentations from among positive, unlabeled positive, and negative samples. We evaluated RelaX-Former on a dataset containing real-world indoor images and human annotated instructions in- cluding complex referring expressions. The experimental results demonstrate that RelaX-Former outperformed existing baseline models across standard image retrieval metrics. Moreover, we performed physical experiments using a DSR to evaluate the performance of our approach in a zero-shot transfer setting. The experiments involved the DSR to carry objects to specific receptacles based on open-vocabulary instructions, achieving an overall success rate of 75%.