Mobile Manipulation Instruction Generation from Multiple Images with Automatic Metric Enhancement
Kei Katsumata, Motonari Kambara, Daichi Yashima, Ryosuke Korekata, Komei Sugiura
AI summary
Problem
Existing image captioning models are optimized for single images and cannot effectively generate the complex, multi-image instructions required for mobile manipulation tasks.
Approach
The authors propose a model using Triplet Qformer to align target object and receptacle visual features with text, trained via a Human Centric Calibration Phase (HCCP) that uses both n-gram and learning-based metrics as rewards.
Key results
- Outperformed baselines including GPT-4o and Gemini on automatic evaluation metrics
- Developed Triplet Qformer to align multiple visual features with text anchors
- Introduced HCCP training to improve word co-occurrence and paraphrasing
- Improved performance of existing multimodal language understanding models through data augmentation in physical experiments
Why it matters
Reduces the labor-intensive effort required to create high-quality instruction datasets for service robots.
Abstract
We consider the problem of generating free-form mobile manipulation instructions based on a target object image and receptacle image. Conventional image captioning models are not able to generate appropriate instructions because their architectures are typically optimized for single-image. In this study, we propose a model that handles both the target object and receptacle to generate free-form instruction sentences for mobile manipulation tasks. Moreover, we introduce a novel training method that effectively incorporates the scores from both learning-based and n-gram based automatic evaluation metrics as rewards. This method enables the model to learn the co-occurrence relationships between words and appropriate paraphrases. Results demonstrate that our proposed method out- performs baseline methods including representative multimodal large language models on standard automatic evaluation metrics. Moreover, physical experiments reveal that using our method to augment data on language instructions improves the performance of an existing multimodal language understanding model for mobile manipulation.