SHOPPER: Practical Insights on Grasp Strategies for Mobile Manipulation in the Wild
Isabella Huang, Richard Cheng, Sangwoon Kim, Daniel Kruse, Carolyn Chen, Lukas Kaul, JC Anne Hancock, Shanmuga Perumal Harikumar, Mark Tjersland, James Borders, Daniel Helmick
AI summary
Problem
Mobile manipulation systems struggle to reliably grasp diverse items in unstructured, real-world environments due to kinematic imprecision, tight spaces, and varied item properties, hindering widespread deployment.
Approach
The authors developed SHOPPER, an integrated mobile manipulator deployed in a real grocery store, using a modular pipeline that classifies items into six grasp strategies and applies runtime corrections and admittance control to handle real-world uncertainties.
Key results
- A fully integrated mobile manipulation pipeline deployed in unmodified real-world grocery stores
- Six generalizable grasp strategies covering the majority of grocery items based on geometric and learned classification
- Extensive failure mode analysis from 1200+ grasp attempts identifying key bottlenecks like kinematic drift and extraction collisions
- A public dataset of 1200+ grasp attempts across 800+ unique items in unseen grocery stores
Why it matters
Provides actionable insights and a benchmark dataset to guide the robotics community in developing more reliable, deployable mobile manipulation systems for unstructured environments.
Abstract
Mobile manipulation systems have advanced sig- nificantly in recent years. However, substantial gaps remain that prevent state-of-the-art platforms from achieving widespread real-world deployment, particularly in reliably grasping items in unstructured environments. To help bridge this gap, we develop SHOPPER, a mobile manipulation robot platform designed to push the boundaries of reliable and generalizable grasp strategies. We develop these grasp strategies and deploy them in a real-world grocery store – an exceptionally challeng- ing setting chosen for its vast diversity of manipulable items, fixtures, and layouts. In this work, we present our detailed approach to designing general grasp strategies towards picking any item in a real grocery store. Additionally, we provide an in- depth analysis of our latest real-world field test, discussing key findings related to fundamental failure modes over hundreds of distinct pick attempts. Through our detailed analysis, we aim to offer valuable practical insights and identify key grasping challenges, which can guide the robotics community towards pressing open problems in the field. Lastly, we provide a dataset of 1200+ grasp attempts in unseen grocery stores.