Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategy
Yuxin He, Ruihao Zhang, Tianao Shen, Cheng Liu, Qiang Nie⋆
AI summary
Problem
Fixed robot cameras frequently suffer from visual occlusion and insufficient sensory data, hindering complex manipulation. This paper addresses the broader challenge of actively gathering information to complete tasks requiring exploration or precise focus.
Approach
The authors introduce the EFM-10 benchmark and a Bimanual Active Perception strategy that repurposes the non-operating arm for eye-in-hand vision while using the operating arm for force sensing, backed by a new expert dataset.
Key results
- Introduced the EFM-10 benchmark with 10 tasks spanning four manipulation categories.
- Developed the Bimanual Active Perception strategy and collected BAPData with 1,850 expert demonstrations.
- Proved that active views must capture both the target area and the end-effector for reliable policy learning.
- Demonstrated that adding force sensing to imitation learning increases success rates and reduces contact forces in delicate tasks.
Why it matters
Offers a practical, neck-free active perception framework and a standardized benchmark to accelerate humanoid manipulation research.
Abstract
Recently, active vision has reemerged as an im- portant concept for manipulation, since visual occlusion occurs more frequently when main cameras are mounted on the robot heads. We reflect on the visual occlusion issue and identify its essence as the absence of information useful for task completion. Inspired by this, we come up with the more fundamental problem of Exploratory and Focused Manipulation (EFM). The proposed problem is about actively collecting information to complete challenging manipulation tasks that require exploration or focus. As an initial attempt to address this problem, we establish the EFM-10 benchmark that consists of 4 categories of tasks that align with our definition (10 tasks in total). We further come up with a Bimanual Active Perception (BAP) strategy, which leverages one arm to provide active vision and another arm to provide force sensing while manipulating. Based on this idea, we collect a dataset named BAPData for the tasks in EFM-10. With the dataset, we successfully verify the effectiveness of the BAP strategy in an imitation learning manner. We hope that the EFM-10 benchmark along with the BAP strategy can become a cornerstone that facilitates future research towards this direction. Project website: EFManipula- tion.github.io