No Need to Look! Locating and Grasping Objects by a Robot Arm Covered with Sensitive Skin
Karel Bartunek∗, Lukas Rustler∗, and Matej Hoffmann
AI summary
Problem
Robot manipulation typically relies on vision, leaving a gap in capabilities for environments where visual sensors are unavailable or obstructed by lighting, dust, or occlusion.
Approach
A two-stage pipeline using whole-body skin for coarse workspace exploration followed by an end-effector force/torque sensor for precise localization and grasping.
Key results
- 85.7% success rate on real robot for single object
- 6x faster than baseline using only end-effector haptic feedback
- 83.1% average success rate in simulation across various positions
- Successful location and grasping of multiple diverse objects without vision
Why it matters
Enables robot manipulation in visually challenging environments, such as picking fruits or vegetables from dense foliage in agriculture.
Abstract
Locating and grasping of objects by robots is typically performed using visual sensors. Haptic feedback from contacts with the environment is only secondary if present at all. In this work, we explored an extreme case of searching for and grasping objects in complete absence of visual input, relying on haptic feedback only. The main novelty lies in the use of contacts over the complete surface of a robot manipulator covered with sensitive skin. The search is divided into two phases: (1) coarse workspace exploration with the complete robot surface, followed by (2) precise localization using the end- effector equipped with a force/torque sensor. We systematically evaluated this method in simulation and on the real robot, demonstrating that diverse objects can be located, grasped, and put in a basket. The overall success rate on the real robot for one object was 85.7% with failures mainly while grasping specific objects. The method using whole-body contacts is six times faster compared to a baseline that uses haptic feedback only on the end-effector. We also show locating and grasping multiple objects on the table. This method is not restricted to our specific setup and can be deployed on any platform with the ability of sensing contacts over the entire body surface. This work holds promise for diverse applications in areas with challenging visual perception (due to lighting, dust, smoke, occlusion) such as in agriculture when fruits or vegetables need to be located inside foliage and picked.