Overcoming Hand and Arm Occlusion in Human-To-Robot Handovers: Predicting Safe Poses with a Multimodal DNN Regression Model

Handovers play a key role in human-robot in- teractions. However, current research focuses on visible-hand handovers, thereby heavily relying on hand detection. Large objects in human-robot interactions present a unique challenge: they inherently block the person’s hands and arms from the robot’s view. This occlusion raises the robot’s risk of unintended physical contact with the person, leading to discomfort and safety concerns. This study aims to develop a model that can determine a pose for the robot that ensures a handover that avoids physical contact with the person, especially in scenarios when hands and arms are occluded. Toward this goal, a three-branch multimodal Deep Neural Network (DNN) regression model was implemented. First, a robust human- pose keypoints detection to calculate shoulder-elbow angles is applied. Secondly, we extract the refined object’s segmented mask. Thirdly, we compute two intrinsic object properties. The concatenated outputs from these branches pass through extra dense layers, resulting in the prediction of the robot’s 14 arms- joint angles. Compared to an only keypoint data processed- based model, our multimodal approach made a 17.7% accuracy improvement. The experiments highlight each pipeline step’s significance, showing important results even when hands and arms were heavily occluded, adjusting to different variations.