A Complementary Approach for Robust and Safety-Oriented Visual Tracking Via Near-Infrared and RGB-D Cameras for Safe Physical Human-Robot Interaction

Safe physical human-robot interaction (pHRI) in industrial settings requires robust and accurate tracking of key points on the human co-worker’s body and the robot structure. However, vision-based single-camera tracking solutions often face many challenges, such as limited field of view (FoV), detection range, occlusions, and inconsistent detection. This paper proposes a complementary multi-sensor tracking scheme that integrates RGB-D and near-infrared (NIR) cameras to improve human motion tracking accuracy while ensuring compliance with ISO/TS 15066 safety requirements. For the first time in pHRI, we deploy an infrared-based tracking system, originally designed for driver assistance and accident prevention, to complement RGB-D cameras, which provide detailed pose estimation at near range but suffer from a narrow FoV. A safety- oriented complementary approach is developed to fuse human tracking data from both systems into robot control, integrating a well-established safety paradigm based on the Safe Motion Unit (SMU) framework. The proposed system is experimentally validated in real-world collaborative robotic workspaces across various pHRI scenarios. Results demonstrate its effectiveness in respecting human safety constraints, even under challenging op- erating conditions, without unnecessary performance restrictions. The complementary vision-based approach improves tracking accuracy, expands FoV, and enhances reliability, making it a promising solution for certifiable, human-aware collaborative robotics in various industrial settings. The video documentation can be seen at https://youtu.be/xWksc_vhuew.