Vision and Tactile-Based Continuous Multimodal Intention and Attention Recognition for Safer Physical Human-Robot Interaction

Employing skin-like tactile sensors on robots enhances both the safety and usability of collaborative robots by adding the capability to detect human contact. Unfortunately, simple binary tactile sensors alone cannot determine the context of the human contact—whether it is a deliberate interaction or an unintended collision that requires safety manoeuvres. Many published methods classify discrete interactions using more advanced tactile sensors or by analysing joint torques. Instead, we propose to augment the intention recognition capabilities of simple binary tactile sensors by adding a robot-mounted camera for human posture analysis. Different interaction characteristics, including touch location, human pose, and gaze direction, are used to train a supervised machine learning algorithm to classify whether a touch is intentional or not with an F1-score of 86%. We demonstrate that multimodal intention recognition is significantly more accurate than monomodal analyses with the collaborative robot Baxter. Furthermore, our method can also continuously monitor interactions that fluidly change between intentional or unintentional by gauging the user’s attention through gaze. If a user stops paying attention mid-task, the proposed intention and attention recognition algorithm can activate safety features to prevent unsafe interactions. We also employ a feature reduction technique that reduces the number of inputs to five to achieve a more generalized low-dimensional classifier. This simplification both reduces the amount of training data required and improves real-world classification accuracy. It also renders the method potentially agnostic to the robot and touch sensor architectures while achieving a high degree of task adaptability. Note to Practitioners—Whenever a user interacts physically with a robot, such as in collaborative manufacturing, the robot may respond to unintended touch inputs from the user. This may be through body collisions or that the user is suddenly distracted and is no longer paying attention to what they are doing. We propose an easy-to-implement method to augment safety of physical human-robot collaboration by determining Manuscript received 16 December 2022; revised 18 April 2023; accepted 7 May 2023. This article was recommended for publication by Associate Editor A. M. Zanchettin and Editor P. Rocco upon evaluation of the reviewers’ comments. This work was supported in part by the Fonds de Recherche du Quebec–Nature et technologies (FRQNT) and in part by the Natural Sciences and Engineering Research Council of Canada (NSERC). (Corresponding author: Christopher Yee Wong.) Christopher Yee Wong and Wael Suleiman are with the Department of Electrical and Computer Engineering, University of Sherbrooke, Sher- brooke, QC J1K 2R1, Canada (e-mail: christopher.wong2@usherbrooke.ca; wael.suleiman@usherbrooke.ca). Lucas Vergez is with Arts et Métiers Institute of Technology, 13617 Aix-en-Provence, France (e-mail: lucas.vergez@ensam.eu). This article has supplementary material provided by the authors and color versions of one or more figures available at https://doi.org/10.1109/TASE.2023.3276856. Digital Object Identifier 10.1109/TASE.2023.3276856 Fig. 1. A human-robot collaborative scenario where a tool held by a robot is guided by having a user physically manipulate the robot. whether the touch from a user is intentional or not through the use of robot-mounted basic touch sensors and computer vision. The algorithm examines the location of the user’s hands relative to the touched sensors in addition to observing where the user is looking. Machine learning is then used to classify in real- time, with an F1-score of 86%, whether a touch is intentional or not such that the robot can react accordingly. The method is particularly applicable in collaborative manufacturing contexts, but can also be applied anywhere where a user physically interacts with a robot. We demonstrate the utility of the method in enhancing safety during human-robot collaboration through a simulated collaborative manufacturing scenario with the robot Baxter, but the method can easily be adapted to other systems.