Hand Tracking System Utilizing Learning Based on Vision Sensing and Ionic Gel Sensor Glove

Hand tracking has attracted considerable interest in fields such as virtual reality and human-robot interaction. However, single-sensor approaches to hand tracking face chal- lenges, particularly with data loss due to occlusions or finger overlap. This paper proposes a multi-sensor system for 3D hand pose estimation that fuses a vision sensor and an ionic gel sensor. We use the Intel RealSense Depth Camera D435 to capture finger joint angles, supplemented by Ionic Gel Sensor Glove that provides continuous measurements. By combining these data streams using a machine learning framework with an autoencoder and LSTM network, we can accurately estimate finger joint angles even in the presence of the missing visual data. The experiments compared the method using both the visual sensor and the glove with the method using only the visual sensor. As a result, it was confirmed that the accuracy of finger joint angle estimation improved significantly, especially in cases where data was missing. Additionally, the method demonstrated consistent improvements in accuracy across different users and types of gloves.