Image-To-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light
Jiayin Wang, mingfeng yao, Yanran Wei, Xiaoyu Guo, Ayong Zheng, Weidong Zhao
AI summary
Problem
Most minimally invasive surgical robots lack direct force sensors due to space and cost constraints, while existing vision-based methods fail to accurately estimate forces on texture-deficient soft tissues during critical pulling tasks.
Approach
The method projects a custom one-shot structured light pattern onto tissue to generate a dense 3D point cloud from a single stereo camera frame, which is then processed by a modified PointNet neural network to predict interaction forces.
Key results
- Dense 3D point cloud reconstruction from a single stereo frame using a DeBruijn structured light pattern
- Modified PointNet network accurately maps tissue deformation to tensile and compressive forces
- Custom dataset of paired 3D point clouds and force measurements enables effective training
- Experimental validation on varying-stiffness silicon materials and a laparoscopic robot confirms high estimation accuracy
Why it matters
Provides a practical, sensorless haptic feedback solution that enhances surgical safety and precision in laparoscopic systems without requiring invasive hardware or complex temporal processing.
Abstract
For Minimally Invasive Surgical (MIS) robots, ac- curate haptic interaction force feedback is essential for ensuring the safety of interacting with soft tissue. However, the major- ity of existing MIS robotic systems cannot facilitate direct mea- surement of the interaction force with hardware sensors due to space limitations. This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue coupled with haptic informa- tion processing through a trained image-to-force neural network. The images captured from the endoscopic stereo camera are an- alyzed to reconstruct high-resolution 3D point clouds for soft tis- sue deformation. The proposed methodology involves a modified PointNet-based force estimation method, which has demonstrated proficiency in accurately representing the intricate mechanical properties of soft tissue. To validate the efficacy of the proposed methodology, numerical force interaction experiments were con- ducted on three silicon materials with varying stiffness levels. The experimental results substantiate the efficacy of the proposed methodology.