Point Cloud Segmentation for Autonomous Clip Positioning in Laparoscopic Cholecystectomy on a Phantom
Balazs Gyenes, Nikolai Franke, Paul Maria Scheikl, Pit Henrich, Rayan Younis, Gerhard Neumann, Martin Wagner, Franziska Mathis-Ullrich
AI summary
Problem
Autonomous robotic surgery requires extreme precision and interpretability, but data scarcity and the limitations of end-to-end learning models hinder reliable task execution. Specifically, accurately positioning surgical clips during laparoscopic cholecystectomy demands sub-millimeter accuracy while remaining verifiable and adjustable by surgeons.
Approach
The system segments target anatomical regions from a colorless point cloud, fits B-splines to the segmented structures, and extracts precise target points via spline interpolation, allowing surgeons to visually validate and adjust the planned trajectories before execution.
Key results
- 95% target localization success within 0.75 mm tolerance on a physical phantom
- 100% autonomous clip positioning success rate in real robot experiments
- Effective fine-tuning with only 60 real samples using 128k synthetic pre-training
- Novel augmentation and logit-space merging improve sim-to-real robustness
Why it matters
This framework demonstrates how data-efficient, interpretable vision systems can enable safe semi-autonomous execution of high-precision surgical tasks, offering a blueprint for other robotic applications requiring human oversight.
Abstract
High-risk applications in robotics, such as robot- assisted surgery, present unique challenges. These systems must be both highly precise and interpretable in order to be deployed in environments with very low tolerance for error or unsafe exploration. We present the first robotic system to demonstrate autonomous clip positioning on a physical phantom in laparo- scopic surgery, one of the most common interventions in general surgery. After segmentation of a colorless point cloud from a single camera, target positions for the clips are extracted using spline interpolation, and can then be adjusted by the human operator. The segmentation model is trained on only 60 hand- labeled real point clouds, reflecting data scarcity in the surgical domain. We overcome this with a combination of pre-training on 128,000 synthetic point clouds and two novel data augmentation techniques. The motion of the end-effector to each target is visu- alized for the operator, satisfying the unique motion constraints of minimally-invasive surgery while ensuring that the robot’s actions are verifiable and interpretable. In real robot experiments, our system localizes targets with the required precision of 0.75 mm at a 95% success rate and executes autonomous clip positioning with a 100% success rate. We provide insights that are applicable to many other surgical and non-surgical tasks that require identifying and navigating to a precise target. Our source code is available at https://github.com/balazsgyenes/kirurc.