SurgIRL: Towards Life-Long Learning for Surgical Automation by Incremental Reinforcement Learning
Yun-Jie Ho, Zih-Yun Chiu, Yuheng Zhi, Michael C. Yip
AI summary
Problem
Current reinforcement learning approaches for surgical automation train policies independently from scratch, making skill reuse difficult and slowing down the automation of complex, sequential tasks.
Approach
The authors introduce SurgIRL, a knowledge-grounded reinforcement learning framework that maintains an expandable library of heterogeneous policies. It uses a novel attention-based navigation algorithm and flexible incremental pipelines to accumulate and reuse skills across diverse surgical tasks.
Key results
- KIAN-ACE outperforms state-of-the-art KGRL in sample efficiency and success rates across ten simulated tasks
- Incremental pipelines successfully accumulate knowledge to solve unseen tasks sequentially
- Successful sim-to-real transfer of trained policies on the da Vinci Research Kit
- Flexible knowledge reuse strategies adapt to varying task similarities and state/action spaces
Why it matters
Provides a scalable pathway for surgical robots to accumulate expertise over time, accelerating the development of autonomous, lifelong learning surgical automation systems.
Abstract
Surgical automation holds immense potential to improve the outcome and accessibility of surgery. Recent studies use reinforcement learning to automate various surgical tasks. However, these policies are developed independently, and their reusability is limited when applied to other scenarios, making it more time-consuming for robots to incrementally solve tasks. Inspired by how human surgeons build their expertise, we pro- pose Surgical Incremental Reinforcement Learning (SurgIRL). SurgIRL aims to (1) acquire new skills by referring to external policies (knowledge) and (2) build an expandable knowledge base and reuse it to solve multiple unseen tasks incrementally (incremental learning). Our SurgIRL framework includes three major components. We first define an expandable knowledge set containing heterogeneous policies that can be helpful for surgical tasks. Then, we propose Knowledge Inclusive Attention Network with mAximum Coverage Exploration (KIAN-ACE), which en- hances learning performance through extensive navigation of the knowledge base. Finally, we develop incremental learning pipelines to expand and reuse a knowledge base and solve multi- ple surgical tasks sequentially. Our simulation experiments show that SurgIRL efficiently learns to automate ten surgical tasks separately or incrementally. We also demonstrate successful sim- to-real transfers of SurgIRL’s policies on the da Vinci Research Kit (dVRK). The results represent an initial step towards lifelong robot learning for surgical automation.