Procedure Recognition by Knowledge-Driven Segmentation in Robotic-Assisted Vitreoretinal Surgery

Internal limiting membrane (ILM) peeling is a vital vitreoretinal surgery procedure. However, due to the thickness of just 1-2 micrometers and the intricacies associated with its varying density and adhesion, the difficulty of manipulation exceeds the physiological limits of human perception and operation. Surgical robot is characterized by high precision and stability. However, navigating intricate intraocular environments and handling minuscule high-precision areas remain enormous challenges. These include issues of uneven lighting, field-of-view loss, and motion blur. This paper proposed a perception method named 'Multimodal Surgical Process Recognition based on Domain Knowledge and Segmentation (MSPR-DKS),' designed to address these challenges and provide input for the precise control of robots. Moreover, a comprehensive dataset focused on ILM peeling during macular hole surgeries was established. Experimental results underscore the efficacy of this approach, with segmentation accuracies exceeding 99.27% for instruments and macular holes and an average accuracy of 98.97% in recognizing surgical processes. This study paves the way for leveraging domain knowledge and image segmentation to improve robot-assisted manipulation of soft tissues in ophthalmology. This research is supported by the National Key Research and Development Program of China (Grant 2022YFB4702900), the National Natural Science Foundation of China (Grant 62027813, U20A20196), the Beijing Science Fund for Distinguished Young Scholars (JQ21016), the Excellent member of CAS Youth Innovation Promotion Association (Y2022054). Zhen Li is with the School of Electronic and Information Engineering, Tongji University, 200092, Shanghai, China, and Institute of Automation, Chinese Academy of Sciences, 100190, Beijing, China (e-mail: zhen.li@ia.ac.cn). Ya-Wen Deng is with the School of Mechatronic Engineering, Beijing Institute of Technology, Beijing, 100081, China and with the Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China (e-mail: 3120235412@bit.edu.cn). Qiang Ye and Gui-Bin Bian are with the Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China (e-mail: qiang.ye@ia.ac.cn, guibin.bian@ia.ac.cn). Wei-Hong Yu is with the Department of Ophthalmology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100005, China (e-mail: yuwh@pumch.cn). Haoxiang Qi and Yaliang Liu are with the School of Mechatronic Engineering, Beijing Institute of Technology, Beijing, 100081, China (e-mail: 3120215098@bit.edu.cn, liuyaliang@bit.edu.cn). Zhangguo Yu is with the School of Mechatronic Engineering, Beijing Institute of Technology, Beijing, 100081, China, with the Key Laboratory of Biomimetic Robots and Systems, Ministry of Education, Beijing 100081, China, with the International Joint Research laboratory of Biomimetic Robots and Systems, Ministry of Education, Beijing 100081, China, and also with the State Key Laboratory of Intelligent Control and Decision of Complex System, Beijing 100081, China (e-mail: yuzg@bit.edu.cn). *Corresponding author: Gui-Bin Bian