Estimating the Joint Angles of a Magnetic Surgical Tool Using Monocular 3D Keypoint Detection and Particle Filtering

Magnetic surgical tools benefit greatly from real- time pose estimation, as this is essential for controlling them safely and effectively. Current pose estimation methods for surgical tools either focus on rigid tools, or are developed specifically for the da Vinci surgical system. In this work, we use computer vision from a monocular endoscopic camera to estimate the pose of an articulated magnetic surgical tool. In particular, we present a deep 3D keypoint estimation framework and a particle filter to achieve this. The former method can be used for any articulated surgical tool, while the latter method is specific to magnetic tools. We show that the deep 3D keypoint estimation framework estimates the surgical tool’s joint angles with an average error of 4.0 degrees and a speed of 29 Hz. In addition, we demonstrate the robustness of the magnetic particle filter and the deep pose estimation method for real- time tool pose estimation.