Perception-Control Coupled Visual Servoing for Textureless Objects Using Keypoint-Based EKF
Allen Tao, Jun Yang, Stanko Oparnica, Wenjie Xue
AI summary
Problem
Traditional visual servoing struggles with textureless objects due to unreliable features, and adverse conditions like occlusions or lighting changes corrupt feedback, causing instability and reduced accuracy.
Approach
The method tightly couples perception and control by using an Extended Kalman Filter to fuse per-frame keypoints with a motion prior for robust 6D pose estimation, which drives a probabilistic controller that outputs both camera velocity and its uncertainty.
Key results
- Closed-loop perception-control framework for textureless object servoing
- Keypoint-based EKF fusing temporal cues and motion priors for robust 6D pose estimation
- Probabilistic control law computing velocity and uncertainty for safe operation
- Outperforms IBVS and PBVS baselines in success rate, accuracy, and robustness under adverse conditions
Why it matters
Enables reliable and safe robotic manipulation of common industrial textureless objects in real-world, unstructured environments.
Abstract
Visual servoing is fundamental to robotic appli- cations, enabling precise positioning and control. However, applying it to textureless objects remains a challenge due to the absence of reliable visual features. Moreover, adverse visual conditions, such as occlusions, often corrupt visual feedback, leading to reduced accuracy and instability in visual servoing. In this work, we build upon learning-based keypoint detection for textureless objects and propose a method that enhances robustness by tightly integrating perception and control in a closed loop. Specifically, we employ an Extended Kalman Filter (EKF) that integrates per-frame keypoint measurements to estimate 6D object pose, which drives pose-based visual servoing (PBVS) for control. The resulting camera motion, in turn, enhances the tracking of subsequent keypoints, effectively clos- ing the perception-control loop. Additionally, unlike standard PBVS, we propose a probabilistic control law that computes both camera velocity and its associated uncertainty, enabling uncertainty-aware control for safe and reliable operation. We validate our approach on real-world robotic platforms using quantitative metrics and grasping experiments, demonstrating that our method outperforms traditional visual servoing tech- niques in both accuracy and practical application.