Vision-Guided Robotic Grinding with Deep Learning-Based Bead Segmentation and Digital Twin Verification
Seong Hyeon Kim, Hyo-Young Kim
AI summary
Problem
Conventional robotic grinding relies on rigid teach-pendant programming that cannot adapt to natural variations in weld bead geometry and position, making it impractical for high-mix manufacturing and exposing workers to hazardous manual grinding conditions.
Approach
The system uses a ResNet34-backed U-Net to segment weld beads from RGB images, transforms 2D contours to 3D robot coordinates via hand-eye calibration, plans collision-free paths with CHOMP, and validates them in an NVIDIA Isaac Sim digital twin before physical execution.
Key results
- U-Net segmentation achieves 0.9311 mean IoU and 0.9641 Dice coefficient on weld bead detection
- Automated pipeline generates adaptive grinding waypoints along bead centerlines without manual re-teaching
- CHOMP trajectory planner produces smooth, collision-free paths verified in a digital twin prior to deployment
- System processes segmentation in 29 ms on an RTX 5070 GPU, enabling real-time adaptive grinding
Why it matters
Provides a flexible, safe, and efficient solution for automated weld bead grinding in high-mix manufacturing, reducing commissioning time and worker exposure to hazardous conditions.
Abstract
Weld bead grinding is a critical post-processing step in metal fabrication, yet conventional robotic grinding based on teach-pendant programming lacks adaptability to variations in bead geometry and position. This paper presents a vision-guided robotic grinding system that combines deep learning-based weld bead segmentation, automated grinding path generation, and digital twin-based pre-verification. A U-Net model with a ResNet34 encoder and ImageNet pre-training segments weld bead regions from RGB images captured by an Intel RealSense D415 camera mounted on a Staubli RX160 manipulator, achieving a mean Intersection over Union (IoU) of 0.9311 and a Dice coefficient of 0.9641. The segmented bead contours are transformed into the robot coordinate frame through hand-eye calibration and forward kinematics, enabling automated generation of grinding waypoints along the bead centerline. The CHOMP algorithm plans collision-free trajectories within MoveIt, and all planned motions are validated in a digital twin environment built on NVIDIA Isaac Sim 5.0, integrated with ROS through a distributed multi-container architecture. Experimental results demonstrate that the proposed system effectively generates adaptive grinding paths for varying weld bead geometries and verifies them in simulation before physical deployment.