Towards Learning Boulder Excavation with Hydraulic Excavators
Jonas Gruetter, Lorenzo Terenzi, Pascal Arturo Egli, Marco Hutter
AI summary
Problem
Autonomous boulder removal with standard digging buckets remains a critical gap in construction robotics, as existing methods require impractical tool changes, detailed 3D models, or fail under harsh outdoor conditions with limited perception.
Approach
A reinforcement learning policy is trained in simulation using rigid-body dynamics and analytical soil models to process sparse LiDAR points and proprioceptive feedback, enabling adaptive control of a standard excavator bucket.
Key results
- RL policy discovers adaptive excavation strategies (surface dragging vs. direct penetration) based on soil resistance
- Sparse perception pipeline using SAM2 segmentation successfully isolates target rocks with only 20 LiDAR points each
- Field deployment on a 12-ton excavator achieves a 70% success rate across diverse rock sizes and soil types
- Performance approaches human operator baselines (83% success) despite challenging outdoor conditions
Why it matters
Demonstrates that heavy construction equipment can perform complex discrete object manipulation without specialized tools, enabling seamless and efficient site automation.
Abstract
Construction sites frequently require removing large rocks before excavation or grading can proceed. Human operators typically extract these boulders using only standard digging buckets, avoiding time-consuming tool changes to spe- cialized grippers. This task demands manipulating irregular ob- jects with unknown geometries in harsh outdoor environments where dust, variable lighting, and occlusions hinder perception. The excavator must adapt to varying soil resistance—dragging along hard-packed surfaces or penetrating soft ground—while coordinating multiple hydraulic joints to secure rocks using a shovel. Current autonomous excavation focuses on continuous media (soil, gravel) or uses specialized grippers with detailed geometric planning for discrete objects. These approaches either cannot handle large irregular rocks or require impractical tool changes that interrupt workflow. We train a reinforcement learning policy in simulation using rigid-body dynamics and analytical soil models. The policy processes sparse LiDAR points (just 20 per rock) from vision-based segmentation and proprioceptive feedback to control standard excavator buckets. The learned agent discovers different strategies based on soil resistance: dragging along the surface in hard soil and penetrating directly in soft conditions. Field tests on a 12- ton excavator achieved 70% success across varied rocks (0.4– 0.7m) and soil types, compared to 83% for human operators. This demonstrates that standard construction equipment can learn complex manipulation despite sparse perception and challenging outdoor conditions.