DexFruit: Dexterous Manipulation and Gaussian Splatting Inspection of Fruit
Aiden Swann, Alex Qiu, Matthew Strong, Angelina Zhang, Samuel Morstein, Kai Rayle, Monroe Kennedy
AI summary
Problem
Automated handling of fragile soft fruits is hindered by high bruising rates and a lack of tactile feedback in manipulation policies, while existing damage assessment methods are either subjective or require expensive, specialized equipment.
Approach
The framework combines optical tactile sensing with a diffusion-based imitation learning policy for gentle pick-and-place manipulation, and introduces FruitSplat to reconstruct and quantify visual damage in 3D using only webcam footage.
Key results
- 92% grasping policy success rate across three fruit types
- Up to 15% reduction in visual bruising compared to baselines
- Up to 31% improvement in grasp success rate on challenging fruit
- Accurate 3D damage quantification from standard webcam video via FruitSplat
Why it matters
Offers a scalable, low-cost solution for automating delicate agricultural handling and post-harvest quality control, directly addressing labor shortages and reducing global food waste.
Abstract
Dexfruit is a robotic manipulation framework that enables gentle, autonomous handling of fragile fruit and precise evaluation of damage. Soft fruits have long faced an issue of produce loss in both the harvesting and post-harvesting processes due to their extreme fragility and susceptibility to bruising, making them one of the hardest produce type to manipulate with automation. In this work, we demonstrate by using optical tactile sensing, autonomous manipulation of fruit with minimal damage can be achieved. We show that our tactile informed diffusion policies outperform baselines in both reduced bruising and pick- and-place success rate across three fruits: strawberries, tomatoes, and blackberries. In addition, we introduce FruitSplat, a novel technique to represent and quantify visual damage in a high- resolution 3D representation via 3D Gaussian Splatting (3DGS). Existing metrics for measuring damage lack quantitative rigor or require expensive equipment. With FruitSplat, we distill a 2D fruit mask as well as a 2D bruise segmentation mask into the 3DGS representation from just a web-cam video. Furthermore, this representation is modular and general, compatible with any relevant 2D model. Overall, we demonstrate a 92% grasping policy success rate, up to a 15% reduction in visual bruising, and up to a 31% improvement in grasp success rate on challenging fruit compared to our baselines across our three tested fruits. We rigorously evaluate this result with over 630 trials. Please checkout our website, which contains our code and datasets at https://dex-fruit.github.io/.