← Back ICRA 2026

A Benchmarking Study of Vision-Based Robotic Grasping Algorithms

Bharath Kumar Ramesh Babu, Sumukh Sreenivasarao Balakrishna, Brian Flynn, Vinayak Kapoor, Adam Norton, Holly Yanco, Berk Calli

PDF

AI summary

Key figure (auto-extracted from paper)

Standardized benchmarking reveals that vision-based grasping performance varies significantly across hardware, lighting, and environments, with notable gaps between simulation and real-world results.

Vision-based grasping performance benchmarking robotic manipulation sim-to-real gap standardized evaluation grasp synthesis

Problem

The lack of standardized benchmarking protocols and common experimental setups makes it difficult to fairly compare grasping algorithms, reproduce results, or systematically improve robotic manipulation systems.

Approach

The authors benchmarked two machine-learning and two analytical vision-based grasping algorithms using a standardized protocol across 5,040 real-world and simulation experiments, systematically varying cameras, grippers, lighting, backgrounds, and robot platforms.

Key results

Evaluated 4 grasping algorithms across 5,040 experiments with 10 YCB objects
Quantified performance impacts of lighting, background textures, and camera noise
Identified significant sim-to-real performance discrepancies
Released open-source benchmarking software and experiment recordings

Why it matters

Provides robotics researchers with a reproducible framework and empirical insights to fairly evaluate and improve vision-based grasping systems under real-world variability.

Abstract

We present a benchmarking study of vision-based robotic grasping algorithms and provide a comparative analysis. In particular, we compare two machine-learning-based and two analytical algorithms using an existing benchmarking protocol from the literature and determine the algorithms’ strengths and weaknesses under different experimental conditions. These conditions include variations in lighting, background textures, cameras with different noise levels, and grippers. We also run analogous experiments in simulations and with real robots and present the discrepancies. Some experiments are also run in two different laboratories using the same protocols to further analyze the repeatability of our results. We believe that this study, comprising 5040 experiments, provides important insights into the role and challenges of systematic experimentation in robotic manipulation and guides the development of new algorithms by considering the factors that could impact the performance. The experiment recordings1 and our benchmarking software2 are publicly available.

Index terms

Grasping Performance Evaluation and Benchmarking Deep Learning in Grasping and Manipulation