Statistical Stratification and Benchmarking of Robotic Grasping Performance

Robotic grasping is fundamental to many real- world applications, and new approaches must be systematically evaluated. However, in most cases, the performance of a specific approach is assessed by simply counting the number of successful attempts in a given task, and this success rate is then compared to those of other solutions, without taking into account the random variability across different experiments (e.g. due to sensor noise, or variations in object placement). In order to address this issue, we classify the observed perfor- mance into qualitatively ordered outcomes, thereby stratifying the results. We then show how to analyse these results, in a statistical framework which accounts for the variability between experiments. The advantages of our approach are demonstrated in the practical comparison of four grasp planning algorithms. In particular, we show that the proposed approach allows us to carry out several distinct evaluations from a single set of experiments, without having to repeat the data collection pro- cess. We demonstrate that differences between the algorithms, which would not be apparent from overall success rates, can be identified and evaluated.