Improving Robot Proficiency Self-Assessment Via Meta-Assessment

Proficiency self-assessment (PSA), which is the abil- ity to estimate how likely one can complete a task, is a beneficial property for autonomous robots. Prior work developed the assumption-alignment tracking (AAT) method for PSA, which estimates the probability that a robot will successfully complete a task. This paper refers to the prediction made by AAT as the first-level assessment (FLA), and further proposes a second-level assessment (SLA) that determines whether the FLA prediction is correct. The probability that the FLA prediction is correct is conditioned on four features: (1) the mean distance from a test sample to its nearest neighbors in the training set; (2) the predicted probability of success made by the FLA; (3) the ratio between the robot’s current performance and its performance standard; and (4) the percentage of the task the robot has already completed. The SLA model is trained on the four features using a Random Forest algorithm. It is evaluated by two metrics: discriminability, measured by the area under the ROC curve, and calibration, measured using expected calibration error. On a simulated navigation task and a manipulation task by a Sawyer robot, results demonstrate that the SLA model not only calibrates the FLA model as well as existing calibration methods (Platt calibration and isotonic regression), but also produces very high discriminability even if the FLA model’s original discriminability is much lower. Results also indicate the usefulness of each of the four features used by the SLA model.