The Role of Real-World Data in Evaluating Causal Bayesian Networks: Data Collection Guidelines and Case Study

Causal Bayesian Networks (CBNs) in robotics are often learned in simulation due to the considerable amount of data required for training. However, discrepancies between simulation and the physical world can cause the learned causal relations to fail in real-world scenarios. Thus, the sim-to-real evaluation is a critical step to deploy a simulation-learned CBN in the real-world. The main challenges in this process are the lack of real-robot evaluation datasets that capture the complexity, noise, and variability of physical environments, which are missing in simulation. In this paper, we propose a set of task-agnostic guidelines for real-robot data collection to evaluate Causal Bayesian Networks (CBNs). The guidelines are generalizable and can be applied to collect real-robot datasets across different robot tasks and platforms. To demonstrate this, we apply them to a robotic platform performing one concrete task, e.g., the robot TIAGo performing a two-cube stacking task, and we collect the real-robot dataset from 100 trials. As a case study, we demonstrate how the dataset can be used to evaluate a simulation-trained CBN on real-robot executions, reporting 10% accuracy drop from sim-to-real transfer. We present this as a first step towards standardized and quantifiable sim-to-real evaluation for CBNs.