CRED: Counterfactual Reasoning and Environment Design for Active Preference Learning
Yi-Shiuan Tung, Gyanig Kumar, Wei Jiang, Bradley Hayes, Alessandro Roncone
AI summary
Problem
Existing active preference learning methods rely on fixed trajectory sets or replay buffers, which limits query diversity and fails to efficiently identify informative comparisons for learning human reward functions.
Approach
CRED generates diverse preference queries by imagining new environments and using counterfactual reasoning to synthesize trajectory pairs that highlight differences between competing reward hypotheses.
Key results
- Higher reward accuracy with fewer human queries across simulation domains
- Superior sample efficiency compared to state-of-the-art baselines
- Lower user mental workload and higher interaction preference in user studies
- Proven contributions from both counterfactual reasoning and environment design components
Why it matters
Enables robots to rapidly and accurately align with complex, context-dependent human preferences without manual reward engineering.
Abstract
As a robot’s operational environment and tasks to perform within it grow in complexity, the explicit specification and balancing of optimization objectives to achieve a preferred behavior profile moves increasingly farther out of reach. These systems benefit strongly by being able to align their behavior to reflect human preferences and respond to corrections, but manually encoding this feedback is infeasible. Active preference learning (APL) learns human reward functions by presenting trajectories for ranking. However, existing methods sample from fixed trajectory sets or replay buffers that limit query diversity and often fail to identify informative comparisons. We propose CRED, a novel trajectory generation method for APL that improves reward inference by jointly optimizing environment design and trajectory selection to efficiently query and extract preferences from users. CRED “imagines” new scenarios through environment design and leverages counterfac- tual reasoning–by sampling possible rewards from its current belief and asking “What if this were the true preference?”– to generate trajectory pairs that expose differences between competing reward functions. Comprehensive experiments and a user study show that CRED significantly outperforms state- of-the-art methods in reward accuracy and sample efficiency and receives higher user ratings.