Safe Explicable Policy Search
Akkamahadevi Hanni, Jonathan Montano, Yu (Tony) Zhang
AI summary
Problem
Users often form expectations of AI agents that conflict with the agent's optimal or safe behaviors, yet existing methods cannot generate safe, explicable policies in continuous, learning-based settings where the user's model is unknown.
Approach
SEPS formulates policy search as a constrained optimization problem that maximizes a surrogate reward capturing user expectations while enforcing task and safety limits, solved via Constrained Policy Optimization in a reinforcement learning framework.
Key results
- Formulates SEPS as a constrained optimization problem bridging explicable planning and safe RL
- Derives an analytic solution for the constrained CMDP using adapted CPO updates
- Validates efficacy in safety-gym environments and a physical robot experiment
- Demonstrates generation of policies that align with user expectations while strictly maintaining safety limits
Why it matters
Enables reliable human-AI teaming in robotic applications by ensuring AI behaviors are both understandable to users and provably safe during learning and deployment.
Abstract
When users work with AI agents, they form conscious or subconscious expectations of them. Meeting user expectations is crucial for such agents to engage in successful in- teractions and teaming. However, users may form expectations of an agent that differ from the agent’s planned behaviors. These differences lead to the consideration of two separate decision models in the planning process to generate explicable behaviors. However, little has been done to incorporate safety considerations, especially in a learning setting. We present Safe Explicable Policy Search (SEPS), which aims to provide a learning approach to explicable behavior generation while minimizing the safety risk, both during and after learning. We formulate SEPS as a constrained optimization problem where the agent aims to maximize an explicability score subject to constraints on safety and a suboptimality criterion based on the agent’s model. SEPS innovatively combines the capabilities of Constrained Policy Optimization and Explicable Policy Search to introduce the capability of generating safe explicable behaviors to domains with continuous state and action spaces, which is critical for robotic applications. We evaluate SEPS in safety-gym environments and with a physical robot experiment to show its efficacy and relevance in human-AI teaming.