AnySafe: Adapting Latent Safety Filters at Runtime Via Safety Constraint Parameterization in the Latent Space
Sankalp, Sunny Agrawal, Junwon Seo, Kensuke Nakamura, Ran Tian, Andrea Bajcsy
AI summary
Problem
Existing latent safety filters assume fixed, pre-defined safety constraints, limiting their ability to adapt to changing environments or user requirements during deployment.
Approach
The method conditions a latent safety filter on an encoded image of a constraint, using a learned latent similarity measure and conformal calibration to dynamically define the failure region, all trained within a world model's imagination.
Key results
- Runtime adaptation to arbitrary image-specified constraints
- Performance matches specialized filters trained on single constraints
- Generalizes to novel constraints beyond specialized filters
- Conformal calibration enables adjustable avoidance conservatism
Why it matters
It enables vision-based robotic systems to safely adapt to dynamic, user-defined safety rules in real-world deployments without retraining.
Abstract
Recent works have shown that foundational safe control methods, such as Hamilton–Jacobi (HJ) reachability analysis, can be applied in the latent space of world models. While this enables the synthesis of latent safety filters for hard-to-model vision-based tasks, they assume that the safety constraint is known a priori and remains fixed during deploy- ment, limiting the safety filter’s adaptability across scenarios. To address this, we propose constraint-parameterized latent safety filters that can adapt to user-specified safety constraints at runtime. Our key idea is to define safety constraints by condi- tioning on an encoding of an image that represents a constraint, using a latent-space similarity measure. The notion of similarity to failure is aligned in a principled way through conformal calibration, which controls how closely the system may ap- proach the constraint representation. The parameterized safety filter is trained entirely within the world model’s imagination, treating any image seen by the model as a potential test-time constraint, thereby enabling runtime adaptation to arbitrary safety constraints. In simulation and hardware experiments on vision-based control tasks with a Franka manipulator, we show that our method adapts at runtime by conditioning on the encoding of user-specified constraint images, without sacrificing performance. Video results can be found on the project website.