Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning
Emanuele Musumeci, Michele Brienza, Francesco Argenziano, Abdel Hakim Drid, Vincenzo Suriani, Daniele Nardi, Domenico Bloisi
AI summary
Problem
Classical planners fail when preconditions are unmet, while LLM-based planners often hallucinate unfeasible actions. Both lack a principled mechanism to adapt user intent to environmental constraints when tasks are initially unreachable.
Approach
The framework iteratively shifts the environment representation and relaxes goal constraints along functionality and feasibility dimensions, using LLMs to propose intent-preserving goal variants and classical planners to validate and execute them.
Key results
- Novel bidimensional goal-relaxation formalism preserving user intent
- ContextMatters framework coupling LLMs with classical planners for feasible 3D planning
- +52.45% success rate improvement over state-of-the-art LLM+PDDL baselines
- Real-world deployment on a TIAGo robot and a new dataset of 141 relaxation-prone tasks
Why it matters
Enables reliable long-horizon task execution for embodied agents in complex, dynamic real-world environments where rigid planning fails.
Abstract
Embodied agents need to plan and act reliably in real and complex 3D environments. Classical planning (e.g., PDDL) offers structure and guarantees, but in practice it fails under noisy perception and incorrect predicate grounding. On the other hand, Large Language Models (LLMs)-based planners leverage commonsense reasoning, yet frequently propose actions that are unfeasible or unsafe. Following recent works that combine the two approaches, we introduce ContextMatters, a framework that fuses LLMs and classical planning to perform hierarchical goal relaxation: the LLM helps ground symbols to the scene and, when the target is unreachable, it proposes functionally equivalent goals that progressively relax constraints, adapting the goal to the context of the agent’s environment. Operating on 3D Scene Graphs, this mechanism turns many nominally unfeasible tasks into tractable plans and enables context-aware partial achievement when full completion is not achievable. Our experimental results show a +52.45% Success Rate improvement over state-of-the-art LLMs+PDDL baseline, demonstrating the effectiveness of our approach. Moreover, we validate the execution of ContextMatters in a real world scenario by deploying it on a TIAGo robot. Code, dataset, and supplementary materials are available to the community at https://lab-rococo-sapienza.github.io/context-matters/.