Mash, Spread, Slice! Learning to Manipulate Object States Via Visual Spatial Progress
Priyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto MartÃn-MartÃn, Kristen Grauman
AI summary
Problem
Most robotic manipulation focuses on rigid-body motion, leaving tasks that progressively transform an object’s physical and visual state underexplored due to the difficulty of representing gradual intra-object changes and designing effective reward signals for learning.
Approach
SPARTA integrates spatially progressing object state change segmentation maps to strip away visual appearance noise and generate dense, progress-aware rewards, powering both a reinforcement learning policy for fine-grained control and a lightweight greedy controller for fast deployment.
Key results
- Sample-efficient real-world RL training in 1.5–3 hours without demonstrations or simulation
- Successful execution of spreading, mashing, and slicing across 10 diverse real-world objects
- Significant improvements in training speed and accuracy over sparse reward and goal-conditioned baselines
- Unified framework supporting both adaptive RL agents and fast heuristic greedy controllers
Why it matters
Provides a scalable, vision-driven foundation for robots to master complex, non-rigid manipulation tasks common in cooking and household chores without relying on simulation or human demonstrations.
Abstract
Most robot manipulation focuses on changing the kinematic state of objects: picking, placing, opening, or rotating them. However, a wide range of real-world manipulation tasks involve a different class of object state change—such as mashing, spreading, or slicing—where the object’s physical and visual state evolve progressively without necessarily changing its position. We present SPARTA, the first unified framework for the family of object state change manipulation tasks. Our key insight is that these tasks share a common structural pattern: they involve spatially-progressing, object-centric changes that can be represented as regions transitioning from an actionable to a transformed state. Building on this insight, SPARTA integrates spatially progressing object change segmentation maps, a visual skill to perceive actionable vs. transformed regions for specific object state change tasks, to generate a) structured policy observations that strip away appearance variability, and b) dense rewards that capture incremental progress over time. These are leveraged in two SPARTA policy variants: reinforcement learning for fine-grained control without demon- strations or simulation; and greedy control for fast, lightweight deployment. We validate SPARTA on a real robot for three challenging tasks across 10 diverse real-world objects, achieving significant improvements in training time and accuracy over sparse rewards and visual goal-conditioned baselines. Our results highlight progress-aware visual representations as a versatile foundation for the broader family of object state manipulation tasks. More information at https://vision. cs.utexas.edu/projects/sparta-robot