IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction
Yingke Wang, Hao Li, Yifeng Zhu, Hong-Xing Yu, Ken Goldberg, Li Fei-Fei, Jiajun Wu, Yunzhu Li, Ruohan Zhang
AI summary
Problem
Reproducing oil paintings with a robot requires precise control of deformable brushes, predicting complex paint interactions, and multi-step planning, typically without access to human demonstrations or accurate simulators.
Approach
The system learns a neural pixel dynamics model from robot self-play to predict canvas updates from stroke actions, then uses receding-horizon model predictive control to plan trajectories and forces for closed-loop painting.
Key results
- Trained a pixel dynamics model from robot self-play that accurately predicts stroke outcomes.
- Outperformed baselines in planning and execution accuracy on expert human brushstrokes.
- Demonstrated closed-loop, multi-step planning to approximate full multi-stroke artworks.
- Integrated low-level force control, learned dynamics, and high-level MPC into a complete robotic painting system.
Why it matters
Enables autonomous, high-fidelity robotic art reproduction without relying on simulators or human demonstrations, advancing force-sensitive manipulation and learned dynamics for real-world robotics.
Abstract
Robotic reproduction of oil paintings using soft brushes and pigments requires force-sensitive control of de- formable tools, prediction of brushstroke effects, and multi-step stroke planning, often without human step-by-step demonstra- tions or faithful simulators. Given only a sequence of target oil painting images, can a robot infer and execute the stroke tra- jectories, forces, and colors needed to reproduce it? We present IMPASTO, a robotic oil-painting system that integrates learned pixel dynamics models with model-based planning. The dynam- ics models predict canvas updates from image observations and parameterized stroke actions; a receding-horizon model predictive control optimizer then plans trajectories and forces, while a force-sensitive controller executes strokes on a 7-DoF robot arm. IMPASTO integrates low-level force control, learned dynamics models, and high-level closed-loop planning, learns solely from robot self-play, and approximates human artists’ single-stroke datasets and multi-stroke artworks, outperforming baselines in reproduction accuracy. Project website and ap- pendix: https://impasto-robopainting.github.io/.