Spline-FRIDA: Towards Diverse, Humanlike Robot Painting Styles with a Sample-Efficient, Differentiable Brush Stroke Model
Lawrence Chen, Peter Schaldenbrand, Tanmay Shankar, Lia Coleman, Jean Oh
AI summary
Problem
Existing robotic painting systems rely on restrictive stroke representations like Bézier curves, which fail to capture the complexity and diversity of human brushwork, limiting stylistic control and human-like output.
Approach
The authors record human drawing trajectories via motion capture, model them with a variational autoencoder (TrajVAE), and integrate a novel differentiable polyline renderer (Traj2Stroke) into the FRIDA platform to enable flexible, gradient-based stroke planning.
Key results
- Successfully captures diverse human brushstroke styles using a sample-efficient variational autoencoder
- Introduces Traj2Stroke, a differentiable renderer that significantly reduces the Sim2Real gap for marker strokes
- Human evaluations confirm Spline-FRIDA produces more human-like, artistic, and semantically accurate paintings than baseline FRIDA
- Enables plug-and-play style adaptation with minimal fine-tuning data (<20 trajectories per style)
Why it matters
Advances human-robot co-creation by giving artists flexible stylistic control over robotic painting, bridging the gap between computational art and expressive human brushwork.
Abstract
A painting is more than just a picture on a wall; a painting is a process comprised of many intentional brush strokes, the shapes of which are an important component of a painting’s overall style and message. Prior work in modeling brush stroke trajectories either does not work with real-world robotics or is not flexible enough to capture the complexity of human-made brush strokes. In this work, we introduce Spline-FRIDA which can model complex human brush stroke trajectories. This is achieved by recording artists drawing using motion capture, modeling the extracted trajectories with an autoencoder, and introducing a novel brush stroke dynamics model to the existing robotic painting platform FRIDA. We conducted a sur- vey and found that our open-source Spline-FRIDA approach successfully captures the stroke styles in human drawings and that Spline-FRIDA’s brush strokes are more human-like, improve semantic planning, and are more artistic compared to existing robot painting systems with restrictive B ́ezier curve strokes.