PICaSo: A Collaborative Robotics System for Inpainting on Physical Canvas Using Marker and Eraser

Robotics collaborative drawing involves the inter- action between humans and robots to create of visual art using a variety of tools and materials, serving various functions such as communication, narration, and emotional representation. A creative technique within the human natural drawing process is known as inpainting, which involves reconstructing or editing elements in a drawing. This paper introduces PICaSo (Physical Inpainting on Canvas Solution), a robotic drawing system that enables multiple users to collaboratively create artwork on a canvas by integrating the inpainting process. PICaSo utilizes a fine-tuned text-to-image model to interpret natural language prompts into artistic renderings on canvas. Users guide the process by simple descriptive text and specifying desired drawing placement, empowering the robotic arm to au- tonomously translate these instructions into physical artworks. Our system’s innovation lies in its effective translation of digital inpainting processes into physical actions. By leveraging our erasing capability that enables selective removal of specific parts on the canvas without impacting neighboring areas, facilitating the creation of sequential drawings. This paper comprehen- sively outlines the capabilities of the proposed system, explores potential applications across various domains, and addresses technical challenges encountered during its development. Project website: shadynasrat.github.io/PICaSo