Bi-KVIL: Keypoints-Based Visual Imitation Learning of Bimanual Manipulation Tasks

the hand/object relationships into a symbolic Hybrid Master- Slave Relationship (HMSR) with (c) sub-symbolic geometric constraints for each master-slave pair to model motion styles. (d) The learned tasks are then reproduced with category-level generalization in cluttered scenes by ARMAR-6. Abstract— Visual imitation learning has achieved impressive progress in learning unimanual manipulation tasks from a small set of visual observations, thanks to the latest advances in computer vision. However, learning bimanual coordination strategies and complex object relations from bimanual visual demonstrations, as well as generalizing them to categorical objects in novel cluttered scenes remain unsolved challenges. In this paper, we extend our previous work on keypoints-based visual imitation learning (K-VIL) [1] to bimanual manipulation tasks. The proposed Bi-KVIL jointly extracts so-called Hybrid Master-Slave Relationships (HMSR) among objects and hands, bimanual coordination strategies, and sub-symbolic task repre- sentations. Our bimanual task representation is object-centric, embodiment-independent, and viewpoint-invariant, thus gener- alizing well to categorical objects in novel scenes. We evaluate our approach in various real-world applications, showcasing its ability to learn fine-grained bimanual manipulation tasks from a small number of human demonstration videos. Videos and source code are available at https://sites.google.com/view/bi-kvil.