ASC: Adaptive Skill Coordination for Robotic Mobile Manipulation

We present Adaptive Skill Coordination (ASC) – an approach for accomplishing long-horizon tasks like mobile pick- and-place (i.e., navigating to an object, picking it, navigating to another location, and placing it). ASC consists of three compo- nents – (1) a library of basic visuomotor skills (navigation, pick, place), (2) a skill coordination policy that chooses which skill to use when, and (3) a corrective policy that adapts pre-trained skills in out-of-distribution states. All components of ASC rely only on onboard visual and proprioceptive sensing, without requiring detailed maps with obstacle layouts or precise object locations, easing real-world deployment. We train ASC in simulated indoor environments, and deploy it zero-shot (without any real-world experience or fine-tuning) on the Boston Dynamics Spot robot in eight novel real-world environments (one apartment, one lab, two microkitchens, two lounges, one office space, one outdoor court- yard). In rigorous quantitative comparisons in two environments, ASC achieves near-perfect performance (59/60 episodes, or 98%), while sequentially executing skills succeeds in only 44/60 (73%) episodes. Extensive perturbation experiments show that ASC is robust to hand-off errors, changes in the environment layout, dynamic obstacles (e.g., people), and unexpected disturbances. Supplementary videos at adaptiveskillcoordination.github.io. Manuscript received: June, 25, 2023; Revised October, 4, 2023; Accepted November, 6, 2023. This paper was recommended for publication by Editor Jens Kober upon evaluation of the Associate Editor and Reviewers’ comments. The Georgia Tech effort was supported in part by ONR YIPs, ARO PECASE, and Korea Evaluation Institute of Industrial Technology (KEIT) funded by the Korea Government (MOTIE) under Grant No.20018216, Devel- opment of mobile intelligence SW for autonomous navigation of legged robots in dynamic and atypical environments for real application. JT was supported by an Apple Scholars in AI/ML PhD Fellowship. The views and conclusions are those of the authors and should not be interpreted as representing the U.S. Government, or any sponsor. 1 NY, JT, SH, and DB are with Georgia Institute of Technology. E-mail: nyokoyama@gatech.edu 2 AC, EU, TY, SA, DB, and AR are with Meta AI. Digital Object Identifier (DOI): see top of this page.