VF Designer: CAD-Guided Virtual Fixtures for Enhanced Robot Teleoperation in Multi-Step Manipulation Tasks

As robots are deployed into new domains, teleop- eration with a human-in-the-loop remains an important method for training new skills. Direct control of a remote robot over a network demands high concentration of the human teleop- erator and remains challenging due to unavoidable network delays and congestion that can degrade performance. In this work, we demonstrate a shared control solution using well established virtual fixtures (VFs) as a teleoperator assistance tool and introduce a novel VF generation pipeline leveraging mate constraints in available CAD data of the manipulated objects. This approach simplifies the definition of VFs by taking parameters from relationships already defined within the CAD data. Using an industry-sponsored task board and a bilateral leader-follower hand-guided robot scenario, we demonstrate how a set of VFs can be constructed and activated in a series to support teleoperators with a multi-step manipulation task including the pressing of buttons, peg-in-hole with the picking and inserting of a Multimeter Probe Plug, and a novel VF task of opening of a hinged door. We present data from a pilot user study with eight teleoperators and 67 trial attempts with two scenarios (with and without VFs enabled) across three test conditions of round-trip network delays of 0 ms, 100 ms, and 250 ms. We found teleoperators had an increased task success rate, lowered the total travel distance, and overall reduced task execution time, in the best case of 0 ms delay by 26 seconds, or 21%, when VFs were enabled versus when they were not. Performance was maintained or improved at higher network delays.