Multi-modal interaction for robotics mules

To lighten the load of dismounted infantry, research is being conducted into the use of "robotic mules" that can help carry equipment and supplies for small units as they conduct their operations. While autonomy and perception features of robotic systems are steadily improving, the way in which people interact with them is still fairly rudimentary. The operator control units (OCUs) for these robotic mules are typically hand-held gaming controllers for tele-operation, or worn ruggedized portable computers or tablets with point-and-click interfaces. These control devices add more weight to the operator, and often require them to hold the controllers in their hands and to spend considerable time looking down at the OCU screen to enter commands or to understand what the robot is doing. Furthermore, these interfaces often require specialized training to understand how to operate the OCUs. This paper describes research aimed at reducing the physical, cognitive, and training burdens that robotic systems place on operators above and beyond the their regular jobs as warfighters. We first present an analysis of relevant infantry communication to identify interaction requirements, and an analysis of technologies that might be used to support these interactions. We then describe a prototype heads-up, hands-free system for controlling robotic mules using a lightweight, worn interaction device that facilitates natural twoway interaction (including speech and gesture input) between the robotic mule and the user. We describe the challenges in building this system and some formative evaluations of the technology