Diegetic Graphical User Interfaces for Robot Control via Eye-gaze

Eye-gaze stands out as an intuitive interface for hands-free control of robotic devices due to its brief training time, fast calibration, low invasiveness, and reduced complexity and cost. However, current approaches are limited by available screen space, excessive wait times, frequent context switching, inconsistent gaze tracker accuracy, and the trade-off between feature-richness and usability. This article presents Diegetic Graphical User Interfaces, a novel, intuitive, and computationally inexpensive approach for gaze-controlled interfaces applied to a robotic arm for precision pick-and-place tasks. By using customizable symbols paired with fiducial markers, interactive buttons are defined and embedded into the robot, which users can trigger via gaze. Twenty-one participants completed the Yale-CMU-Berkeley (YCB) Block Pick and Place Protocol, reporting good usability and user experience, while achieving comparable workload to similar systems. The resulting system is fast to learn, does not restrain the user’s head, and mitigates context switching, while demonstrating intuitive control continuous Cartesian control of a robot arm in precision tasks.