Gaze-It: Intelligent Multimodal AR Cross-Reality Control System

AR HMDs are considered auspicious devices and may substitute mobile phones as the next generation of intelligent interactive devices. Nowadays, the Internet of Things (IoT) has entered our lives. However, in smart home, the common control method now is to use apps on the smartphone. Hence, in this work, we design an AR HMD based control system for cross-reality intelligent IoT devices. We focus on the real-life living room scene, and try to compactly integrate the remote control on many smart electric appliance (such as desk lamps, air conditioners, and water bottles) via AR HMD and natural multi-modal interaction including gazing, gesture, voice and typing. This AR HMD based remote controller is called Gaze-It. It enables to automatically recognize the user's current state and adaptively receive multimodal input. Gaze-It can coexist with the current smart home ecosystem dominated by voice control, bring benefits to human intelligent life concurrently, and contribute to the development and popularization of AR HMDs.