E-Gaze: Gaze Estimation With Event Camera

Near-eye gaze estimation is a task that maps the recording of an eye captured by an adjacent camera to the direction of a person's gaze in space. In contrast to frame-based cameras, event cameras are characterized by high sensing rates, low latency, sparse asynchronous data outputs, and high dynamic range, which are well suited for recording the fast eye movements. However, algorithms and system designs that operate on frame-based cameras are not applicable to event-based data, due to the natural differences in the data characteristics. In this work, we study the pattern of near-eye event-based data streams and extract eye features to estimate gaze. First, by analyzing eye parts and movements, and harnessing the polar, spatial, and temporal distribution of the events, we introduce a real-time pipeline to extract pupil features. Second, we present a recurrent neural network with a proposed coordinate-to-angle loss function to accurately estimate gaze from pupil feature sequence. We demonstrated that our system achieves accurate real-time estimation with angular accuracy of 0.46 ^° and update rates of 950 Hz, thus opening up avenues for novel applications. To our knowledge, this is the first system that operates only on event-based data to perform gaze estimation.