Where Am I Looking: Localizing Gaze In Reconstructed 3D Space

We propose a method to estimate the 3D gaze of the observer onto the scene using a portable eye tracker with a monocular camera. We reconstruct the 3D scene using Structure from Motion (SfM) and use camera pose and 3D reconstruction information of the scene to localize the position and pose of the observer in the 3D reconstructed space. Along with the position, we can obtain the 2D gaze of the observer using an eye-tracker. Each person may have a different perspective of the same 3D object in the scene, observing it from different positions. We use this information to fuse these multiple perspectives in 3D space to get a better understanding of how differently each observer perceives the same scene, compared to others. In the entire system, we developed a convo-lutional neural network to detect and track eye movement and a camera re-localization method to localize the camera in 3D environment. Based on our novel eye tracking and camera re-localization methods, we can accurately localize the gaze in the 3D reconstructed environment.