A hand-eye calibration method for augmented reality applied to computer-assisted orthopedic surgery

Recent technology advances in Augmented Reality (AR) is creating an opportunity for a paradigm shift in the field of Computer Assisted Surgery (CAS). Despite having the potential of being a promising technology in clinical applications, its accuracy, robustness, and performances under specific environmental conditions have not been fully investigated and evaluated. In an attempt to address the limitations and uncertainties that arise from using computer vision, we propose a Hand-Eye (HE) based approach to determine an invariant mapping between the HoloLens headset (i.e., rigid body defined by a unique set of passive markers) and the OHMDs virtual camera. In order to achieve the required precision for using the Microsoft HoloLens in a surgical application, an invariant mapping was estimated. First, a set of non-collinear retroreflective markers were rigidly attached to the Microsoft HoloLens headset. Consequently, the OHMDs's intrinsic parameters, as well as their radial and tangential lenses distortion coefficients, were determined by solving simultaneously a set of homogeneous system of equations by using nonlinear optimization techniques. The accuracy of the proposed HE based method has been evaluated by using a retro-projection error estimation method. Initially, a surgical frame has been anchored to the organ of interest. After loading the patient-specific dataset, a hologram of the organ of interest appears in front of the user's initial position. Correspondences between the hologram and the fiducial markers can then be established by using the tracked surgical pointer and then be mapped with respect to the surgical frame. After finishing this process, an orthonormal basis is created in the physical space, which has its counterpart in the image space and, therefore, a transformation can be determined to map the hologram with respect to the surgical frame. OHMDs may be able to create an opportunity for a paradigm shift in the field of CAS, since all this detailed information could easily be displayed on virtual and interactive screens in operating rooms on the same time and in real time. A very important aspect of using such OHMDs is that it can be very intuitive, especially among inexperienced surgeon, since surgical instruments and maneuvers motions are represented with respect to the surgeon's visual system and, therefore, attention disruptions are more likely to be reduced in terms of the surgeon's gaze, considering that the surgeon would not need to be searching for a conventional display located at static location in the operating room.