BX2S-Net: Learning to reconstruct 3D spinal structures from bi-planar X-ray images

Grasping good understanding of the weight-bearing spatial structure of the spine of a human subject in a standing position is critical for the treatment of spinal disorders. Such disorders are commonly diagnosed via 2D X-ray imaging of the human subject in a standing position. However, 3D reconstruction techniques based on bi-planar X-ray imaging can enable better exploration and analysis of the spinal structure. In particular, compared to earlier deformable modeling approaches, the recently-developed deep-learning-based 3D reconstruction methods exhibit higher efficiency and generalizability. But these methods usually employ simple architectures with 2D encoders and 3D decoders. Thus, these methods have several drawbacks, namely, the existence of a semantic gap between dimensionally-inconsistent feature maps, the difficulty of jointly handling multi-view inputs, and the information source limitations for the decoding process. In order to better assist clinicians and tackle these problems, we propose a novel convolutional neural network framework, which we call BX2S-Net, to effectively achieve 3D spine reconstruction based on bi-planar X-ray images. In particular, a dimensionally-consistent encoder-decoder architecture is designed in conjunction with a dimensionality enhancement method in order to reduce the semantic gap between feature maps and achieve information fusion for multi-view inputs. A feature-guided progressive decoding process is developed on the decoder side, where a full-scale feature attention guidance (FFAG) module is introduced to efficiently aggregate image features and guide the decoding process at each level. In addition, a class augmentation method and a spatially-weighted cross-entropy loss function are used for network training with improved reconstruction quality for the vertebral edge region. The experimental results demonstrate the effectiveness of our model in reconstructing high-quality 3D spinal structures from bi-planar X-ray images. The code is available at https://github.com/NBU-CVMI/bx2s-net.