A Dual-View Fusion Network for Automatic Spinal Keypoint Detection in Biplane X-ray Images

Accurate keypoint detection in medical images of the spine is critical for the assessment, diagnosis, treatment planning, and clinical investigation of spinal deformities. However, due to severe occlusions of spinal structures in lateral X-ray images, accurate keypoint detection can be hardly achieved in lateral X-ray images based on single-view information. Thus, methods based on both the anterior-posterior (AP) and lateral (LAT) X-ray image views have been proposed to alleviate occlusion problems and achieve better keypoint detection performance. Although some progress has been made with these dual-view methods, they do not effectively exploit a priori knowledge of the spine and hence cannot adequately account for the structural correlation of the vertebrae across views. In this paper, a new dual-view fusion network (DVFNet) framework is proposed for keypoint detection in spinal X-ray images. This framework obtains structural correlations between AP and LAT views of the spine based on a priori spine knowledge represented by high-level semantic features. Meanwhile, the proposed framework combines local and global features extracted respectively by a local subnetwork and a global subnetwork. On the one hand, the local subnetwork is constructed as an enhanced codec structure based on both the AP and LAT views. This subnetwork is trained to output local features that contain both joint semantic features of the two views and independent fine-grained features of each individual view. This scheme leads to accurate keypoint estimation locally. On the other hand, the global subnetwork utilizes a self-attention mechanism to extract view-specific global features based on either the AP view or the LAT view in order to eliminate ambiguity, and reduce confusion on keypoint locations. Further, we propose a weighted feature fusion (WFF) module for adaptive fusion of the local and global features. We evaluated the DVFNet model on a private dataset and found that our proposed method achieves more accurate spinal keypoint detection compared to other state-of-the-art methods, and thus our method can provide reliable assistance to clinicians.