Clinically Coherent Radiology Report Generation with Imbalanced Chest X-rays

Automatically generating radiology reports given radiographs has considerable promise for easing clinical workflows, reducing diagnostic errors, and ultimately streamlining clinical care. Therefore, this work has attracted much attention and early studies mainly employed cutting-edge techniques from computer vision and natural language generation to improve the readability of generated reports. However, those methods often fail to accurately convey critical information such as the presence status of disease, which is the first priority of medical image-to-text generation. Additionally, the class imbalance issues are frequently found in a number of radiographic datasets, resulting in missed diagnosis of some uncommon diseases. In this manuscript, we propose a clinically coherent radiology report generation method that introduces an additional memory-enhanced feature-wise affine transformation (MFAT) layer and an imbalance-aware clinically accurate reward (ICAR) to tackle the aforementioned issues. We evaluate the proposed method by comparing with state-of-the-art methods over both natural language generation (NLG) metrics and clinical efficacy results on two datasets, namely IU X-Ray and MIMIC-CXR. The results demonstrate that our method achieves significantly higher clinical efficacy accuracy for both common or uncommon disease annotations while still maintaining acceptably high NLG metrics for readability.