Adversarial Training with Comprehensive Objective for Medical Image Report Generation

Medical Image Report Generation aims to automatically generate medical reports based on radiology images, thus freeing radiologists from the tedious task of writing reports. Generating report texts that match the content of a given medical image, focus on local anomalies, and fluently conform to professional reporting norms presents the challenge of linking visual patterns to informative human-language descriptions. In this paper, we propose a novel generative adversarial framework (MIRGAN) to guide the generator to generate medical reports that are indistinguishable from those written by professional radiologists. MIRGAN introduces a multimodal discriminator to evaluate the performance of the report generator on the comprehensive objective containing three sub-objectives when generating each token. MIRGAN then uses it as the reward in reinforcement learning to guide the generator to optimize toward the desired objective in real time. We conduct sufficient experiments to evaluate MIRGAN on two widely used datasets for chest-related diseases. The experimental results show that MIRGAN can significantly improve the performance of the generator on most metrics.