From Observation to Concept: A Flexible Multi-View Paradigm for Medical Report Generation

Automated radiology report generation aims to generate accurate and radiologist-like descriptions for the patient's images, which can greatly relieve the workload of radiologists. However, due to the data bias and long report problems, medical report generation has been a challenging task. In this article, we propose a F lexible M ulti- v iew P aradigm (FMVP) for medical report generation in a novel observation-to-concept manner. It first makes some medical observations automatically or with the help of a radiologist on the patient's image to obtain patient-related priori knowledge, just as radiologists do in practice. Furthermore, to bridge the gap between pretrain and generation phases, the hierarchical alignment is proposed to jointly conduct the implicit alignment between region-tag and the explicit global alignment of the image-report pair. Finally, a compatible decoder towards decoding the fused multi-view knowledge is proposed to capture more complementary information for the report generation, which breaks the traditional entrenched decoding mechanism guided by visual information. Extensive quantitative and qualitative experiments on the public MIMIC-CXR and IU-Xray datasets show that our model achieves competitive performance compared to state-of-the-art methods.