Semantic-Driven Global-Local Cooperative Contrastive Learning for Medical Report Generation

As the demand for radiology continues to increase, the shortage of specialized professionals becomes a challenging issue. Such shortage highlights the need to utilize advancements in artificial intelligence (AI) to automatically generate draft medical reports from radiology images. Recently, the application of contrastive learning has been leveraged in image-to-text generation because it allows the model to learn meaningful representations in latent space by contrasting similar and dissimilar image-text pairs. However, existing approaches to applying contrastive learning in medical report generation are limited by the following: 1) they are performed as an independent pretraining step, which hinders the cooperation between contrastive learning and the subsequent report generation step; 2) these methods are contingent on pairing images with their corresponding reports, thus establishing similarity based solely on this association. Such contingency inadvertently overlooks the situation where unpaired reports could also be relevant to a given image, thereby failing to accurately capture and understand the semantic relationships within the data; and 3) existing contrastive learning in medical report generation only utilizes the global representation, which cannot capture subtle but crucial local visual information. To address these limitations, we propose a Semantic-Driven Global-Local Cooperative Contrastive Learning Network (SGLCCNet), which integrates contrastive learning into the training process of report generation, enriched with semantic information extracted from reports and enhanced by the inclusion of local feature exploration. Extensive experiments on the IU-Xray dataset demonstrate that our method achieved the state-of-the-art. Further, we demonstrate how each of our proposed steps adds to the overall performance.