Cross-Modal Audio-Text Retrieval with Soft Alignment

Audio-text retrieval is a critical task in multimodal learning. However, aligning heterogeneous modalities remains challenging due to inherent disparities between audio and text features. Traditional methods typically rely on hard alignment, imposing a strict one-to-one correspondence between modality-specific features. However, such methods often distort modality-specific structures and struggle to bridge semantic gaps effectively. To address these limitations, we propose SoftAlign-GAN, a novel framework that integrates multimodal generative adversarial networks with metric learning to achieve softened alignment between the modality embedding spaces. This approach combines adversarial training with contrastive loss functions, leveraging pre-trained audio encoders to enhance cross-modal learning. By aligning modality spaces through adversarial training, our method facilitates soft alignment, mitigating inherent modality differences while preserving the structural integrity of each modality's representation. Moreover, unlike other soft alignment methods, our approach is unsupervised and does not require labeled data. We evaluated SoftAlign-GAN on the AudioCaps and Clotho datasets. The results show that SoftAlign-GAN with NT-Xent loss outperforms baseline models in terms of R@1, R@5, and R@10 metrics, achieving significant improvements in modality alignment and recall rates, thus advancing audio-text retrieval. Visualizations confirmed that the model effectively reduces inter-modal gaps while preserving modality-specific characteristics. Fine-tuning experiments further demonstrated the framework’s adaptability, showing marked improvements on the WavCaps dataset.