Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

Metric learning projects samples into an embedded space, where similarities and dissimilarities are quantified based on their learned representations. However, existing methods often rely on label-guided representation learning, where representations of different modalities, such as audio and visual data, are aligned based on annotated labels. This approach tends to underutilize latent complex features and potential relationships inherent in the distributions of audio and visual data that are not directly tied to the labels, resulting in suboptimal performance in audio-visual embedding learning. To address this issue, we propose a novel architecture that integrates cross-modal triplet loss with progressive self-distillation. Our method enhances representation learning by leveraging inherent distributions and dynamically refining soft audio-visual alignments—probabilistic aligns between audio and visual data that capture the inherent relationships beyond explicit labels. Specifically, the model distills audio-visual distribution-based knowledge from annotated labels in a subset of each batch. This self-distilled knowledge is used to automatically generate soft-alignment labels for the remaining audio-visual samples. These soft-alignment labels are used to construct soft cross-modal triplets, which in turn are employed to fine-tune the model’s parameters. Experimental results on two audio-visual benchmark datasets demonstrate the effectiveness of our proposed method in the cross-modal retrieval task, achieving state-of-the-art performance with improvements of 2.13% and 1.82% on the AVE and VEGAS datasets, respectively, in terms of average MAP metrics.