On the Role of Hard Negatives in Cross-Modal Contrastive Representation Learning Algorithms

Cross-modal retrieval (CMR) has long grappled with a challenging trade-off between performance and efficiency. Mainstream approaches typically depend on computationally intensive dual-encoder architectures, such as ResNet-50 and BERT, which lead to high inference latency and significant deployment costs. To tackle this issue, the present paper introduces a lightweight cross-modal retrieval framework grounded in Hard Negative Sample Mining (HNM). This framework utilizes MobileNetV3 and DistilBERT as image and text encoders, respectively, thereby preserving robust representational capabilities while markedly reducing computational complexity. Specifically, the HNM contrastive loss dynamically identifies and emphasizes the most hard negative samples within each batch. This mechanism allows the model to concentrate on optimizing decision boundaries within a shared semantic space, effectively compensating for the limited discriminative power of lightweight encoders. Experimental results indicate that this framework achieves approximately 46.7 % reduction in parameters and 17.9 % decrease in inference latency on the Flickr30k dataset while surpassing traditional heavyweight baseline models across multiple Recall@K metrics. This work illustrates that synergistic optimization of lightweight architectures alongside sophisticated negative sample mining techniques facilitates efficient and accurate cross-modal retrieval under resource-constrained conditions, offering an effective pathway for future research aimed at balancing performance with deployability.