Multimodal Contrastive Learning for Zero-Shot HS Code Classification in Container Screening

Harmonized System (HS) Code classification is critical for securing maritime trade amid rising global trade volumes. Previous studies mainly address HS Code classification in the context of noisy textual declarations and the limited discriminative power of container scans. However, the classification of unseen or rare goods, which reflects the long-tail distribution of trade goods, remains understudied. This study addresses the problem by proposing a multimodal contrastive learning framework capable of zero-shot HS Code classification for both seen and unseen goods. A batch of multimodal data comprising gamma-ray container scans and HS Code description pairs is processed by a Vision Transformer (ViT) image encoder and a state-of-the-art text encoder, respectively. A projection head for each modality maps encoder outputs into a unified latent space using a contrastive learning objective, thereby maximizing the cosine similarity between positive pairs and minimizing it between negative pairs. Zero-shot classification is enabled by using semantic anchors through 5,675 prompt-engineered Six-Digit HS Code (HS6) descriptions. Evaluation on multi-class container screening data with 123,782 instances and 504 HS6 classes, curated from the Port of Tanjung Perak, Indonesia, demonstrates the proposed framework’s ability to bridge the gap between the visual density of gamma-ray container scans and the trade semantics of HS Code descriptions. Experimental results across various transformer-based architectures, including ViT with BGE-Large, MPNet, Gemma, and CLIP, show that the proposed framework maintains performance competitive with ResNet baselines, with ViT-BGE achieving Top-10 Accuracy of $93.16\pm 0.16$ and $23.99\pm 0.64$ on the seen and unseen classes, respectively.