Real-time shipping container code spotting plays a vital role in enabling intelligent inspection to optimize smart port logistics. Recent methods suffer from error accumulation and have complex pipelines, resulting in high computational complexity that limits the generalization ability for multi-directional, including horizontal, vertical, and multi-line (HVM) codes. Moreover, fixing the parameters of visual sensors (e.g., angle, position, and focal length) to avoid noise interference limits deployment on mobile devices. To address them, we propose an end-to-end framework for HVM code spotting. First, an inter-domain feature fusion network that includes the Self-Spatial Enhancement Module (SSEM) and Self-Channel Enhancement Module (SCEM) is proposed to reduce noise interference and improve the efficiency of the lightweight model. Next, a transformer-based branch with Character Contrastive Learning (CCL) loss is designed to enhance the representation of character features. Experimental results indicate that the proposed method achieves state-of-the-art performance, in which the F-measure and recognition accuracy reach 91.2% and 93.2%, respectively, in real-time.