A Semi-Supervised Learning Framework Combining CNN and Multiscale Transformer for Traffic Sign Detection and Recognition

The accurate extraction of traffic signs is of great significance to the digitization of traffic information and the fine management of traffic. This article introduces an innovative approach to address the challenges associated with recognizing and detecting traffic signs, considering their vulnerability to complex backgrounds, variations in illumination, and motion blur. The proposed method utilizes a semi-supervised learning (SSL) strategy, combining convolutional neural networks (CNNs) with a transformer encoder–decoder architecture, to extract traffic sign features from vehicle panoramic images. To enhance feature extraction, a hierarchical sampling method (HSM) is introduced, which facilitates the extraction of multiscale self-attention features in the transformer encoder–decoder structure. Additionally, a network module called local and global information aggregator (LGIA) is designed based on HSM, enabling the incorporation of both local and global context information. Furthermore, a SSL strategy is adopted to simultaneously train our model using both labeled and unlabeled data samples. This strategy aims to improve the extraction of traffic signs by capitalizing on the broader data set available through unlabeled data. Experimental results demonstrate the effectiveness and robustness of the proposed method in improving the detection and recognition of traffic signs. The approach showcases significant improvements in overcoming the challenges posed by complex backgrounds, variations in illumination, and motion blur. Our approach achieved a 0.9% improvement in the F1-score evaluation over the current classical object detection algorithm on the public data set Tsinghua-Tencent 100K and a 1.1% improvement on the SSW data set.