Automatic curtain wall frame detection based on deep learning and cross-modal feature fusion

The curtain wall construction industry is one of the most popular industries with excellent development prospects. On the other hand, curtain wall installation is mainly performed manually, which has the disadvantages of great danger and low efficiency. Therefore, this study designed a method for curtain wall frame detection based on computer vision to assist curtain wall installation in completing positioning and installation tasks. This paper presents a deep learning method with two input streams and cross-modal feature fusion based on the encoder-decoder structure (CWFD-net) to detect curtain wall frames accurately. In particular, the high-level semantic features of the RGB and Depth streams in the encoder stage are fused to generate RGB-D features to achieve preliminary cross-modal feature fusion, which makes input information include more curtain wall frame features. The coordinate attention mechanism enables the network to focus more on the position information of the curtain wall frame. A cross-stage feature fusion strategy was adopted in the decoder stage to enhance the features further and suppress interference factors. A dataset containing curtain wall frame images of different styles in various curtain wall construction scenarios was established to verify the effectiveness of this method, which is trained, validated, and tested with this dataset. The experimental results show that the detection performance of the proposed method is superior to the commonly used segmentation or detection methods, which achieves the highest mIoU 87.33%, Accuracy 96.98%, Recall 92.28%, F1-Score 87.66%, and the lowest 95-HD 6.13. This model is expected to be deployed and applied to curtain wall installation robots.