Dual-stream transformer-based activity classification for off-site construction productivity analysis

This paper proposes a computer vision-based framework for automated productivity analysis in modular construction. A dual-stream transformer model classifies module installation activities based on on-site video data. The framework involves three main steps: object detection, activity classification, and productivity analysis. A convolutional neural network (CNN)-based object detector identifies key resources — cranes, modules, and workers — while the Transformer model analyzes the spatiotemporal changes in their movements. Six detailed installation activities are classified with high accuracy, achieving F1-scores above 0.98 across all classes. In contrast to previous rule-based or static image-based approaches, the proposed model captures the continuous and dynamic nature of operations, including transitions between activities. Productivity is evaluated by aggregating activity durations, which subsequently helps identify process bottlenecks. Post hoc video analysis further reveals the causes of delays. The proposed method supports real-time and data-driven monitoring, offering practical insights for improving operational efficiency. This framework provides a basis for future applications such as productivity forecasting and cross-site benchmarking. • Dual-stream Transformer-based activity classification model is developed. • Model accurately classifies module installation tasks (F1-score ⩾ 0.98). • Object detection and Transformer integrated for real-time productivity analysis. • Framework identifies operational delays and productivity bottlenecks. • Method validated across diverse modular construction site conditions.