The skylight process represents a pivotal element of railway maintenance work; however, it encompasses substantial safety risks. The prompt identification of trespassing behaviors during the skylight process is crucial for maintaining railway stability and ensuring worker safety. Existing trespassing detection research is mainly based on railway operation scenes, with no prior studies focusing on trespassing detection in the context of the skylight process in railway maintenance. Addressing these challenges, this study proposes an algorithm for detecting trespassing by railway workers during the skylight process, utilizing the You Only Look Once instance segmentation model trained on the manually annotated data set with augmentation. This trespassing detection algorithm includes innovative elements, such as the horizontal line detection method, customized width ratio, and a customized rule for identifying the railway under maintenance. The proposed algorithm is designed to aid safety personnel at skylight process sites by providing enhanced supervision and effectively detecting instances of worker trespassing. With a detection precision exceeding 97% in the experimental results, this algorithm proved its efficiency in the railway skylight process, underscoring its extensive potential for diverse trespassing detection applications.