AR-assisted assembly method based on instance segmentation

AR-assisted assembly refers to the overlaying of virtual models, annotations, and other AR instructions in a genuine scene to help workers perform assembly tasks. However, most AR-aided assembly processes lack scene awareness and require frequent interaction to complete the assembly guidance process. To achieve intelligent AR-assisted assembly, this paper firstly uses an instance segmentation method based on depth learning to process the RGB-D data of the assembly scene, segment the instance of the assembly object, and segment the corresponding depth information according to the instance mask to reconstruct the point cloud instance of the assembly object. Next, the Iterative Closest Point (ICP) algorithm is employed to register all recognized assembly objects to the 3D model of the assembly, allowing for pose estimation and assembly status perception of the objects in the scene. Based on this, the current assembly step can be determined, and AR instructions can be automatically triggered to reduce user interaction burden. Finally, the proposed AR-assisted system was evaluated through quantitative and qualitative experiments, and the experimental results showed that the proposed method effectively improved assembly efficiency and reduced the occurrence of assembly errors.