Real Time Object Detection in Digital Twin with Point-Cloud Perception for a Robotic Manufacturing Station

The present work aims to develop a digital twin system for a small-scale robot workstation for intelligent manufacturing, based on ROS and Unity 3D. Such digital twin system can be used to remotely visualize, monitor and control the manufacturing process, which is of great significance in the development of industrial automation and intelligent manufacturing. In the present work, the system is preliminarily developed for a pick-and-place task. To extend this framework enabling it to be more intelligent, we have considered integrating, in our framework, the 3D vision perception system with deep learning based vision algorithms, especially for perception of complex objects. The purpose is primarily for real time monitoring of dynamic manufacturing processes such as detecting moving workpiece, 3D formation of complex workpieces in 3D printing, etc., the data of which cannot be obtained from controllers of manufacturing stations. The 3D vision system and the developed algorithm are based on point cloud perception.