We proposed a deep learning-based net-breaking detection and recognition system for net-breaking detection and recognition of underwater unmanned vehicles. The system was developed to assist aquaculture farmers in checking the condition of offshore fish cages. Using a Remotely Operated underwater Vehicle (ROV), underwater images were captured to identify whether the fish cage was damaged or not. The system assisted in reducing fish losses and minimizing the cost and risk of employing divers. The ROV used a single RGB camera combined with deep learning for detection. The deep learning algorithm used in the system was You Only Look Once (YOLO), which allowed the machine to learn a large number of features of various categories first and then use the resulting model to detect net damage in the fish cages.