Abstract The current YOLO model faces the following challenges in cup defect detection tasks: (1) traditional convolutional downsampling leads to the loss of small defect features; (2) the standard IoU loss is not sensitive enough to the positions of small targets; (3) the accuracy of detecting tiny defects is low; and (4) certain types of defects are difficult to obtain. This paper proposes a cascade online detection method for cup defects based on an improved YOLOv11 (CS-YOLO). This approach provides a new technological paradigm for the precise detection of small and rare defects in industrial quality inspection settings, significantly enhancing the level of intelligence in the production line of disposable cups. The specific innovations include: (1) the creation and publication of a typical defect dataset from the paper cup production process, serving as the foundation for this research and providing a reference for subsequent studies on paper cup defect detection; (2) replacing the down-sampling convolution operation in the original model with ADown convolution, which reduces the loss of fine-grained information and improves the model’s efficiency and performance; (3) using the NWD-CIOU loss function to address the sensitivity of IOU to positional deviations in small targets; (4) adding a detection head for tiny objects to further enhance the detection effectiveness of small target defects; (5) introducing Slicing Aided Hyper Inference to improve detection accuracy during the inference phase and reduce missed detections; (6) to tackle the issue of the scarcity of rim curl defect samples in the dataset, this study adopts a detection method based on adaptive curvature analysis to address the shortcomings of deep learning detection, thereby further enhancing the comprehensiveness and robustness of the detection process. Experimental results show that the improved CS-YOLO model performs exceptionally well in paper cup defect detection tasks: mAP50 reaches 88.9%, and mAP50–95 reaches 70.9%, representing improvements of 4.0% and 4.3% over the original model, respectively. Under the improved YOLOv11 combined with the SAHI framework, the model effectively reduces missed and false detections, significantly enhancing its ability to accurately identify small-sized and less distinguishable defects.