Enhancing Large Vision–Language Models for Multimodal Defect Detection via SFT–GRPO Reinforcement Learning

Large Vision Language Models (LVLMs) offer strong visual reasoning capabilities but their direct application to industrial defect inspection remains limited due to domain complexity, diverse defect modes, and the need for structured reporting. This paper presents a unified fine-tuning framework that combines Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO) to adapt open-source LVLMs for multimodal defect inspection in Liquefied Natural Gas (LNG) tank manufacturing. Using a balanced dataset derived from 188,631 inspection images, the proposed method enables each LVLM to perform joint defect localization, attribute prediction, and automatic generation of structured JSON inspection reports. The GRPO stage incorporates verifiable reward signals that enforce JSON validity, schema compliance, bounding box accuracy, and metadata consistency. Experimental results on four LVLM architectures demonstrate substantial performance gains, with mean Average Precision at IoU 0.5 improving from$35-39$percent to$84-89$percent and$F 1$scores for defect labels increasing from approximately 14 percent to above 84 percent. The best-performing model, Qwen2.5-VL-7B, achieves 88.77 percent mAP at IoU 0.5 and over 90 percent$F 1$in key metadata fields. These findings indicate that SFT and GRPO provide complementary benefits, enabling LVLMs to deliver accurate, interpretable, and computationally efficient inspection for next-generation manufacturing environments.