Research on vision-sensor-language multimodal model for knee joint risk early warning in squat training

As a core movement for lower limb strength training, improper execution of squats is prone to sports injuries such as knee ligament strain and meniscus wear, and real-time and accurate risk early warning is of great significance for sports safety. Aiming at the problems of one-sided features, insufficient robustness, and semantic disconnection in knee joint risk identification with single-modal data (vision, sensor, or language), this paper proposes a vision-sensor-language multimodal fusion model to realize real-time early warning and grade classification of knee joint risks during squats. Firstly, squat motion images are collected by visual cameras, and human skeletal key points are extracted by an improved YOLOv8 model to capture kinematic features such as knee joint angle and displacement. Secondly, inertial measurement units (IMU) and pressure sensors are used to collect dynamic features such as knee joint force and acceleration, and noise interference is eliminated by an adaptive filtering algorithm. Then, a domain-specific language model is constructed to parse text information such as squat motion specification descriptions and risk types, extract semantic features, and realize cross-modal alignment with visual and sensor features. Finally, a multimodal feature fusion module is designed, which strengthens the weight of key risk features combined with the attention mechanism, and completes the classification and early warning of risk levels (safe, mild risk, moderate risk, high risk) through a deep neural network. Experiments selected 120 subjects (covering different training levels) for squat tests, collected multimodal data to build a dataset, and compared the performance of the single-modal model with the proposed multimodal model in this paper. The results show that the risk early warning accuracy of the proposed model reaches 96.8%, the recall rate is 95.7%, and the F1-score is 96.2%, which are significantly better than those of the single-modal model (the accuracy is improved by 10.3%-18.6%), and the early warning delay is controlled within 50 ms, meeting the real-time requirements. The research results can provide technical support for the intelligent risk prevention and control of squat training, and also provide new ideas and methods for the application of multimodal fusion in the field of sports injury early warning.