A Spatiotemporal Behavior & Metabolic lag Effects-Infused Multimodal Model for fish oxygen consumption prediction: FishBehavMet-TCN

Accurate and non-invasive estimation of fish oxygen consumption is crucial for precision aquaculture and efficient water environment management. However, achieving this remains a challenge due to the complex coupling between fish behavior, physiology, and water environments. In this study, a spatiotemporal behavior & metabolic lag effects-infused multimodal oxygen consumption prediction model (FishBehavMet-TCN) was proposed. Behavioral phenotypes were first extracted from video streams using DeepLabCut and then fused with key physiological and environmental parameters. The core of this model is a spatiotemporal dual-branch architecture built upon temporal convolutional network (TCN). The model combines temporal enhancement (TEM) and global attention mechanism (GAM) to capture interdependencies among behavioral, environmental, and physiological variables. Following this, an oxygen debt equation (ODE) constraint was introduced into the loss function, while an input injection block (IIB) integrated historical behavioral data to quantify metabolic lag effects. Results obtained on a self-built dataset demonstrated that FishBehavMet-TCN achieved R 2 values of 86.92%, 90.60%, 94.08%, and 94.90% at four window sizes, outperforming 18 benchmark models (best R 2 : 83.84%, 87.97%, 88.99%, and 91.39%). The model's applicability was further explored in multi-fish scenarios, providing a basis for future group-level studies. These results indicate that incorporating multi-scale behavioral features with metabolic lag effects enables non-invasive oxygen consumption prediction.