The accurate prediction of subway passenger flow is crucial for managing urban transportation systems. This research introduces a hybrid forecasting approach that combines an enhanced TimesNet model, Seasonal Autoregressive Integrated Moving Average (SARIMA), and Variational Mode Decomposition (VMD) to improve passenger flow prediction. The method decomposes time series data into Intrinsic Mode Functions (IMFs) using VMD, followed by adaptive predictions for each IMF with TimesNet and SARIMA. The dataset spans from 1 January to 25 January 2019, encompassing 70 million records processed into five-minute intervals. The results show that the VMD preprocessing effectively extracts features, enhancing prediction performance (13.25% MAE, 19.7% RMSE improvements). The hybrid method excels during peak times (52.75% MAE, 50.61% RMSE improvements) and outperforms baseline models like Informer and Crossformer, achieving 66.14% and 63.24% improvements in the MAE and RMSE, respectively. This research offers a reliable tool for predicting subway passenger flow, supporting the smart evolution of urban transport systems.