Abstract Objective . Magnetic particle imaging (MPI) is an emerging imaging technique based on superparamagnetic iron oxide nanoparticles, offering high sensitivity and rapid imaging. However, in measurement-based MPI, image quality is degraded by noise arising during both the system matrix (SM) calibration procedure and the signal acquisition process. This study aims to develop a deep learning-based model for efficient noise suppression to enhance MPI image quality. Approach . We propose a hybrid encoder–decoder network integrating residual blocks (Res-Blocks) and swin transformer modules. The model employs a multi-scale feature extraction strategy to disentangle noise from valid signals, coupled with cross-level feature fusion to optimize frequency-domain recovery. Main results . Model performance was evaluated on simulated dataset, OpenMPI dataset, and dataset acquired from in-house MPI systems. The denoised SM achieved an average 12 dB improvement in signal-to-noise ratio (SNR). Reconstructed images showed better visual quality, with a peak SNR of 29.11 dB and a structural similarity index of 0.93, which outperformed the compared approaches. Significance . This work provides a robust solution for noise suppression in SM to enhance MPI image quality. The noise suppression framework is extensible to other SM-based medical imaging modalities.