Pathogenic bacteria pose serious threats to public health and environmental safety. Conventional colony counting, a standard method for bacterial detection, is time-consuming and unsuitable for rapid on-site detection. In this work, a flexible ACH/Ag substrate as a surface-enhanced Raman scattering (SERS) platform was constructed by loading silver nanoparticles (Ag NPs) onto a three-dimensional porous agarose/chitosan (ACH) base via interfacial self-assembly for bacteria detection. The 3D porous structure and enhanced electrostatic interactions boosted its bacterial capture. Meanwhile, the interfacial self-assembly of Ag NPs enabled a uniform distribution of high-density SERS hot spots. Owing to the integrated effect of improved bacterial affinity and dense SERS hot spots, this label-free SERS platform exhibited significantly enhanced sensitivity for bacterial detection. The detection limits reached 8.32, 50.6, 20.8, and 21.2 CFU/mL for Staphylococcus aureus (S. aureus), Escherichia coli (E. coli), Salmonellaenterica (S. enterica), and Pseudomonas aeruginosa (P. aeruginosa), respectively. Assisted by machine learning, the method enabled 100% classification accuracy and accurate concentration prediction of mixed bacterial samples. In real samples, including chicken, milk, and pond water, the detection recoveries ranged from 93% to 118%, and the bacteria could be rapidly enriched without long-term incubation, thereby significantly improving detection efficiency. To our knowledge, few studies have reported flexible substrates as a label-free SERS platform for the sensitive detection and discrimination of multiple pathogens. This work provides a strategy for the design of high-performance SERS substrates with promising practical applications.