Virtual multiplex immunofluorescence identifies lymphocyte subsets predictive of response to neoadjuvant therapy

Background: Hematoxylin and eosin (H&E) staining is routine in pathology but lacks cellular specificity. Multiplex immunofluorescence (mIF) captures spatial immune relationships in tumors, but cost and complexity limit clinical application. Novel approaches to yield similar information from readily available tumor histology are needed. Objectives: Develop and validate a novel deep learning tool capable of translating standard H&E-stained histopathology images into high-fidelity synthetic mIF images that preserve immune cell information predictive of treatment response in breast cancer. Design: Comparative model evaluation and predictive modeling in a retrospective breast cancer cohort. Methods: Core-needle biopsies from 17 triple-negative breast cancer cases underwent mIF imaging. Hematoxylin and eosin and mIF images for DAPI (nuclei), pan-CK (tumor), CD3/CD4/CD8 (T-cells), and CD20 (B cells) were aligned. A pipeline outperforming standard Pix2Pix and CycleGAN image translation networks was developed, “multiplex Synthetic Immunofluoresence Generated through H&E Translation” (mSIGHT), which integrates a registration network to overcome misalignment between the input and target images. Generated images were evaluated with pixel-level metrics and biological metrics, including cell density and cell-to-cell adjacency. The pipeline was then applied to an external cohort to assess associations between predicted immune features and pathologic response to neoadjuvant chemotherapy. Results: Generated images preserved immune cell distributions and proximity metrics correlated to the ground truth cell counts. In a cohort of 218 breast cancer cases treated with neoadjuvant chemotherapy, predicted density of CD8+ T cells was significantly associated with complete response (adjusted odds ratio 1.89, 95% confidence interval 1.23–2.80, p = 0.002), independent of receptor status, grade, and pathologist TIL annotations. Conclusion: The mSIGHT pipeline enables translation of routine H&E slides into virtual mIF images with interpretable immune biomarkers, offering a scalable and affordable alternative to multiplex imaging. It also identifies immune features predictive of therapeutic response and has the potential to assist in the personalization of neoadjuvant therapy.