Abstract Introduction Lymphopenia after radiotherapy has an adverse effect on the patient’s outcome. However, the relationship between radiotherapy dose delivery and lymphopenia is not fully understood. This work used image-based data mining to identify anatomical regions where the received dose is correlated with severe lymphopenia. Methods A total of 901 patients with lung cancer were analyzed. A Cox model was used to assess prognostic factors of overall survival (OS). Two matched groups were defined—patients with lymphopenia of grade 3 or higher and patients without lymphopenia of grade 3—based on tumor volume, baseline lymphocytes, and prescribed dose. Then, data mining was used to identify regions where dose correlates significantly with lymphopenia of grade 3 or higher. For this, dose matrices were aligned using registration of the computed tomography images to one reference patient. Mean dose distributions were obtained for the two groups, and organs of significance were detected. Dosimetric parameters from the identified organs that had the highest correlation with lymphocytes at nadir were selected. Multivariable analysis was conducted for lymphopenia of grade 3 or higher on the full lung cohort, and the model was tested on 305 patients with esophageal cancer. Results Adjusted Cox regression revealed that lymphopenia of grade 3 or higher is an independent factor of OS. The anatomical regions identified were the heart, lung, and thoracic vertebrae. Dosimetric parameters for lymphopenia included thoracic vertebrae V20, mean lung dose, and mean heart dose, which were further validated in the esophageal cancer cohort. Conclusions We report that severe lymphopenia during radiotherapy is a poor prognostic factor for OS in patients with lung cancer and could be mitigated by minimizing thoracic vertebrae V20, mean lung dose, and mean heart dose to limit the irradiation of stem cells and blood pool.