Purpose To identify primary sociodemographic factors contributing to cancer health literacy disparities in cancer and non-cancer samples. Design Cross-sectional study. Setting Cancer and primary care clinics, healthcare events, community centers, and churches in the Mid-Atlantic region. Subjects We recruited U.S. adults with confirmed cancer diagnosis (n = 1306) and adults without cancer (n = 512). Measures The Cancer Health Literacy Test–30 and a demographic questionnaire were administered using touchscreen tablets. Analysis One-factor model and measurement invariance were tested. Structural equation modeling was used in univariate and multivariable prediction models. Results One-factor and scalar invariance models were uniformly supported. Race/ethnicity, education, and income were the strongest predictors of cancer health literacy. Age effect was non-significant in univariate analysis, but it was significant with a small effect size in multivariable analysis. A significant but small marital status effect in univariate analysis became non-significant after adjustment, similar to sex effect in non-cancer sample. A significant but small sex effect in cancer sample became non-significant after adjustment. Multivariable models explained 57%-59% variability in cancer health literacy. Conclusions Race/ethnicity, educational attainment, and income are the primary drivers of disparities in cancer health literacy. Health promotion activities should target cancer health literacy in subpopulations self-identified as Non-Hispanic Black, low income, and low education, and adopt practical health promotion strategies to improve the health outcomes and wellbeing targeting these patients.