Purpose: Left-hemisphere stroke survivors who score above the Western Aphasia Battery–Revised (WAB-R) cutoff, but have self-reported aphasia, have been called not aphasic by WAB-R (NABW). Our prior work found letter fluency tasks to be sensitive to individual cases of very mild aphasia in the NABW group. Letter fluency tasks rely on both lexical retrieval and executive functioning (EF), which leaves open the question of which domain(s) are driving the impaired letter fluency in the NABW group. Method: Twenty-seven NABW and 63 healthy controls completed two-letter fluency tasks, timed picture naming, and an EF task called Antelopes and Cantaloupes (A&C). A&C is a test of switching control, semantic control, and phonological control. We used two-tailed Mann–Whitney U tests to compare groups. We calculated correlations between each measure and the letter fluency tasks. Receiver operating characteristic curve analysis was used to determine if any of these measures outperformed letter fluency tasks in identifying cases of very mild aphasia in the NABW group. Results: Six measures had group differences and were correlated with the letter fluency tasks. The measures included naming reaction times (RTs), A&C measures that required rapid lexical retrieval, and an A&C measure of switching control involving lexical retrieval without phonological or semantic conflict. None of the measures had higher sensitivity and specificity for individual cases of very mild aphasia compared to letter fluency tasks. Conclusion: Our findings suggest that deficits in rapid lexical retrieval and switching with lexical retrieval underlie impaired letter fluency in the NABW group.