Background Despite the association between sleep disorders and mental impairments, controversies persist over the connection between sleep duration and cognitive impairment and the underlying mechanisms remain unexplored. Objective We investigated this controversial association in a Chinese population and explored the potential mediators. Methods Participants (aged 50–75 years) were recruited from a Chinese population-based cohort study, the PRECISE (PolyvasculaR Evaluation for Cognitive Impairment and vaScular Events) study. Sleep duration and Montreal Cognitive Assessment (MoCA) scores were collected during the baseline survey. Participants were divided by sleep duration into short sleep (<7 h), normal sleep (7–9 h), and long sleep (>9 h) groups. The association between sleep duration and MoCA scores was assessed by mediation analysis. Results Of the 3028 participants (mean age: 61.1 ± 6.7 years; females: 53.4%), 362 (12.0%) participants were enrolled in the short-sleep group, 2163 (71.4%) in the normal-sleep group, and 503 (16.6%) in the long-sleep group. Long sleep duration was negatively associated with MoCA scores (long versus normal: β = −0.82, 95% CI: −1.21 to −0.43, p < 0.001), after the adjustment for age, sex, education, body mass index, current smoking, and current drinking. The observed association was mediated by cardiometabolic factors (systolic blood pressure and fasting plasma glucose) and intracranial atherosclerotic plaque, with a cumulative mediating proportion of 12.4%. Conclusions A long sleep duration may be associated with cognitive impairment, which is partially mediated by blood pressure, fasting plasma glucose, and intracranial atherosclerosis. Therefore, individuals who sleep excessively should be monitored for abnormal cardiometabolic factors.