The mechanism of cognitive impairment in patients with cerebral small vessel disease (CSVD) remains unknown. The glymphatic system dysfunction, which has been demonstrated to influence cognitive impairment, can be evaluated by diffusion tensor image analysis along the perivascular space (ALPS index). We explored whether cognitive impairment in CSVD is associated with glymphatic clearance dysfunction. In this study, 133 patients with CSVD were enrolled and underwent neuropsychological test batteries as well as magnetic resonance imaging (MRI). They were then categorized into a CSVD with cognitive impairment (CSVD-CI) group and a cognitively normal CSVD (CSVD-CN) group. The ALPS index and four CSVD markers [white matter lesions (WMLs), cerebral microbleeds (CMBs), lacunes, and perivascular spaces (PVSs)] were also assessed. Univariate analysis showed that the ALPS index was significantly different between the CSVD-CN (n = 50) and CSVD-CI groups (n = 83) (p < 0.001). This difference remained significant (95% CI < 0.001-0.133) after adjusting for six common risk factors (age, education, hypertension, diabetes, smoking, and alcohol abuse) as well as CSVD markers. The ALPS index was independently linearly correlated with global cognitive function, executive function, attention function, and memory after adjusting for the aforementioned six risk factors or CSVD markers. Our results suggest that glymphatic system impairment is independently related to cognitive impairment in patients with CSVD.