Introduction: Cerebral small vessel disease (CSVD) is a significant burden of morbidity and mortality among elderly people around the world. Epidemiological data with complete CSVD evaluations and a large sample size in the general population are still limited. Methods: Community-dwelling residents in Lishui city in China from the cross-sectional survey of the Polyvascular Evaluation for Cognitive Impairment and Vascular Events (PRECISE) study were included in this study from 2017 to 2019. All participants underwent 3 Tesla brain magnetic resonance images to assess CSVD imaging markers. Demographic and risk factor data were collected. The general and age-specific prevalence of lacune, confluent white matter hyperintensity (WMH), moderate-severe enlarged perivascular spaces (EPVS), cerebral microbleed (CMB), and total CSVD score (an ordinal scale from 0 to 4, counting the presence of four imaging markers of CSVD) was evaluated. Associations between vascular risk factors and these markers were analyzed by multivariable logistic regression. Results: A total of 3,063 participants were enrolled. The mean age was 61.2 years and 46.5% were men. The most prevalent CSVD marker was confluent WMH (16.7%), followed by CMB (10.2%), moderate-severe EPVS in the basal ganglia (BG-EPVS) (9.8%), and lacune (5.6%). 30.5% of the participants have at least one of the four markers (total CSVD score ≥1 points). The prevalence of CSVD markers increases as age increases. Age and hypertension were independent risk factors for four CSVD markers and the total CSVD score. Conclusions: In this Chinese cohort with community-based adults aged 50–75 years, our findings showed a prevalence of 30.5% for CSVD. The most prevalent CSVD marker was confluent WMH, followed by CMB, moderate-severe BG-EPVS, and lacune. The risk factors for CSVD must be strictly screened and controlled in adults living in the community.