Validity and Reliability of Outcome Measurement Instruments for Cognitive Function in Alzheimer’s Disease: A Systematic Review Free

An aging global population has resulted in a rise in the incidence of neurological diseases. Of these diseases, Alzheimer’s disease (AD) is one of the most common dementias and has placed a considerable burden on patients and their families [1]. AD is a progressive neurodegenerative disease characterized by memory loss and cognitive impairment that severely affects patients’ ability to function in daily life. To date, the etiology and pathogenesis of AD have not been fully elucidated [2]. The National Institute on Aging-Alzheimer’s Association workgroup diagnostic guidelines state that patients with AD exhibit poor performance in one or more cognitive domains, and these impairments are expected to markedly deteriorate over time. Such alterations can manifest across a range of cognitive domains, such as memory, executive function, attention, language, and visuospatial skills. Impairment in episodic memory (the capacity to learn and retain new information) is most prevalent among patients with mild cognitive impairment (MCI) [3]. The initial symptoms of AD are often subtle and difficult to detect by caregivers. Therefore, patients are often in the moderate to severe stage of AD by the time caregivers become aware of abnormalities and have thus missed the opportunity for early and effective treatment. This can negatively impact patients’ health-related outcomes and quality of life. As the disease progresses, cognitive impairment gradually worsens, and patients encounter increasingly more difficulties in daily life.

Outcome measurement instruments (OMIs) constitute a standardized mechanism designed to quantify specific health-related constructs within defined clinical domains. The taxonomy of OMIs encompasses diverse modalities, including but not limited to: questionnaires (including patient-reported outcome measurement or clinician-reported outcomes), clinician-administered rating scales, biomarker analyses, and so on [4]. OMIs are used to monitor the health status of patients (including psychological and cognitive health) and evaluate treatment methods in research and clinical practice [5, 6]. OMIs are widely used in AD research and clinical practice to obtain subjective cognitive function data via systematic scales and scoring criteria. By assessing different aspects of cognitive function, these instruments can help determine the extent, type, and progression of a patient’s cognitive deficits. Their standardized implementation facilitates comparative outcome analyses while informing evidence-based clinical decision-making processes.

Cognitive measurement instruments represent economically viable, time-efficient screening solutions, demonstrating particular utility in preclinical screening paradigms for detecting individuals at elevated risk for developing symptomatic AD manifestations. Although extensive investigations and summaries have been conducted on the measurement characteristics such as reliability, validity, sensitivity, and responsiveness of cognitive assessment tools for AD, their measurement has not yet been used the internationally recognized methodology for structured quality rating of the scale. The comprehensive evaluation results of measurement attributes have not been integrated to form a clinical recommendation level [7, 8]. Determining whether existing scales for evaluating cognitive function require updating is vital. Therefore, this systematic review aimed to identify the most commonly used cognitive assessment instruments in AD screening, diagnosis, and evaluation. We assessed and compared the quality of measurement properties (i.e., reliability and validity) and the characteristics of cognitive function assessment scales to examine the quality of studies on MCI and AD according to the guidelines of Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) and the consensus-based standards for the selection of health measurement instruments (COSMIN). PRISMA-COSMIN (2024) is an expert consensus developed by the Enhancing the Quality and Transparency Of health Research (EQUATOR) guidelines and included a literature search, expert consultations, a Delphi study, a hybrid workgroup meeting, pilot testing, and an end-of-project meeting, with integrated patient/public involvement. The PRISMA-COSMIN methodology for OMIs, offering a detailed, standardized, and transparent framework, facilitates the selection and use of high-quality OMIs in both research and clinical practice [9‒12].

The systematic review was conducted and reported in accordance with the recommendations of the PRISMA statement. The methodological quality of the included studies was evaluated using the COSMIN checklist. We examined the psychometric properties of each instrument. The PRISMA checklist is provided in the Supplementary Material.

Instruments were included in the review if they satisfied the following criteria: (1) the study described the development (i.e., inauguration paper) and/or the evaluation of the psychometric properties and their relationships with other relevant cognitive constructs (i.e., validation paper) in AD patients; (2) the study population were adults with AD; (3) the instrument was designed to evaluate the degree of cognitive impairment for AD specifically; (4) the study only documented OMIs (the format is limited to cognitive measurement instruments such as questionnaires, Q&A tools, scales, and tests, which can obtain scores in a short period of time without the assistance of medical imaging equipment); and (5) studies were written in English or Chinese. Instruments were excluded if they met any of the following criteria: (1) nonscale formats (the types include biomarker analyses, laboratory tests, scores obtained through a physical examination or observations of an image, and so on); (2) randomized controlled studies using OMIs only as a measurement tool (i.e., clinical trials without any validation of the measurement tool); (3) dissertations, editorials, conference proceedings, and reports; (4) republished papers; and (5) papers with abstracts only or unavailable full texts.

Articles were identified by conducting searches of two electronic databases: PubMed and China National Knowledge Infrastructure (CNKI). Searches were performed from inception to February 1, 2024. All identified articles were screened. We used 5 combined sets of search terms, which are provided in the Supplementary Material.

The retrieved articles were organized using NoteExpress3.9.0 (Beijing Aegean Lezhi Technology Co., Ltd., Beijing, China). For studies that met the inclusion criteria, two reviewers independently extracted the following data: name of the instrument, author, country, number of samples, measurement domains, number of items, and evidence of reliability and validity. Data were summarized using Excel2019 (Product ID: 00405-32257-26357-AAOEM).

The methodological quality of the studies was assessed using the COSMIN checklist [9, 13, 14], which is a standardized tool for assessing the measurement properties of instruments across nine domains: internal consistency, reliability, measurement error, content validity, construct validity (subdivided into structural validity, hypothesis testing, and cross-cultural validity), convergent validity, and responsiveness. Each domain is rated using 5–18 items. The methodological quality of each item was rated as “very good,” “adequate,” “doubtful,” or “inadequate.” A methodological quality score for each measurement property was obtained by taking the lowest rating of any item (i.e., the “worst score counts” principle). The measurement property was then rated as sufficient (+), insufficient (−), inconsistent (±), or indeterminate (?). The quality of the instruments was graded as high, moderate, low, or very low using a modified grading of recommendations assessment, development, and evaluation (GRADE) approach. Finally, 4 recommendation levels (A, B, C, and D) were used to represent high, moderate, low, and very low research quality, respectively. Two independent reviewers assessed the methodological quality of the included studies using the checklist. Any disagreements in scoring were resolved through discussion with a third reviewer.

In addition, according to the GRADE approach for evaluating measurement properties in systematic reviews of OMIs, we considered the following 4 factors: (1) risk of bias (i.e., the methodological quality of the studies), (2) inconsistency (i.e., unexplained inconsistency of results across studies), (3) imprecision (i.e., total sample size of the available studies), and (4) indirectness (i.e., evidence from populations that differ from the current population of interest).

We obtained 804 articles from the database search. Of these, 32 were obtained from CNKI, and 772 were retrieved from PubMed. We excluded 376 articles because they described the development (inauguration paper) and/or assessment (validation paper) of psychometric properties that were not relevant to cognitive function in patients with AD or were not related to other constructs. We excluded 81 articles because the full text was unavailable. A further 26 review articles and 132 articles reporting unqualified measurement studies (e.g., studies using instruments only as clinical trial outcomes, studies without any validation, and studies missing important measurement properties) were excluded. We also excluded 58 studies that only used OMIs as a measurement tool or validation studies that used OMIs for other instruments. Fifteen studies that were not OMIs or specific OMIs were excluded. After excluding 28 non-Chinese or English articles, 62 valid articles were finally included. The article selection process is shown in Figure 1.

The studies collectively used a total of 49 scales. Our data analysis revealed that the AD Assessment Scale-cognitive subscale (ADAS-Cog) was the most frequently used scale, where nine articles evaluated the measurement properties of the ADAS-Cog. Four articles evaluating the Montreal Cognitive Assessment (MoCA) were included in English and Chinese. Three articles assessed the Baylor Profound Mental Status Examination (BPMSE), and two studies evaluated the Clinical Dementia Rating (CDR) scale. We included only one article on the measurement properties of the Mini-Mental State Examination (MMSE). Based on the grading criteria outlined in the COSMIN guidelines, we categorized the recommendation levels into 4 grades (A, B, C, and D) and compiled the recommendation levels of the 49 assessment tools into the Table 1.

Table 2 shows the 6 properties of the cognitive assessment instruments for evaluating AD patients. Sample sizes ranged from 15 to 5,704 patients. Studies were conducted in various countries, including China (n = 19), the USA (n = 10), and South Korea (n = 5). Instruments typically contained 3 to 10 dimensions (e.g., memory, number processing, orientation, language ability, and behavioral ability), with the number of items ranging from 6 to 103. More than half of the articles (n > 30) involved tests of memory. Some instruments (10 or so) emphasized changes in language abilities, and other instruments were focused on attentional and perceptual dysfunctions. Two articles did not describe specific measurement dimensions of the scale used [15, 16].

We have listed the articles and instruments used in online supplementary Table 2 (for all online suppl. material, see https://doi.org/10.1159/000545907), including instruments that were only used in one study. For this review, we specifically considered the clinical application of the instrument and thus focused on the measurement tools that appeared in more than one study. We placed the review results (Measurement Characteristic Evaluation Form and Bias Risk Evaluation Form) in the supplementary table.

The nine studies that evaluated the measurement properties of the ADAS-Cog are listed in Table 3. The ADAS-Cog was developed in 1984 and is a widely used neuropsychological screening tool for cognitive impairment in patients with AD [76]. According to Rosen et al., the scale was not designed as a diagnostic instrument but rather as a valuable tool for evaluating functional impairments and facilitating research on AD and other forms of dementia. Its applications encompass psychiatric medication studies, assessments of nursing environments, and longitudinal investigations, underscoring its multifaceted potential in the field [76]. The ADAS comprises 44 items. The maximum score of the ADAS-Cog is 70, and a higher score indicates greater cognitive impairment. The study by Weng et al. [27] did not examine the convergent validity of the ADAS-Cog. The other eight studies measured and evaluated the reliability and validity of the ADAS-Cog and considered the MMSE as the gold standard to determine the convergent validity of the ADAS-Cog. The Pearson and Spearman correlation coefficient showed a significant negative correlation between ADAS-Cog scores and MMSE scores (|r| > 0.6), which indicated a “very good” methodological quality rating. The structural validity of the ADAS-Cog was evaluated in only two studies [21, 27], which yielded a sufficient (+) rating for measurement properties and an inadequate rating for methodological quality. Internal consistency (as measured by Cronbach’s alpha) of the other seven articles was rated as sufficient, and methodological quality was rated as “very good.” The studies by Ben et al. [21] and Mavioglu et al. [15] achieved a sufficient (+) rating for test-retest validity and a “very good” rating for methodological quality. The methodological quality of the study by Cano et al. [25] was rated as “doubtful” because the timing of the retest was not stated. Although Li et al. [28] used the Spier correlation coefficient to measure retest effectiveness, no relevant data were found. Therefore, the methodological quality of the study was rated as “inadequate.” Four articles measured ADAS-Cog reliability using the intragroup correlation coefficient (ICC). The results were rated as sufficient (+), and methodological quality was rated as “very good.” The final recommended level for the ADAS-Cog was A.

The four articles that measured the characteristics of the MoCA are described in Table 4. The MoCA, developed by Nasreddine et al. [77] in 2005, is a short and useful screening tool with high sensitivity and specificity for detecting cognitive dysfunction in people who score normally on the MMSE. Because the MoCA can effectively detect MCI, it is suited to clinical use when time is limited (<10 min) [77]. The MoCA is a single-page assessment that evaluates seven cognitive domains: visuospatial/executive functions, naming, verbal memory registration and learning, attention, abstraction, 5-min delayed verbal memory, and orientation. The scores of the MoCA range from 0 to 30, and a higher score indicates higher cognitive function. In all four papers, the convergent validity of the MoCA was measured by the Pearson correlation coefficient or area under the curve AUC, and Cronbach’s alpha coefficient was used to measure the internal consistency of the MoCA. The measurement properties were all sufficient (+), and the method quality rating was “very good.” Pinto et al. [57] and Janelidze et al. [60] measured retest reliability using ICC and Cronbach’s alpha, respectively, and both achieved a sufficient (+) rating for measurement property, with an explicit reference to retest time, and a “very good” rating for methodological quality. Hu and colleagues achieved a methodological quality rating of “doubtful” because the timing of the retest was not explicitly stated. The final recommended level for the MoCA was A.

The three articles that assessed the measurement characteristics of the BPMSE are described in Table 5. Our group (R.S.D. and E.P.F.) developed the BPMSE in 1988, which consists of a 25-question cognition subscale (BPMSE-cog), a 10-item dichotomous behavior subscale (BPMSE-behav), and 2 descriptive questions concerning communication and social interactions [35]. The cognition subscale is divided into 4 subareas: orientation, language, attention, and motor functioning. The test can be administered in 5 min and uses common items (e.g., watches, pencils, and keys) for the assessment, which allows administration of the test in almost any setting without the need for designated testing rooms or specialized auditor training. All three papers examined the test-retest reliability (using ICC or correlation coefficients) and convergent validity (using Pearson correlation coefficients) of the BPMSE. For both test-retest reliability and convergent validity, the measurement feature rating was rated as sufficient (+), and methodological quality was rated as “very good.” It is worth noting that although Kørner used Wilcoxon’s test to measure the reliability of the BPMSE, the results were not reported and thus the measurement properties were rated indeterminate (?), and the methodological quality rating was “doubtful.” In addition, Salmeron et al. used factor analysis to measure the structural validity of the BPMSE, and although the measurement properties were rated as sufficient (+), the methodological quality was rated as “inadequate.” The BPMSE received an A level Table 6.

The CDR scale was evaluated in two studies. The CDR scale was initially developed as a staging instrument for AD severity and encompasses six categories: memory, orientation, judgment and problem solving, community affairs, home and hobbies, and personal care. The global rating for the CDR scale, which is calculated using a complex algorithm, ranges from 0 (no dementia) to 3 (severe dementia) [78]. The CDR scale focuses solely on the evaluation of cognitive function and meticulously excludes the confounding effects of noncognitive factors. Therefore, the CDR scale is invaluable to a diverse array of multidisciplinary specialists for conducting comprehensive and accurate global assessments of cognitive function in older adults across a wide spectrum of mild to severe dementia [79]. Convergent validity was measured by AUC, and internal consistency was measured by Cronbach’s alpha. For both of these measures, the measurement properties were rated as sufficient (+), and methodological quality was rated as “very good.” Coley et al. [39] used factor analysis to measure the structural validity of the CDR scale. Although the measurement property was rated as sufficient (+), the amount of evidence was insufficient and the methodological quality was “inadequate.” The final recommended level for the CDR scale was A.

This review aimed to examine the quality of OMIs used to assess cognitive impairment in patients with AD. The ADAS-Cog, MoCA, BPMSE, CDR, and other 28 scales were rigorously evaluated according to COSMIN guidelines, and all instruments demonstrated satisfactory levels of validity (i.e., structural validity) and reliability (i.e., internal consistency and test-retest reliability). They were all assigned the highest recommended grade of A, which highlighted their credibility in assessing the severity of cognitive impairment. The COSMIN guidelines were applied to ensure the rationality of the scales rather than to assess measurement bias. However, the scales do use various formats to assess the same cognitive dimension. The methodological quality of the four scales was under control, and the recommended levels were comparable across all four scales. Therefore, the specific setting and contents of the scale could be considered when conducting AD-related clinical research to ensure the most appropriate assessment scales are used.

When evaluating the same cognitive dimension, the use of different items from different scales can significantly impact performance in terms of sensitivity and coverage. For example, for immediate memory testing, the MMSE [56] and MoCA [57] involve the oral presentation of 3 or 5 words, whereas the ADAS-Cog [21] evaluates delayed memory by using word recognition. Therefore, the MMSE and MoCA are more suited to illiterate patients. Furthermore, the cultural background of subjects has a significant impact on ADAS-Cog performance. Future research could prioritize the development of culturally adapted OMIs and validate these instruments across diverse populations, given the documented impact of cultural factors on measurement validity and applicability.

In terms of usage scenarios and conditions, the ADAS-Cog boasts versatility; it can be applied to a range of scenarios, such as AD and other dementia studies, psychotropic medication trials, care setting evaluations, and longitudinal research [76]. The MoCA, renowned for its brevity, ease of use, and heightened sensitivity toward MCI patients, has widespread clinical application [77]. In contrast, the BPMSE is distinct in that it requires no test kit, specialized reviewer training, or informant dependency, which simplifies the administration process. Thus, the BPMSE is an effective tool for evaluating cognitive function in patients with severe AD in both research and clinical settings [35, 37, 36].

In terms of research subject, the efficacy of the ADAS-Cog for detecting MCI is relatively low. It is more suitable for evaluating patients with moderate to severe AD [25, 80]. The MoCA is more appropriate for patients with MCI during the early stage of AD [57‒60, 81]. The CDR scale encompasses a broad spectrum of dementia severity and integrates multiple patient characteristics, which makes it a practical tool for neurologists, psychiatrists, and other specialists alike. However, to accurately evaluate the cognitive function of patients, the CDR scale requires that users have a certain knowledge base (e.g., neurologists and psychiatrists) [38, 39, 79]. The content described above is also in line with current diagnostic and treatment guidelines and previous literature [82‒84]. It is evident that each existing scale has its relative limitations and shortcomings. However, with technological advancements, an emerging approach – digital cognitive assessment platforms and adaptive testing methodologies – has demonstrated considerable potential. This technology can systematically identify and categorize issues within existing scales while dynamically integrating assessment content based on individual patient characteristics (e.g., cognitive ability, cultural background, and clinical needs). As a result, it generates highly personalized, tailored scales. The field is rapidly evolving through expanded validation initiatives incorporating larger-scale cohorts and longitudinal investigation designs [85‒87]. We suggest that our future research could focus on rigorous longitudinal validation of these technologies to establish their psychometric robustness in tracking preclinical AD progression trajectories and predicting clinical conversion patterns.

Additionally, Hu et al. [58] examined the measurement characteristics of the MoCA in eastern Chinese communities and discovered that cultural span affects the response rate of MoCA testing and proposed suggestions for improvement. Kaya et al. [59] and Janelidze et al. [60] studied the measurement properties of the Turkish and Georgian versions of the MoCA, respectively, and reported that both versions had good reliability and validity. In addition, owing to the change of times and differences of region, the use and elimination of paper letters vary among regions with diverse historical backgrounds. In the current version of the ADAS-Cog, there is an operational issue regarding how patients can send paper letters to post office. Therefore, the universal applicability and compatibility must be carefully considered when translating or applying these evaluation tools. We suggest that clinical researchers studying AD could consider time, linguistic, and cultural variations when selecting instruments because scale translation, adaptability, and cultural sensitivity can influence the accuracy and effectiveness of the scales.

To test the convergent validity of instruments, 27 articles calculated Pearson correlation coefficients between the study instrument score and the MMSE score. Although we only screened one study examining the relevant measurement properties of the MMSE [88], it is well established that the MMSE has high clinical significance in the assessment of cognitive function in AD patients. The MMSE [56] has become the most commonly used screening tool in clinical and community settings that provides an overall measure of cognitive impairment. The maximum score for the MMSE is 30, with a lower score indicating greater cognitive impairment [56]. However, there has been limited systematic evaluation of the diagnostic accuracy and measurement properties of the MMSE [89]. Harrell et al. argued that the MMSE score range is too limited to distinguish between moderate and severe AD and subsequently developed a new instrument based on the MMSE: the Severe Mini-mental State Examination (SMMSE). The SMMSE is simple, easy to administer, and requires no specialized equipment. Moreover, it has a high tolerance for patients with severe cognitive impairment.

Despite retrieving a large number of studies on measurement attributes, the methodological quality and structural validity of almost all studies were rated “inadequate” and “doubtful,” respectively. Although few studies measured structural validity according to the COSMIN guidelines, most provided evidence that convergent validity could have achieved a sufficient (+) rating. However, the association criteria for each study could not be controlled, and the results of the instrument measurements may have been influenced by the subjective decisions of the researchers. The included articles provided little evidence of content validity, although there was also no compelling evidence to suggest inadequate content validity. Therefore, we did not include content validity as a measurement property in our review. The COSMIN rating method ensures the feasibility of validity rating, even when information on the content effectiveness is insufficient. Therefore, conclusions regarding content validity are based largely on limited evidence, and thus, our results, as guided by the COSMIN, should be interpreted with caution [85‒87].

We conducted a systematic evaluation of assessment tools for evaluating the cognitive function of AD patients by rating the measurement characteristics of the instruments and analyzing the methodological quality of each measurement method. We found four commonly used scales with high therapeutic efficacy: the ADAS Cog, MoCA, BPMSE, CDR, and MMSE scale. However, for clinical research on AD, the research objectives, usage scenarios, scale dimensions, and subtle differences in scale items must be carefully considered. Furthermore, the impact of time, region, and culture on the reliability and validity of the scales could be examined.

The author extended heartfelt gratitude to Professor Lyv, the corresponding author, for her invaluable guidance and profound insights shared by all mentors. Additionally, the author would like to thank the literature review and screening team for their significant contributions. We thank Sarina Iwabuchi, PhD, from Liwen Bianji (Edanz) (https://www.liwenbianji.cn) for editing the English language of a draft of this manuscript.

The research design of this review did not involve human participants. Thus, no ethical review was required.

The authors have no conflicts of interest to declare.

This work was supported by the 15th batch of free exploration projects of the Institute of Clinical Basic Medicine of Chinese Academy of Traditional Chinese Medicine: performance study of outcome measurement tool for patient-reported outcome measures of Alzheimer’s disease based on clinical measurement.

Zhe-Zuo Zhang: literature analysis, and writing-original draft; Xiao-Ying Lyu: conceptualizaton, writing-review and editing, supervision, and project administration; Xiang-Wei Dai, Jian-Ni Cong, and Fu-Xia Yang: providing revision suggestions and editing from the perspetives of clinical medicine and scales methodilogy.

All data generated or analyzed during this study are included in this article and its supplementary material files. Further inquiries can be directed to the corresponding author.