Abstract
Introduction: The cognitive screening usually requires a face-to-face format, which might limit its use in many circumstances. We aimed to develop a new application-based cognitive screening test (ACST) to serve as an accessible and valid tool in the community. Methods: The ACST was developed by using paired association and digit span tests. This test was administered to 70 cognitively normal participants, 62 participants with MCI, and 64 participants with dementia. The 2nd edition of the Mini-Mental State Examination (MMSE-2) and the Montreal Cognitive Assessment (MoCA) were collected by certified psychologists. The ACST was self-administered by the participants, with a clinician providing instructions for those with dementia or technological limitations. The diagnosis was made according to DSM-5 criteria by an experienced geriatric neurologist blinded to the application score. Content validity, test-retest reliability, interrater reliability, and correlations between application scores and MMSE-2 and MoCA scores were analyzed. Results: The sensitivity and specificity for distinguishing cognitively normal participants from non-normal participants were 92.9% and 70%, respectively (cutoff point ≤7). The sensitivity and specificity for distinguishing between the cognitively normal group and the MCI group were 87.1% and 70%, respectively (cut point ≤7). The sensitivity and specificity for distinguishing cognitively normal participants from participants with dementia were 93.8% and 82.9%, respectively (cut point ≤6). A cutoff point ≤6 was considered suitable for participants aged 75 years or older or with 6 or fewer years of education. Discussion: The ACST is an easy-to-use and valid tool for cognitive screening in older Thai adults in clinical practice. Patients with an application score ≤7 are considered to be at risk of cognitive impairment and to require further evaluation.
Introduction
The proportion of older people in the general population has increased substantially. The share of the global population aged 65 years or older is projected to increase from 10% in 2022 to 16% in 2050 [1]. Consequently, dementia is becoming a global public health priority [2]. The World Health Organization (WHO) estimated that the number of individuals with dementia worldwide was approximately 55 million in 2019, and this number is expected to reach approximately 78 million by 2030 and 139 million by 2050 [3].
Additionally, dementia is the seventh leading cause of death and disability and dependency among older people worldwide; it has physical, psychological, social, and economic impacts on carers, families, and society at large [4]. Despite its high prevalence, dementia is underdiagnosed by clinicians in approximately 39.5% of patients [5] because of a lack of awareness and inaccessibility to diagnosis. At present, diagnosis requires history taking, cognitive assessment, and brain imaging, which need face-to-face evaluation by a healthcare professional.
Cognitive assessment plays a crucial role in the early and accurate diagnosis of dementia, facilitating timely interventions and aiding in differential diagnosis among various types of dementia. Dementia is primarily a cognitive disorder, and cognitive assessment provides a structured approach to evaluate cognitive domains commonly affected in dementia, including memory, attention, executive function, language, and visuospatial abilities. The integration of cognitive assessment results with clinical, neuroimaging, and laboratory findings enhances diagnostic precision and allows for a comprehensive understanding of the patient’s cognitive status. However, the process of cognitive assessment is time-consuming and is not accessible in many circumstances.
A paired associated test is one of the most useful cognitive tests. The paired associated test primarily assesses associative memory, a key component of episodic memory. This cognitive domain involves the ability to form and recall associations between pairs of unrelated items, such as a word and an image or two unrelated words. Additionally, the tasks can indirectly test attention and visual memory, especially in versions that involve visual pairings. Several tests have used this principle, including the Wechsler Memory Scale [6], the Signed Paired Associates Test [7], and the Face-Name Associative Memory Exam (FNAME) [8].
The Face-Name Associative Memory Exam (FNAME) [8] is a cross-modal associative memory test that was initially developed in 2011. It includes 16 face-name pairs and 16 face-occupation pairs to detect subtle memory changes that are commonly found in patients with dementia. FNAME has been validated in American [9] and Spanish [10] populations. In 2014, a modified version of the task, called FNAME-12 [11], was developed. The FNAME-12 exhibited psychometric equivalence with the original FNAME (r = 0.77, p < 0.001) and was subsequently validated in American [11], Latino American [12], and Greek [13] populations. This shortened version can be used to differentiate between normal aging, mild cognitive impairment (MCI), and dementia.
Computerized versions of paired associated tests, including the Brain Health Assessment (BHA) [14], Paired Associates Learning (PAL) [15], and FACEmemory® [16], are also available. In 2020, FACEmemory® was developed by transforming the FNAME-12 into a self-administered test on tablets with voice recognition, touchscreen answers, and automatic scoring. FACEmemory® has a short-term memory task and a long-term memory task that includes face, name, and occupation recognition, with a sensitivity of 80.5% and specificity of 80.3% for detecting MCI.
While associative memory is a critical component of dementia assessment, attention is another essential cognitive domain to consider. Tools such as the digit span test, with its forward and backward components, offer a practical method for evaluating attention and immediate memory, further supporting the identification of cognitive impairment. This tool is less time-consuming, and it is valid for detecting cognitive impairment. The digit forward test demonstrated a sensitivity of 63% and a specificity of 69% for detecting MCI. Digit backward had a sensitivity of 77% and a specificity of 57% [17] for detecting MCI. Digit span test had a sensitivity of 79% and a specificity of 65% [18] for detecting dementia. Thus, simultaneous testing using FNAME and digit span would increase the net sensitivity of the tool.
To make cognitive assessment even more accessible, integrating smartphone and tablet technology into these tools represents a forward-looking approach. With smartphone and tablet use among older adults rapidly increasing, application-based cognitive screening tests could provide efficient, self-administered options for early dementia detection outside traditional clinical settings. In addition, the smartphone and tablet revolution has also affected older adults. Smartphone ownership increased from only 10% of older adults in 2011 to 61% in 2021. Tablet ownership increased from 4% in 2011 to 44% in 2021 [19]. Given the growing availability and ease of use of mobile devices among older populations, developing application-based cognitive screening tools could address key barriers to early dementia diagnosis. We aimed to develop and validate an application-based cognitive screening test (ACST) as an accessible, self-administered and valid tool used with tablets and smartphones in the community. By facilitating early intervention and management, such tools have the potential to significantly impact public health outcomes in dementia care.
Methods
Study Design
This study is a diagnostic study to explore the validity of the ACST compared to a gold standard of the MCI and dementia diagnosis with the DSM-5 criteria.
Setting and Participant
Participants were recruited from the Geriatric Clinic of Siriraj Hospital, Mahidol University, from August 2023 to January 2024. The inclusion criteria were as follows: (1) older individuals aged 60 years or older who attended outpatient visits at the Geriatric Clinic of Siriraj Hospital; (2) participants who met the diagnostic criteria for dementia and MCI according to DSM-5 criteria [20] and Clinical Dementia Rating (CDR) [21] as well as individuals considered cognitively normal; (3) the ability to communicate and understand the Thai language.
The exclusion criteria were as follows: (1) individuals with abnormal psychiatric or neurological conditions and behavioral problems, such as uncontrolled depression (TGDS >12) [22], or severe behavioral and psychological symptoms of dementia (BPSD) that affect communication; (2) individuals with moderate to severe dementia (global CDR >1) were excluded, as the study included only participants with early to mild dementia (global CDR 0.5–1), who often exhibited mild symptoms and underwent ACST. Furthermore, in clinical practice, patients with obvious symptoms are usually readily recognized by their families and already have a medical diagnosis; (3) individuals with acute medical conditions such as hospitalization within the past 3 months who might not have stable physical and cognitive status due to recent illness; (4) disabled older adults who had physical limitation affecting the assessment such as motor weakness, joint deformities, or visual impairment; (5) relatives or caregivers who refused to participate in the CDR questionnaire; (6) individuals who could not use the application-based cognitive screening test (ACST) because of the technological limitation. Voluntary withdrawal from the study was permitted based on the wishes of the participants and their family caregivers.
Participant Recruitment Method
All participants in the clinic who met the inclusion/exclusion criteria were approached. Details of the study were provided by a research assistant. Written informed consent was obtained from the participants and from the legal representatives of those with dementia before they joined the study.
Application Development
Test Development
The ACST was developed by using paired association tests to assess short-term memory, a common cognitive domain that is impaired in dementia, and digit span to assess attention. Both written and verbal instructions were provided for illiterate or hearing-impaired individuals. The ACST is a touch screen system without voice detection, drawing, or typewriting. The ACST is composed of two parts.
Part 1 consists of six pairs of people and their favorite fruit. Each pair has 10 s to remember. After finishing the six pairs, the system moves to part 2.
Part 2 includes the digit span, which comprises the following:
- -
Digit forward: two groups of numbers (5 and 6 digits)
- -
Digit backward: two groups of numbers (3 and 4 digits)
- -
The individuals subsequently must describe the favorite fruit for each person. Next, the screen shows the total score, the score in each part, and the time used. Moreover, advice and a VDO clip is provided for individuals with normal and abnormal scores to follow. There is an example demonstrating how to perform each task separately (one example for each task before starting the actual task).
Quality of the Test
The content validity of the ACST was assessed using an item-level content validity index (I-CVI) by four specialists who did not participate in the study, including one neurologist, one geriatrician, one geriatric psychologist, and one nurse who specializes in gerontology. The findings indicated an I-CVI of 1.0.
Pilot Study
A pilot study of 20 subjects (5 cognitively normal individuals, 5 individuals with MCI, 5 individuals with dementia, and 5 individuals with caregivers) who attended the Geriatric Clinic of Siriraj Hospital, Mahidol University, was performed. The subjects provided feedback on the clarity, difficulty, appropriateness, and satisfaction of the ACST using a 4-point scale (1 = completely agree, 2 = moderately agree, 3 = mildly agree, and 4 = disagree).
Eighty percent (80%) of participants were highly satisfied/satisfied with this application, and 70% found it practical to use. Some participants complained about the amount of time given to answer, the size of the alphabet, and the volume of the voice instructions. This feedback was used to adjust the ACST before the validation process.
Validation Process
Participants were initially categorized into three groups by a research assistant by the diagnosis of previous cognitive status: cognitively normal, MCI, and dementia. Study participants then were stratified by age and educational level among the groups because these demographic characteristics could affect the application score independently of cognitive performance. Participants who met the eligibility criteria were included.
General information about the participants was collected, including sex, age, occupation, educational level, underlying diseases, medications, history of alcohol consumption, hearing impairment, traumatic brain injury, neurological conditions, and psychiatric disorders such as depression. All participants underwent the following assessments administered by certified psychologists who were blinded to the diagnosis: the Thai version of Mini-Mental State Examination-2 (MMSE-2 Thai) [23], Montreal Cognitive Assessment (MoCA-Thai) [24, 25], Thai Geriatric Depression Scale (TGDS-30) [22], Activities of Daily Livings (ADLs) using the Barthel index [26], and Lawton-Brody ADL [27].
The MMSE-2 is an updated version of the original MMSE, developed to address limitations in detecting early cognitive impairment and to reduce cultural and language translation difficulties in diverse populations. We used the standard version of MMSE-2, which is similar to the original MMSE; this 30-item version assesses additional cognitive domains such as attention, language, and visual construction, with a typical administration time of 20 min. The MMSE-2 has exhibited high sensitivity (84%) in differentiating Alzheimer’s disease (AD) from normal cognitive aging and subcortical dementia, enhancing its utility in early detection and monitoring. A total score is 30 points. Cutoff points vary based on clinical and cultural norms, often around 24–26 for the standard version to differentiate between normal cognition and cognitive impairment. Specific cutoff scores may also be adjusted depending on the population and clinical objectives [23].
The MoCA is used to evaluate multiple cognitive domains, including executive function, visuospatial ability, memory, attention, language, and orientation. This multi-domain structure enables the MoCA to detect subtle cognitive changes, enhancing its sensitivity compared to single-domain assessments. With a maximum score of 30, the standard MoCA cutoff for identifying cognitive impairment is 25/26. Studies demonstrate that the MoCA offers high sensitivity and specificity in distinguishing between normal aging, MCI, and mild dementia, especially in populations where early detection of cognitive decline is critical [24, 25].
The Geriatric Depression Scale (GDS) is used to detect depression in elderly individuals who may have difficulty expressing or recognizing their emotional distress due to age-related changes or cognitive decline. The scale consists of a series of yes/no questions that assess mood, behavior, and physical symptoms commonly associated with depression. A cutoff score for depression in the 30-item Thai GDS version is a score of 12 or higher, though this may vary depending on the population and clinical context [22].
The Barthel index (20-point version) is a simplified tool designed to assess an individual’s ability to perform Activities of Daily Living (ADLs), which are essential self-care tasks. The scale evaluates functional independence by scoring a patient’s performance on 10 ADLs with a total score out of 20 points. Sum the patient’s scores for each item. Total possible scores range from 0 to 20, with lower scores indicating increased disability [26, 27].
The Lawton and Brody IADL Scale is a widely used assessment tool designed to measure the Instrumental Activities of Daily Living (IADLs) in older adults. The scale evaluates eight domains of IADL function: using the telephone, shopping, food preparation, housekeeping, laundry, transportation, medication management, and financial management [27, 28]. The IADL scale is particularly useful in evaluating cognitive or physical decline in aging populations, making it valuable in research, geriatrics, and neuropsychology. It is commonly used to differentiate between normal aging and more serious cognitive impairments, such as dementia. The scale is designed for use by caregivers, family members, or healthcare providers to assess and monitor the functional abilities of older adults.
During the same visit, the ACST was self-administered by the cognitively normal and MCI participants with a researcher instructed how to start using the ACST application in those with dementia or in those with physical and technological limitation, followed by diagnosis according to DSM-5 [20] criteria by an experienced geriatric neurologist who was blinded to the application score using clinical interviews, the Clinical Dementia Rating Scale (CDR) [21], and neuroimaging data (CT/MRI brain) in patients with dementia.
This study was conducted with participants previously diagnosed with MCI or dementia. Since MCI may progress to dementia, clinical information was obtained through interviews with both patients and their caregivers. Additionally, cognitively normal elderly participants underwent a new clinical interview. The psychologist and an experienced geriatric neurologist, both blinded to the ACST results, conducted the assessments.
Moreover, 10 nonparticipants were evaluated for interrater reliability by a clinician and a geriatric clinic personnel member, and test-retest reliability was conducted 2 weeks apart. The details are summarized in Figure 1.
Sample Size Calculation
In this study, the participants were categorized into three groups: cognitively normal, MCI, and dementia. According to the FACEmemory [16], a computerized version of the Short Form of the Face-Name Associative Memory Exam had a sensitivity and specificity for MCI of 80.5% and 80.3%, respectively. In addition, digit span [14] had a sensitivity of 79% and a specificity of 65% for detecting dementia.
The sample size for each group was calculated using the principle of estimating a single proportion with a margin of error (d = 0.10). The total sample size was 196 participants, including 70 cognitively normal participants, 62 participants with MCI, and 64 participants with dementia.
Statistical Analysis
All the data were tested for normality using the Shapiro-Wilk test. We reported the mean ± standard deviation for continuous variables and the median for the CDR. Pearson correlation analysis was conducted to assess the correlation between the application score and the Thai MMSE-2 and MoCA-Thai scores. The intraclass correlation coefficient was calculated to assess the interrater reliability and test-retest reliability of the ACST. To evaluate concurrent validity, receiver operating characteristic curves were plotted, and the area under the curve (AUC) with the 95% confidence interval (CI) was calculated.
Additionally, we analyzed the sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio (+LR), and negative likelihood ratio (−LR) for determining the appropriate cutoff points for the diagnosis of MCI and dementia. SPSS version 26 was used for the statistical analysis.
Results
Overall, 196 participants were included in this study, including 70 cognitively normal participants, 62 participants with MCI, and 64 participants with dementia. Among the participants, 64.8% were female, the mean age was 74.6 ± 6.7 years, and the mean duration of education was 11.7 ± 4.9 years. The average age and education level in the cognitively normal, MCI, and dementia groups were 72.3 ± 6.7, 75.0 ± 6.4, 76.9 ± 6.4 years, and 12.8 ± 4.6, 10.4 ± 5.1, 11.6 ± 4.9, respectively. There were no significant differences among the groups.
In terms of comorbidities, we found that dyslipidemia was the most common comorbidity, followed by hypertension and diabetes mellitus. A total of 13.8% of the participants had cerebrovascular disease, and 9.7% had depression. The average Thai MMSE-2 score and MoCA-Thai score were 23.5 ± 4.0 and 20.8 ± 5.3, respectively. For the reference standard of the study, the CDR-SB ranged from 0 to 9 (median 0.5), and the global CDR ranged from 0 to 1 (median 0.5). The details are provided in Table 1 and the current medications are shown in online supplementary Table 1 (for all online suppl. material, see https://doi.org/10.1159/000543309).
Baseline characteristics of the participants
. | Participants (N = 196) . | Normal (n = 70) . | MCI (n = 62) . | Dementia (n = 64) . | p value . |
---|---|---|---|---|---|
Female, n (%) | 127 (64.8) | 51 (73) | 32 (51.6) | 44 (68.8) | 0.028 |
Age, mean±SD (min–max) | 74.6±6.7 (60–94) | 72.3±6.7 (60–94) | 75.0±6.4 (60–91) | 76.9±6.4 (60–93) | 0.037 |
BMI, mean±SD (min–max) | 22.5±3.8 (14–35) | 21.5±3.7 (14–31) | 23.0±3.7 (17–33) | 23.0±4.0 (15–35) | 0.080 |
Education year, mean±SD (min–max) | 11.7±4.9 (4–20) | 12.8±4.6 (4–20) | 10.4±5.1 (4–18) | 11.6±4.9 (4–20) | 0.319 |
Barthel index, mean±SD (min–max) | 19.9±0.5 (17–20) | 20±0 (20) | 19.9±0.3 (18–20) | 19.6±0.8 (17–20) | <0.001 |
Lawton IADL, mean±SD (min–max) | 10.6±2.3 (1–12) | 11.9±0.3 (10–12) | 11.6±1.1 (6–12) | 8.1±2.3 (1–12) | <0.001 |
Underlying diseases | |||||
Dyslipidemia | 120 (61.2%) | 38 (54.3%) | 40 (64.5%) | 42 (65.6%) | 0.329 |
Hypertension | 99 (50.5%) | 28 (40%) | 34 (54.8%) | 37 (57.8%) | 0.085 |
Diabetes mellitus | 37 (18.9%) | 6 (8.6%) | 10 (16.1%) | 21 (32.8%) | 0.001 |
Cerebrovascular disease | 27 (13.8%) | 8 (11.4%) | 10 (16.1%) | 9 (14%) | 0.734 |
Depression | 19 (9.7%) | 5 (7.1%) | 7 (11.3%) | 7 (10.9%) | 0.666 |
Coronary artery disease | 9 (4.6%) | 2 (2.9%) | 2 (3.2%) | 5 (7.8%) | 0.323 |
Parkinson’s disease | 4 (2.0%) | 0 | 2 (3.2%) | 2 (3.1%) | 0.321 |
MoCA-Thai, mean±SD (min–max) | 20.8±5.3 (6–30) | 25.2±2.7 (19–30) | 20.7±3.6 (11–27) | 16.1±4.8 (6–27) | <0.001 |
Thai MMSE-2, mean±SD (min–max) | 23.5±4.0 (12–30) | 26.7±2.1 (22–30) | 24.0±2.5 (18–29) | 19.6±3.3 (12–27) | <0.001 |
CDR-SB, min–max (median) | 0–9 (0.5) | 0 | 0.5–2 (0.5) | 1–9 (3.5) | <0.001 |
Global CDR, min–max (median) | 0–1 (0.5) | 0 | 0.5 | 0.5–1 (0.5) | <0.001 |
. | Participants (N = 196) . | Normal (n = 70) . | MCI (n = 62) . | Dementia (n = 64) . | p value . |
---|---|---|---|---|---|
Female, n (%) | 127 (64.8) | 51 (73) | 32 (51.6) | 44 (68.8) | 0.028 |
Age, mean±SD (min–max) | 74.6±6.7 (60–94) | 72.3±6.7 (60–94) | 75.0±6.4 (60–91) | 76.9±6.4 (60–93) | 0.037 |
BMI, mean±SD (min–max) | 22.5±3.8 (14–35) | 21.5±3.7 (14–31) | 23.0±3.7 (17–33) | 23.0±4.0 (15–35) | 0.080 |
Education year, mean±SD (min–max) | 11.7±4.9 (4–20) | 12.8±4.6 (4–20) | 10.4±5.1 (4–18) | 11.6±4.9 (4–20) | 0.319 |
Barthel index, mean±SD (min–max) | 19.9±0.5 (17–20) | 20±0 (20) | 19.9±0.3 (18–20) | 19.6±0.8 (17–20) | <0.001 |
Lawton IADL, mean±SD (min–max) | 10.6±2.3 (1–12) | 11.9±0.3 (10–12) | 11.6±1.1 (6–12) | 8.1±2.3 (1–12) | <0.001 |
Underlying diseases | |||||
Dyslipidemia | 120 (61.2%) | 38 (54.3%) | 40 (64.5%) | 42 (65.6%) | 0.329 |
Hypertension | 99 (50.5%) | 28 (40%) | 34 (54.8%) | 37 (57.8%) | 0.085 |
Diabetes mellitus | 37 (18.9%) | 6 (8.6%) | 10 (16.1%) | 21 (32.8%) | 0.001 |
Cerebrovascular disease | 27 (13.8%) | 8 (11.4%) | 10 (16.1%) | 9 (14%) | 0.734 |
Depression | 19 (9.7%) | 5 (7.1%) | 7 (11.3%) | 7 (10.9%) | 0.666 |
Coronary artery disease | 9 (4.6%) | 2 (2.9%) | 2 (3.2%) | 5 (7.8%) | 0.323 |
Parkinson’s disease | 4 (2.0%) | 0 | 2 (3.2%) | 2 (3.1%) | 0.321 |
MoCA-Thai, mean±SD (min–max) | 20.8±5.3 (6–30) | 25.2±2.7 (19–30) | 20.7±3.6 (11–27) | 16.1±4.8 (6–27) | <0.001 |
Thai MMSE-2, mean±SD (min–max) | 23.5±4.0 (12–30) | 26.7±2.1 (22–30) | 24.0±2.5 (18–29) | 19.6±3.3 (12–27) | <0.001 |
CDR-SB, min–max (median) | 0–9 (0.5) | 0 | 0.5–2 (0.5) | 1–9 (3.5) | <0.001 |
Global CDR, min–max (median) | 0–1 (0.5) | 0 | 0.5 | 0.5–1 (0.5) | <0.001 |
BMI, body mass index; SD, standard deviation; IADL, Instrumental Activity of Daily Living; MoCA, Montreal Cognitive Assessment; MMSE-2, Mini-Mental State Examination-second edition; CDR-SB, Clinical Dementia Rating-Sum of Boxes.
Most of the participants with dementia were diagnosed with AD (73.4%), followed by AD with small vessel disease (10.9%), AD with cerebrovascular disease (6.3%), vascular dementia (6.3%), and AD with vascular dementia (3.1%). The average application score was 5.8 ± 2.6, and the average time to complete the application test was 186.3 ± 34.2 s or approximately 3.1 min. The details are shown in online supplementary Table 2. The correlation between the total score of ACST and MMSE-2 and the total score of ACST and MoCA and the total score of MMSE-2 and MoCA are shown in Figure 2 and online supplementary Figure 1.
Scatterplot showing the correlation between the total scores of the ACST and MMSE-2 (a) and the ACST and MoCA (b).
Scatterplot showing the correlation between the total scores of the ACST and MMSE-2 (a) and the ACST and MoCA (b).
Table 2 shows the concurrent validity of the ACST. A cutoff point ≤7 was the most appropriate score for distinguishing cognitively normal from non-normal patients, with a sensitivity of 92.9% and a specificity of 70% (AUC of 0.890, 95% CI = 0.842–0.938; Fig. 3) and for distinguishing cognitively normal from MCI, with a sensitivity of 87.1% and a specificity of 70% (AUC of 0.832, 95% CI = 0.763–0.901; online suppl. Fig. 2). A cutoff point of ≤6 was considered suitable for distinguishing cognitively normal individuals from patients with dementia, with a sensitivity of 93.8% and a specificity of 82.9% (AUC = 0.946, 95% CI = 0.911–0.982; online suppl. Fig. 3). Furthermore, a cutoff point of ≤4 was considered suitable for distinguishing MCI from dementia, with a sensitivity of 71.9% and a specificity of 72.6% (AUC = 0.783, 95% CI = 0.703–0.863; online suppl. Fig. 4).
Concurrent validity of the ACST
ACST score . | Sensitivity . | Specificity . | PPV . | NPV . | +LR . | −LR . |
---|---|---|---|---|---|---|
Cognitively normal versus non-normal (AUC = 0.890, 95% CI = 0.842–0.938) | ||||||
5/10 | 65.1% | 87.2% | 90.1% | 58.1% | 5.1 | 0.4 |
6/10 | 78.6% | 82.9% | 89.2% | 68.2% | 4.6 | 0.3 |
7/10 | 92.9% | 70.0% | 84.8% | 84.5% | 3.1 | 0.1 |
8/10 | 96.8% | 50.0% | 77.7% | 89.7% | 1.9 | 0.1 |
Cognitively normal versus MCI (AUC = 0.832, 95% CI = 0.763–0.901) | ||||||
5/10 | 48.4% | 87.1% | 76.9% | 65.6% | 3.8 | 0.6 |
6/10 | 62.9% | 82.9% | 76.5% | 71.6% | 3.7 | 0.5 |
7/10 | 87.1% | 70.0% | 72.0% | 86.0% | 2.9 | 0.2 |
8/10 | 93.5% | 50.0% | 62.4% | 89.7% | 1.9 | 0.1 |
Cognitively normal versus dementia (AUC = 0.946, 95% CI = 0.911–0.982) | ||||||
4/10 | 71.9% | 95.7% | 93.9% | 78.8% | 16.7 | 0.3 |
5/10 | 81.3% | 87.1% | 85.2% | 83.6% | 6.3 | 0.2 |
6/10 | 93.8% | 82.9% | 83.3% | 93.5% | 5.5 | 0.1 |
7/10 | 98.4% | 70.0% | 75% | 98% | 3.3 | 0.02 |
MCI versus dementia (AUC = 0.783, 95% CI = 0.703–0.863) | ||||||
3/10 | 54.7% | 87.1% | 81.4% | 65.0% | 4.2 | 0.5 |
4/10 | 71.9% | 72.6% | 73.0% | 71.4% | 2.6 | 0.4 |
5/10 | 81.3% | 51.6% | 63.4% | 72.7% | 1.7 | 0.4 |
6/10 | 93.8% | 37.1% | 60.6% | 85.2% | 1.5 | 0.2 |
ACST score . | Sensitivity . | Specificity . | PPV . | NPV . | +LR . | −LR . |
---|---|---|---|---|---|---|
Cognitively normal versus non-normal (AUC = 0.890, 95% CI = 0.842–0.938) | ||||||
5/10 | 65.1% | 87.2% | 90.1% | 58.1% | 5.1 | 0.4 |
6/10 | 78.6% | 82.9% | 89.2% | 68.2% | 4.6 | 0.3 |
7/10 | 92.9% | 70.0% | 84.8% | 84.5% | 3.1 | 0.1 |
8/10 | 96.8% | 50.0% | 77.7% | 89.7% | 1.9 | 0.1 |
Cognitively normal versus MCI (AUC = 0.832, 95% CI = 0.763–0.901) | ||||||
5/10 | 48.4% | 87.1% | 76.9% | 65.6% | 3.8 | 0.6 |
6/10 | 62.9% | 82.9% | 76.5% | 71.6% | 3.7 | 0.5 |
7/10 | 87.1% | 70.0% | 72.0% | 86.0% | 2.9 | 0.2 |
8/10 | 93.5% | 50.0% | 62.4% | 89.7% | 1.9 | 0.1 |
Cognitively normal versus dementia (AUC = 0.946, 95% CI = 0.911–0.982) | ||||||
4/10 | 71.9% | 95.7% | 93.9% | 78.8% | 16.7 | 0.3 |
5/10 | 81.3% | 87.1% | 85.2% | 83.6% | 6.3 | 0.2 |
6/10 | 93.8% | 82.9% | 83.3% | 93.5% | 5.5 | 0.1 |
7/10 | 98.4% | 70.0% | 75% | 98% | 3.3 | 0.02 |
MCI versus dementia (AUC = 0.783, 95% CI = 0.703–0.863) | ||||||
3/10 | 54.7% | 87.1% | 81.4% | 65.0% | 4.2 | 0.5 |
4/10 | 71.9% | 72.6% | 73.0% | 71.4% | 2.6 | 0.4 |
5/10 | 81.3% | 51.6% | 63.4% | 72.7% | 1.7 | 0.4 |
6/10 | 93.8% | 37.1% | 60.6% | 85.2% | 1.5 | 0.2 |
Bold font indicates the best cut-off value.
MCI, mild cognitive impairment; ACST, application-based cognitive screening test; AUC, area under the curve; PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio; CI, confidence interval.
ROC curve comparing cognitively normal patients with non-normal patients at a cutoff point ≤7.
ROC curve comparing cognitively normal patients with non-normal patients at a cutoff point ≤7.
Subgroup Analysis
A subgroup analysis was conducted by age-group and education level. The age groups were 60–74 years and ≥75 years. Education level was categorized based on an education duration of ≤6 years, 7–12 years, and >12 years. A cutoff point ≤7 was the most appropriate score for distinguishing cognitively normal from non-normal participants aged 60–74 years, with a sensitivity of 92.6% and specificity of 80% (AUC of 0.925, 95% CI = 0.875–0.975); an education duration of 7–12 years, with a sensitivity of 87.5% and specificity of 52.6% (AUC of 0.753, 95% CI = 0.617–0.888); and an education duration >12 years, with a sensitivity of 96.2% and specificity of 82.5% (AUC of 0.959, 95% CI = 0.923–0.994). A cutoff point ≤6 was the most appropriate score for distinguishing cognitively normal from non-normal participants aged ≥75 years, with a sensitivity of 86.1% and a specificity of 68.2% (AUC of 0.818, 95% CI = 0.716–0.920), and an education duration of ≤6 years, with a sensitivity of 87.9% and a specificity of 72.7% (AUC of 0.854, 95% CI = 0.712–0.996).
In addition, the interrater and test-retest reliability of the ACST were excellent, with intraclass correlation coefficients of 0.911 and 0.878, respectively. The application score showed a convergent validity with the scores obtained for the Thai MMSE-2 (Pearson correlation coefficient of 0.698, p < 0.001) and MoCA-Thai (Pearson correlation coefficient of 0.684, p < 0.001).
User preference for the ACST was assessed by 10 participants who were cognitively normal, 10 participants with MCI, and 10 participants with dementia. Among the cognitively normal participants, 100% were highly satisfied/satisfied with this application, and 90% found it to be practical for use. In contrast, 90% of the participants with MCI were highly satisfied/satisfied with the application and 80% considered it practical for use; 90% of the patients with dementia were highly satisfied/satisfied with this application, and 80% considered it practical for use.
Discussion
The application-based cognitive screening test (ACST) has been shown to be an accessible, self-administered, brief, and valid tool for detecting cognitive dysfunction in older Thai adults. It needs further evaluation by standard evaluation if the test result is abnormal.
In comparison to other available Thai cognitive screening tests, ACST demonstrated a sensitivity of 87.1% and a specificity of 70% for detecting MCI and a sensitivity of 93.8% and a specificity of 82.9% for detecting dementia. The Mini-Cog [27] is a brief and valid tool, which was validated in a community setting. This tool had a sensitivity of 64.1% and a specificity of 93.5% for detecting MCI and a sensitivity of 90% and a specificity of 93.5% for detecting dementia. However, it requires face-to-face evaluation by healthcare professionals and has limited accuracy in people with no formal education or illiterate. The ACST contains higher sensitivity in MCI detection and can be self-administered.
The TMSE, which was validated at a geriatric clinic in Thailand, has demonstrated a sensitivity of 86.7% and a specificity of 80% for detecting dementia [29]. However, the screening performance in patients with MCI is limited in clinical practice, and no available data have been validated in patients with MCI, which are considered likely to be similar to MMSE in MCI. MMSE validated at a clinic in Thailand demonstrated a sensitivity of 13.8% and specificity of 100% for detecting MCI [30] and of 78.7% and 66.3% for detecting dementia [31]. The MMSE, validated at the Geriatric Clinic in the Netherlands [32], had a sensitivity of 64.2% and a specificity of 77.6% for detecting MCI and a sensitivity of 81% and a specificity of 92.4% for detecting dementia. The poor performance of MMSE in patients with MCI limited its use in this population. The MMSE score was previously reported to provide low sensitivity in patients with a low education level or who were illiterate. The MMSE is widely used in clinical practice; however, data on the validity of computerized versions of the MMSE remain limited. This study aimed to emphasize the utility of the application-based cognitive screening test (ACST) as a viable option for self-administered cognitive screening when traditional administration is not feasible.
MoCA-T, which was validated at a memory clinic, had a sensitivity of 80% and a specificity of 80% for detecting MCI and a sensitivity of 100% and a specificity of 98% for detecting dementia [25]. Systematic and meta-analysis data [33] showed that MoCA had a sensitivity of 83.8% and a specificity of 70.8% for detecting MCI (at a cutoff <25). Data from a geriatric clinic in Korea revealed that at a cutoff <25, MoCA had a sensitivity of 98% and a specificity of 61% for detecting dementia [34]. However, computer-based version of MoCA requires time to complete the test.
The results of our study align well with recent literature on the effectiveness of computerized cognitive assessments in early dementia diagnosis. Henkel et al. [35] emphasized that computerized tools not only improve diagnostic accuracy through repeated assessments and longitudinal data collection but also reduce variability and manual scoring errors common in traditional assessments. Their review highlighted that the highest-performing tools achieved sensitivity rates up to 91%, demonstrating their utility in clinical settings for detecting early cognitive changes [35]. Similarly, Zhang et al. [36] demonstrated the reliability of the Computerized Neurocognitive Battery for Chinese-Speaking participants (CNBC) as a culturally adapted, automated screening tool for Chinese-speaking populations, achieving strong correlations between computerized and traditional assessments in episodic memory and executive function. These findings support the utility of culturally adapted computerized assessments like the ACST in diverse populations. Furthermore, Chan et al. [37] showed that computerized assessments offer comparable accuracy to paper-and-pencil tests across verbal and visual memory domains, while also offering advantages in standardization and usability. Their meta-analysis reported diagnostic accuracies for MCI detection comparable to conventional methods, supporting our findings on the ACST’s effectiveness in distinguishing between cognitively normal, MCI, and dementia groups [37]. Together, these studies validate the role of computerized assessments in dementia care, highlighting their potential to facilitate early intervention and improve access to screening in diverse and resource-limited settings.
Compared with other computerized cognitive screening tests, the validity of the ACST was similar to that of other tests, and it required less time, with an average time to application test completion of approximately 3.1 min. For example, the Brain Health Assessment (BHA) [14] had a sensitivity of 84% and a specificity of 85% for detecting MCI and a sensitivity of 100% and a specificity of 85% for detecting dementia, requiring approximately 10 min to complete. The FACEmemory® [16] had a sensitivity of 80.5% and a specificity of 80.3% for detecting MCI. A novel computerized cognitive function test [38] had a sensitivity of 70.6% and a specificity of 87.5% for detecting MCI and a sensitivity of 100% and a specificity of 87.5% for detecting dementia. Both the FACEmemory® and novel computerized cognitive function tests require approximately 30 min to complete.
Our study has several strengths. First, this study was validated against the DSM-5 and CDR by an experienced geriatric neurologist to ensure accurate diagnosis. Second, the ACST has good validity for detecting MCI and dementia. Third, the ACST is a brief tool with an average time of only 3.1 min and can be accessed via smartphones and tablets. Fourth, the instructions are provided in both written and verbal formats to enhance their use in illiterate and hearing-impaired individuals.
Despite these strengths, our study has several limitations that warrant consideration. First, the participants in our study had relatively high educational levels. As a result, generalizability of the findings to populations with lower educational levels in the community may be restricted. Second, digital literacy is needed. Thus, some older adults cannot use this tool. However, this limitation can be found in most studies using technology-based applications. Third, the ACST cannot evaluate all domains affected in dementia because of limitations in the use of technology-based applications. However, in a cognitive screening test, memory and executive function are the most essential components. The paired associate test primarily assesses associative memory, a key aspect of episodic memory. The digit span test can be used to assess aspects of executive function, although it is primarily a test of working memory. The test involves recalling a series of digits in both forward and backward orders, tapping into various cognitive processes related to executive functions. The backward digit span, in particular, can evaluate attention, mental flexibility, and inhibition. Moreover, we intend to screen for common dementia syndromes, with AD, vascular dementia, and mixed AD with vascular lesions being the three most prevalent types of dementia in the general population. Moreover, the good validity in the detection of both MCI and dementia supports the use of this tool in clinical practice.
Conclusion
The application-based cognitive screening test (ACST) is an easy-to-use test that is not time-consuming, with an average time to complete of 3.1 min, exhibits good interrater/test-retest reliability, and is a valid tool for cognitive screening in older Thai adults. Thus, the ACST could be used in clinical practice when face-to-face evaluation is limited. A cutoff point ≤7 was the most appropriate score for distinguishing cognitively normal from non-normal patients. A cutoff point ≤6 was the most appropriate score for distinguishing cognitively normal from non-normal individuals among participants aged ≥75 years or ≤6 years.
Statement of Ethics
This study protocol was reviewed and approved by the Siriraj Institutional Review Board, Faculty of Medicine, Siriraj Hospital, Mahidol University, Bangkok, Thailand, Approval No. 314/2566 (IRB1), License number Si 397/2023. All participants received a detailed explanation of the study before participating in the study. Written informed consent was obtained from the participants and from the legal representatives of all patients with dementia before they joined the study.
Conflict of Interest Statement
The authors have no conflicts of interest to declare.
Funding Sources
This research project was supported by Siriraj Foundation, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok, Thailand. The funder had no role in the design, data collection, data analysis, and reporting of this study.
Author Contributions
Yangyuensathaporn B. and Muangpaisan W. involved in conceptualization, methodology, data analysis, and writing of final paper. Yangyuensathaporn B., Chansaengpetch S., Jongsawadipatana A., and Muangpaisan W. involved in data collection and validation process. All co-authors involved in writing of final paper. Muangpaisan W. involved in project administration and supervision.
Data Availability Statement
All data generated or analyzed during this study are included in this article and its online supplementary material files. Further inquiries can be directed to the corresponding author.