Impact of Different Operational Definitions on Mild Cognitive Impairment Rate and MMSE and MoCA Performance in Transient Ischaemic Attack and Stroke

In the first year after stroke, around a quarter of patients will have dementia, rates being lowest in those with first ever stroke and highest in those with recurrent events [1,2]. Mild cognitive impairment (MCI) is at least as prevalent as dementia after transient ischaemic attack (TIA)/stroke [3] and is increasingly recognised as an important outcome in observational studies and randomised trials. However, there is no consensus on how impairment should be defined, and numerous different criteria exist. Previous studies have shown that the use of different criteria to define cognitive impairment without dementia affects reported prevalence rates in epidemiological studies [4,5,6,7], memory clinic subjects [8] and Parkinson's disease [9]. This is analogous to the findings of varying dementia estimates obtained across different dementia diagnostic criteria in patients with cerebrovascular disease [10,11].

However, even within established criteria for cognitive impairment such as those described by Petersen and colleagues [12], there is uncertainty around how such criteria should be applied in practice including the threshold at which abnormality should be set, i.e. 1 standard deviation (SD) or greater below normal performance [9], which norms are used as the standard, the definition of cognitive domains (i.e. which tests should be used and whether single or multiple tests are required) [9] and in defining subjective memory (or cognitive) impairment or even whether this should be a requirement [4].

Lack of consensus on how to operationalise established criteria is important since it is likely to result in lack of comparability between studies and conflicting results regarding the performance of short cognitive tests which is problematic since such short tests are increasingly used as outcome measures in large pragmatic studies and clinical trials. We hypothesised that lower thresholds for defining impairment would result in higher measured rates of cognitive impairment in patients with TIA and stroke, higher optimal cut-offs on short cognitive tests and superiority of the Montreal Cognitive Assessment (MoCA) [13], which is sensitive to milder cognitive deficits, over the Mini Mental State Examination (MMSE) which was designed to detect moderate/severe cognitive impairment [14]. Conversely, higher thresholds would result in lower measured rates and similar performance of MoCA and MMSE in detecting impairment. We therefore aimed to determine the effect of different operational definitions within the Petersen-MCI criteria [12], a widely used definition of cognitive impairment without dementia, on rates of MCI and reliability of short tests (MMSE and MoCA) ≥1 year after TIA and stroke.

Patients were participants in the Oxford Vascular Study (OXVASC 2002 -), a prospective population-based cohort study of all acute vascular events occurring within a defined population of about 91,000 [15,16]. The study was approved by the local ethics committee (Oxfordshire Clinical Research Ethics Committee, CO.043), and informed consent was obtained. Between August 2009 and November 2010, consecutive patients were invited at their routine 1- or 5-year follow-up to undergo further cognitive testing with the National Institute of Neurological Disorders and Stroke-Canadian Stroke Network (NINDS-CSN) Vascular Cognitive Impairment Harmonization Standards Neuropsychological Battery [17] in addition to the MMSE [14] and MoCA [13] which were done routinely at the follow-up appointment. Some data on this cohort have been reported previously [3,18]. Further cognitive testing was performed by investigators blinded to the MMSE and MoCA results. Patients who had problems that interfered with testing, such as poor vision, severe hearing impairment, inability to use the right arm, dysphasia, poor English or acute illness, were excluded as described previously [19].

The NINDS-CSN Harmonization Standards [17] recognised that there are no perfect neuropsychological tests but recommended tests based on their wide usage, availability of published norms and ability to detect impairment in ‘executive/activation function, processing speed, word retrieval and episodic memory' of particular importance in vascular cognitive impairment. The battery took around 50-60 min to administer and included:

(i) Trail Test (parts A and B) [20,21];

(ii) Symbol Digit Modalities Test [22];

(iii) Boston Naming Test (30-item version) [21,23];

(iv) Rey-Osterrieth complex figure copy [24,25];

(v) Hopkins Verbal Learning Test-Revised [26,27];

(vi) letter (Controlled Oral Word Association Test [21,28] and category (animals) fluency [29].

The presence of a subjective memory complaint was examined using the question: ‘Do you think you have more problems with your memory than most?' Although ‘subjective memory complaint' was broadened to ‘subjective cognitive complaint' in the 2004 publication on the MCI criteria [12], we chose to identify memory rather than ‘cognitive' complaints owing to uncertainties in defining the latter in practice.

For MCI by the Petersen method, 4 subtypes of MCI were distinguished [12]:

(i) amnestic single domain: objective impairment of memory only;

(ii) amnestic multiple domain: memory and at least one other cognitive domain impaired;

(iii) non-amnestic single domain: one single domain other than memory impaired;

(iv) non-amnestic multiple domain: at least two cognitive domains impaired but not memory.

We examined the effect of different operational methodologies as follows:

(i) different thresholds (1, 1.5 and 2 SD below published norms as used in previous studies [9]) for defining impairment on a given test corrected for age and education;

(ii) using abnormal performance on a single test versus combinations of tests to define impairment in a given cognitive domain [4,9] (online suppl. methods and table S1; for all online suppl. material, see www.karger.com/doi/10.1159/000355496); test combinations for a given cognitive domain were selected on the basis of the NINDS-CSN Harmonization Standards recommendations [17];

(iii) requiring multiple- versus single-domain impairment;

(iv) with (Petersen-original criteria) and without (Petersen-modified criteria) requirement for a subjective memory complaint [4].

For all designations of cognitive impairment, subjects had no impairment of basic functional activities of daily living as measured by the Barthel index, and did not fulfil the DSM-IV dementia diagnostic criteria [30].

An additional point for low education (≤12 years) was added to the MoCA score in line with the original publication [13]. Reliability of MMSE and MoCA for MCI diagnosis by the different operational criteria was assessed using the area under the receiver-operating characteristic curve (AUC). AUC were calculated for all patients with any MCI (both single- and multiple-domain MCI) versus no MCI and then separately for those with single-domain MCI versus no MCI, multiple-domain MCI versus no MCI + single-domain MCI, multiple-domain MCI versus single-domain MCI and multiple-domain MCI versus no MCI for each operational definition. Sensitivities and specificities for various MoCA and MMSE cut-offs for identifying cognitive impairment were determined from the AUC curves.

Ninety-one non-demented patients completed MMSE, MoCA and Neuropsychological Battery assessments at a mean of 3.1 ± 1.9 years after the index event. TIA (n = 42) and stroke (n = 49) patients were similar in age (69.8 ± 9.0 vs. 69.7 ± 13.6 years, p = 0.97), male sex (55 vs. 63%, p = 0.52) and education <12 years (67 vs. 59%, p = 0.52).

Rates of cognitive impairment ranged from 14/91 (15%) for the most stringently defined cognitive impairment (MCI-original at 2 SD cut-off) to 61/91 (67%) for the least stringently defined impairment (MCI-modified, single test definition, at the 1 SD cut-off; table 1). MCI was more likely to be classed as multiple domain when criteria for impairment were less stringent, e.g. within the MCI-modified, multiple test definition, multiple-domain MCI was 24/46 (52%) at 1 SD cut-off compared to only 5/27 (20%) at 2 SD cut-off (table 1). Amnestic single-domain impairment was relatively rare irrespective of methodology ranging from 0 to 20% of total MCI (fig. 1; table 1). Regardless of the chosen cut points for defining abnormality (i.e. number of SD below norm), requirement for a subjective memory complaint at least halved the measured rate of impairment using the single test definition: 17 (19%) MCI-original versus 39 (43%) MCI-modified, at 1.5 SD cut-off. In contrast, use of a single test versus multiple tests to define cognitive domain impairment had less impact, although rates were lower when multiple tests were used (table 1, fig. 1).

Looking at TIA and stroke patients separately, MCI was more likely in stroke versus TIA patients: 27/49 (55%) versus 12/42 (29%), with OR = 3.06, 95% CI = 1.28-7.36, p = 0.01 for MCI-modified, 1.5 SD cut-off and single test definition, and 12/49 (24%) versus 5/42 (12%), with OR = 2.40, 95% CI = 0.77-7.50, p = 0.13 for MCI-original, 1.5 SD cut-off (online suppl. table S2). Multiple-domain MCI was more likely than single-domain MCI in stroke patients, whereas the opposite was seen in TIA patients.

AUC were generally higher for the MoCA than the MMSE for detecting MCI where subjective memory complaint was not required (MCI-modified), whereas the MMSE was more reliable at identifying MCI-original. More stringent cut-offs (1.5 or 2 SD) for defining MCI usually resulted in higher AUC. AUC were also higher for identifying multiple-domain MCI than for identifying single-domain MCI and were highest for multiple-domain MCI versus no MCI, i.e. when the difference in cognitive performance between the two groups being compared was greatest (table 2). Both MMSE and MoCA performed least well in discriminating between single-domain MCI and no MCI and between single- and multiple-domain MCI, i.e. when the difference between the two groups being compared was small, although AUC for MoCA were higher than for MMSE for MCI-modified. Although numbers were too small to be precise about the relative performance of the MMSE and MoCA for detecting MCI separately in TIA and stroke patients, AUC values were broadly similar (online suppl. table S3).

Optimal cut-offs ranged from MMSE <30 to <26 and MoCA <27 to <23 and were lower when requirements for cognitive impairment were more stringent, i.e. for MCI defined using 1.5 or 2 rather than 1 SD cut-offs below the norm or as multiple-domain impairment (table 3). The MMSE had low sensitivity for mild impairment compared to the MoCA with evidence of a marked ceiling effect.

Operational differences in applying Petersen-MCI criteria in patients after TIA and stroke have a major impact on measured rates of impairment and on the relative proportion of single- versus multiple-domain MCI and also on performance of short cognitive tests. Varying the threshold used to define abnormality and requiring a subjective memory complaint had greater impact than the use of a single versus multiple tests to define abnormality in a given domain.

Previous studies have described variable estimates of (vascular) dementia and cognitive impairment across different diagnostic criteria [4,5,6,7,8,9,10,11], but our study provides the first data on the impact of operational methodology on the measured rate of cognitive impairment without dementia within a given set of criteria (Petersen-MCI) in patients with cerebrovascular disease and on optimal cut-offs on short cognitive tests. We observed that the measured rate of MCI after TIA and stroke was highly dependent on operational methodology with lowest rates seen when cut-offs were most stringent, in keeping with studies in Parkinson's disease [9].

Amnestic single-domain impairment, common in memory research cohorts, was rare, in keeping with findings from clinical practice [8], whereas non-amnestic single-domain impairment was prevalent regardless of operational methodology. However, the relative proportions of MCI subtypes and in particular of multiple- versus single-domain impairment depended on the operational criteria used to define MCI with more stringent cut-offs reducing the proportion of multi-domain impairment. Although multiple- versus single-domain impairment has been shown to be associated with a greater risk of conversion to dementia on follow-up [31,32], our results suggest that this risk will vary according to the stringency of the MCI definition.

Regardless of operational definition, multiple-domain cognitive impairment was generally detected more reliably than single-domain impairment by short cognitive tests particularly by the MMSE perhaps unsurprisingly, since the MMSE was developed to detect moderate/severe cognitive impairment. The MMSE also performed well in detecting MCI-original (subjective memory complaint required) suggesting sensitivity to Alzheimer-type impairment presumably owing to an emphasis on memory, language and orientation. When subjective memory complaint was not a requirement, MCI rates were much higher and the MoCA performed well in detecting both single- and multiple-domain impairment consistent with its sensitivity to mild deficits not detected by the MMSE [9,20]. These findings suggest that use of MCI-modified and the MoCA rather than the MMSE are appropriate for studies of vascular cognitive impairment.

Given the reliability of short cognitive tests for detection of multi-domain impairment, short tests may be equally as good as the formal Neuropsychological Battery in identifying those at high risk of conversion to dementia. Further, it has been shown that some subjects with global cognitive impairment defined on short tests do not have MCI by formal criteria but are nevertheless at high risk of dementia [33]. If short cognitive tests are shown to have good prognostic value, this will be of great value in routine clinical practice and large studies where long neuropsychological batteries are not feasible.

Our results provide some explanation for conflicting results from previous studies regarding optimal cut-offs and relative performance of the MMSE and MoCA for detection of cognitive impairment after stroke. Godefroy et al. [34] performed the MoCA in the acute phase of stroke and compared it with the Neuropsychological Battery around 3 weeks later and found optimal cut-offs in the low 20s and that the specificity of the MoCA was moderate, the latter performing no better overall than the MMSE. Dong et al. [35] found optimal MoCA cut-offs of 21/22 when administered acutely for predicting cognitive impairment at 3-6 months. However, we found, in stable patients with TIA and stroke, that optimal MoCA cut-offs were around 25 and that the MoCA was more sensitive than the MMSE for milder deficits whilst retaining reasonable specificity [3].

The relatively low MoCA cut-offs and similar MMSE and MoCA performance in the studies by Godefroy et al. [34] and Dong et al. [35] could be explained in part by the requirement for multiple-domain/moderate to severe impairment, whereas we included single-domain impairment - our optimal MoCA and MMSE cut-offs for detecting multiple-domain impairment are not that dissimilar. Also, the Godefroy and Dong studies used the MoCA acutely with neuropsychological evaluation some weeks/months later, and significant cognitive recovery is likely to have occurred between testing points - even minor cerebrovascular events are known to cause transient cognitive impairment even in the absence of overt delirium [36]. Lastly, we included TIA patients as well as those with stroke, whereas the Dong and Godefroy studies were confined to stroke patients.

Our study has some limitations. First, we used a relatively small sample of stable patients with TIA and stroke, and patients with dysphasia and severe stroke were not included. Second, we did not determine the effect of informant report of cognitive decline on the measured rate of impairment. Third, we were not able to examine the effects of different psychological tests and norms within a given domain owing to the need to be pragmatic in testing older and sometimes frail patients, but this is likely to be an additional factor in comparing results between studies. Finally, we selected tests according to the Harmonisation Standards recommendations [17] which are biased towards verbal functioning and thus non-verbal functions, particularly memory, were not assessed in detail in our study and this may have impacted on the findings.

In conclusion, even within established criteria for MCI, differences in operational definitions resulted in 4-fold variation in MCI estimates. In addition, optimal MMSE and MoCA cut-offs were lower, and reliability of the two tests was more similar, when requirements for MCI were more stringent. Our findings have implications for measuring cognitive impairment outcomes in randomised trials and observational studies in terms of sample size and adjusted relative risk calculations and choice of cut-offs on short cognitive tests. Our data should also facilitate the interpretation of reported MCI rates from across different studies.

The Oxford Vascular Study is funded by the Wellcome Trust, Wolfson Foundation, British Heart Foundation, UK Stroke Association, the Dunhill Medical Trust, the National Institute of Health Research (NIHR), the Medical Research Council, and the NIHR Oxford Biomedical Research Centre. Dr. Pendlebury is supported by the NIHR Biomedical Research Centre, Oxford. Prof. Rothwell is an NIHR Senior Investigator and a Wellcome Trust Senior Investigator.

Competing interests: none declared.