Objective: The aim of this study is to assess the utility of the Cogstate self-administered computerized neuropsychological battery in a large population of older men. Methods: We invited 7,167 men (mean age of 75 years) from the Health Professionals Follow-up Study, a prospective cohort of male health professionals. We considered individual Cogstate scores and composite scores measuring psychomotor speed and attention, learning and working memory and overall cognition. Multivariate linear regression was used to assess the association between risk factors measured 4 and 28 years prior to cognitive testing and each outcome. Results: The 1,866 men who agreed to complete Cogstate testing were similar to the 5,301 non-responders. Many expected risk factors were associated with Cogstate scores in multivariate adjusted models. Increasing age was significantly associated with worse performance on all outcomes (p < 0.001). For risk factors measured 4 years prior to testing and overall cognition, a history of hypertension was significantly associated with worse performance (mean difference of -0.08 standard units (95% CI -0.16, 0.00)) and higher consumption of nuts was significantly associated with better performance (>2 servings/week vs. <1 serving/month: 0.15 (0.03, 0.27)). Conclusions: The self-administered Cogstate battery showed significant associations with several risk factors known to be associated with cognitive function. Future studies of cognitive aging may benefit from the numerous advantages of self-administered computerized testing.
With an aging global population, the public health burden of dementia is expected to rise rapidly in the near future. Increasing attention must be placed on dementia research to identify new risk factors and interventions. While technological advances in medicine, particularly in neuroimaging and genetics, have already made valuable contributions to our understanding of the disease [1,2,3], similar advances in the effective measurement of cognitive outcomes have not progressed as quickly.
Epidemiologic studies of cognitive aging typically rely on neuropsychological tests, which can not only provide a breadth of data on cognitive function, but also require trained interviewers (introducing both inter- and intra-interviewer variability) as well as substantial time and cost on the part of both investigators and study participants. In contrast, computerized cognitive testing offers numerous advantages over traditional neuropsychological testing such as substantially increased cost-efficiency and convenience, accurate response time measurement and decreased susceptibility to sources of human error such as interviewer bias .
The Cogstate brief battery, a computerized series of neuropsychological tests , has demonstrated good validity and high test-retest reliability in cognitively normal older adults as well as in those with mild cognitive impairment (MCI) or dementia [5,6,7,8]. The Cogstate battery is also sensitive enough to detect subtle cognitive decline over 12 months in a population of older adults with MCI  and to differentiate between normal cognitive function, MCI and Alzheimer's disease . However, prior studies using the Cogstate battery in older populations are in predominately small samples and often used only in supervised clinical or research settings. While unsupervised self-administration (e.g. from the participant's home) can maximize efficiency and convenience, to our knowledge, no prior studies have involved unsupervised self-administration in a large population. As a valuable future supplement or alternative to traditional methods of neuropsychological testing, it remains of interest to examine the feasibility and performance of self-administered computerized testing in a large-scale setting, particularly in older populations. We, therefore, aimed to evaluate the usability and distribution of scores of the Cogstate brief battery in a population of older adults.
The Health Professionals Follow-up Study is an ongoing longitudinal study that began in 1986, when we recruited 51,529 men aged 40-75 years in allied health professions (dentists, pharmacists, optometrists, osteopath physicians, podiatrists and veterinarians). Participants were originally recruited via mailed questionnaires, with follow-up data collected using biennial questionnaires. Health and lifestyle data for the present study was collected using the 2010 questionnaire, allowing a slight lag between risk factor evaluation and cognitive assessment to reduce the possibility of reverse causation. Computerized cognitive testing was conducted in 2014, when email invitations to complete testing were sent out to the 7,167 men who had completed the 2014 mailed questionnaire and had email addresses available.
Covariates were chosen a priori based on risk factors known to be associated with cognitive function in prior literature. Age was calculated from self-reported date of birth. Body mass index (BMI, kg/m2) was calculated from self-reported height and weight (<22, 22-24.9, 25-29.9 and ≥30). Lifestyle factors included smoking status (never, former or current) and physical activity measured as estimated mean energy expended per week (quartiles of metabolic equivalents per week) using a validated physical activity questionnaire . Dietary factors, recorded using a validated semi-quantitative food frequency questionnaire [11,12], included current multivitamin use, alcohol intake (none, 1-2 servings/day, >2 servings/day), nut intake (<1 serving/month, 1-3 servings/month, 1-2 servings/week, >2 servings/week), fish intake (<1 serving/month, 1-3 servings/month, 1-2 servings/week, >2 servings/week) and total energy intake (kcal/day). Because extensive dietary data were not available from the 2010 questionnaire, alcohol and nut intake were recorded using data from the 2006 questionnaire. Comorbidities included a history of self-reported physician diagnosis of diabetes, hypertension and myocardial infarction. Because of the potential importance of mid-life factors, we also collected information on all covariates from the 1986 questionnaire.
Cognitive function was measured using the self-administered Cogstate computerized battery . Participants used a desktop or laptop computer to complete testing on the Cogstate web site using a Flash-based application (Adobe Systems, San Jose, Calif., USA). The Cogstate battery comprises 4 tasks taking approximately 15-20 min to complete in total. At the beginning of each task, participants view instructions for each task, perform a practice trial for that task and then are given the actual task to complete. All tasks involve images of playing cards, due to their familiarity to most ages and cultures. Each task requires participants to respond to the playing cards, using the ‘K' and ‘D' keys on their computer keyboard, which correspond to a ‘Yes' or ‘No' response, respectively. Descriptions of each of the 4 tasks are presented below, with participants performing the tasks in the order presented.
The Detection Task (DET) measures psychomotor function and information processing speed. The participant views a series of joker playing cards on the screen turn over. When a card turns over, the participant must then press the ‘Yes' key as quickly as possible.
The Identification Task (IDN) measures visual attention and vigilance. The screen shows red or black joker cards flipping over, and participants press the ‘Yes' and ‘No' keys as quickly as possible to note the red cards (i.e. ‘Yes' if red, ‘No' if black).
The One Card Learning Task (OCL) measures visual learning and short-term memory. A series of playing cards is flipped over on the screen one at a time. Each time a card is revealed, the participant must then respond ‘Yes' or ‘No' to note whether that card has been previously shown at any time during the task.
The One Back Task (ONB) is designed to measure attention and working memory. A series of playing cards is flipped over on the screen one at a time. When each card is revealed, the participants respond with ‘Yes' or ‘No' to note whether the card is the same as the previous card.
For the DET, IDN and ONB tasks, scores are the log10 transformed mean response times of correct trials, while for the OCL task, scores are the arcsine of the square root of the proportion of correct responses (transformations are applied to normalize the distribution). Therefore, for the DET, IDN and ONB tasks, a lower score indicates better performance, while for the OCL task, a higher score indicates better performance. Response times are computed on the participant's local computer and sent remotely after testing is completed. In addition to assessing each task individually, we also created composite scores since composite measurements may increase power through increased precision and sensitivity [13,14]. Composite scores were created by averaging the z-scores of scores from individual tasks. Three composite scores were created: (1) DET and IDN, to measure psychomotor speed and attention; (2) OCL and ONB, to measure learning and working memory; and (3) all 4 tests, as a measure of overall cognition.
We created summary statistics to describe the mean, variability and range of scores on each Cogstate task. Univariate analyses were done to assess associations between response or non-response and the risk factors associated with cognitive function. The chi-square test was used for categorical variables. We used the Kruskal-Wallis test for continuous variables because all continuous variables were non-normally distributed. To evaluate the association between risk factors and scores on each task or composite score, we conducted a separate multivariate linear regression model for each cognitive outcome. Covariates for all models included age, smoking status, BMI, physical activity, alcohol intake, nut intake, total energy intake, diabetes, hypertension and myocardial infraction. Because mid-life factors may be important predictors for late life cognitive function, we conducted an additional analysis using risk factors measured at mid-life in 1986. In addition, for primary analysis, we used integrity criteria to exclude any Cogstate scores that were below the established thresholds , which were (in percent of trials correct) 80% for the DET, 80% for IDN, 50% for OCL and 70% for ONB tasks. In a secondary analysis, we used more inclusive integrity criteria, excluding only those scores for which participants scored 0% correct on a task. Linear tests of trend for ordinal variables (physical activity, nut intake and fish intake) were conducted by modeling the median value of each category as a continuous variable. All analyses were performed using the SAS 9.3 (SAS Institute Inc., Cary, N.C., USA).
Baseline characteristics of all participants invited to complete Cogstate testing are shown in table 1. Men ranged in age from 63 to 95 years (mean of 71 years). Among the 7,167 men who were invited to participate, 1,866 men (26%) conducted Cogstate testing. Overall, the differences were small between men who responded and those who did not respond. On average, men who did not respond compared to those who did respond were significantly more likely to be older (mean = 71.3 and 70.2 years, respectively) and have a history of hypertension (prevalence = 58.7% and 54.5%, respectively). In addition, among those who responded, 15% (279 men) did not pass the integrity criteria for at least 1 task (99 did not pass integrity criteria on all 4 tasks and were excluded from analysis). The number of integrity failures were generally greater for the more difficult tasks (DET: n = 123, IDN: n = 154, OCL: n = 197, ONB: n = 168). When using more inclusive integrity criteria (flagging participants who scored 0% correct on a task), this pattern reversed (DET: n = 94, IDN: n = 28, OCL: n = 0, ONB: n = 8). Compared to men who passed integrity criteria on all tasks, men who scored below integrity criteria on at least 1 task were on average, older, reported lower nut consumption and had a history of hypertension and diabetes.
The distribution of scores for each task, by age at testing, is shown in figure 1. For the entire sample, mean scores for each task were 2.59 ± 0.09 for DET, 2.73 ± 0.07 for IDN, 1.00 ± 0.11 for OCL and 2.92 ± 0.09 for ONB. Ranges for each score were 2.34-3.17 for DET, 2.55-3.18 for IDN, 0.79-1.38 for OCL and 2.70-3.36 for ONB. Scores for the reaction time-based scores of DET, IDN and ONB tasks demonstrated a slight positive skew.
Association between Participant Characteristics and Cogstate Scores
Results showing the associations between risk factors and scores on individual tasks are shown in table 2. As expected, older men had significantly worse mean scores on all cognitive outcomes (p < 0.001). Men with higher BMI had significantly worse mean scores on the OCL task (30+ vs. 22-24.9 kg/m2: mean difference -0.021 points (95% CI -0.040, -0.002); 25-29.9 vs. 22-24.9 kg/m2: mean difference -0.016 (-0.029, -0.002)) and better mean scores on the ONB task (30+ vs. 22-24.9 kg/m2: mean difference -0.023 (-0.037, -0.008); 25-29.9 vs. 22-24.9 kg/m2: mean difference -0.012 (-0.022, -0.002)). Men who reported to consuming 1-2 drinks/day had significantly better mean scores on the DET task compared to non-drinkers (mean difference -0.012 (-0.022, -0.002)). On an average, men with a history of hypertension had worse scores on the IDN task compared to those without a history of hypertension (mean difference 0.007 (0.000, 0.013)). Men who reported more frequent nut intake generally had better mean scores on the OCL task. Although the p value for a linear trend was not significant (p = 0.59), there was a borderline significant threshold effect (1-3 servings/month vs. <1 serving/month: mean difference 0.017 (-0.007, 0.041); 1-2 servings/week vs. <1 serving/month: mean difference 0.022 (0.003, 0.042); >2 servings/week vs. <1 serving/month: mean difference 0.018 (-0.001, 0.038)). Results remained similar when using more inclusive integrity criteria.
Associations between risk factors and composite scores are shown in table 3. Increased nut intake was associated with higher mean scores on overall cognition (>2 servings/week vs. <1 serving/month: 0.15 standard units (0.03, 0.27); 1-2 servings/week vs. <1 serving/month: 0.12 standard units (0.00, 0.25); 1-3 vs. <1 serving/month: 0.04 standard units (-0.11, 0.18); p trend = 0.02). For men with the most frequent nut intake compared to men with the least nut intake, this difference was equivalent to an approximately 5-year difference in age. On an average, men with a history of diabetes had significantly worse scores for the composite outcome of psychomotor speed and attention (-0.16 standard units (-0.32, 0.00)), equivalent to an approximately 5-year difference in age compared to men without diabetes. Lastly, men reporting a history of hypertension had, on average, significantly worse scores on the composite outcomes of learning and working memory (-0.08 standard units (-0.16, 0.00)) and overall cognition (-0.08 standard units (-0.15, -0.01)), approximately equal to a 2-year increase in age.
When assessing the association between mid-life risk factors measured in 1986 and composite Cogstate scores, associations for some risk factors changed (table 4). In contrast to the main analysis, men with higher levels of physical activity had better mean scores on learning and working memory (p trend = 0.02) and overall cognition (p trend = 0.049). In addition, we observed a threshold effect where men who consumed fish more frequently had higher mean scores on the composite outcome of psychomotor speed and attention (>2 servings/week vs. <1 serving/month: 0.28 standard units (0.09, 0.46); 1-2 servings/week vs. <1 serving/month: 0.31 standard units (0.13, 0.49); 1-3 servings/month vs. <1 serving/month: 0.29 standard units (0.08, 0.49)). We observed a similar threshold effect for overall cognition, with men who consumed the most fish compared to men who consumed the least having on average better scores equivalent to a 5-year difference in age (>2 servings/week vs. <1 serving/month: 0.15 standard units (0.00, 0.29); 1-2 servings/week vs. <1 serving/month: 0.15 standard units (0.01,0.29); 1-3 servings/month vs. <1 serving/month: 0.13 standard units (-0.03, 0.29)). Similar to the main analysis, men who reported the most frequent nut intake (>2 servings/week) had better mean scores for overall cognition compared to men who reported the least frequent nut intake (<1 serving/month), but this association only reached borderline significance (0.08 standard units (-0.02, 0.19)). When using more inclusive integrity criteria, results were similar.
To our knowledge, this is the first large, population-based study to conduct unsupervised, self-administered, computerized cognitive testing in older adults. Although the participation rate was low in these older men, characteristics of participants who responded were generally similar to those who did not, suggesting that low participation is not differentially attributed to risk factors for cognitive decline; this is important since it suggests that non-participation would reduce sample size but would not introduce meaningful bias into research findings. In addition, several factors known to be associated with cognitive function were significantly associated with Cogstate scores, supporting the validity of the battery in measuring several cognitive domains.
In addition to the low participation, the proportion of participants who did not complete a Cogstate task above integrity criteria (15%), as well as participant feedback during data collection (e.g. confusion regarding task instructions), suggest the need for clear and unambiguous instructions and a user interface properly optimized for an older population in self-administered testing. As prior studies in older adults suggest a willingness or even preference for digital interfaces in primary data collection [15,16,17], an appropriate interface and instructions can be vital to maximize the task completion for a population who may be limited by sensory impairments and/or a low level of computer literacy. This should be a clear priority in future research involving self-administered cognitive evaluation in older adults.
Nonetheless, the distribution of scores on Cogstate tasks was generally similar to those reported in prior studies using trained, cognitively normal older participants. Mean scores on the IDN task in our study (2.73 ± 0.07) were very similar compared to those in prior studies (Fredrickson et al. : n = 301, mean age = 61.9 ± 7.2, mean score = 2.72 ± 0.07; Lim et al. : n = 15, mean age = 73.6 ± 6.9, mean score = 2.73 ± 0.06; and Hammers et al. : n = 23, mean age = 68.4 ± 9.5, mean score = 2.73 ± 0.08). The OCL and ONB tasks, despite being the most difficult and thus likely to have greater variability, also had very similar mean scores compared to prior studies [5,6,18]. In contrast, mean scores for the DET task in our study (2.59 ± 0.09) were slightly worse than those reported in other studies of cognitively normal older adults (Hammers et al. : mean score = 2.50 ± 0.11, Fredrickson et al. : mean score = 2.52 ± 0.11, Lim et al. : mean score = 2.56 ± 0.10) and more similar to scores among participants with MCI (Hammers et al. : n = 20, mean age = 73.5 ± 5.9, mean score = 2.52 ± 0.08; Lim et al. : n = 47, mean age = 78.9 ± 6.9, mean score = 2.59 ± 0.12). Although such differences may be attributed to random chance, it is also possible that worse performance on DET, the simplest task, reflected difficulties with task comprehension, given that this task was both administered first and had the highest proportion of participants scoring 0% correct.
A possible limitation of this study is the relative homogeneity and high education level of the population. Thus, participation and performance on the Cogstate battery and participants' level of task comprehension may differ in other older populations. It is also possible that the low participation and somewhat low task comprehension may not be generalizable to somewhat younger populations. However, task comprehension will likely improve in future cohorts as computer literacy increases in older adults . Additionally, although prior studies of the Cogstate battery in older adults demonstrate good correlation with performance on other neuropsychological test instruments [8,20], the Cogstate battery may not adequately measure some cognitive domains such as executive functions or semantic and verbal fluency.
The Cogstate self-administered test showed promising results, with performance on Cogstate tasks significantly associated with several known risk factors for cognitive decline. Further studies to establish psychometric standards and normative data in different populations would be helpful to promote more widespread application in clinical and research settings.
Acknowledgments and Funding
This work was supported by the National Institute of Health (UM1 CA167552, T32 MH017119) and the International Nut and Dried Fruit Council Federation.