Superior Test-Retest Reliability of Cognitive Assessment with Cognivue® vs Slums During an 18-Month Longitudinal Study View PDF

The US population living with cognitive impairment conditions is expected to grow in the years to come, as the population ≥65 years of age balloons from 53 million in 2018 to 88 million by 2050 [1]. The presence of these conditions increases the burden on the health system due to higher rates of long-term care utilization, per-member payments for insurance beneficiaries, and out-of-pocket spending [1]. Cognitive issues may be related to a number of causes, including life-style-related risk factors, underlying disease states, and prescription drug side effects [1-5].

Neuropsychological testing allows healthcare providers an opportunity to intervene in some cases of cognitive issues [1]. When testing uncovers mild cognitive impairment, healthcare providers can take steps to optimize existing cognitive function by accessing therapies that may preserve cognition; manage symptoms, medications, and comorbid conditions; and plan for future care [1-3,5]. Early detection of cognitive issues is key to successful intervention, with research indicating approximately 35% of dementias are attributable to modifiable risk factors [3]. By the time functional impairment manifests in people with cognitive issues, it may be too late to intervene and treat underlying disease processes [6].

Research has shown that structured cognitive assessment tools are more effective in detecting mild cognitive impairment or dementia vs spontaneous detection by primary care providers [7]. Traditional cognitive assessment methods such as the St. Louis University Mental Status (SLUMS) examination, Mini-Mental Status Examination (MMSE), and Montreal Cognitive Assessment (MoCA) rely on a pen and paper format, but these tests have limitations that may affect test scores, including subjectivity, the potential for respondents to learn answers with repeated use, and an inability to track small changes in cognitive ability over time [8-10].

Cognivue is a computerized testing tool rooted in adaptive psychophysics that is US Food and Drug Administration (FDA)-cleared for use as an adjunctive aid to assess cognitive function in individuals between 55 and 95 years of age [11]. Cognivue is not intended to be used alone for diagnostic purposes.

Cognivue bypasses the limitations of other structured neuropsychological tests. The 10- minute, self-administered test uses scores from a sequence of tasks to provide clinicians and patients with a simple, easy-to-read 2-page report with an overall score and a subsequent breakdown into 6 key cognitive domains and 2 speed parameters [11] (Figure 1). The overall Cognivue^® score as well as those from the individual sub-tests of the perception and memory sub-batteries are expressed as a percentage of correct responses (0 to 100%) [11].

In an FDA-regulated clinical study for device clearance, Cognivue^® demonstrated good agreement with the St. Louis University Mental Status (SLUMS) and other neuropsychological tests, and superior test re-test reliability compared to SLUMS across 2 sessions, 1 to 2 weeks apart (Cognivue regression fit: R² = 0.81, r = 0.90); SLUMS regression fit: R² = 0.67, r = 0.82) [12].

In the current analysis, we sought to determine the test-retest reliability of Cognivue^® vs SLUMS across multiple sessions over the course of 18 months. A separate sub-analysis of medical records sought to determine the impact, if any, of co-morbidities and/or medication usage on cognitive function.

Study Design

Two datasets, those from the FDA pivotal trial and the longitudinal study, were compared in this analysis.

The FDA pivotal trial data set consisted of 358 subjects who completed Cognivue and a battery of pen and pencil tests across 2 separate sessions 1 to 2 weeks apart [12]. Pen and pencil tests included SLUMS, SLUMS-clock drawing, SLUMS-animal naming, the Geriatric Depression Scale (GDS), and others [12]. Subjects in the FDA pivotal trial were adults (55-95 y) from independent-living communities at risk for age-related cognitive decline or dementia [12]. Exclusion criteria included limiting motor or visual disabilities and/or the ability to provide informed consent [12]. Rank linear regression analysis and factor analysis were performed to assess score psychometrics vs other neuropsychological tests [12].

The longitudinal data set consisted of 238 unique subjects that had participated in the FDA pivotal trial. Subjects in the longitudinal data set underwent the Cognivue test and the SLUMS, SLUMS-clock drawing, SLUMS-animal naming, and GDS tests at up to 5 sessions over the course of 18 months. Sessions 1 and 2 were 1 week apart and were in addition to the FDA sessions. These were followed by sessions at 6, 12, and 18 months. The same regression analysis methods used in the FDA pivotal trial were applied to results in the longitudinal study.

SLUMS was considered the reference standard. SLUMS consist of an 11-item questionnaire with scores ranging from 0 to 30 and is designed to measure orientation, memory, attention, and executive functions [12]. The Cognivue quantitative assessment tool includes 3 sub- batteries (visuo-motor ability, perceptual processing, and memory processing) presented in automated sequence over 10 minutes [12]. Additional descriptive aspects and sample screens of the individual sub-tests are shown in Table 1.

A sub-analysis of medical records was also performed to determine the correlation, if any, between comorbidities or medication usage and Cognivue score. Patients in this data set participated in a follow-up study after participating in the FDA-regulated clinical study for device clearance (1 or 2 visits) and/or the longitudinal study (up to 5 visits). The earliest visit from both studies was considered the subject’s baseline and was then compared to their follow- up study test.

Adapted from Cahn-Hidalgo D, et al. (2020) [12]

Subjects sent medical records for supplemental data. After this was cleaned and formatted, the following data were analyzed:

• Co-morbidities, organized into 5 categories (endocrine-, cardiopulmonary-, mood disorder-, sleep disorder-, or bone-related in nature).
• Medications taken, organized into 14 categories.
• Surgeries undergone, organized into 20 categories.
• Standard statistics, including body mass index, blood pressure, and blood work.

Test-retest reliability was determined through analysis of the correlation (r) and linear regression (r²) between variables. A Deming’s regression technique was also used to gauge variability between dependent and independent variables. Analyses compared test-retest reliability for Cognivue and SLUMS for test sessions 1 and 2, 2 and 3, 3 and 4, 1 and 3, and 1 and 4. Additional comparisons of Cognivue results between test sessions 1 and 5 and 4 and 5 were performed.

For the medical records sub-analysis, paired t-tests were used to determine statistical significance between mean original and follow-up Cognivue scores. Analyses also sought to determine correlation between comorbidities/medication usage and Cognivue scores.

Test-Retest Reliability Analyses

In progressive test session comparisons, test-retest reliability analyses showed similar scores across repeated testing with Cognivue (Table 2; test session 1&2: regression fit: R² = 0.76; r = 0.87; test session 2&3: regression fit: R² = 0.73; r = 0.85; test session 3&4: regression fit: R² = 0.78; r = 0.88). The SLUMS test demonstrated greater variability in regression scores across test sessions (test session 1&2: regression fit: R² = 0.63; r = 0.79; test session 2&3: regression fit: R² = 0.53 r = 0.73; test session 3&4: regression fit: R² = 0.60; r = 0.77). In an additional comparison of Cognivue results between test sessions 4 and 5, Cognivue demonstrated regression fit of R² = 0.74 and correlation of r = 0.86. In comparisons between Cognivue and SLUMS, Cognivue demonstrated superior test-retest reliability by a significant margin (Figure 2).

Using test session 1 as a baseline, Cognivue showed similar scores in test re-test reliability analysis (Table 3; test session 1&3: regression fit: R² = 0.72; r = 0.85; test session 1&4: regression fit: R² = 0.73; r = 0.86). The SLUMS showed greater variability in regression scores vs test session 1 (test session 1&3: regression fit: R² = 0.43 r = 0.65; test session 1&4: regression fit: R² = 0.64; r = 0.80). In an additional comparison of Cognivue results between test sessions 4 and 5, Cognivue demonstrated regression fit of R² = 0.74 and correlation of r = 0.86. In an additional comparison of Cognivue results between test sessions 1 and 5, Cognivue demonstrated regression fit of R² = 0.68 and correlation of r = 0.83 (Figure 3).

Table 2: Correlation and regression across test sessions.

Table 3: Correlation and regression vs first test session.

Medical Records Sub Analysis

In the sub analysis, medical records of 203 subjects who participated in the longitudinal study and/or the FDA pivotal trial were analyzed. Subjects had an average number of 5.0 comorbidities, with cardiopulmonary comorbidities being the most common (Table 4).

Overall, an increased co-morbidity count significantly decreased subjects’ Cognivue scores (correlation -0.21; P=0.01; Table 5). Negative correlation values indicate that if a subject has a co-morbidity in that category, their Cognivue score decreases. Among comorbidity categories, cardiopulmonary comorbidities had the largest impact on a patient’s Cognivue score (78.0 average score for those without this comorbidity vs 67.1 average score for those with; P<0.001).

Subjects in the sub analysis took an average number of 6.1 medications. Of these medications, use of anti-HTN medications was significantly correlated with a decrease in subjects’ Cognivue scores (correlation -0.2; P=0.02).

Table 4: Comorbidity categories by subject count.

Table 5: Correlation between comorbidities and Cognivue score.

This study demonstrated that Cognivue yields similar test scores when administered over the course of 18 months. When variability in Cognivue scores was compared with that of SLUMS, Cognivue demonstrated superior test-retest reliability.

Cognitive impairment is typically progressive, beginning with age-related cognitive decline and moving across a continuum to mild cognitive impairment on to dementia [13]. Because age-related diseases affect the structure and function of the brain and neural networks, the risk of dementias, including Alzheimer’s disease, and other cognitive dysfunctions rises as a person ages [1,14]. The aging US population can be expected to increase the rate of these conditions, which can result in more hospital stays, increased long-term care utilization, and higher spending [1].

Early neuropsychological testing allows healthcare professionals to identify cognitive impairment and intervene to maintain cognitive function [1]. Interventions such as stroke prevention and reduction of vascular risk factors may reduce risk of mild cognitive impairment progressing to dementia, while other conditions such as depression and sleep apnea can be addressed to minimize negative impacts on cognitive function [1]. Identification of cognitive impairment may also prompt reassessment of pharmacotherapies that may result in cognitive issues [1-3,5]. Research has shown that non-pharmacologic interventions can significantly decrease odds of institutionalization for people with dementia and significantly increase time to institutionalization for these patients [15].

In the clinical setting, repeated neuropsychological testing over time allows healthcare providers the opportunity to track worsening or improvement of cognitive function [16]. When coupled with therapeutic intervention, results from these tests may reveal the efficacy-or lack thereof-of the current treatment plan [16]. Unfortunately, multiple studies have demonstrated that limitations in available neuropsychological tests may result in practice effects, in which patients’ performance improves with repetitions over time [16,17]. Research suggests that these effects are related to the format of the tests themselves, which may result in test-retest variability that reduces clinician’s ability to objectively assess change in a patient’s cognitive function [17].

Cognivue uses a proprietary algorithm that dynamically adapts to user response [11]. This eliminates the possibility of memorization for subjects, thus delivering reliable test results over time. Because the device is uniformly calibrated and automatically scores performance, Cognivue also helps healthcare providers avoid subjectivity due to human error in scoring.

Patient scores are archived in a database, and can be recalled for direct comparison across repeated testing sessions.

This longitudinal study demonstrated that Cognivue maintained superior test re-test reliability compared to SLUMS over up to 5 test sessions in a period of 18 months after the FDA-regulated trial for device clearance. A sub analysis of medical records showed that an increased comorbidity count and cardiopulmonary comorbidities significantly decreased a subject’s Cognivue score.

DEN130033.

The study was reviewed and approved by the Western Institutional Review Board (WIRB). All study participants provided informed written consent about personal and medical data collection prior to study enrolment.

1. Alzheimer’s Association (2018) 2018 Alzheimer's disease facts and figures. Alzheimers Dement 14: 367-429. https://doi.org/10.1016/j.jalz.2018.02.001
2. World Health Organization (2019) Risk reduction of cognitive decline and dementia: WHO guidelines. Switzerland.
3. Livingston G, Sommerlad A, Orgeta V, Costafreda SG, Huntley J, et al. (2017) Dementia prevention, intervention, and care. Lancet 390: 2673-2734.https://doi.org/10.1016/S0140-6736(17)31363-6
4. American Diabetes Association (2019) 12. Older adults: standards of medical care in diabetes—2019. Diabetes Care 42: S139-S247. https://doi.org/10.2337/dc19-S012
5. Foster NL, Bondi MW, Das R, Foss M, Hershey LA, et al. (2019) Quality improvement in neurology: Mild cognitive impairment quality measurement set. Neurology 93: 705-713. https://doi.org/10.1212/WNL.0000000000008259
6. Petersen RC, Roberts RO, Knopman DS, Boeve BF, Geda YE, et al. (2009) Mild cognitive impairment: ten years later. Arch Neurol 66: 1447-1455. https://doi.org/10.1001/archneurol.2009.266
7. Cordell CB, Borson S, Boustani M, Chodosh J, Reuben D, et al. (2013) Alzheimer’s Association recommendations for operationalizing the detection of cognitive impairment during the Medicare Annual Wellness Visit in a primary care setting. Alzheimers Dement 9: 141-150. https://doi.org/10.1016/j.jalz.2012.09.011
8. Connor DJ, Jenkins CW, Carpenter D, Crean R, Perera P (2018) Detection of rater errors on cognitive instruments in a clinical trial setting. J Prev Alzheimers Dis 5: 188-196. https://doi.org/10.14283/jpad.2018.20
9. Collie A, Darby D, Maruff P (2001) Computerised cognitive assessment of athletes with sports related head injury. Br J Sports Med 35: 297-302. http://dx.doi.org/10.1136/bjsm.35.5.297
10. Freddi Segai-Gidan PA (2013) Cognitive screening tools. Clin Rev 23: 12-18.
11. US Food and Drug Administration (2013) De novo classification request for cognivue. De Novo Summary (DEN130033). United States.
12. Cahn-Hidalgo D, Estes PW, Benabou R (2020) Validity, reliability, and psychometric properties of a computerized, cognitive assessment test (Cognivue®). World J Psychiatry 10: 1-11. https://dx.doi.org/10.5498/wjp.v10.i1.1
13. Pottie K, Rahal R, Jaramillo A, Birtwhistle R, Thombs BD, et al. (2016) Recommendations on screening for cognitive impairment in older adults. CMAJ 188: 37-46. https://doi.org/10.1503/cmaj.141165
14. Murman DL (2015) The impact of age on cognition. Semin Hear 36: 111-121. https://doi.org/10.1055/s-0035-1555115
15. Spijker A, Vernooij?Dassen M, Vasse E, Adang E, Wollersheim H, et al. (2008) Effectiveness of nonpharmacological interventions in delaying the institutionalization of patients with dementia: a meta?analysis. J Am Ger Soc 56: 1116-1128. https://doi.org/10.1111/j.1532-5415.2008.01705.x
16. Bartels C, Wegrzyn M, Wiedl A, Ackermann V, Ehrenreich H (2010) Practice effects in healthy adults: a longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience 11: 118. https://doi.org/10.1186/1471-2202-11-118
17. Wilson RS, Li Y, Bienias L, Bennett DA (2006) Cognitive decline in old age: separating retest effects from the effects of growing older. Psych Aging 21: 774-789. https://doi.org/10.1037/0882-7974.21.4.774