Computerized Neurocognitive Tests in Clinical Practice

There are 3 ways to assess a patient’s cognitive status: an easy way that is fast but unreliable; a formal approach that is definitive but expensive; and a number of alternatives in between. The first is based on a psychiatrist’s observations of a patient’s memory, attention, and thought processes during the mental status examination, complemented sometimes by symptom checklists for the patient or knowledgeable observers. The second is a gold standard formal neuropsychological evaluation. It is hard to get sometimes and is not always well covered by insurance, including Medicaid and Medicare.

The alternatives include an array of “mini-mental state” measures, from the ubiquitous Mini-Mental State Evaluation (MMSE), the Montreal Cognitive Assessment (MoCA), and the Saint Louis University Mental Status used by physicians to the Wechsler Abbreviated Scale of Intelligence, the Repeatable Battery for the Assessment of Neuropsychological Status, and the Neurobehavioral Cognitive Status Examination, primarily used by psychologists.

Theoretically at least, computerized neurocognitive tests (CNTs) have the advantage of all 3 approaches: they can be administered relatively quickly and do not require a physician’s time; they can be tailored to a specific clinical issue (eg, ADHD), concussion, or mild cognitive impairment, or they can be broad-spectrum and comprehensive; and they are self-scoring and generate a report as soon as the test is done. They are accurate to the millisecond and data can be stored for serial comparison of a patient’s results. When such tests are reported in the medical literature, the developers assure us of their reliability and validity.

Discriminant validity, however, may not be a meaningful standard for a clinician. Test A, for example, can distinguish between patients with ADHD and normal controls. However, can it distinguish among the conditions in the differential diagnosis of ADHD (eg, anxiety, depression, OCD)? Similar questions arise when CNTs are used for dementia screening or concussion management. Such questions are rarely addressed, although that is really what the clinician needs.

There are many CNTs available for clinical assessment. Most are commercial products, with costs ranging from $7.50 per test to several hundred dollars for a license with per-test costs on top of that. Several are research instruments that are not available to clinicians but are used in academic centers or in clinical trials. At least one is free and available on the Internet for qualified users. There is even a CNT code that covers “computerized tests” (99120), but reimbursement is never ensured.

Cognitive assessment by computer might solve many of the problems physicians have in accurately appraising a patient’s cognitive abilities if, in fact, the tests are reliable and valid in the clinical setting. The first CNT used in the North Carolina Neuropsychiatry Clinics was the neurobehavioral evaluation system (NES)-II, developed by the late Richard Letz in the early 1980s for measuring the cognitive effects of industrial neurotoxins.¹ We found it useful for evaluating patients with ADHD and mild brain injury, but because it was a DOS-based program, it is only a fossil now. A number of specific ADHD tests-the Conners Continuous Performance Test (CPT) and the Test of Variables of Attention as well as computerized versions of single tests such as Categories, the Stroop Test, and the Computerized Assessment of Response Bias (CARB)-were useful for detecting malingerers but are now also extinct.

Another CNT we used was MicroCog. It was developed at the behest of the Risk Management Foundation of the Harvard Medical Institutions, which insured about 5000 physicians at the time. The purpose of the test was to improve malpractice underwriting by identifying impaired physicians. A CNT would be less threatening to physicians, it was thought, than a battery administered by one of their colleagues.

Since the early days, when CNTs ran on Commodores or Apple 2e’s, dozens of computerized tests and test batteries have been developed-1 or 2 new ones every year. Physicians and psychologists in research settings like CNTs it seems, but they have been slow to catch on in clinical practice. In 2001, my colleagues and I developed our own test battery-CNS Vital Signs.

Are all computerized tests the same?

CNTs are more alike than different, but some differences are important. Most CNTs have taken conventional neuropsychological tests and adapted them for computer administration. Others, notably the Cambridge Neuropsychological Test Automated Battery (CANTAB), use a battery of novel tests. Some new tests use multimedia presentations that are said to have better “ecological validity.” As a rule, the closer a CNT is to established tests, the easier it is to learn and to interpret in a clinical setting.

Most CNTs can be self-administered by a patient who can read and maintain his or her attention for a suitable length of time. Others require the patient to be supervised by a technician. The latter is preferable but is costly and defeats the purpose of computerization. In our clinics, technicians sit with patients during test sessions in special circumstances. Most patients can take a CNT by themselves or with the assistance of a family member.

In general, the test batteries include a number (3 to 20) of specific tests. In some CNTs, all the tests must be administered at one time; others allow the tester to choose individual tests to administer. The venerable CPT is a component of many test batteries but is also available as a stand-alone test.

Many of the wide-spectrum tests take about 30 minutes to complete; the screening batteries are shorter and some of the more complicated tests take more than an hour. Thirty minutes is the ideal time.

Some test batteries were designed for specific applications, such as concussion management (Immediate Post-Concussion Assessment and Cognitive Testing [ImPACT]) or dementia screening (ComputerAdministered Neuropsychological Screen for Mild Cognitive Impairment [CANS-MCI]). Others were designed for a broad spectrum of clinical applications. Since the test batteries tend to be similar, emphasis on a specific clinical application usually reflects the particular interest of the test developer, the research base supporting the test, and marketing decisions made by test developers.

Applications for computerized tests

The list of potential applications for computerized testing is quite long and includes screening children for learning disability; evaluating fit-ness for driving or returning to work; gauging impairment; and measuring the impact of environmental toxins, postoperative cognitive dysfunction, and rheumatic disease. In the Neuropsychiatry Clinics, every new patient takes a broad-spectrum CNT as part of an initial evaluation.

Even patients with what appears to be a simple and straightforward psychiatric condition, such as anxiety or depression, may have an underlying neurocognitive deficit. The CNT can shed light on the underlying basis of their problems, the appropriate diagnosis, and treatment. There is a high degree of comorbidity between virtually all of the psychiatric disorders and learning disabilities; similarly, the presentation of patients with toxic exposure or early dementia is not infrequently psychiatric.

ADHD. Variations of the CPT are probably the most commonly used CNTs in assessing ADHD. The assumption is that deficits in sustained attention (or vigilance) are central to the disorder, which may be true for children. Deficits in executive function and processing speed, however, are more pertinent to the assessment of ADHD in adolescents and adults. In the Neuropsychiatry Clinics, we use a broad-spectrum CNT in the “test-dose” paradigm. The patient is tested at baseline and an hour following a dose of short-acting methylphenidate or dextroamphetamine. Not everyone responds favorably to a stimulant.²

Concussion. The ImPACT battery is widely used by professional, college, and high school teams. When a player is concussed, his performance must return to baseline before he is allowed to return to the game.

The Automated Neuropsychological Assessment Metrics (ANAM) battery of tests was developed for concussion management in the battlefield. Serious questions, however, have been raised about the reliability of both ImPACT and ANAM; it is dubious practice to rely on an athletic trainer and a CNT to diagnose or manage concussions. Most CNTs are sensitive to mild brain injuries, and all of them are sensitive to the effects of moderate or severe brain injuries. In the clinical setting, they are useful for serial patient evaluation.

Dementia screening. Many tests may be useful for the early diagnosis of dementia and mild cognitive impairment, a sometime precursor to dementia. Some may distinguish between dementia and depression.³ We have used several CNTs for that purpose over the years; however, a CNT alone will usually generate an unacceptable level of both false positives and false negatives. If a CNT is used in conjunction with other tests, however, one’s confidence in the early diagnosis improves. The MMSE, MoCA, tests of verbal and visual memory, and tests of visual-spatial ability are necessary before the patient is subjected to a full dementia workup or drug treatment.

Psychiatric disorders. Patients with schizophrenia almost always do poorly on CNTs; testing does not necessarily improve diagnosis. Serial testing, however, may improve management. Mild deficits in processing speed are commonly seen in patients with anxiety or mood disorders and the Stroop test is often difficult for patients with OCD. These observations are based on clinical observations and have not been confirmed.

A more interesting observation is that more substantial cognitive deficits occur in about 20% of patients with generalized anxiety or major depression and in 30% of patients with bipolar disorder. Most patients are cognitively intact, but a substantial minority are not and they tend to be patients with more severe disorders who are more difficult to treat.

Other disorders. Sleep disorders are almost always associated with diffuse cognitive impairment, as is chronic pain. Some pain patients who are receiving opiate drugs do so poorly on a CNT that one has to question whether they should be allowed to drive. Processing speed deficits are common in patients with multiple sclerosis, epilepsy, and cerebrovascular white matter disease.⁴ Epilepsy has many different neuropathological substrates, however, and antiepileptic drugs may themselves have cognitive toxicity. Substance abusers and alcoholics will also generate variable but impaired profiles on a broad-based CNT. Hepatic encephalopathy is an underdiagnosed condition that is associated with deficits in processing speed and attention.

The limitations of computerized testing

Computerized tests have been around as long as personal computers. In 1983, Joseph Matarazzo,⁵ an eminent neuropsychologist, wrote:

Thousands of relatively low-cost microcomputers and associated software are being purchased for use in psychological testing by employers, physicians, psychologists, social workers, counselors, nurse practitioners, and other licensed health care providers. . . . There is a danger that wholesale use of automated tests by people without a knowledge of their limitations will be a disservice to the public.

As with any test in medicine or psychology, the results of a CNT are only meaningful if the examiner understands the limitations of the test and the differential diagnosis attendant on particular test profiles. CNTs generate enormous quantities of precise data, but interpreting the data correctly depends on training and experience. Some commercial test batteries include diagnostic recommendations (a dubious practice in the author’s opinion). CNTs generate measurements, just as a battery of blood tests does. Not knowing the functions of a specific test precludes its use.

Performance on a CNT is compromised in patients who are not well educated or who are not familiar with computers. Thus, results of a test must be interpreted relative to the patient’s estimated IQ and familiarity with computers.⁶ Only one CNT includes those parameters in its report.

It is assumed that if a CNT is a replica of a conventional neuropsychological test, then the CNT is measuring the same thing. This assumption is wrong. Computerized tests are primarily measures of central processing speed and general mental ability; even when they are supposed to measure memory or executive function, they are usually assessing the patient’s abilities in the context of his processing speed.⁷

As with most tests in medicine, the results of a CNT are not diagnostic, but they are useful adjuncts to the diagnostic process. Many neuropsychologists use CNTs as part of a formal assessment. They are particularly helpful in evaluating the course of certain conditions; for example, to interpret the nature and cause of a gradual or abrupt deterioration. Many test batteries have the capacity to generate serial reports of a patient’s status over time.

CNTs introduce an additional measure of objectivity to the assessment and treatment process in psychiatry. The cost of testing is low but not unsubstantial. The availability of CNT freeware should make it more appealing, especially since third-party coverage is unpredictable. Patients can take some of the newer Internet-based tests at home, using a tablet or a smartphone. One hopes that such improvements in test availability will lead to increased use and a corresponding improvement in clinical care. Computerized testing is not a new technology, but it has yet to achieve its clinical potential in psychiatry, neurology, and general medicine.

Disclosures:

Dr Gualtieri is Medical Director of the North Carolina Neuropsychiatry Clinics in Chapel Hill, Charlotte, and Raleigh. He reports no conflicts concerning the subject matter of this article.

References:

1. Baker EL, Letz RE, Fidler AT, et al. A computer-based neurobehavioral evaluation system for occupational and environmental epidemiology: methodology and validation studies. Neurobehav Toxicol Teratol. 1985;7:369-377.

2. Gualtieri CT, Johnson LG. ADHD: is objective diagnosis possible? Psychiatry (Edgmont). 2005;2: 44-53.

3. Elwood RW. MicroCog: assessment of cognitive functioning. Neuropsychol Rev. 2001;11:89-100.

4. Alpherts WC, Aldenkamp AP. Computerized neuropsychological assessment of cognitive functioning in children with epilepsy. Epilepsia. 1990;31(suppl 4): S35-S40.

5. Matarazzo JD. Behavioral health: a 1990 chal-lenge for the health sciences professions. In: Matarazzo JD, Weiss SM, Herd A, et al, eds. Behavioral Health: A Handbook of Health Enhancement and Disease Prevention. New York: John Wiley and Sons; 1984:3-40.

6. Iverson GL, Brooks BL, Ashton VL, et al. Does familiarity with computers affect computerized neuropsychological test performance? J Clin Exp Neuropsychol. 2009;31:594-604.

7. Robbins TW, James M, Owen AM, et al. Cam-bridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. Dementia. 1994;5: 266-281.