Lies, Damn Lies, and Statistics

©VictorTondee/Shutterstock

Introduction: Lies, Dam Lies, and Statistics, by Allan Tasman, MD

I titled this column using the famous quote from Benjamin Disraeli, not because this column is about lying, but to illustrate the complexity of determining the validity of a statistical analysis. I’m referring specifically to a recent change implemented by NIMH regarding its assessment of the percentage of the population with schizophrenia, discussed in the accompanying analysis of the change by Dr. Fuller Torrey and Elizabeth Sinclair. While this might seem like an arcane academic issue, there are potentially very distressing real-world consequences.

Imagine, for example, if federal and state or local officials who fund already financially strapped community mental health centers (CMHCs) decided to make significant reductions in funding if they believe the actual numbers of patients with schizophrenia are much lower than those on which they calculated their original budget allocations. Since CMHC’s bear much of the burden of ambulatory care of those with schizophrenia, the impact could be devastating.

Dr. Torrey’s piece is followed by a commentary by NIMH Director, Joshua A. Gordon, MD, PHD, who explains the context of the NIMH decision-making process and the next steps.

See: HOCUS POCUS: How the NIMH Made 2 Million People With Schizophrenia Disappear, by E. Fuller Torrey, MD and Elizabeth Sinclair

See: A RESPONSE FROM THE NIMH DIRECTOR: On the Prevalence of Schizophrenia and Transforming Care Through Research, by  Joshua A. Gordon, MD, PhD

HOCUS POCUS: How the NIMH Made 2 Million People With Schizophrenia Disappear, by E. Fuller Torrey, MD and Elizabeth Sinclair

Dr. Torrey is Founder of the Treatment Advocacy Center and Associate Director for Research at the Stanley Medical Research Institute. Ms Sinclair is a Research Associate at the Treatment Advocacy Center.

Although he has been the Director of the National Institute of Mental Health (NIMH) for less than 2 years, Dr. Joshua A. Gordon has made 2 million individuals with schizophrenia disappear. NIMH has not said where they went but officially, they no longer exist. This is a remarkable accomplishment, especially given the lack of attention NIMH has given serious mental illness

Until November 2017, NIMH had claimed that the 1-year prevalence of schizophrenia in adults in the US was 1.1%. In other words, in any given year approximately 11 out of every 1,000 adults would be diagnosed with this disease of the brain. The Bureau of the Census estimated in 2017 that there were 252.1 million adults in the US; thus approximately 2.8 million would have been diagnosed with schizophrenia.

This estimate was based on the last major epidemiological study of serious mental illness in the US, the Epidemiologic Catchment Area (ECA) study. This NIMH-funded study sampled 5 areas (Los Angeles, St. Louis, New Haven, Baltimore, and rural counties in North Carolina) in the early 1980s. At least 3,000 adults living in households and 500 individuals living in mental hospitals, prisons, nursing homes and other institutions were directly interviewed in each area, then surveyed by telephone 6-months later and re-interviewed another 6-months after that. The results are considered to be the most accurate mental illness prevalence data available for the US. In 1993, at the request of the US Senate Committee on Appropriations, the ECA data were given to the Senate so that they could be used to assess the costs of these disorders. The ECA data have thus been the standard in the US for the past 3 decades.

Inexplicably, in November 2017, NIMH changed its website and declared that the 1-year prevalence of schizophrenia is now 0.3%, or 3 cases in 1,000 adults. Based on the current census estimates, this means that there are now only approximately 750,000 adults with schizophrenia each year in the US. Since there were 2.8 million previously, just over 2 million have been made to disappear.

How does NIMH justify this remarkable feat? They no longer make any mention of the ECA study and instead cite as the basis of their estimate a National Comorbidity Survey carried out in 2001-2003. This survey was a sampling of 9,282 people living at home and specifically excluded individuals in institutions, including in psychiatric or other hospitals, nursing homes, jails, prisons, homeless shelters, or individuals living on the street. Thus the official count of individuals with schizophrenia in the US now excludes anyone not living at home. The refusal rate for participating in the survey was 29%, and the authors cite a study reporting that people with paranoid symptoms are more likely to refuse to participate in such surveys. For screening their original sample of 9,282 people living at home, the study used the Composite International Diagnostic Interview (CIDI), a screening tool known to be poor in identifying individuals with schizophrenia. Among the original 9,282 persons, 2,382 had some psychiatric symptoms and were then screened using an instrument for non-affective psychosis (schizophrenia, schizoaffective disorder, delusional disorder, non-affective psychosis not otherwise specified). The final psychiatric diagnoses for this survey were based on telephone interviews with 73 people.

NIMH then tries to justify the use of a schizophrenia adult prevalence rate of 0.3% per year by citing 2 summaries of a worldwide prevalence study. This study included 188 prevalence studies, including the ECA study, carried out in 46 countries, some as early as 1935. The main finding of this study was the very wide range of prevalence in different countries, with the US being among the highest. Many of the studies were carried out in developing and rural countries that have reported a low prevalence of schizophrenia until recent years. Taking the median prevalence from these 188 studies to justify a current estimate for the US is scientifically irresponsible.

The new estimates from NIMH are also egregiously inconsistent with the majority of other prevalence estimates, even within the federal government. The latest data from the Substance Abuse and Mental Health Services Administration (SAMHSA) state that schizophrenia accounts for 22% of people with serious mental illness, or about 2.3 million adults based on current estimates. The Centers for Medicare and Medicaid Services reports that 4% of all Medicare beneficiaries have schizophrenia, increasing to 11.2% all individuals who are dual-eligible for Medicare and Medicaid. In addition, widely-cited academic literature has prevalence estimates consistently between 0.9%–1.1%.

Why did NIMH wish to massively reduce the official prevalence of schizophrenia in the US? In the past 2 years, members of Congress have become much more interested in problems associated with untreated severe mental illness. With the passage of the 21st Century Cures Act and appointment of the nation’s first Assistant Secretary for Mental Health in the Department of Health and Human Services, it seems likely that federal mental health agencies like NIMH may be held accountable in a way that they have not been previously. Having to account for 750,000 people with schizophrenia is much easier than having to account for 2.8 million. In addition, NIMH has already been widely criticized for ignoring the needs of individuals with schizophrenia, specifically by failing to support research on the development of new medications and other effective treatments.

It should also be noted that NIMH has not funded a major study to determine the current prevalence of schizophrenia since the ECA study in the early 1980s-more than 30 years ago. Thus we really have no idea whether the prevalence of schizophrenia has increased, decreased, or stayed the same. The lack of data in the US on this question stands in sharp contrast to almost every European country. In England, for example, 7 major studies on the prevalence of schizophrenia were carried out between 1986 and 2009, including studies of whether the prevalence is increasing; why the prevalence is so high among certain immigrant groups; and whether there are regional differences in prevalence. A measure of England’s interest in these issues is a 258-page report, “Systematic Review of the Incidence and Prevalence of Schizophrenia and Other Psychoses in England,” prepared for the Department of Health in 2012. The Scandinavian countries, of course, have even more sophisticated prevalence studies based on their national case registers. Compared with these countries, the data available in the US on the prevalence of schizophrenia are equivalent to the data available in some developing countries.

Summary

The last major study of the prevalence of schizophrenia in the US was carried out in the early 1980s. The results from that study have been used for 3 decades to estimate the annual prevalence of schizophrenia at 1.1% (11 in 1,000 adults). For unknown reasons, NIMH decided in November 2017 to no longer use that data. Instead, it adopted a 2001–2003 study that only assessed schizophrenia in people living at home. Individuals who are hospitalized, in nursing homes, in jail or prison, or homeless are no longer included in the measure of prevalence, which NIMH claims to now be 0.3% (3 in 1,000 adults). This decreases the number of people estimated to have schizophrenia in the US from approximately 2.8 million to 750,000. The very fact that NIMH would make such a change is a measure of the institute’s lack of interest and expertise on this important issue.

We have 2 recommendations that the agency should take to address this major discrepancy. First, it should restore the previous estimate of the prevalence of schizophrenia (1.1% annual rate for adults) until more updated data become available. Second, NIMH, perhaps in collaboration with SAMHSA, should undertake a major new study of the prevalence of schizophrenia, including individuals in psychiatric hospitals, in jails or prisons, and who are homeless. When Dr. Gordon became the Director of NIMH, he stated that his highest priority was to support good science. Let us hope that making individuals disappear is not representative.

See: A RESPONSE FROM THE NIMH DIRECTOR: On the Prevalence of Schizophrenia and Transforming Care Through Research, by  Joshua A. Gordon, MD, PhD

A RESPONSE FROM THE NIMH DIRECTOR: On the Prevalence of Schizophrenia and Transforming Care Through Research, by  Joshua A. Gordon, MD, PhD

Mental illnesses affect tens of millions of people in the US each year. As the lead federal agency for research on mental illnesses, the mission of the National Institute of Mental Health (NIMH) is to transform the understanding and treatment of mental illnesses through basic and clinical research, paving the way for prevention, recovery, and cure. NIMH, part of the National Institutes of Health (NIH), strives to provide the public with up-to-date, accurate information on our understanding of mental illnesses and the tremendous burden these illnesses pose to individuals and their families. To this end, we include prevalence statistics on our public-facing website to provide a framework for understanding the scope and impact of mental illnesses.

NIMH recently undertook an effort to refresh the statistics pages on our website, with respect to both design and content. For all statistics we report, our goal is to use the most accurate data available, which includes consideration of the most recent findings, as well as quality of the data. To help users understand and interpret the statistics we present, we aim to be rigorous and transparent by providing adequate background and context, as well as information on relevant strengths and limitations of the data sources. In the case of the refreshed schizophrenia statistics page, we used findings from several papers to approximate an estimate for the prevalence of schizophrenia. We now recognize, in retrospect, that we did not provide sufficient context to understand or justify the selected prevalence estimate, nor did the number we settled on reflect the full spectrum of knowledge available in the relevant literature.

Accurate and stable estimates of the prevalence of schizophrenia are made challenging by many factors. Comparisons across studies are hindered by inclusion of varying combinations of diagnostic criteria (eg, schizophrenia/schizophreniform; schizophrenia and other psychotic disorders; non-affective psychotic symptoms). The very definition of schizophrenia itself has changed over time since the DSM-III-based Epidemiologic Catchment Area (ECA) paper reported the prevalence of 1.1% in 1981.1 More recent surveys might under-count those with serious mental illness for many reasons, including the fact that many of these individuals decline to participate in surveys, or are homeless or incarcerated or otherwise difficult to reach. However, in the years since the ECA, a large number of studies, including several in the US, have clearly demonstrated that the 1.1% figure is no longer scientifically defensible.

But that does not mean the new figure we initially arrived upon – 0.3% – necessarily reflects the full picture. We are in the process of expeditiously reviewing and updating the information presented on our webpage to more fully reflect the available data on schizophrenia prevalence. We anticipate providing an estimated range, rather than a single point estimate; reporting such results will require inclusion of relevant caveats and limitations. More broadly, timely and accurate information on the prevalence of schizophrenia and other mental illnesses is important to multiple Department of Health and Human Services agencies beyond NIMH, including the Substance Abuse and Mental Health Services Administration (SAMHSA) and the Centers for Disease Control and Prevention. By working together toward a solution to address the difficulties in estimating prevalence for schizophrenia and other serious mental illnesses, we can accomplish far more for our key stakeholders. NIMH looks forward to continuing to work collaboratively to advance these important objectives. I thank Dr. Torrey and Ms. Sinclair for bringing the need for this more thorough treatment of schizophrenia prevalence to our attention.

At the same time, I feel compelled to counter the notion that NIMH does not devote sufficient attention to serious mental illness. NIMH supports a diverse portfolio of research aimed at improving care for, and the lives of, individuals living with a serious mental illness. This research spans basic and translational sciences aimed at identifying and developing the breakthrough treatments of tomorrow, as well as services and intervention sciences aimed at optimizing and implementing the effective treatments of today.

For example, neurocognitive dysfunction is a hallmark of schizophrenia and represents an important, and untreated, component of the disability associated with the disorder. NIMH supports several cognitive remediation studies for individuals with psychosis aimed at improving cognition and functional outcomes. We are also supporting the development and testing of 3 novel compounds for their long-term potential to improve cognition in patients with schizophrenia.

NIMH funds several large-scale effectiveness trials of practical interventions to reduce premature mortality in people with serious mental illness by addressing health risk factors. Recognizing that modifiable health risks associated with premature mortality appear early, NIMH also funds research capitalizing on opportunities for early intervention with young people at high risk. Additional efforts test interventions aimed at improving overall functioning by focusing on employment, social engagement, and community integration. These are but a sampling of the many research projects of relevance to serious mental illness supported by NIMH.

Notably, NIMH-supported research is impacting people now. For example, the NIMH-funded Recovery After an Initial Schizophrenia Episode (RAISE) project, a large-scale comparative effectiveness research study, demonstrated that coordinated specialty care early in the course of treatment for first episode psychosis was superior to usual care in reducing symptoms, improving school and work functioning, and increasing quality of life. Importantly, the intervention tested in RAISE was found to be cost-effective. Based on results from RAISE, the Centers for Medicaid & Medicare Services extended Medicaid coverage for coordinated specialty care and SAMHSA set aside 10% of its Mental Health Block Grant allocation for each state to support evidence-based programs that target first episode psychosis. Today over 200 coordinated specialty care programs operate in 49 states, representing an unprecedented translation of scientific findings into services that directly benefit persons with early serious mental illness.

Most importantly, we must keep in mind that statistics, while useful, do not tell the full story of the tremendous burden shouldered by individuals and families living with schizophrenia, and the outsized impact this burden has on our society. NIMH will continue to aggressively support novel approaches to biological understanding, therapeutics development, and services and intervention delivery, all aimed at reducing this burden.

Reference:

1. Regier DA, Narrow WE, Rae DS, et al. The de facto US mental and addictive disorders service system. Epidemiologic catchment area prospective 1-year prevalence rates of disorders and services. Arch Gen Psychiatry. 1993;50:85-94.