The DSM-5 Field Trials (Part 2): Wrong Goals and Wrong Methods Make Them Irrelevant

[Editor's note: Please click here to read part 1 of this blog.]

The ideal field test would study how the diagnostic manual will eventually perform under conditions most closely approximating its future everyday use. The goal is to avoid unpleasant surprises in translation from what has been written on paper to what is practiced in real life. No field test can ever approach the ideal. The very act of studying the DSM's performance always improves it beyond what can be achieved when no one is looking. Field tests also fail to account for the  pressures that will lead to systematic, future misuse-especially the drug company marketing of mental disorders that leads to over-diagnosis.

It is thus wise always to regard field trial results as the best possible case and to assume that the DSM will perform worse, possibly much worse, in actual practice. The DSM-IV field trials were meticulously conducted, but completely failed to predict the later false epidemics in attention deficit, bipolar, and autistic disorder. Field trials can smoke out some problems, but will inevitably miss others.   

So, we have absolutely no right to expect DSM-5 to mount anything approaching an ideal field test. But we should expect a trial that will produce at least some useful data. Remarkably, the DSM-5 field trial fails this most minimalist expectation. For reasons that defy understanding, DSM-5 is conducting an elaborate and expensive project that perversely fails to ask (and therefore will be unable to answer) the only questions about its proposals that really count.

What do I mean? DSM-5 has made a number of radical suggestions for change, particularly the inclusion of many new diagnoses at the threshold of normality. These have the potential to reclassify as mentally disordered tens of millions of people currently considered normal. The only relevant questions are the overall rates of these disorders in the general population and the risks of false negative over-diagnosis.

Several of the new diagnoses-particularly Mixed Anxiety Depression and Binge Eating-would go from nowhere to join the most common of the psychiatric disorders. We need to know just how common and how specific these diagnoses are. Otherwise, we may be labeling as mental disorders what are no more than the aches, pains, and sufferings of everyday life. DSM-5 is completely avoiding the measurement of rates in the primary care settings in which these diagnoses will most be used. At the end of the DSM-5 field trials, we will have no idea whatever whether its suggestions will create false epidemics of misidentified pseudo-patients.

The DSM-5 Field trials are intensive, extensive, expensive, complicated, time consuming- -and almost a complete waste of time. They are perversely designed to measure only feasibility, clinical utility, and reliability. These were important questions 30 years ago when DSM-III was first introduced. They are stale and irrelevant now, not really informative to decisions that have to be made on the DSM-5 proposals for new diagnoses. Experience has taught us that busy clinicians will use only the very simplest measures required for their reporting. The complicated dimensional scale development proposed by DSM-5 will never be embraced by any but the tiniest percentage of clinicians. We don't need a fancy field trial to tell us this. Moreover, the DSM-5 dimensional measures are ad hoc and lack coherence and face validity. They are  far too unrefined and cumbersome to deserve extensive field testing.

The DSM-5 reliability testing is almost as futile. Thirty five years of extensive study teach us that reliability does not inhere to the diagnosis- -instead it depends on how the diagnosis is evaluated. Highly trained raters using structured interviews in selected populations (that have an enriched prevalence) can achieve excellent reliability. Average, rushed clinicians using brief,unstructured interviews with unselected patients have very much lower levels of agreement. The reliabilities of the new subthreshold diagnoses are likely to be particularly low in the primary care settings where they will be most diagnosed.

The DSM-5 field trials should not have focused on reliability. The appropriate goal would have been to measure the rates of the proposed new disorders in the general population and the level of impairment associated with them. The appropriate method of data collection would have been telephone survey. This would be only a pale surrogate of clinical practice and could not predict future drug company exploitation. But it would provide information that would count in decision making-done at low cost, in timely fashion. The DSM-5 field trials are an irrelevant and expensive distraction.