A paper recently published in the Journal of the American Medical Association has gotten a lot of press. From a prominent story in The New York Times to a cover story in Newsweek, print and web publication were filled with the news that the latest study of antidepressants showed no effect on mild to moderate depression. There's only one problem. The study showed nothing of the kind. How did journalists get it so wrong? They relied on the press release instead of reading and analyzing the paper itself.

Before we consider what the study showed, let’s think about what kind of evidence we’d need to conclude that antidepressants don’t work.

First, although there are different types of antidepressants, the term used colloquially refers to antidepressants of a specific type, SSRI’s or selective serotonin reuptake inhibitors. There are other, older types of antidepressants that are rarely used today because of their unpleasant side effects. Hence any study that claims to show that “antidepressants” are ineffective, must look at SSRIs.

Second, there are literally thousands of studies of SSRIs, and it would be helpful to aggregate the results. Aggregating results can be done in a type of paper known as metaanalysis. Metaanlysis adds the results of multiple similar studies to find trends that might not be apparent in individual small studies. But a metaanlysis is subject to several important limitations that must always be considered. The most important limitation is that the authors of the metaanalysis choose the papers to be included. Bias can be introduced by examining only papers that have a desired outcome; that can be accomplished by restricting the inclusion criteria in arbitrary ways.

Let’s look at the study, Antidepressant Drug Effects and Depression Severity. According to the abstract:

Randomized placebo-controlled trials of antidepressants approved by the Food and Drug Administration in the treatment of major or minor depressive disorder were selected. Studies were included if their authors provided the requisite original data, they comprised adult outpatients, they included a medication vs placebo comparison for at least 6 weeks, they did not exclude patients on the basis of a placebo washout period, and they used the Hamilton Depression Rating Scale (HDRS). Data from 6 studies (718 patients) were included…

… The magnitude of benefit of antidepressant medication compared with placebo increases with severity of depression symptoms and may be minimal or nonexistent, on average, in patients with mild or moderate symptoms. For patients with very severe depression, the benefit of medications over placebo is substantial.

According to one of the authors was interviewed for the NYTimes article, Popular Drugs May Aid Only Severe Depression:

“The message for patients with mild to moderate depression,” Dr. DeRubeis said, “is, ‘Look, medications are always an option, but there’s little evidence that they add to other efforts to shake the depression — whether it’s exercise, seeing the doctor, reading about the disorder or going for psychotherapy.’ ”

Let’s go back and compare the paper to the criteria we identified above. The first criterion was to look at the antidepressants currently used in clinical practice. But out of the 6 studies in the metaanalysis, 3 looked at imipramine, a tricyclic antidrepressant that has not been the standard of care for over a decade because of its unpleasant side effects. The other three studies looked at Paxil (paroxitene). Paxil is an SSRI, but it is only one member of the class of SSRIs. Although all SSRIs share the same mechanism of action, they have different profiles of effectiveness and side effects. Therefore, generalizing from Paxil to all SSRIs cannot be justified.

So in terms of clinically relevant information, the paper included only 3 studies of an SSRI. How did the authors whittle down thousands of papers on SSRI effectiveness to only 3? According to the authors:

The criteria for inclusion required studies to be randomized placebo controlled trials of an FDA-approved antidepressant in the treatment of the full range of patients with major or minor depressive disorder … In addition, the studies had to include an ADM/placebo comparison of at least 6 weeks’ duration and HDRS scores at intake and at the end of treatment. Studies were excluded if they excluded patients on the basis of a placebo washout period. The final inclusion criterion was that individual patient-level data had to be available for analysis.

Are these criteria relevant? Certainly, the inclusion of only RCTs is a reasonable criterion. That left the authors with 141 studies to analyze. They then decided to remove all studies that included a placebo washout period. Using that criterion, the authors excluded fully 118 of the 141 papers.

A placebo washout period involves treating everyone in both arms of the study with placebo for an initial period of time, often 3 weeks. People who respond to placebo are then excluded from the study. The theory is that excluding known placebo responders makes it easier to identify real effects. There is disagreement among psychiatric researchers about whether a placebo washout period introduces bias into a study. While it is theoretical defensible to exclude studies with a placebo washout period, such an exclusion means that the authors are no longer reviewing "the literature." They are reviewing only a tiny, non-representative subset of the literature.

Finally, there is one indisputably arbitrary criterion that is acknowledged by the authors. The initial analysis identified 23 studies, but they could only gain access to the data in 6 studies, so they simply ignored the other 17.  As it happnes, fully 15 out of 17 of the excluded studies showed antidepressants to be effective in mild to moderate depression. Indeed, out of the 23 original studies, ALL the studies that demonstrated the effectiveness of antidepressants in mild to moderate depression were deliberately left out.

In summary, then, by using questionable exclusion criteria, the authors analyzed only 3 clinically relevant studies (the Paxil studies), involving only one SSRI. It is not clear that these studies are representative of existing studies on SSRIs, or even if they can be generalized to other SSRIs.  Dr. Rubeis’ public assertion that for patients with mild to moderate depression there is little evidence that “medications” add to efforts to treat the depression cannot be justified by the findings in his study.

You'd never know that, though, by reading the news stories about the paper. The medical writers simply accepted the authors' claims wholesale and never bothered to check if the data justified the claims.