A big part of being a physician is figuring out how to navigate reference class problems. The conundrum is that, even though there’s no limit to the ways that a patient can be described, clinicians have to somehow suss out the single most useful depiction. Accuracy is necessary, but not sufficient. It might be just as accurate, for instance, to categorize a patient as a diabetic as it is to classify him as a diabetic male over the age of 65 with a history of heart disease and no insurance. Even if those constructions are both correct, they’re far from equivalent; one of them, obviously, is a lot more specific than the other. And that distinction is critical because, try as they might, doctors usually aren’t able to select a course of action based on direct knowledge or previous experience dictating how their particular subject will respond; medicine, sadly, is nothing like the movie Groundhog Day.
As a result, treatment plans are instead based on how people who resemble the patient — that is, the reference class — are known to behave. So while endocrinologists don’t treat patients as generic diabetics, they also don’t approach them as singularly unique individuals with no similarities to anyone else. Rather, doctors stake out a middle ground by identifying the mix of attributes that can place the patient into a reference class that we know how to treat. The trick (or one of them, anyway) is to slot each patient into the most useful class — or, put differently, the grouping that best supports a high degree of confidence that a potential treatment will be successful. It can be a tough balancing act. The ideal cohort has to be both broad enough to have produced solid data on the various interventions, but still narrow enough to imply that the patient will respond in lockstep with rest of the crowd. In the end, solving reference class problems is mostly a matter of finding just the right level of specificity.
When the Food and Drug Administration assesses a new medication, it’s also wrestling with a reference class problem. The FDA’s basic task is to approve new treatments, if and only if the benefits outweigh the risks. But you can’t run that analysis without, at least implicitly, deciding who the relevant population is. Should they consider a complete cross-section of society? Only elderly diabetics with heart problems? Something in between? Presumably each different slice of the population will exhibit its own unique potency and safety profile, and, because of that, the decision to approve a drug or not can easily turn on how exactly the reference class lines are drawn.
In practice though, since hefty sample sizes are necessary to detect potential threats and benefits alike, the FDA tends to mostly consider expansive reference classes. Take the controversial 2016 approval of the serotonin antagonist Nuplazid, for example. Nuplazid takes a novel approach to combatting Parkinson’s psychosis, which is a secondary symptom that affects close to half of Parkinson’s sufferers [2]. In support of its application, the drug manufacturer ran a clinical trial looking at the class of Parkinson’s psychosis patients over the age of 40 with a stable medication profile and no recent history of deep brain stimulation [1] — which the FDA, in turn, used to weigh the medication’s risks and benefits.
The drug was approved, but touched off a firestorm in the process. The loudest dissenting voice was from one of the FDA’s medical reviewers, a psychiatrist named Paul Andreason. Dr. Andreason made two separate, but related arguments [3]. First, he pointed out that even though the trial found that Nuplazid was statistically more effective at preventing hallucinations and other symptoms of psychosis than a placebo, the difference was small, and not necessarily clinically meaningful. Second, he claimed that, given Nuplazid’s somewhat marginal efficacy, its risk profile — which indicated that it more that doubled the odds of serious adverse events, including death — suggested that it should not be approved.
Dr. Andreason’s grievances started to look prophetic earlier this year when CNN reported that, since its approval, Nuplazid had been fingered as a “suspect medication” in hundreds of deaths reported to the FDA [4]. And, to make matters worse, reporters also uncovered more than a thousand claims of patients continuing to experience hallucinations despite taking the drug. Put together, it made for a tidy story: not only did Nuplazid not cure Parkinson’s psychosis, it was actually killing people. The FDA appeared concerned, and promised to reopen its investigation.
Ultimately the agency “did not identify any new or unexpected safety findings with Nuplazid, or findings that are inconsistent with the established safety profile” [5], and declined to pull the drug from the market. In other words, the FDA decided that the deluge after-market complaints didn’t alter its view of Nuplazid’s effectiveness or safety. Fair enough. For one thing, patients who take Nuplazid have to contend not just with Parkinson’s but also the ever-present dangers of polypharmacy — the median user took ten other medications [6] — so it’s not especially surprising that none of the reports could definitively show that Nuplazid was at fault. Plus, the manufacturer maintained especially close contact with the specialty pharmacies that distributed the drug and probably became aware of a higher-than-usual proportion of the negative outcomes as a result [2]. All in all, it wasn’t hard for the FDA to make a plausible case for keeping Nuplazid on the market.
But in doing so they managed to miss the real reason why Nuplazid should’ve been left alone. There are two things that we know for sure about Nuplazid’s effectiveness. One is that, on average, patients who use the drug see a marginal improvement in their psychosis symptoms; that was the conclusion of the clinical trial that got the medication approved in the first place. The other is that, as CNN reported, thousands of people went out of their way to complain that Nuplazid didn’t work as advertised [4][7]. On the surface, that’s disappointing. No drug manufacturer wants to be inundated with criticism from its customers. But on closer examination it’s not nearly as unfortunate as it first appears: since patients on average tend to see an improvement with Nuplazid, for every suboptimal outcome there must be another patient (or more) realizing a better than expected reaction. Thousands of complaints indicate, at minimum, thousands of success stories.
The point is that because the reference class outlined by the FDA was nonspecific, it couldn’t also be homogenous. Variance was inevitable. Some patients in the class were bound to have superlative responses to the medication, and others not so much. In fairness, it’s difficult to expect the FDA or a company-sponsored clinical trial to operate at a greater level of specificity; they’re required to construct enormous sample sizes and simply don’t have access to granular or intimate information about any given patient. Individual doctors, on the other hand, don’t have any of those constraints. In fact, physicians are perfectly positioned to identify the factors — medical, demographic, psychosocial, whatever — that might or might not place a patient in a specific reference class that portends favorable outcomes from Nuplazid, or some other cutting-edge therapy. Doctors are masters of reference class specificity. The FDA doesn’t even try.
It all comes down to a fairly straightforward case of informational asymmetry: The FDA doesn’t know all that much about any given patient whose pharmaceutical options it controls, whereas doctors know (or at least can know) nearly everything. As a consequence, to the FDA, the variance inherent in a large reference class is a nuisance to be bludgeoned away with the largest sample size it can possibly muster. But to doctors that same variance is a golden opportunity to identify the particular patients who are most likely to benefit from a controversial treatment. Seen from that perspective, it’s clear that doctors, and not the FDA, should determine their patient’s access to Nuplazid and other similarly high-variance therapies.
And, incredibly, the FDA agrees. In Dr. Andreason’s dissenting letter, he ends a section called “Benefit-Risk Summary and Assessment” with the suggestion that Nuplazid “be developed for another indication … [so] it may be used by the neurological treatment community until both approvable safe and effective treatments for Parkinson’s psychosis are developed. [3]” Even Dr. Andreason, the fearless whistleblowing Cassandra, believes that neurologists ought to be allowed to prescribe Nuplazid, regardless of whether the drug meets the FDA standards for the entire reference class. Doctors, he seems to be saying, can see each patient for precisely who they really are — even when the FDA can’t.
Notes
[1] “A Study of the Safety and Efficacy of Pimavanserin in Patients With Parkinson’s Disease Psychosis – Full Text View.” Full Text View – ClinicalTrials.gov, clinicaltrials.gov/ct2/show/study/NCT01174004?sect=X01256.
[2] Hicken, Melanie. “ACADIA Response.” DocumentCloud, http://www.documentcloud.org/documents/4433424-ACADIA-Response.html.
[3] Andreason, Paul. “Medical Review.” FDA, https://www.accessdata.fda.gov/drugsatfda_docs/nda/2016/207318Orig1s000MedR.pdf
[4] Ellis, Blake, and Melanie Hicken. “Reports of Deaths Spark Concern over New Drug.” CNN, Cable News Network, 9 Apr. 2018, http://www.cnn.com/2018/04/09/health/parkinsons-drug-nuplazid-invs/index.html.
[5] Center for Drug Evaluation and Research. “Drug Safety and Availability – FDA Analysis Finds No New or Unexpected Safety Risks Associated with Nuplazid (Pimavanserin), a Medication to Treat the Hallucinations and Delusions of Parkinson’s Disease Psychosis.” U S Food and Drug Administration Home Page, Center for Drug Evaluation and Research, http://www.fda.gov/Drugs/DrugSafety/ucm621160.htm.
[6] Institution for Safe Medication Practices. Safety Signals for Two Novel Drugs. 1 Nov. 2017, http://www.ismp.org/sites/default/files/attachments/2018-01/2017Q1_0.pdf.
[7] Ellis, Blake, and Melanie Hicken. “FDA Re-Examines Safety of Parkinson’s Drug .” CNN, Cable News Network, 25 Apr. 2018, http://www.cnn.com/2018/04/25/health/fda-nuplazid-safety-evaluation-invs/index.html.
There Are No Fours 