Enlarge /. FDA Commissioner Stephen Hahn spoke at the press conference where he badly maimed statistics.
View more stories
After several days of rumors with steadily growing hype, the Trump administration announced on Sunday that the Food and Drug Administration would issue an Emergency Use Authorization (EUA) for COVID-19 treatment. The move was controversial from the start. A number of health professionals reportedly opposed the EEA, including Francis Collins, director of the National Institute of Health and Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases. The press conference failed to resolve the issue, and a growing chorus of scholars said the data presented in support of the EEA had been misrepresented.
FDA Commissioner Stephen Hahn admitted on Monday night that he had made a significant mistake in portraying the benefits of the treatment and apologized on Tuesday at the statement. But Hahn backed away against indications that approval of treatment on the eve of the Republican National Convention was motivated by political pressure.
In the treatment in question, the antibody-containing plasma is taken from people who have recovered from SARS-CoV-2 (convalescent plasma) infection and given to people currently suffering from COVID-19 symptoms. At the press conference on Sunday, the basic justification for approving this treatment under an EEA was a 35 percent decrease in mortality for those who received plasma in the first three days of treatment. In particular, Hahn said 35 out of 100 people were "saved" by treatment.
There were a number of obvious problems with this statement, which was purportedly based on a draft paper from a study of people who had received this convalescent plasma treatment. The most obvious problem is that the paper has no control group. All received convalescent plasma, but there were differences in the rate at which treatment was initiated. Without controls, it is impossible to say how many people would have died without treatment.
While it was possible to find some of the figures mentioned in the paper that could be interpreted to produce a 35 percent difference between different types of treatments, as the press conference progressed things got more complicated and those in attendance added additional numbers to the number Added caveats: only these applied to people under 80 years of age who did not need a ventilator, and it required the identification of plasma with high levels of virus-specific antibodies. Since these numbers were not part of the analysis performed by the draft authors, it was clear that the 35 percent number had been generated by working with the raw data discussed in the study. Hence it was impossible to say how it was derived.
With Hahn's approval, things are a little clearer now. Hahn spoke of a number called relative risk, which, as the name suggests, compares the risks two different groups face. He then discussed this risk as if building his 35 out of 100 numbers were an absolute risk – a very fundamental statistical error and a science journalist is often warned not to do so.
To see why this is a problem, imagine a case where only three in 100 people are affected, and an experimental condition changes that number to one in 100. The relative risk differs by a factor of three – a difference of 300 percent. The absolute risk, however, is the slight change from three percent to one percent.
The actual difference in absolute risk in the study was 4.8 percent, not 35 percent. This means the treatment could save an additional 4.8 lives per 100 COVID-19 patients.
The politics of everything
Hahn made a rookie mistake that made a treatment sound much better than it actually was, and that mistake added to the perception that politics was a major driver of the EEA. At the same time as he was admitting his mistake, Hahn was forcibly backing off suspicions of political interference. He stated that the decision to approve the EEA was made by a career FDA scientist, not a policy officer.
In fact, that statement is in complete agreement with previous reporting that suggested the FDA was ready to issue the EEA, but health officials outside the agency urged the decision to be delayed until a randomized clinical trial could be completed. According to the New York Times, Francis Collins, Anthony Fauci and Clifford Lane from the National Institute of Allergy and Infectious Diseases, among others, should be noted. According to a report last week, their opposition had put the EEA on hold.
But Trump appeared to hold the FDA responsible for unspecified delays, accusing its staff of sabotaging his re-election shortly before the EEA was approved. If there had been political pressure, it would not have resulted in the original FDA approval decision. instead, it would have overcome opposition to moving from outside the agency.
Trump's pointer to the FDA was simply the result of his confusion over the exact roles of the FDA and NIH, according to one source.
In any case, the currently limited evidence is enough to meet the legal threshold for an EEA. That mistake, the timing of the EEA, and its presentation at a press conference laden with praise from President Trump, have undermined the FDA's credibility. And with several vaccines currently in the later stages of testing, this is not an ideal time for this harm.