A lot of you have reached out to me because you don’t know what information to trust, and what not to trust. This is not new in science. I have written about this problem in the past. We are all familiar with “eggs are good for you”; “no, eggs are bad for you”; “no, it turns out eggs are good for you”. In fact, I have made it my career to help you discern good science from bad science. The truth is that most science is bad science; and that’s when times are good. Science is probably worse when times are bad because fear, uncertainly, and desperation make us more willing to lower our thresholds for evidence.
But, bad science is bad science. The rules of science and evidence-based medicine (EBM) do not change because of the Coronavirus. So let me give you a crash course in how to evaluate a trial. As a concrete example, I will evaluate studies behind the anti-malarial medicine chloroquine and it’s relative hydroxychloroqine (Plaquenil brand name). Specifically, the recent French trial that spread over the internet and social media yesterday.
I would like to point out, I do not intend to make any specific recommendations regarding whether you should, or should not, use this medicine. I would like to remain neutral. Each person is different, and each person has different risks and fears. Additionally, I may critique the study, but I am not critiquing the individuals that performed the study; I have the deepest respect and admiration for their bravery and tenacity - they are simply making do with what little resources and time they have.
Now, lets answer some critical questions about the study:
Is there a risk of bias?
In many trials, the most obvious risk of bias comes from a financial conflict of interest, such as a sponsoring pharmaceutical company. Chloroquine and hydroxychloraquine are cheap and generic and there is no pharmaceutical company behind this. However, there is another source of bias - pride, fame, and fortune for the researcher. Imagine being the researcher who proves hydroxychloroquine works - the medicine that cured the COVID-19 pandemic. The stakes are high and the rewards are huge - you have saved the world, or at least the economy.
Is it a multi-center trial?
People bring their own personal beliefs to their research. There is always a maverick scientist somewhere that claims they have the cure to (INSERT DISEASE HERE). Overconfidence and confirmation bias are a bad combination. Independent groups of people doing research together is always better. The more people involved in a trial, the less likely it is to be biased, and the less likely there is to be manipulation of data, confabulation, and researcher fraud.
The French study was conducted by a single team, from a single hospital, run by one dominant famous researcher. Not a good sign.
Is it peer reviewed and published in a high impact factor journal?
The paper that people saw yesterday is likely the rough draft from the author. It was probably sent straight to a journal. This is just the beginning of the peer review process. The editors of the journal will try to determine if the research is legit. They will comment on the deficiencies, ask for raw data, ask for calculations, and assess for conflict of interest. Papers are often corrected and sent back several times before publication.
Journal editors are very serious about peer review. Especially the ones with a high impact factor. They do not want to get egg on their face publishing a bad paper - they need to protect themselves. The Lancet, for instance, was severely chastised when they did not properly peer review the famous anti-vaccine paper from Andrew Wakefield in 1999. Peer review does not guarantee the validity of a paper, but it is a start. It is necessary, but not sufficient. The French paper has, as of yet, not been peer reviewed nor has it been published in a journal.
Did patients drop out of the study and why?
It is very important to keep track of patients that initially enroll in a trail and then drop out - and the reasons why they drop out. Drop out can skew results. The French study began with 16 patients in the control group and 26 patients in the treatment group. Apparently all 16 controls remained in the study; however, 6 of the 26 treated patients left the study and were not included in the studies results; 3 of these 6 patients were transferred to the ICU because they got worse and ONE DIED. This is very important. They decided to exclude from the results the only patient that really matters - the patient that died - WHO WAS TAKING HYDROXYCHLOROQUINE. (Making me wonder if anyone has actually read this study - See below taken directly from study website)
Are there hard outcomes?
This infection kills people. Ultimately a medicine needs to demonstrate that it prevents death. Surrogate outcomes are a start, but not the end. In the French trial, the main outcome was a surrogate marker - a measurement of the virus in the nose. The hydroxyquinone group had a faster clearance of the virus from the nose than the control group.
That's nice, but what we need is a hard outcome. Prevention of death. That's all we really care about. Although, death was not the primary endpoint of the trial , the only death that did occur was in the hydroxychloroquine treated group (as mentioned above). Hence, the trial can be summarized as such: hydroxyquinone helps clear the virus from the nose but increases death. This brings new meaning to the expression, "The operation was a success, but the patient died".
Is there a control group?
As I mentioned in my last email, because this virus has an over 95% recovery rate, if someone got better after a treatment we do not know if they got better spontaneously. Imagine, if you will, that the case fatality was actually close to 100%. If a patient survived after given a medicine, this would be good evidence (even without a control group) that the medicine works. However, this is not the case with COVID-19. There MUST be a control group - a group of patients matching the treatment group that does not get the treatment. In the French study, the control group were patients in another hospital and patients who refused treatment - this is not a control group. Patients in another hospital and patients who refuse treatments are probably different from the treatment group. In a well controlled trial, a group of similar patients are randomly allocated to the treatment group or the control group.
Is it a large sample size?
As mentioned above, most people with COVID-19 survive. If we take a sample of only 20 patients, only one or two of them are expected to die. Even if the treatment was helpful in preventing death these numbers are too small to detect a statistical difference between groups.
The French study had another factor that made the sample sizes even smaller. In the treatment group, 14/20 patients got Hydroxychloroquine alone and 6/20 patients got Hydroxychloroquine + Zithromax. Multiple treatments, dilutes the sample size even more and makes analysis more complicated.
Are there side effects to the medicine?
Medications developed after the 1980's had to carefully document side effects in large clinical trials and enroll in postmarking surveillance. Hydroxychloroquine was developed in the 1930's. It was grandfathered. We don't have detailed studies of its true side effects.
Nonetheless, its known side effects include nausea, injury to the retina, and prolongation of QT interval. Prolongation of the QT interval puts you at risk for lethal cardiac arrhythmia; especially when combined with other drugs that increase QT interval. Many common drugs increase the QT interval; here is a comprehensive list. Beware of this drug interaction. Ironically, azithromycin - the trials other medication to combine with hydroxychloroquine - is on that list. It too, increases QT interval.
Does FDA approval mean that it works
There were rumors yesterday that hydroxychloroquine was already approved by the FDA for COVID-19. This turned out to be false. But let’s say the FDA did approve hydroxychloroquine. Does that mean it works? The FDA has approved many drugs that only work on surrogate markers. In fact, some may argue that the majority of drugs approved by the FDA rely on studies that improve only surrogate endpoints. For example, most chemotherapy drugs approved in the last couple of decades have only shown response rate or progression-free survival. They have not produced evidence that they reduce death or improve quality of life. Additionally, in dire circumstances such as rare diseases where research is limited, drugs with little evidence get an accelerated approval. This is known as “compassionate use".
I would not be surprised if the FDA approves hydroxyquinine very shortly. With our current crisis, the FDA is likely to be even more lenient with drug approval than it usual is. Do they have a choice? China and Korea have hydroxychloroquine on their official treatment protocols, and as we speak, American hospitals have started using it for their sick COVID patients.
This can be deceiving for you, the patient. Agencies approving a drug, doctors and hospitals using it, is not proof that it works. What we do out of desperation, and what actually works, are two different things.The problem is that we all want a treatment right now; the government wants a treatment, the patients want a treatment, and the doctors want a treatment. Doing something is always better than doing nothing - this has always been the unwritten credo in medicine. Consequently, like blood-letting for the Plague in the Middle Ages, we will get our treatment - whether it works or not.
The real irony is that I can barely get my patients to take chloroquine for an application it actually works - Malaria prophylaxis with travel to an endemic area - and Malaria kills more than one million people a year. Now, everyone is breaking down pharmacy doors to get some for COVID-19. I've even seen it prescribed for children by pediatricians in Miami Beach.
I desperately want hydroxycholoroqine to work. But the current evidence falls short. I am wishful that it will prove itself in the larger trials underway. Unfortunately, what we wish to be true and what is actually true are often not the same thing.