In medicine, it is common to see many reasearch papers on the same topic. The evidence gets produced at such a fast pace it is hard for laypersons, and even doctors, to keep up. The purpose of this email is to produce a systematic review of the evidence for hydroxychloroquine, incorporate this new evidence, and update our beliefs.
What is a systematic review?
A systematic review is an up-to-date and unbiased review of an idea or treatment. The core concepts of a systematic review are as follows:
Unbiased - Start from scratch without preconceived notions. Do not bring personal beliefs into the review.
Asses the quality of evidence - there is a hierarchy of the evidence. Some evidence is better than other evidence.
Comprehensive - Look at all the evidence. All the studies. Evidence that confirms the treatment and evidence that disconfirms the treatment.
Ultimately, the main objective of the systematic review is to avoid the human tendency to confirmation bias. Eloquently explained by Voltaire,
"The human brain is a complex organ with the wonderful power of enabling man to find reasons for continuing to believe whatever it is that he wants to believe."
A systematic review of hydroxychlorquine
I will review the evidence for hydroxychloroquine and sort it by different evidence types. Bear in mind, the evidence is coming very quickly and many studies have not been peer reviewed. I will do my best:
In vitro studies - in vitro literally means “in glass”; studies done in a test tube or in the lab. Some in vitro studies try to find compounds that can reduce SARS-CoV-2 viral replication. Others try to identify protein targets for the virus, and suggest compounds known to block or disturb that protein. These high-throughput screening studies search libraries of thousands of compounds. They can generate a lot of potential drug candidates. One study for example, generated over 69 different agents, and this was after limiting the search to drugs already in use. This high hit rate is not evidence for the reliability of in vitro studies, but quite the contrary - in vitro evidence is not specific and generates mainly false positives. Hydroxychlorquine is one of the many compounds mentioned in these studies. This is weak evidence at best.
Mechanism of action - It's nice to know how a drug works. It does not prove that a drug works - but it helps. Moreover, it's not essential to know the mechanism of action - there are many drugs that work, but we don’t really know their mechanism of action. Nonetheless, in vitro studies of hydroxychloroquine demonstrate that it has anti-viral properties; it can prevent viral replication of SARS-CoV-2. It's easy to see how this could help the outcome of the infection. Hydroxchloroquine also has anti-inflammatory properties; it is used in autoimmune diseases to suppress the immune system. How could this help COVID-19? It has been hypothesized that the real problem with COVID-19 is not the infection, but the immune system response. SARS-CoV-2 stimulates the immune system too much, causing swelling in the lungs. Hydroxychloroquine could potentially mute this response. (If this is true by the way, you may want to avoid things that claim to “boost your immune system”?). However, none of these claims are specific to hydroxychloroquine; there are many compounds that are anti-virals and many compounds that are anti-inflammatory. Very few of them have proven to save lives with other similar viral infections. Therefore, finding a mechanism of action is not specific to hydroxychloroquine, and is weak evidence that a medication works.
Anecdote and uncontrolled trials - As discussed previously, a control group is essential when studying a disease that has a >95% recovery rate. Most patients with COVID-19 will recover with no treatment at all. Consequently, if someone gets better after a treatment, it is much more likely to be the natural history of the disease than the treatment. Because of these characteristics, COVID-19 is the perfect set-up for false anecdotes. Treatments that look like they work but don't. For example, the alternative medicine doctor, David Brownstein MD, is recommending breathing treatments of iodine and hydrogen peroxide in the lungs. (Don't try this at home kids!!). Iodine and hydrogen peroxide can kill the virus in a Petri dish - why not breathe them into your lungs with a nebulizer? In the video below, he sits with one of his patients who relays the harrowing story of his COVID-19 disease, the nebulizer treatment, and how it saved his life.
Dr Brownstein takes full credit for the recovery, of course, painfully reminding us of another famous quote by Voltaire, “Let nature do the healing, and the doctor takes the fee”. In a bizarre twist of events, the patient admits in the video that he actually never got tested for COVID. He just assumed he had it because he was “really sick”, highlighting the need to fully interrogate any anecdote you hear. If you are willing to accept the anecdotes of hydroxychloroquine, you should be willing to accept the anecdotes of nebulized iodine and hydrogen peroxide. There is nothing that makes the anecdotes of hydroxycholorquine any more credible than the anecdotes of nebulized iodine and hydrogen peroxide. Either all of these treatments work, or anecdotes are misleading. You decide. According to evidence-based medicine, anecdotes are very poor quality evidence.
The French trial - this trial from a couple a weeks ago gained a lot of media attention. It was very small and improperly controlled trial with many shortcomings. Please see my review and others.
The French trial part 2 - the authors of the above trial released a purely observational trial with no control group. It followed a series of 80 patients treated with hydroxychlorquine +/- zithromax. Many of these patients had decreased virus counts in nasal samples after a week of treatment. Without a control group, it is essentially a series of anecdotes - the data is pretty much meaningless.
Chinese controlled trial from last week - There was a small controlled trial from China last week. (I am unable to read the details as it is was posted in the Chinese language). Apparently, 15 patients received hydroxychloroqine, and 15 patients did not. It showed that hydroxychloroquine did NOT clear the virus from nasal samples any faster than the control group.
The new Chinese controlled trial - 62 moderately ill patients were randomized to either hydroxychloroquine 200mg twice a day for 5 days or standard care (the control group). The hydroxychloroquine group had a more impressive reduction in fever, cough, and pneumonia on CT scan. However, the trial has some shortcomings. It only measured surrogate markers of infection (fever, cough, CT results). They did not measure the most important outcome - death. Since hydroxychloroquine has anti-inflammatory properties, improvement in signs of inflammation like fever and cough are not surprising. An aspirin (an anti-inflammatory) may reduce fever in any infection, but this does not translate to reduction in death. Other shortcomings include: not peer reviewed, small study (68 patients), and no reports of statistical significance (p-values).
Summary - The in vitro data, mechanisms of action, anecdotes, and uncontrolled trials are low quality evidence. Positive results from these types of evidence are pervasive and not specific to hydroxychloroquine. The two controlled trails of hydroxychloroquine were small, measured surrogate markers, and had conflicting findings.
Updating beliefs with Bayes Theorem
Bayes theorem or Bayesian inference is a probability theory we use in statistics and medicine. It is not as complicated as it sound - so stay with me. Bayes Theorem helps update our beliefs when we are presented with new evidence. Let’s say we have a certain belief about a treatment working or not. This is called the "prior probability". A new trial is done which shows the treatment works. Bayes theorem allows us to update our beliefs. How much higher is the probability given this new positive evidence?
We can calculate the updated belief by knowing the three following variables:
The prior probability of the treatment working p(H)
The strength of the new trial p(E|H)
The likelihood of bias in the new trial p(E)
Bayes Theorem written in mathematical form:
(you can ignore it, I just put it there as a shout out to my fellow geeks):
p(H|E) = p(H) X p(E|H)
p(E)
H=Hypothesis
E=Evidence
p(H|E) = The probability a hypothesis is true given the evidence
p(H) = The prior probability the hypothesis is true
p(E|H) = The probability of getting the evidence given hypothesis is true
p(E) = The probability of getting positive evidence
It's most important to determine the prior probability p(H) first. Before we learned about the new Chinese study, what was the likelihood that hydroxychloroquine worked? I believe about 1%. (To be clear, what I mean by "working" is preventing death - not just reducing viral counts, fever, or cough). When speaking with my friends, relatives, and patients, they guessed a prior probability of 50%-70%. Very different than 1%. Why? Unfortunately, non-doctors are not privy to our failures in medicine as much as they are our successes. Every successful drug you see lies on a mountain of failures. Only about one in a hundred drugs that look good in vitro and supported by anecdotes, turn out to be both safe and effective in large trials. 1% is our success rate for drug discovery - and that's when we have time and things are good.
The next thing to determine is the strength of the new study p(E|H). If the new study is very very strong, it can compensate for a low prior probability. In this case, the new Chinese Hydroxychloroquine study does not report statistical strength, p-values are not given. Furthermore, the study was small in size and only looked at surrogate markers. Being kind, I would call it low-to-moderate strength evidence.
Finally, we need to evaluate the likelihood of bias in the new study p(E). We know there is a tendency for researchers to bias studies to show positive results - this is known as p-hacking. We also know that positive studies tend to be published more that negative one - this is known as publication bias. These two factors lead to an excess of false positives. How do we know this? The replication crisis. When basic and classic experiments have been repeated in fields such as psychology and biomedicine, the replication rate has been less than 50%.
When we plug these estimates into Bayes theorem, we get a 1% prior probability, multiplied by a weak-to-moderately strong study, divided by a moderate chance of positive bias. If I were to be kind to this new study, I could increase the likelihood that hydroxychloroqine works from 1% to 5%.
Why is Bayes theorem helpful in medicine?
It is easy to hijack evidence-based medicine. One can make even the most implausible treatments look like they work in clinical trials. Just design trials with small numbers of subjects, do not blind the subjects or the experimenters, measure many surrogate outcomes, and decide after the experiment which outcomes you want to present. It's easy to squeeze out positive results.
Let's take Homeopathy for example. Homeopathy is not what you think it is. It is not medicine with herbs. It is a system of medicine invented in the 1700's by a German doctor, Samuel Hahnemann. It involves the practice of diluting medicines past Avogadro’s number. Diluted so much so, that not a single molecule of active substance remains. Homeopaths acknowledge there is nothing but water in their nostrums. They claim, their medicines work because “water has memory”. In deed, there are many small clinical trials showing that homeopathy works. Are we to believe these trials?
This is when Bayes Theorem comes to the rescue. For homeopathy to work, everything we know about physics, chemistry, and biology must be wrong. The prior probability of homeopathy working is a as close to zero as any treatment I can think of. According to Bayes theorem we would require very very strong evidence to compensate for this. For instance, multiple very large well-designed and unbiased trials - which we do not have.
The essence of Bayes theorem is captured best by Carl Sagan's famous quote,
“Extraordinary claims require extraordinary evidence.”
Conclusion
This new study from China is promising for hydroxychloroquine. However, it is moderately weak evidence. It measures only surrogate marker outcomes and its results conflict with the only other controlled trial from China. Even if I give the study more credit than it deserves, it increases my prior probability from 1% to 5% that hydroxychloroquine works.
According to clinicaltrials.gov, there are at least 44 trials in the works testing hydroxychloroquine for use in COVID-19. As these results come in, I will be performing systematic reviews and updating my beliefs with Bayes theorem.
Once again, I am not writing this as a recommendation to take or not to take hydroxychloroquine. That is up to you. The threshold one has, to take a medicine or not, changes from person to person, and situation to situation. I am only trying to assign a probability of whether it works or not.
(In my next email, I plan to discuss what it means to take a medicine that has a 5% chance of working)