Kaletra is an anti-viral medicine used for HIV. It has had some success with other Coronaviruses, SARS and MERS, in uncontrolled trials. It is now being studied with the new Coronavirus virus (SARS-CoV--2). In vitro studies have shown promising results - meaning it can reduce replication of the virus in a test tube. Finally, we understand its mechanism of action - it is a protease inhibitor for viruses. Promising. However, until recently, there were no large randomized controlled trials to prove if it works or not.
Four days ago, the LOTUS trial was published; this is the first moderately large and well-controlled trial of a drug against COVID-19. One hundred and ninety nine patients with severe COVID-19 infections were randomized to get Kaletra or standard care. Despite the promising preliminary evidence, the results of the trial were disappointing. The main outcome, time it took to get better, was the same in the Kaletra group and the control group.
The goal of this email is to review the key features of the Kaletra trial and compare it with the French hydroxychloroquine trial that I recently reviewed. Ultimately, I want to teach you how to judge the quality of a trial. How likely is it to answer the question, "does this treatment work or not?"
Is there a risk of bias in the trial?
When a trial has negative results, like the Kaletra trial, it's probably not biased. Everybody wants their medicine to work - everyone wants their trial to be positive. So looking for a source of bias in a negative trial is irrelevant.
The hydroxychloroquine trial yielded positive results. Any positive trial for COVID-19 carries with it a huge potential for bias. Imagine being the first one to find the cure for the virus of the century?
Are there lots of researchers involved?
Yes. It was started in the beginning of the outbreak in January and was conducted at a single hospital, the Jin Yin-Tan Hospital, Wuhan, Hubei Province, China. However, there were over 70 authors listed. This helps to counter-balance any one large source of bias. The hydroxychloroquine trial had only three authors.
Is it peer reviewed and published in an high impact factor journal?
Yes. It has been peer-reviewed, editorialized, and published in the New England Journal of Medicine. This is what a published study looks like. The French hydroxychlorquine study is yet to be peer review and published, and was sent to most of us in a Word document.
Did patients drop out of the study and why?
No. There was no drop out. (This is China we're talking about). 199 patients entered the trial and all 199 patients were accounted for by the end of the trial. The French hydroxychloroquine trial had 6/26 patients drop out of the treatment group mainly due to worsening conditions and death. Conveniently, these worsening patients were not inclu in the final numbers (intention-to-treat analysis).
Was the primary outcome a hard outcome?
Yes. The main outcome of this trial was time to get better. This is a pretty hard outcome. Death is a harder outcome - and all we really care about. It was not chosen as the primary outcome of this study because the sample size was not large enough.
The main outcome of the French hydroxychlorquine study was measurement of the virus in the nose - a surrogate marker. Ironically, if one teases out the death rate in the study, the hydroxychloroquine treated patients had one death, control group zero.
Is there a control group?
Yes. The trial started with 199 patients. Patients were randomly assigned to receive Kaletra (99 patients) and the control group (100 patients) received only standard care. However, there was no placebo given to the control group because the study was rushed. Therefore, patients knew if they were getting the treatment or not - we call this an open-label study. Consequently, it is possible that placebo effects may have improved the results of the Kaletra group.
Moreover, the doctors, researchers and nurses knew which patients were getting the treatment and which were not. This may have also improved the results of the Kaletra group. A phenomenon known as experimenter-expectancy effect and why we like double blind studies (blind the experimenters). However, as this was a negative study, all of this is irrelevant. With all of these advantages, the Kaletra group was still no better than the control.
In the French hydroxychloroquine study, the control group were patients in another hospital and patients who refused treatment - this is not a control group. Patients in another hospital and patients who refuse treatments are probably different from the treatment group.
Is it a large sample size?
Yes. Moderately large - 199 patients. This number of patients is certainly large enough to determine large effects of the drug. A larger sample size would be required to pick up smaller effects of the drug. Yet, for this crisis, most would argue, we don't need a drug with a small effect.
The French hydorxychloroquine trial had 14 patients on the drug and 20 in the control group. A very small trial in deed.
Lessons learned
The Kaletra trial was a large randomized controlled trial with little risk of bias published in a high impact journal. The results are very useful; it certainly helps answer the question, "does this medicine work or not?". Kaletra probably does not work.
The trial also confirms one of the most important principles of evidence-based medicine. Medicines like Kaletra, that are supported by anecdotal evidence, uncontrolled trials, and perform well in test tubes, most often fail in larger controlled trials. This is the pillar of evidence-based medicine. It is known as the hierarchy of evidence. Not all forms of evidence are equally reliable. Lower forms of evidence tend to produce false positives which are often refuted when put to the test with more rigorous forms of evidence. When trials are designed with a control group, blinding of both subjects and experimenter, a large sample size, and a hard outcome (like death), treatments often fail.
The French hydroxychlorquinine study was a small uncontrolled trail with surrogate end points, a high risk of bias, unpublished, and not peer reviewed. Hence, the results of the trail cannot be trusted and do not help us answer the question, "does this medicine work?". At this point, we can be as confident of hydroxychloroquine working as we did about Kaletra before the Lotus trial. Unfortunately, the rules of evidence-based medicine and the results of the Lotus trial, predict a high probability of failure. Larger well controlled trials, like the Lotus trial, are currently being conducted for hyroxychloroquine. I am not very confident. I hope I am wrong.
One note of encouragement is that death at 28 days was lower in the Kaletra group (19%) compared with the control group (25%). However, this was not a predefined primary outcome measure and it did not reach statistical significance. It is possible that a larger trial would show statistical significance. It is also possible Kaletra needs to be started at an earlier point in the infection to make a difference. Does this mean more Kaletra trials are needed?
Not really, if Kaletra was working one would expect the viral loads in the treated patients to go down.Yet, viral loads did not differ at all between the Kaletra and the control group. (See graph below of viral loads over the 28 days). It is important to note that reduction in viral loads does not guarantee a treatment will work, it is only a nice start.
More to come
The fact that the Chinese government was able to put this trial together so close to the beginning of the outbreak, with all the pandemonium in their hospitals, is impressive. This is just the beginning of trials like these - there will be many more over the weeks and month to come.
I will continue to search for, and evaluate, potential COVID-19 treatments for you. If you see a trial and want me to evaluate it, or you want me to teach you how to evaluate it - it would be my pleasure. To start with, try and answer the following questions about the trial:
Is there a risk of bias?
Are there lots of researchers involved?
Is it peer reviewed and published in a high impact journal?
Did patients drop out of the study?
Was the primary outcome a hard outcome?
Is there a control group?
Is it a large sample size?
Comments