Hierarchy of evidence and sleep apnea
Updated: May 2, 2020
Medical decision making in the 21st century has changed from a patient’s perspective because of two very powerful forces - 1. the exponential rise of biomedical studies, and 2. the internet/social media. This has led to an explosion of thought-provoking ideas, a democratization of information (anyone can access any study - pubmed), and empowerment of patients to be involved in their own medical decisions. But at the same time patients are now, more than ever, exposed to bad information synthesized by non-experts or experts with a conflict of interest. ‘Biohackers’ like Tim Ferris are examples of non-experts that popularize and sell untested claims with surprising confidence. And business entrepreneur Dave Asprey who started his popular Bulletproof empire after noticing his fit and thin Sherpas in Tibet drank coffee with yak butter. But what is really troubling is that the internet/social media seems to be significantly weighted to the outlandish untested claims than to the sites that espouse critical thinking.
A medical Rosetta Stone
The combination of patient empowerment and prevalence of bad information makes it essential to teach patients how to critically evaluate a medical claim. This is one of my primary goals as a physician. It TRULY empowers patients - it is like having a medical Rosetta Stone - a tool that deciphers the infinite biomedical knowledge contained within the internet and allows for the ranking of ideas and treatments. Otherwise it's just hundreds of thousands of claims with very convincing sounding testimonials from slick TV docs, yoga instructors, conspiracy theorists, and business entrepreneurs. How do you know which ones are true - especially when some of the claims contradict one another?
The Hierarchy of Evidence
Evidence is not all the same. Some types of evidence have been found to be more reliable and accurate than others. The randomized placebo controlled trial is the most rigorous form of evidence. It is hard to fool. The other forms of evidence, such as anedote, observational study, or physiology experiments often lead us to think a treatment works when, in reality, it does not. In this post, I will be using a recently published sleep apnea study to demonstrate this hierarchy of evidence. It turns out that although evidence from observational studies, physiology experiments, and surrogate markers indicated that sleep apnea would prevent heart disease, this was not supported by randomized controlled trials. Treating sleep apnea does not appear to help heart disease. I will briefly try to explain in the following sections the differences and pitfalls of these types of evidence. But the lesson... be careful drawing conclusions from weaker forms of evidence before randomized controlled trials are performed. (below is the evidence pyramid - the strongest forms of evidence are near the top of the pyramid)
What is Sleep Apnea?
Sleep apnea is a prevalent disease that starts in middle age and is more common in males. Risk factors include diabetes and obesity. Most people with sleep apnea snore - but not everyone that snores has sleep apnea. If you don’t snore you likely do not have sleep apnea. The physiology of the disease involves the relaxation of ligaments, tendons, and muscles in throat during sleep, with the help of excess adipose tissue the airway becomes obstructed and breathing stops for several seconds. The build up of carbon dioxide and lack of oxygen triggers the brain to wake you up and forces you to breath. This can disrupt important phases of sleep, like REM, leading to poor sleep quality. See video for demonstration. Most patients with sleep apnea are completely unaware that they have these events. Sometimes a bed partner may witness these events and this can be helpful in establishing the diagnosis. The most common symptom is daytime hypersomnolence - being able to fall asleep easily during daytime. After lunch you can easily take a nap, on a long car ride, etc. You can take the Epworth test to see how symptomatic you are. Patient’s with sleep apnea have several options for treatment but the most popular and effective is CPAP - a mask that gently blows air into the mouth during sleep.
It has been hypothesized that sleep apnea also leads to heart disease and treating it will reduce heart disease. Why?
Cohort studies, case-control studies, and cross sectional studies are all forms of observational studies. If you take a group of patients with sleep apnea and compare them to a group of patients that do not have sleep apnea - the sleep apnea group will have more heart disease. These results have been consistently replicated. There is a definite ASSOCIATION between sleep apnea and heart disease. Unfortunately, an observational study cannot determine CAUSATION. It cannot tell you if sleep apnea CAUSES heart disease? In fact, the diabetes and obesity (confounding factors) associated with sleep apnea could cause the heart disease. Or possibly heart disease causes sleep apnea (reverse causality). A very important distinction - because if sleep apnea does not CAUSE heart disease, treating it will not prevent heart disease.
In other words, association is not causation. Assuming when something occurs together with something else that it causes that something else is a very common logical fallacy (Cum Hoc, Ergo Propter Hoc). One of my favorite examples of association is not causation is the following: If you go to the woods and see lots of bear scat, it is ASSOCIATED with bears and a higher likelihood of running into a bear. If you clean away the bear scat it won't reduce your chances of running into a bear.
Reasoning with evidence from physiology (in vitro studies)
Sleep apnea can result in episodic hypoxemia (low oxygen) that leads to elevated blood pressure, elevated stress hormones (adrenaline and cortisol) oxidative stress, inflammation, and hypercoagulation (thicker blood). One can easily argue how this may cause heart disease.
But.. the problem with reasoning from physiology is that human physiology is so complex and has so many facets that almost any argument can be both supported OR refuted with an explanation from physiology. Sleep apnea, for example, can even be viewed as beneficial from a physiologic basis. A sleep apnea event is similar to exercise. Both cause a decrease in oxygen leading to a release of free radicals and stress hormones and an increase in blood pressure. This negative or ‘toxic’ stimulus leads to cell signaling that grows new blood vessels and regenerates mitochonidria and strengthens the body - aka hormesis - what doesn’t kill you makes you stronger. So reasoning from physiology or in vitro studies is not a strong form of evidence and can lead to spurious conclustions.
Treating sleep apnea with CPAP can improve biomarkers in patients - lower blood pressure, improve insulin resistance, enhance endothelial function.These are known as surrogate markers of the disease. For example, high blood pressure is not heart disease, but we know that it is strongly associated with heart disease. Therefore, if treating sleep apnea reduces blood pressure than it could help heart disease. Researchers love surrogate markers because it makes studies less expensive and shorter in duration. Instead of measuring heart attacks, just measure if blood pressure improves.
But... surrogate markers are an indirect form of evidence and suffer from the same 'association is not causation' problem above. In fact, the medical literature is littered with countless examples of how the use of surrogate markers have led to wrong conclusions. It is preferable to find sleep apnea treatment studies that measure HARD ENDPOINTS like heart attacks or death.
Randomized Controlled Trial (RCT)
To formally test the above ideas we need to perform an RCT. This involves taking a large group a patients that have sleep apnea and heart disease. Treat half of the group with CPAP and the other half with fake CPAP. Follow the patients for several years and see if there is a difference in hard endpoints (heart attacks). If CPAP works then the group getting CPAP should have less heart attacks. This is the best test of a treatment but, as you can imagine, it is very difficult to organize and expensive to perform. The recent sleep apnea study pools the results of 10 clinical trials including a very large one from 2016 with 2300 patients - and the results... treating sleep apnea does not reduce heart disease. (It is important to mention that there are problems with RCTs - they are not perfect. And the above sleep apnea clinical trials have been criticized. However, that goes beyond the scope of this correspondence.)
For decades most of the weaker forms of evidence were pointing to a cardiac benefit from treating sleep apnea with CPAP - it does not appear to be the case. Although patients often feel fantastic and have improved energy from treating sleep apnea, I don’t think I can support the notion, at this point, that treating sleep apnea will have an impact on heart disease. I do not think the case is closed on this topic and I could be easily convinced to change my opinion given new evidence.
It has been stated that the greatest medical advancement of the 20th century was not the discovery of antibiotics or the mapping of the human genome but the development of evidence-based medicine - the use of randomized controlled trials and placing them into a hierarchy of evidence to evaluate medical treatments and claims.