I had previously suggested that you should seek out a few concrete examples of how scientific data and information are used in medicine. In the upcoming blog we’ll clarify many of the concepts. If you have already done your homework you are ahead of the game, because you will be able to follow even better.
You will have noticed that „Evidence Based Medicine“ (EBM) assumes a hierarchy of evidence. Its lowest levels are the clinical anecdotes, the uncontrolled clinical experience, and the experts’ opinions. Above these lie case series and single case studies, which are somewhat more systematic but are still isolated cases. Above that are the studies done with groups of patients, so-called cohort studies, in which patients treated with one method are compared with those treated with a different method. These groups have chosen their treatment themselves, or the doctors did. At the pinnacle are randomised controlled trials (RCTs). These are comparative studies, where the patients or their physicians haven’t selected the treatment method. Such RCTs are more sophisticated and true clinical experiments because these allocations of the patients to the treatment condition were made randomly. In theory, the random assignment of patients has the advantage that the patients’ characteristics are then randomly distributed, making the two groups more comparable. Thus it is more likely that the difference found between the groups is caused by the intervention being tested and is not due to random fluctuations in the population. The common assumption is that „randomized trials“ in which the computer does the allocation are better scientific studies and, therefore, they are held in higher regard by scientists and editors of scientific journals. In some ways this approach makes sense, but I will talk more about that later. Finally, the results of several studies on the same topic can be combined into a meta-analysis, a study that summarizes all the single studies, and is then used to estimate the therapeutic effect. Thus, the evidence gained is thought to be better, the diamond of evidence, as it were. Many of the guidelines that you will have found in your homework are based on such meta-analysis derived data.
What is behind it?
The model of the hierarchy of evidence
The superiority of meta-analysis derived data is explicitly based in a hierarchical understanding of knowledge or evidence. The more controlled a study is and the more experimental the procedure, the more reliable are the conclusions that can be drawn even from a single study. This is undoubtedly true. Therefore, it is assumed that the only really important thing is the reliability of these conclusions. In scientific jargon it is also called the internal validity, i.e. the validity of a study due to its methodological characteristics of control, randomization, and blinding.
If one accepts the premise that the reliability of the conclusion is the only really interesting and important feature of scientific studies, then it can be reasoned that all other studies of lower internal validity are not as „good“ and are only precursors of and preparations for definitive studies. These studies can be ignored in the presence of high „quality“ studies, i.e. randomized trials. As a result, only randomized studies are summarized in forming a medical guideline, despite the fact that even the results of these high quality studies often fluctuate. It is difficult to have an overview of all the findings. This is where meta-analysis comes into play by proposing a statistical procedure that mathematically summarizes the results of various studies and estimates the likely „true“ therapeutic effect hidden behind the interventions in the individual populations. Ideally, the effects of the individual studies are spread out estimates of this „true“ effect.
The approach of creating „evidence“ in the modern, scientific sense is derived from the theory that each study can be ranked by its worth based on its formal features: such as whether or not the study is experimental, among other evaluation criteria. If there are many (sometimes contradictory) findings, the situation becomes confusing. Scientific reviewers, or meta-analysts, like to use the following procedure: they formulate inclusion criteria for their analysis, for example „only randomized, blinded studies written in English and with at least 50 patients per group.“ This reduces considerably the effort required to obtain and screen the scientific literature. The complexity of the topic is reduced and it is easier to arrive at a conclusion. We will ignore for now the individual steps and the implications of such findings. Our current main interests are the requirements that lie underneath and the plausibility of this general approach.
It is clear that this approach is based on the idea of the „hierarchy“ or „pyramid“ of evidence. The „true effect“ of an intervention was distilled from the many individual randomized trials (the „high quality“ ones) to create the summit of this evidence pyramid. The rest is unimportant for it is only scientific mash serving the distillery of the truth in the same way 1000-liter barrels cherries are needed to burn a few liters of cherry brandy. Pigs get the mash, the liquor is sold or consumed.
How plausible and valid is this theory?
We will analyze this approach in three steps. First, we will take a closer look at the paradigmatic assumptions. Second, we will analyze the methodological background, the decisions made, and consequences created. Third, we will return to our example of guideline-based treatments and evaluate if this approach leads to useful results.
1. Is reality hierarchical?
The formulation of medical guidelines is based on the hierarchy of evidence: reality, including knowledge and perception, is assumed to be also hierarchical. We typically see hierarchies, pyramids, etc. in all feudal systems of the past. At the top is a king or emperor, a Pharaoh or God, who delegates rights and obligations to others, and the others have to report back and be accountable to this leader. History has shown that feudal systems were not very flexible. They are marked by the weaknesses of the individuals on top. Such a system can only flourish when a very wise, very strong and very healthy leader is at the top. Then a huge burden is carried by only one, and perhaps all can still flourish in the system. Historical examples of great kings and leaders are those under whose reign peace prevailed: such as Frederick the Great of Prussia, Austria under the rule of Maria Theresa, France under St. Louis or Louis XIV, and Rome under Augustus or Hadrian. If a leader is weak, sickly, mentally disturbed, dies young or is just not very bright, the position of the entire system becomes hairy. Nazi Germany is a prime example of the direction of hierarchical systems under wrong leadership. Modern, functional political systems are indeed hierarchically structured, but they also have control mechanisms in place to restrict absolute power. Hierarchy in such systems facilitates pathways of gathering information, of arriving at a consensus and of communication to the outside, but these pathways always have a cyclical feedback loop and control mechanisms to and from “below” which result in it staying connected with its basis.
What is the structure of big business organizations? Here also it is clear that over the last few decades the most successful companies have become less hierarchical and more modular in their structures. The CEO, of course, remains as a leader with great power, but a good and modern CEO will refrain from decision making without knowing all the aspects of his business. The hierarchy here is also “flat” due to feedback procedures to the basis where all individuals participate in decision-making and access to power.
What about other important areas in life? For example, does all non romantic love (for parents, friends, children) only serve to approximate you to your only true romantic love (for your wife, your husband, partner etc.)? Of course not. It is clear that the more love we experience in one area of our life (such as non romantic), the more we have the capacity for love in other areas of our life (e.g. romantic). The reverse is also true; if a love relationship with a partner works well, this love also reflects on our children. Obviously this is a cyclically influencing and enriching process.
Does the same hold true for organic systems? Let’s look at the immune system. Is there a central control of immune recognition? No, obviously not. The immune system is a highly cross linked, extremely efficient and decentralized system. If anything, only the brain could be considered its headquarters by coordinating conscious behavior in the case of an illness, such as resting when we have a fever. Otherwise, the system is cyclically organized and a hierarchy only in the sense that control processes that come from above are only activated in emergencies.
How about consciousness and our neural system? This is an interesting example. In the early days of brain research, namely the phrenology of the 19th century, scientists believed in a hierarchical structure with a special center of „consciousness neurons“ that gathered all the information from our awareness. More specifically this would have meant, for example, that my brain has „grandmother neurons“ that serve to combine all the information I need to identify my grandmother and to create a conscious recognition: „This is my Grandmother.“ This hierarchical notion paralyzed research for a long time. We know now that the organization of the neural system is highly decentralized and modular, and that consciousness is somehow created everywhere through the coordinated work of many neuronal centers.
Actually, the information of my perception of my grandmother (her appearance, her smell, the sound of her voice, her specific way of moving, her body size) is processed across different brain centers, and only through their coordinated effort do I arrive at the conclusion that this is my grandmother. Hierarchy does exist in the brain, namely in the form of consciously chosen goals, values and wishes „from above“, i.e. from the regions of the cerebral cortex. These goals, values, and wishes can have an inhibitory effect on the other centers to promote unity and harmony of conflicting processes. Therefore, the brain also has a very delicate mixture of circular and modular processes that are moderately hierarchical but always work in feedback with deeper lying centers.
I could mention more examples to show that no area of our known reality is constructed purely hierarchically, except, perhaps, hospitals, the Catholic Church and other outdated institutions that through their adherence to outdated structures show themselves as candidates for the evolutionary dead end. It seems to me that the preferential medical methodology comes from the implicit medical hierarchy which is transmitted without further reflection.
If nature and reality are taken as a paradigmatic example, then one would have to use a more complex metaphor in the creation of knowledge. The cyclical feedback loops would have to be included in the methodological process of creating evidence.
2. A circular alternative
The symbol or metaphor I propose is a circle or a spiral. Certainly, knowledge and insight is the result of many individual steps and thus has a kind of hierarchical structure. An insight is more than the individual steps that have contributed to making it. The key point here is that we only get to a realization if we bring all the steps in a cyclical form, i.e. in relation to each other, to a supposed synthesis processing of the information. Technically speaking, if the supposed correct effect size was identified from all the existing studies, then the question should arise: How does this effect size we have gained from experimental studies hold up in real life settings? Can we expect to see that same effect size (and the same side effects as in the studies) in all potential patient groups, even in those not included in the trials? Will a treatment that worked in an experimental study of patients with no preference for any method also work in patients who are skeptical, desperate, or have lost their faith in themselves and medicine? Can an intervention tested in patients with „simple“ diseases without multiple diagnoses be applied to „real“ patients that not only suffer from high blood pressure but also depression, anxiety and diabetes?
Conclusively, the experimental studies testing the effectiveness of new drugs and interventions favor internal validity, the replicability of the conclusions, instead of the external validity in the form of the generalizability of the results. Patients are carefully selected for experimental studies. Patients with complicated diseases due to multiple diagnoses, etc. are never included. This is how the current hierarchical model works. It follows that the result gained from such an approach has to be linked back to the basis and to practice. This feedback to reality has to be studied in a different way. New approaches to clinical trials are needed if one wants to know whether other patients also benefit from an intervention. One has to conduct a study without exclusion criteria, and eventually simply keep a record of general practice documentation, a register. Alternatively, one can compare naturally occurring groups, one treated with the intervention and one treated differently. The currently favoured method of randomized studies also excludes and ignores the most important element of the treatment, the patient’s conscious decision to do something for his or her health.
So I plead for a cyclic process of evidence gaining. Those who require hierarchies could imagine a hierarchy of cyclic processes which refine and improve our knowledge in the manner of a spiral. I will explain more about how this cyclic model of evidence works in a later post. For more information on how I argue and support this theory, see the literature listed below. Interestingly enough, by now even the leading methodologists agree with me, to some extent.
- Tinetti, M.E., & Studenski, S.A. (2011). Comparative effectiveness research and patients with multiple chronic conditions. New England Journal of Medicine, 364, 2478-2481.
- Walach, H. (2006). Zirkulär statt hierarchisch – methodische Überlegungen zur Evaluation komplexer therapeutischer Massnahmen [Circular instead of hierarchical – methodological considerations for evaluating complex therapeutic interventions]. In Boedeker, W. & Kreis, J. (Eds.), Evidenzbasierung in Gesundheitsförderung und Prävention [Evidence base in Health Promotion and Disease Prevention] (pp. 33-56). Bremerhaven: Wirschaftsverlag NW.
- Walach, H., Falkenberg, T., Fonnebo, V., Lewith, G., & Jonas, W. (2006). Circular instead of hierarchical – Methodological principles for the evaluation of complex interventions. BMC Medical Research Methodology, 6 (29).