- Open Access
Coherence of evidence from systematic reviews as a basis for evidence strength - a case study in support of an epistemological proposition
BMC Research Notesvolume 5, Article number: 26 (2012)
This article aims to offer, on the basis of Coherence theory, the epistemological proposition that mutually supportive evidence from multiple systematic reviews may successfully refute radical, philosophical scepticism.
A case study including seven systematic reviews is presented with the objective of refuting radical philosophical scepticism towards the belief that glass-ionomer cements (GIC) are beneficial in tooth caries therapy. The case study illustrates how principles of logical and empirical coherence may be applied as evidence in support of specific beliefs in healthcare.
The results show that radical scepticism may epistemologically be refuted on the basis of logical and empirical coherence. For success, several systematic reviews covering interconnected beliefs are needed. In praxis, these systematic reviews would also need to be of high quality and its conclusions based on reviewed high quality trials.
A refutation of radical philosophical scepticism to clinical evidence may be achieved, if and only if such evidence is based on the logical and empirical coherence of multiple systematic review results. Practical application also requires focus on the quality of the systematic reviews and reviewed trials.
Epistemology is described as the branch of philosophy that concerns itself with questions regarding human knowledge . One particular epistemological question relates to the Object - Subject distinction and asks whether the objective world is subjectively knowable at all. It has been proposed that the real (objective) world and our (subjective) perception of the real world are not the same. Descartes (1641) argued that all that we can know of the real world is tainted by our senses and abilities of understanding . Kant (1998 ) distinguished between the unknowable Ding an sich (German: The thing in itself) and the knowable Erscheinung (German: Phenomenon) . Postmodernism contends that the notion of reality is an illusion . Quine (1964) described physical objects as mere cultural posits , and cognitive psychology and neuroscience have presented evidence that sense experience, the 'bedrock of empirical knowledge', is actively edited by human perception . Such 'editing' renders sense experience subjective to the particular observer .
All these examples [2–6] share the common concept that some form of bias determines subjective perception concerning real world events. Bias, also called systematic error, manifests in various forms and number, and constitutes any factor in the knowledge acquisition process that systematically diverts its outcomes from true values . The concept of knowledge has been defined as true and justified belief . Systematic error, therefore, limits the trueness and thus justifiability of such belief (thus its value to be considered as knowledge) and so widens the gap between the real (objective) world and our (subjective) perception of it. In order to limit the influence of some forms of systematic errors in the field of healthcare, methodological interventions for clinical trials, such as randomisation, blinding and intention-to-treat analysis, are proposed  and have been shown to be effective [9, 10]. Clinical trials that employ such methodological interventions are identified and synthesized through systematic reviews. Systematic reviews present: (i) a synthesis of the acquired knowledge regarding one particular clinical question derived from all relevant studies that are identifiable at one point in time, (ii) identify the level of validity and the subsequent potential systematic error risk associated with the acquired knowledge and (iii) provide recommendations for improving any identified shortcoming related to internal validity, for further research. Owing to continued further research, systematic reviews also provide continued updates of their synthesis . For this reason, in most cases systematic reviews provide the most objective knowledge possible about real world events; such as answers to clinical questions concerning causalities, with the least possible systematic error.
However, the described distinction between the real (objective) world and (subjective) perception of the real world [2–6] has also given ground to radical, philosophical scepticism about whether it is possible to gain knowledge of real (objective) world events, at all . Radical, philosophical scepticism does not just question current standards of evidence: it completely denies that any causality existing independently from our perception in the real world is knowable at all; regardless of the availability and strength of evidence, e.g. as presented in systematic reviews that supports knowledge of such causality. By disregarding such evidence, radical, philosophical scepticism denies any justification for or against any perception of objective knowledge. Among the main modes of argument utilized in radical, philosophical scepticism are the five modes of Agrippa [1, 11]. These modes of argument state: (i) for each thesis there is a possible antithesis but without the option of Fichte's synthesis through thesis and antithesis  - because: (ii) each point of view is relative to each particular claimant and thus cannot hold general truth; because: (iii) proof of any point of view or thesis is not possible, as every proof, evidence or justification requires for its own proof, evidence or justification, thus entering into an infinite regress of proof that is incapable of proving anything; (iv) to end such infinite regress of proof by refusing further proof shows lack of proof; and (v) to reason in circles by referring back to already stated proof also shows lack of proof .
In accordance with the Agrippean argument, evidence elicited from single systematic reviews as proof in clinical practice is vulnerable to radical scepticism. An example is the argument that population-based research (research conducted, e.g. through randomised control trials (RCT) and subsequently appraised through systematic reviews) cannot be applicable to the treatment of individual patients . The validity of population-based results, so it is argued, may only be applied to the population as such but remains irrelevant to the single patient, as individual patient data and a population average are never the same . Such criticism denies the ability of systematic review evidence to describe objective causality. It employs the Agrippean argument that: (i) like any other thesis, the thesis of objective evidence by systematic reviews is contradictable; (ii) the thesis of objective evidence by systematic reviews (the population average) is relative and thus cannot be valid for the individual patient because the general validity of such thesis cannot be proven (see Agrippean modes of argument iii - v).
The aim of this article is to offer the epistemological proposition that, with reference to the Coherence theory and illustrated by one case study, evidence from multiple systematic reviews, if mutually supportive (= logically and empirically coherent) may be able in principle and on philosophical ground to terminate the infinite regress of proof and thus refute the Agrippean argument against it. This article further aims to highlight some aspects concerning the translation of such proposition from philosophical grounds into health care praxis.
The key to refuting the Agrippean argument is to terminate the infinite regression of proof. The radical sceptic rejects refusal of further proof and circular reference to proof, preventing formation of a successful refutation. A possible epistemological solution to this problem is to assert that not just one single belief but many, are expressed in a thesis. Such beliefs do not exist independently from each other but form a system or web of mutually supporting beliefs that justify each other. This line of argument against radical scepticism has been called the Coherence theory [14–16]. The Coherence theory states that beliefs are justified if they are systematically interconnected and provide mutual proof for each other . Such mutual proofs do not constitute mere circularity but logical coherence, thus terminating the need for infinite regression of proof. According to Coherence theory in epistemology, justification of proof of a belief depends on how well it fits into the web of beliefs, thus stating that individual beliefs are justified by virtue of belonging to a coherent view. This argument has been criticized on the grounds that it allows any form of belief, no matter how unlikely or likely; to be considered as justified as long as it is surrounded by suitable supporting beliefs . Even if such supporting beliefs are as unlikely as the main belief, it may still form a coherent system. Against such background, the Coherence theory is unable to make a clear distinction between aspects of the real world (causality) and delusions. However, it has been argued that the Coherence theory is not a theory of truth but a theory of justification . The distinction between a coherent real world system and a coherent system of delusion can be made only on the basis of its relation to evidence. Such evidence has to be empirical: facts about the real world ascertained from observation and experiment. For a coherent system to be proof of truth, it not only needs to be: (i) logically coherent in terms of its interconnecting beliefs, but also (ii) coherent in terms of the empirical evidence on which all interconnecting beliefs are based (empirical coherence).
In order to show how the above principles of Coherence theory may be applied in support of specific beliefs in healthcare, a case study including seven systematic reviews [17–23], all co-written and published by the author, was conducted. These systematic reviews related to various aspects of the clinical use of glass-ionomer cements (GIC) and dental caries. The format of a case study was chosen in order to explore without high methodological restrictions the potential usefulness of Coherence theory in this context and to contribute new insights for discussion and further study. As Coherence theory is rooted in philosophical/epistemological thinking, this article limits its line of argument mainly on philosophical/epistemological grounds. However, some practical implications, relevant to the field of health care are also discussed.
The specific objective of this case study was to illustrate, as example, the refutation of radical scepticism towards the belief that GICs are beneficial in tooth caries therapy on the basis of logical and empirical coherence, and included the following steps:
Conducting and publication of systematic review articles addressing different review questions concerning the main belief regarding clinical benefits of GICs in tooth caries therapy;
Extraction of 'web'components, separated by common categories, that are contained in the systematic reviews;
Construction of a 'web-of-beliefs' frame from the extracted components;
Setting of conditions that need to be in place in the constructed 'web-of-beliefs' in order for it to be considered coherent;
Discussion about whether the specific information provided by the systematic reviews fulfils the conditions for (a) logical coherence and (b) empirical coherence.
Systematic review sample
The format of systematic reviews of trials was chosen above other forms of scientific investigation (i.e. randomised control trials, observational studies, clinical case reports or narrative reviews/expert opinion), as these are considered to provide the most comprehensive answers to clinical questions, with the least possible systematic error . All systematic reviews were of quantitative (as opposed to qualitative) nature. Some included meta-analyses for precision of results. The systematic reviews were undertaken in order to answer separate clinical review questions, but all with relevance to the main belief that GIC is beneficial in caries therapy. All systematic reviews were published in MEDLINE- listed peer-reviewed dental journals between September 2009 and February 2011. The language of publication was English.
Of the seven systematic reviews, three included conventional GIC (C-GIC) [17, 18, 22] and three, resin-modified GIC (RM-GIC) [19, 20, 23] as the test material. One systematic review investigated both versions of GIC, in comparison . Two systematic reviews investigated the fissure sealant performance of GIC [17, 19]. The other five investigated its performance as a material for tooth restorations [18, 20–23]. The investigated outcome measures of the seven systematic reviews were tooth remineralisation , absence of tooth caries [17–19, 21, 23] and long-time survival of tooth restorations .
The strength of systematic review evidence relies on (i) the quality of trials reviewed and (ii) the quality of the systematic review itself. The AMSTAR (Assessment of Multiple SysTemAtic Reviews) tool [24–26] was used for critical self-assessment of the quality of the systematic review articles. The results are shown in Table 1. On the basis of this assessment, the AMSTAR score for three systematic reviews was 9 [19, 21, 23], three systematic reviews scored 8 [18, 20, 22] and one scored 7 . An AMSTAR score between 8 and 11 has been suggested to indicate a high level of systematic review quality and a score between 4 and 7, a medium one . According to such interpretation, six of the seven systematic reviews were of high quality and one of medium quality. However, the quality of systematic review results can only be as good as the quality of its reviewed trials.
The use of the AMSTAR tool does not establish whether systematic reviews judged the quality of trials according to whether or not bias was successfully prevented. Selection bias, besides e.g. detection/performance-, attrition- or publication bias, is one of the most important types of bias in clinical trials and effectively diverts trial results away from the truth, even in absence of any other types of bias risk . Therefore, selection bias due to lack of adequate concealment of the random sequence allocation was assessed in all reviewed trials. Allocation, using for example central randomisation, was considered to be adequate concealment. The result showed that none of the 47 trials accepted for review included adequate allocation concealment in their methodology.
Web-of-beliefs and components
All beliefs investigated by the seven systematic reviews: belief related to GICs remineralising effect (Belief 1), caries-preventive effect (Belief 2), restoration survival (Belief 3), are mutual to the main belief regarding GIC's benefit in caries therapy. All systematic reviews contained the component categories: Test material (Tn); Control material (Cn); Treatment type (TNn) and Outcome measure (OMn). Details about the specific components per category are shown in Table 2. These various component categories interconnect by virtue of the methodology employed in the systematic reviews, thus forming strings, such as:
Through the various components present in each systematic review per category, various different strings may thus be generated (Table 3). The various beliefs are expressed and tested along these constructed methodological strings. As each string makes use of at least one of the various components, these strings can be graphically merged and thus illustrate the outline structure of a 'web-of-beliefs' frame, relevant to the overall topic as to whether GIC is beneficial in caries therapy (Figure 1). The illustration of such frame in the form of a graph gives an overview of how the various beliefs are connected and form into a 'web'. Figure 1 also serves as a map to graphically discern the interconnections between the various requirements of coherence.
Requirements for coherence
Whether this web-of-beliefs is indeed 'coherent' depends on how well its underlying concepts are logically coherent, as well as on the empirical coherence of its underlying clinical evidence.
Within the framework of this case study, logical coherence was based on the tautologies that:
Tooth remineralisation (OM1) is conceptually closer related to absence of carious tooth lesions (OM2) than to lesion presence;
Absence of carious tooth lesions (OM2) on restoration margins is conceptually closer related to tooth restoration survival (OM3) than to failure.
In biological/clinical terms, these tautologies are oversimplified, as the process of the caries disease is rather complex. However, within the philosophical context of this case study, these tautologies mean that the higher the tooth remineralisation effect, the less the occurrence of carious tooth lesions is expected to be and the less the occurrence of lesions on restoration margins, the higher the survival of tooth restorations is expected to be.
Coherence of clinical evidence (Empirical coherence) will mean that the following conditions are fulfilled, within the framework of this case study:
The remineralisation effect (OM1) associated with test materials (Tn) is at least not lower than with control materials (Cn), regardless of the type of treatment (TNn);
Absence of carious tooth lesions (OM2) associated with test materials (Tn) is at least not lower than with control materials (Cn), regardless of the type of treatment (TNn);
Tooth restoration survival (OM3) associated with test materials (Tn) is at least not lower than with control materials (Cn), for tooth restoration as treatment type (TN1);
There are no differences in the clinical outcomes between the different versions of the test materials (T1 and 2) regardless of type of treatment (TNn) and outcome measure (OMn);
There is no difference between the clinical outcomes of the same version of the test material (T1 or 2) in comparison to different control materials (Cn), (under condition of same type of treatment (TNn) and outcome measure (OMn);
There is no difference between the clinical outcomes of the different version of test material (Tn) in comparison to the same control material (C1,2,3 or 4) (under condition of same type of treatment (TNn) and outcome measure (OMn).
The information provided by the systematic reviews were discussed in relation to these conditions, to establish whether it complied with the requirements for coherence:
In order to fulfil the condition of logical coherence the tautologies that: (i) the higher the tooth remineralisation effect, the less the occurrence of carious tooth lesions and (ii) the less the occurrence of carious tooth lesions on restoration margins, the higher will be the occurrence of the survival of tooth restorations, were adopted. Tautology is a philosophical term that describes a statement that is true in every circumstance, thus forming a necessary truth . This necessary truth is circumscribed in the commonly accepted definition of carious tooth lesions  and its occurrence on restorative margins as a common cause of tooth restoration failure .
The evidence provided by the seven systematic reviews complies with all set conditions for coherence of evidence (Table 4). No non-compliance that would be in contradiction of coherence within the established web-of-beliefs has been identified: Remineralisation in association with GIC has not been found lower than with the control . Such remineralising effect supports the belief that GIC is not associated with a higher caries risk (= logical coherence tautology) and that belief has been confirmed [17–19, 23]. Such caries resistance in turn supports the belief that the survival rate of GIC-based (ART) tooth restorations is at least not lower than that of the control (= logical coherence tautology). That belief was shown to be correct . In addition, the different versions of GIC do not give different clinical results, because they both contain the same active ingredient [17, 19, 21] and the anti-cariogenic efficacy of GIC was shown, even when compared against those of different control materials [17–19, 23]. Against this background and within the set web-of-beliefs, empirical coherence appears to be confirmed.
Further aspects of evidence
In line with the Coherence theory, the logical and empirical coherence presented in this case study may be regarded as sufficient to terminate the infinite regression of proof imposed by radical scepticism. This means that radical philosophical scepticism towards the clinical benefit of GIC regarding caries may be successfully refuted in epistemological terms. However, it is important to note that such epistemological proposition can only be translated successfully into practice if and only if no general scepticism concerning the validity of the underlying empirical evidence arises. In the presented case study, empirical evidence is provided by seven systematic reviews. Systematic reviews are regarded as providing the highest form of evidence to healthcare questions . Nonetheless, systematic reviews do not by virtue of their nature provide high empirical evidence. Rather, such evidence can be regarded as valid only when the systematic reviews and the trials reviewed are of high quality. The AMSTAR tool has been proposed as one method for measuring the quality of systematic reviews. Critical self-assessment by use of the AMSTAR tool showed a general high quality in the systematic reviews included in the case study (Table 1). However, even if empirical evidence is provided by high-quality systematic reviews, this evidence can only be as valid as the evidence provided by the clinical trials reviewed. Assessment of all 47 trials found reason for doubt (= lack of reporting adequate random allocation concealment) of evidence validity, owing to the high risk of selection bias. It has been emphasized that selection bias can be successfully prevented only if the allocation sequence remains truly random and free from potential interference throughout the trial [31, 32]. Thus, it is important that trials should include an effective process for concealing the random allocation sequence and that by the end of each trial this process has indeed prevented direct observation and prediction of the random sequence allocation [31, 32]. Egger et al., 2003 reported a treatment effect overestimation of between 21% and 54% due to selection bias, solely caused by lack of allocation concealment during the randomisation process . The seriousness of such overestimation becomes clear under consideration that in a condition of a 50% overestimation, the actual result for a test treatment would be a 20% higher relative risk (RR 1.20) than for the control, while the trial report would claim a 20% lower relative risk (RR 0.80). Thus, in this example, the true result of the trial would constitute the complete opposite of the reported trial result. Such high percentages of over-estimation due to bias may therefore lead to situations where ineffective treatment procedures are presented as effective. For this reason, all trial results identified in the seven systematic reviews need to be interpreted with caution.
Radical, philosophical scepticism towards the benefit of clinical procedures in healthcare may epistemologically be refuted on the basis of logical and empirical coherence. Such coherence indicates causality in the real (objective) world - in this example GIC related to caries-therapeutic benefit - that exists independently of (subjective) perceptions or scepticism. Empirical coherence is provided from clinical evidence. Such evidence is acquired through clinical trials and subsequently synthesised by (quantitative) systematic reviews. A single systematic review covering one topic may be insufficient for refuting the Agrippean argument. For this, several systematic reviews covering interconnected beliefs may be needed. It is stressed that a successful application of such epistemological proposition into practice regarding refutation of radical, philosophical scepticism towards the benefit of a particular clinical procedure is possible only if these systematic reviews are of high quality and the quality of reviewed clinical trials is high enough to withstand general scepticism concerning trial validity.
Williams M: Problems of Knowledge - A Critical Introduction to Epistemology. 2001, Oxford: Oxford University Press
Descartes R: Meditations on First Philosophy. Paris: Michael Soly, 1641
Kant I: The Critique of Pure Reason. Trans. by P. Guyer and A.W. Wood 1781. 1998, Cambridge University Press
Auyang SY: What Do You See, and How? The Cognitive Infrastructure of Vision. 1999, Sydney: Talk presented in the Department of Philosophy, University of Sydney, (accessed 19. 02. 2011), [http://www.creatingtechnology.org/papers/vision.pdf]
Quine WVO: Two dogmas of empiricism. From a Logical Point of View. 1964, Cambridge: Harvard University Press
Dennett DC: Consciousness Explained. 1999, Boston: Little, Brown and Company
Murphy EA: The Logic of Medicine. 1976, Baltimore: John Hopkins University Press
Jüni P, Altman DG, Egger M: Systematic reviews in health care. Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.
Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J: How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003, 7: 1-
Mickenautsch S: Systematic reviews, systematic error and the acquisition of clinical knowledge. BMC Med Res Methodol. 2010, 10: 53-10.1186/1471-2288-10-53.
Empiricus S: Outlines of Pyrrhonism. Volume 1 of Sextus Empiricus. 1933, London: Heinemann
Fichte JG: Attempt at a Critique of All Revelations. Trans. by G. Green 1792/1793. 1978, Cambridge University Press
Parker M: False dichotomies: EBM, clinical freedom, and the art of medicine. Med Humanities. 2005, 31: 23-30. 10.1136/jmh.2004.000195.
Davidson D: Inquiries into Truth and Interpretation. 1984, Oxford: Oxford University Press
BonJour L: The Structure of Empirical Knowledge. 1985, Cambridge, Mass.: Harvard University Press
Clay M, Lehre K: Knowledge and Scepticism. 1989, Westview: Boulder, Col
Yengopal V, Mickenautsch S, Bezerra AC, Leal SC: Caries-preventive effect of glass ionomer and resin-based fissure sealants on permanent teeth - a meta analysis. J Oral Sci. 2009, 51: 373-382. 10.2334/josnusd.51.373.
Mickenautsch S, Yengopal V, Leal SC, Oliveira LB, Bezerra AC, Bönecker M: Absence of carious lesions at margins of glass-ionomer and amalgam restorations: a meta-analysis. Eur J Paediatr Dent. 2009, 10: 41-46.
Yengopal V, Mickenautsch S: Resin-modified glass-ionomer cements versus resin-based materials as fissure sealants - a meta-analysis of clinical trials. Eur Arch Paediatr Dent. 2010, 11: 18-25.
Mickenautsch S, Yengopal V: Demineralization of hard tooth tissue adjacent to resin-modified glass-ionomers and composite resins: a quantitative systematic review. J Oral Sci. 2010, 52: 347-357. 10.2334/josnusd.52.347.
Mickenautsch S, Tyas MJ, Yengopal V, Oliveira LB, Bönecker M: Absence of carious lesions at margins of glass-ionomer cement (GIC) and resin-modified GIC restorations: a systematic review. Eur J Prosthodont Rest Dent. 2010, 18: 139-145.
Mickenautsch S, Yengopal V, Banerjee A: Atraumatic restorative treatment versus amalgam restoration longevity: a systematic review. Clin Oral Investig. 2010, 14: 233-240. 10.1007/s00784-009-0335-8.
Yengopal V, Mickenautsch S: Caries-preventive effect of resin-modified glass-ionomer cement (RM-GIC) versus composite resin - a quantitative systematic review. Eur Arch Paediatr Dent. 2011, 12: 5-14.
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007, 7: 10-10.1186/1471-2288-7-10.
Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, Ramsay T, Bai A, Shukla VK, Grimshaw JM: External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007, 2: e1350-10.1371/journal.pone.0001350.
Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M: AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2007, 62: 1013-1020.
Suebnukarn S, Ngamboonsirisingh S, Rattanabanlang A: A systematic evaluation of the quality of meta-analyses in endodontics. J Endod. 2010, 36: 602-608. 10.1016/j.joen.2009.12.019.
Djulbegovic B, Guyatt GH, Ashcroft RE: Epistemologic inquiries in evidence-based medicine. Cancer Control. 2009, 16: 158-168.
Feierskov O: Concepts of dental caries and their consequences for understanding the disease. Community Dent Oral Epidemiol. 1997, 25: 5-12. 10.1111/j.1600-0528.1997.tb00894.x.
Opdam NJ, Bronkhorst EM, Loomans BA, Huysmans MC: 12-year survival of composite vs. amalgam restorations. J Dent Res. 2010, 89: 1063-1067. 10.1177/0022034510376071.
Berger VW: Selection Bias and Covariate Imbalances in Randomised Clinical Trials. 2005, Chichester: Wiley
Berger VW, Alperson SY: A general framework for the evaluation of clinical trial quality. Rev Recent Clin Trials. 2009, 4: 79-88. 10.2174/157488709788186021.
SM developed the concept and outline and wrote this paper.
The author contributes to the conduct and publication of systematic reviews concerned with topics related to Minimum Intervention (MI) in dentistry.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.