A system for rating the stability and strength of medical evidence.
BACKGROUND
Methods for describing one's confidence in the available evidence are useful for end-users of evidence reviews. Analysts inevitably make judgments about the quality, quantity consistency, robustness, and magnitude of effects observed in the studies identified. The subjectivity of these judgments in several areas underscores the need for transparency in judgments.


CONCLUSIONS
This paper introduces a new system for rating medical evidence. The system requires explicit judgments and provides explicit rules for balancing these judgments. Unlike other systems for rating the strength of evidence, our system draws a distinction between two types of conclusions: quantitative and qualitative. A quantitative conclusion addresses the question, "How well does it work?", whereas a qualitative conclusion addresses the question, "Does it work?" In our system, quantitative conclusions are tied to stability ratings, and qualitative conclusions are tied to strength ratings. Our system emphasizes extensive a priori criteria for judgments to reduce the potential for bias. Further, the system makes explicit the impact of heterogeneity testing, meta-analysis, and sensitivity analyses on evidence ratings. This article provides details of our system, including graphical depictions of how the numerous judgments that an analyst makes can be combined. We also describe two worked examples of how the system can be applied to both interventional and diagnostic technologies.


CONCLUSIONS
Although explicit judgments and formal combination rules are two important steps on the path to a comprehensive system for rating medical evidence, many additional steps must also be taken. Foremost among these are the distinction between quantitative and qualitative conclusions, an extensive set of a priori criteria for making judgments, and the direct impact of analytic results on evidence ratings. These attributes form the basis for a logically consistent system that can improve the usefulness of evidence reviews.
Treadwell
Jonathan R
JR
ECRI Evidence-Based Practice Center and Health Technology Assessment Group, 5200 Butler Pike, Plymouth Meeting, Pennsylvania 19462, USA. jtreadwell@ecri.org
Tregear
Stephen J
SJ
Reston
James T
JT
Turkelson
Charles M
CM
D005069:Evaluation Studies as Topic; D019317:Evidence-Based Medicine / Q000379:methods* / Q000592:standards; D006801:Humans; D007600:Judgment; D015201:Meta-Analysis as Topic; D017410:Practice Guidelines as Topic; D036301:Qualitative Research; D018570:Risk Assessment; D012680:Sensitivity and Specificity; D013673:Technology Assessment, Biomedical / Q000379:methods* / Q000592:standards