title: Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations.
abstract: For health information technology tools to fully inform evidence-based decisions, recommendations must be reliably assessed for quality and strength of evidence. We aimed to create an annotation framework for grading recommendations regarding appropriate use of diagnostic imaging examinations.
  
  
  
  The annotation framework was created by an expert panel (clinicians in three medical specialties, medical librarians, and biomedical scientists) who developed a process for achieving consensus in assessing recommendations, and evaluated by measuring agreement in grading the strength of evidence for 120 empirically selected recommendations using the Oxford Levels of Evidence.
  
  
  
  Eighty-two percent of recommendations were assigned to Level 5 (expert opinion). Inter-annotator agreement was 0.70 on initial grading (κ = 0.35, 95% CI, 0.23-0.48). After systematic discussion utilizing the annotation framework, agreement increased significantly to 0.97 (κ = 0.88, 95% CI, 0.77-0.99).
  
  
  
  A novel annotation framework was effective for grading the strength of evidence supporting appropriate use criteria for diagnostic imaging exams.
authors.affiliation: Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA Harvard Medical School, Boston, MA 02115, USA rlacson@partners.org.
authors.forename: Ronilda
authors.initials: R
authors.lastname: Lacson
authors.affiliation: Harvard Medical School, Boston, MA 02115, USA Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA 02114, USA.
authors.forename: Ali S
authors.initials: AS
authors.lastname: Raja
authors.affiliation: Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA.
authors.forename: David
authors.initials: D
authors.lastname: Osterbur
authors.affiliation: Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA Harvard Medical School, Boston, MA 02115, USA Department of Medicine, Brigham and Women's Hospital, Boston, MA 02115, USA.
authors.forename: Ivan
authors.initials: I
authors.lastname: Ip
authors.affiliation: Harvard Medical School, Boston, MA 02115, USA Department of Medicine, Brigham and Women's Hospital, Boston, MA 02115, USA.
authors.forename: Louise
authors.initials: L
authors.lastname: Schneider
authors.affiliation: Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA.
authors.forename: Paul
authors.initials: P
authors.lastname: Bain
authors.affiliation: Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA.
authors.forename: Carol
authors.initials: C
authors.lastname: Mita
authors.affiliation: Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA.
authors.forename: Julia
authors.initials: J
authors.lastname: Whelan
authors.affiliation: Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA.
authors.forename: Patricia
authors.initials: P
authors.lastname: Silveira
authors.affiliation: Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA.
authors.forename: David
authors.initials: D
authors.lastname: Dement
authors.affiliation: Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA Harvard Medical School, Boston, MA 02115, USA.
authors.forename: Ramin
authors.initials: R
authors.lastname: Khorasani
mesh_terms: D020000:Decision Support Systems, Clinical; D003952:Diagnostic Imaging / Q000592:standards*; D019317:Evidence-Based Medicine; D006801:Humans
keywords: Clinical decision support system; clinical practice guidelines; diagnostic imaging; evidence-based practice