title: Is visual evaluation of aneurysm coiling a reliable study end point? Systematic review and meta-analysis.
abstract: OBJECTIVE
  Angiographic occlusion as a surrogate marker of satisfactory aneurysm treatment is commonly used in clinical trials although some pitfalls have to be considered. To investigate the inter-rater reliability of visual rating of aneurysm occlusion as study end point, we performed a systematic review and meta-analysis.
  
  
  METHODS
  Electronic databases (MEDLINE, EMBASE, PubMed, and the Cochrane Library) were searched up to June 2014. Assessment of risk for bias was based on the Quality Appraisal Tool for Studies of Diagnostic Reliability and the Guidelines for Reporting Reliability and Agreement studies. Inter-rater reliability estimates were pooled across studies using meta-analysis, and the influence of several factors (eg, imaging methods, grading scales, and occlusion rate) was tested with meta-regression.
  
  
  RESULTS
  From 1193 titles, 644 abstracts and 87 full-text versions were reviewed. Twenty-six articles met the inclusion criteria and provided 77 reliability estimates. Twenty-one different rating scales were used, and statistical analysis varied. Mean inter-rater agreement of the pooled studies was substantial (κ=0.65; 95% confidence interval, 0.60-0.69). Reliability varied significantly as a function of imaging methods, grading scales, occlusion rates, and their interaction. Observer agreement substantially increased with increasing occlusion rate in digital subtraction angiography but not in MR angiography. Reliability was higher in studies using 2- or 3-value grading scales than in studies with 4-value grading scales.
  
  
  CONCLUSIONS
  There is significant heterogeneity between studies evaluating the reliability of visual evaluation of aneurysm coiling. On the basis of our analysis, we found that the combination of magnetic resonance angiography, 3-value grading scale, and 2 trained raters seems most promising for usage as surrogate study end points.
authors.affiliation: From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.). m.ernst@uke.de.
authors.forename: Marielle
authors.initials: M
authors.lastname: Ernst
authors.affiliation: From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.).
authors.forename: Albert J
authors.initials: AJ
authors.lastname: Yoo
authors.affiliation: From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.).
authors.forename: Levente
authors.initials: L
authors.lastname: Kriston
authors.affiliation: From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.).
authors.forename: Michael H
authors.initials: MH
authors.lastname: Schönfeld
authors.affiliation: From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.).
authors.forename: Eik
authors.initials: E
authors.lastname: Vettorazzi
authors.affiliation: From the Department of Diagnostic and Interventional Neuroradiology (M.E., M.H.S., J.F.), Department of Medical Psychology (L.K.), and Department of Medical Biometry and Epidemiology (E.V.), University Medical Center Hamburg-Eppendorf, Hamburg, Germany; and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston (A.J.Y.).
authors.forename: Jens
authors.initials: J
authors.lastname: Fiehler
mesh_terms: D002533:Cerebral Angiography / Q000379:methods*; D005260:Female; D006801:Humans; D002532:Intracranial Aneurysm / Q000000981:diagnostic imaging* / Q000628:therapy*; D016239:MEDLINE; D018810:Magnetic Resonance Angiography / Q000379:methods*; D008297:Male
keywords: intracranial aneurysm; meta-analysis; reliability of results