title: Reporting health care decision models: a prospective reliability study of a multidimensional evaluation framework.
abstract: The aim of this study was to evaluate the inter-rater reliability of the Phillips-checklist, a proposed framework for the quality assessment of modeling studies. Six raters evaluated nine modeling studies from three different medical specialties. Intra-class correlation (ICC) and corresponding variance components were estimated from these studies. Raters were asked to comment on their experience with the framework. While overall the mean inter-rater reliability showed no significant rater-effect (ICC = 0.69, p = 0.064), there was - presumably as a result of a lower study variability - a significant rater effect for clopidogrel only (p < 0.001). The framework allowed a more structured methodological assessment but several items remained unclear. Regarding the quality assessment of modeling studies with the proposed framework, the rater variability is similar or even higher than variability because of studies or residual effects. Several scoring items can and should be improved to ease interpretation.
authors.affiliation: a Institute for Health Economics and Clinical Epidemiology , The University Hospital of Cologne (AöR) , Cologne , Germany.
authors.forename: Dirk
authors.initials: D
authors.lastname: Müller
authors.affiliation: a Institute for Health Economics and Clinical Epidemiology , The University Hospital of Cologne (AöR) , Cologne , Germany.
authors.forename: Andreas
authors.initials: A
authors.lastname: Gerber-Grote
authors.affiliation: b Helmholtz-Zentrum München, German Research Center for Environmental Health Care Management , Munich , Germany.
authors.forename: Björn
authors.initials: B
authors.lastname: Stollenwerk
authors.affiliation: a Institute for Health Economics and Clinical Epidemiology , The University Hospital of Cologne (AöR) , Cologne , Germany.
authors.forename: Stephanie
authors.initials: S
authors.lastname: Stock
authors.affiliation: a Institute for Health Economics and Clinical Epidemiology , The University Hospital of Cologne (AöR) , Cologne , Germany.
authors.forename: Philipp W P
authors.initials: PW
authors.lastname: Auweiler
authors.affiliation: c Hamburg Center for Health Economics , University of Hamburg , Hamburg , Germany.
authors.forename: Simon
authors.initials: S
authors.lastname: Frey
authors.affiliation: d Department of General Practice/Family Medicine , University of Marburg , Marburg , Germany.|e Department of Health Services Research , Maastricht University , Maastricht , The Netherlands.
authors.forename: Charles Christian
authors.initials: CC
authors.lastname: Adarkwah
authors.affiliation: e Department of Health Services Research , Maastricht University , Maastricht , The Netherlands.|f Department of Research and Development , Epilepsy Center Kempenhaeghe , Heeze , The Netherlands.
authors.forename: Reina
authors.initials: R
authors.lastname: de Kinderen
authors.affiliation: g Institute of Medical Statistics, Informatics and Epidemiology , University of Cologne , Cologne , Germany.
authors.forename: Martin
authors.initials: M
authors.lastname: Hellmich
mesh_terms: D057189:Checklist; D003661:Decision Support Techniques; D003695:Delivery of Health Care / Q000191:economics / Q000379:methods*; D006801:Humans; D008962:Models, Theoretical; D015588:Observer Variation; D011446:Prospective Studies; D015203:Reproducibility of Results
keywords: Inter-rater reliability; cost–effectiveness; intra class-correlation; methodological assessment; modeling studies