title: Are expert panel judgments of medical benefits reliable? An evaluation of emergency medical service programs.
abstract: OBJECTIVE
  We have used multidisciplinary expert panels to assess the health benefits from two different emergency medical service programs in Norway. This gave the opportunity to study the reliability of the expert panel method.
  
  
  METHODS
  Two panels assessed case reports for 18 children, and two other panels assessed case reports for 64 adult patients. The assessments of each case report were compared. These assessments were also compared with assessments of the same case reports, done by the same panels 1 and 9 years earlier.
  
  
  RESULTS
  Two different panels agreed on the benefit/no benefit conclusion in at least 75% of the patients, both for children and adult patients (kappa 0.88-0.50). For groups of patients assessed to have some health benefit, the magnitude of the benefit estimates differed by 25% between the panels. When the same panels assessed the same patient groups twice, 1 and 9 years apart, their estimates of total benefit differed up to 30%. However, estimates for single patients, as well as estimates from single panel members, varied considerably more.
  
  
  CONCLUSIONS
  Use of multidisciplinary expert panels is a useful method for estimating health benefits on program level or for groups of patients. But assessments from single panelists, and for single patients may be seriously biased.
authors.affiliation: Institute of Community Medicine, University of Tromsø, Norway. ragnar.hotvedt@ism.uit.no
authors.forename: Ragnar
authors.initials: R
authors.lastname: Hotvedt
authors.forename: Hans Morten
authors.initials: HM
authors.lastname: Lossius
authors.forename: Ival Sønbø
authors.initials: IS
authors.lastname: Kristiansen
authors.forename: Petter Andreas
authors.initials: PA
authors.lastname: Steen
authors.forename: Eldar
authors.initials: E
authors.lastname: Søreide
authors.forename: Olav Helge
authors.initials: OH
authors.lastname: Førde
mesh_terms: D000293:Adolescent; D000328:Adult; D017732:Air Ambulances; D000552:Ambulances; D002648:Child; D002675:Child, Preschool; D032921:Consensus; D004632:Emergency Medical Services / Q000458:organization & administration* / Q000592:standards / Q000706:statistics & numerical data; D005260:Female; D006302:Health Services Research / Q000379:methods*; D006801:Humans; D007231:Infant, Newborn; D008297:Male; D008875:Middle Aged; D009664:Norway; D019370:Observation