Are expert panel judgments of medical benefits reliable? An evaluation of emergency medical service programs.
OBJECTIVE
We have used multidisciplinary expert panels to assess the health benefits from two different emergency medical service programs in Norway. This gave the opportunity to study the reliability of the expert panel method.


METHODS
Two panels assessed case reports for 18 children, and two other panels assessed case reports for 64 adult patients. The assessments of each case report were compared. These assessments were also compared with assessments of the same case reports, done by the same panels 1 and 9 years earlier.


RESULTS
Two different panels agreed on the benefit/no benefit conclusion in at least 75% of the patients, both for children and adult patients (kappa 0.88-0.50). For groups of patients assessed to have some health benefit, the magnitude of the benefit estimates differed by 25% between the panels. When the same panels assessed the same patient groups twice, 1 and 9 years apart, their estimates of total benefit differed up to 30%. However, estimates for single patients, as well as estimates from single panel members, varied considerably more.


CONCLUSIONS
Use of multidisciplinary expert panels is a useful method for estimating health benefits on program level or for groups of patients. But assessments from single panelists, and for single patients may be seriously biased.
Hotvedt
Ragnar
R
Institute of Community Medicine, University of Tromsø, Norway. ragnar.hotvedt@ism.uit.no
Lossius
Hans Morten
HM
Kristiansen
Ival Sønbø
IS
Steen
Petter Andreas
PA
Søreide
Eldar
E
Førde
Olav Helge
OH
D000293:Adolescent; D000328:Adult; D017732:Air Ambulances; D000552:Ambulances; D002648:Child; D002675:Child, Preschool; D032921:Consensus; D004632:Emergency Medical Services / Q000458:organization & administration* / Q000592:standards / Q000706:statistics & numerical data; D005260:Female; D006302:Health Services Research / Q000379:methods*; D006801:Humans; D007231:Infant, Newborn; D008297:Male; D008875:Middle Aged; D009664:Norway; D019370:Observation