title: Individual and joint expert judgments as reference standards in artifact detection.
abstract: OBJECTIVE
  To investigate the agreement among clinical experts in their judgments of monitoring data with respect to artifacts, and to examine the effect of reference standards that consist of individual and joint expert judgments on the performance of artifact filters.
  
  
  METHODS
  Individual judgments of four physicians, a majority vote judgment, and a consensus judgment were obtained for 30 time series of three monitoring variables: mean arterial blood pressure (ABPm), central venous pressure (CVP), and heart rate (HR). The individual and joint judgments were used to tune three existing automated filtering methods and to evaluate the performance of the resulting filters.
  
  
  METHODS
  The interrater agreement was calculated in terms of positive specific agreement (PSA). The performance of the artifact filters was quantified in terms of sensitivity and positive predictive value (PPV).
  
  
  RESULTS
  PSA values between 0.33 and 0.85 were observed among clinical experts in their selection of artifacts, with relatively high values for CVP data. Artifact filters developed using judgments of individual experts were found to moderately generalize to new time series and other experts; sensitivity values ranged from 0.40 to 0.60 for ABPm and HR filters (PPV: 0.57-0.84), and from 0.63 to 0.80 for CVP filters (PPV: 0.71-0.86). A higher performance value for the filters was found for the three variable types when joint judgments were used for tuning the filtering methods.
  
  
  CONCLUSIONS
  Given the disagreement among experts in their individual judgment of monitoring data with respect to artifacts, the use of joint reference standards obtained from multiple experts is recommended for development of automatic artifact filters.
authors: Department of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands. m.verduijn@amc.uva.nl
authors: Marion
authors: M
authors: Verduijn
authors: Niels
authors: N
authors: Peek
authors: Nicolette F
authors: NF
authors: de Keizer
authors: Erik-Jan
authors: EJ
authors: van Lieshout
authors: Anne-Cornelie J M
authors: AC
authors: de Pont
authors: Marcus J
authors: MJ
authors: Schultz
authors: Evert
authors: E
authors: de Jonge
authors: Bas A J M
authors: BA
authors: de Mol
mesh_terms: D016477:Artifacts; D001795:Blood Pressure Determination / Q000592:standards*; D002496:Central Venous Pressure; D006339:Heart Rate; D006801:Humans; D007600:Judgment; D008991:Monitoring, Physiologic / Q000592:standards*; D015588:Observer Variation; D010820:Physicians; D012015:Reference Standards; D015203:Reproducibility of Results