Individual and joint expert judgments as reference standards in artifact detection.
OBJECTIVE
To investigate the agreement among clinical experts in their judgments of monitoring data with respect to artifacts, and to examine the effect of reference standards that consist of individual and joint expert judgments on the performance of artifact filters.


METHODS
Individual judgments of four physicians, a majority vote judgment, and a consensus judgment were obtained for 30 time series of three monitoring variables: mean arterial blood pressure (ABPm), central venous pressure (CVP), and heart rate (HR). The individual and joint judgments were used to tune three existing automated filtering methods and to evaluate the performance of the resulting filters.


METHODS
The interrater agreement was calculated in terms of positive specific agreement (PSA). The performance of the artifact filters was quantified in terms of sensitivity and positive predictive value (PPV).


RESULTS
PSA values between 0.33 and 0.85 were observed among clinical experts in their selection of artifacts, with relatively high values for CVP data. Artifact filters developed using judgments of individual experts were found to moderately generalize to new time series and other experts; sensitivity values ranged from 0.40 to 0.60 for ABPm and HR filters (PPV: 0.57-0.84), and from 0.63 to 0.80 for CVP filters (PPV: 0.71-0.86). A higher performance value for the filters was found for the three variable types when joint judgments were used for tuning the filtering methods.


CONCLUSIONS
Given the disagreement among experts in their individual judgment of monitoring data with respect to artifacts, the use of joint reference standards obtained from multiple experts is recommended for development of automatic artifact filters.
Verduijn
Marion
M
Department of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands. m.verduijn@amc.uva.nl
Peek
Niels
N
de Keizer
Nicolette F
NF
van Lieshout
Erik-Jan
EJ
de Pont
Anne-Cornelie J M
AC
Schultz
Marcus J
MJ
de Jonge
Evert
E
de Mol
Bas A J M
BA
D016477:Artifacts; D001795:Blood Pressure Determination / Q000592:standards*; D002496:Central Venous Pressure; D006339:Heart Rate; D006801:Humans; D007600:Judgment; D008991:Monitoring, Physiologic / Q000592:standards*; D015588:Observer Variation; D010820:Physicians; D012015:Reference Standards; D015203:Reproducibility of Results