title: Evaluating consensus among physicians in medical knowledge base construction.
abstract: This study evaluates inter-author variability in knowledge base construction. Seven board-certified internists independently profiled "acute perinephric abscess", using as reference material a set of 109 peer-reviewed articles. Each participant created a list of findings associated with the disease, estimated the predictive value and sensitivity of each finding, and assessed the pertinence of each article for making each judgment. Agreement in finding selection was significantly different from chance: seven, six, and five participants selected the same finding 78.6, 9.8, and 1.6 times more often than predicted by chance. Findings with the highest sensitivity were most likely to be included by all participants. The selection of supporting evidence from the medical literature was significantly related to each physician's agreement with the majority. The study shows that, with appropriate guidance, physicians can reproducibly extract information from the medical literature, and thus established a foundation for multi-author knowledge base construction.
authors.affiliation: Department of Medicine, University of Pittsburgh.
authors.forename: N B
authors.initials: NB
authors.lastname: Giuse
authors.forename: D A
authors.initials: DA
authors.lastname: Giuse
authors.forename: R A
authors.initials: RA
authors.lastname: Miller
authors.forename: R A
authors.initials: RA
authors.lastname: Bankowitz
authors.forename: J E
authors.initials: JE
authors.lastname: Janosky
authors.forename: F
authors.initials: F
authors.lastname: Davidoff
authors.forename: B E
authors.initials: BE
authors.lastname: Hillner
authors.forename: G
authors.initials: G
authors.lastname: Hripcsak
authors.forename: M J
authors.initials: MJ
authors.lastname: Lincoln
authors.forename: B
authors.initials: B
authors.lastname: Middleton
mesh_terms: D000038:Abscess / Q000175:diagnosis; D001185:Artificial Intelligence; D003625:Data Collection; D003627:Data Interpretation, Statistical; D005103:Expert Systems; D006801:Humans; D007388:Internal Medicine; D008491:Medical Informatics Applications; D008498:Medical Record Linkage; D010501:Perinephritis / Q000175:diagnosis