title: Controlling for chance agreement in the validation of medical expert systems with no gold standard: PNEUMON-IA and RENOIR revisited.
abstract: In the validation of medical expert systems, agreement among different human specialists on a random sample of cases may be taken as a substitute to a missing gold standard. Distance measures between pairs of experts, extensively described in previous studies, do not take into account the influence of chance-expected agreement. A weighted kappa index, with three different weighting schemes, is proposed as an alternative to be applied in the general situation of N cases assessed by E experts about K possible diagnoses, each of them qualified with one of G ordinal categories. A hierarchical cluster analysis, applied to the kappa matrices generated, allows for the classification of the expert system among clinical specialists, providing a relative assessment of its diagnostic ability. The above methodology is applied to the validation of two medical expert systems, PNEUMON-IA and RENOIR.
authors.affiliation: Department of Medical Informatics, Institut Municipal d'Investigacio Mèdica (IMIM), Universitat Autònoma de Barcelona, Dr. Aiguader 80, E-08003 Barcelona, Spain. Montse.Martin@chcr.scs.es
authors.forename: M
authors.initials: M
authors.lastname: Martín-Baranera
authors.forename: J J
authors.initials: JJ
authors.lastname: Sancho
authors.forename: F
authors.initials: F
authors.lastname: Sanz
mesh_terms: D016000:Cluster Analysis; D003428:Cross Infection / Q000382:microbiology*; D003936:Diagnosis, Computer-Assisted / Q000592:standards*; D005103:Expert Systems; D006801:Humans; D011014:Pneumonia / Q000382:microbiology*; D015203:Reproducibility of Results; D012216:Rheumatic Diseases / Q000175:diagnosis; D012680:Sensitivity and Specificity