title: A protocol for testing expert-system reliability.
abstract: We set out a protocol for giving a computerised medical decision system the label of expert-system meaning that its responses are of the same standard as those of an expert. We apply it to the validation of an expert-system, MENINGE, which is applied to diagnosis of meningitis in children. The proposed protocol can be widely used since it doesn't refer to the existence of correct responses for therapy or diagnosis, and since it involves simple statistical computations. First this validation rests on a comparison of agreement among experts and among physicians new to medical practice. Secondly, the agreement of the system with the experts is compared to the agreement of the system with physicians new to medical practice. In order to perform this comparison we recommend a standard agreement measure, the choice of which is an important issue in this paper.
authors.affiliation: Département de pédiatrie, CHU de Grenoble, 38700 La Tronche, France.
authors.forename: P
authors.initials: P
authors.lastname: François
authors.affiliation: Service d'information et informatique médicale, CHU de Grenoble, France.
authors.forename: C
authors.initials: C
authors.lastname: Robert
authors.affiliation: Service d'information et informatique médicale, CHU de Grenoble, France.
authors.forename: G
authors.initials: G
authors.lastname: Weil
authors.affiliation: Service d'information et informatique médicale, CHU de Grenoble, France.
authors.forename: J
authors.initials: J
authors.lastname: Demongeot
keywords: Agreement; Convex functions; Entropy; Expert-system; Kappa-coefficient