Validation of the AI/RHEUM knowledge base with data from consecutive rheumatological outpatients.
The diagnostic accuracy of AI/RHEUM, an experimental expert system for support in the diagnosis of rheumatic diseases, was assessed using a collection of data in a cohort of 1,570 consecutive outpatients of a Dutch rheumatological clinic. Computer diagnoses based on these data and diagnostic predictions made by rheumatologists were compared with reference diagnoses that had been obtained by consensus of rheumatologists after 6-12 months follow-up. Performance of the tested version of the AI/RHEUM knowledge base is presented by various methods. Sensitivity varied between 29% and 100% for different rheumatological diseases. Average sensitivity and specificity for all 26 diagnoses present in the knowledge base were 67% and 98%, respectively. Performance according to the level of confidence indicated that 78% of the "definite", 65% of the "probable", and 33% of the "possible" conclusions made by AI/RHEUM were in agreement with the reference diagnoses. These results approximated the predictions made by rheumatologists after a single, initial examination. The system was less accurate than it had appeared in previous evaluation studies with complex clinical cases. The AI/RHEUM knowledge base needed refining to diagnose early rheumatic complaints. This study further illustrates the need for objective and informative parameters for expressing accuracy of diagnostic support systems.
Bernelot Moens
H J
HJ
Jan van Breemen Institute, Amsterdam, The Netherlands.
D000293:Adolescent; D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D001185:Artificial Intelligence; D002648:Child; D002675:Child, Preschool; D003936:Diagnosis, Computer-Assisted; D005103:Expert Systems; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D012017:Referral and Consultation; D012216:Rheumatic Diseases / Q000175:diagnosis; D012219:Rheumatology / Q000379:methods*; D012680:Sensitivity and Specificity