title: Detecting and resolving inconsistencies between domain experts' different perspectives on (classification) tasks.
abstract: OBJECTIVE
  The work reported here focuses on developing novel techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. The high level task which the experts (physicians) had set themselves was to classify, on a 5-point severity scale (A-E), the hourly reports produced by an intensive care unit's patient management system.
  
  
  METHODS
  The INSIGHT system has been developed to support domain experts exploring, and removing inconsistencies in their conceptualization of a task. We report here a study of intensive care physicians reconciling 2 perspectives on their patients. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, or changing the assigned categories) and the actual rule-set.
  
  
  RESULTS
  Each of the 3 experts achieved a very high degree of consensus (~97%) between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). We then had the experts produce a common rule-set and then refine their several sets of annotations against it; this again resulted in inter-expert agreements of ~97%. The resulting rule-set can then be used in applications with considerable confidence.
  
  
  CONCLUSIONS
  This study has shown that under some circumstances, it is possible for domain experts to achieve a high degree of correlation between 2 perspectives of the same task. The experts agreed that the immediate feedback provided by INSIGHT was a significant contribution to this successful outcome.
authors.affiliation: Department of Computing Science, University of Aberdeen, Aberdeen AB24 3UE, UK. d.sleeman@abdn.ac.uk
authors.forename: Derek
authors.initials: D
authors.lastname: Sleeman
authors.forename: Laura
authors.initials: L
authors.lastname: Moss
authors.forename: Andy
authors.initials: A
authors.lastname: Aiken
authors.forename: Martin
authors.initials: M
authors.lastname: Hughes
authors.forename: John
authors.initials: J
authors.lastname: Kinsella
authors.forename: Malcolm
authors.initials: M
authors.lastname: Sim
mesh_terms: D001185:Artificial Intelligence; D002965:Classification / Q000379:methods; D003628:Database Management Systems / Q000295:instrumentation*; D003936:Diagnosis, Computer-Assisted / Q000379:methods; D057286:Electronic Health Records / Q000295:instrumentation*; D005104:Expert Testimony; D016247:Information Storage and Retrieval / Q000379:methods; D007362:Intensive Care Units