title: Using statistical text classification to identify health information technology incidents.
abstract: OBJECTIVE
  To examine the feasibility of using statistical text classification to automatically identify health information technology (HIT) incidents in the USA Food and Drug Administration (FDA) Manufacturer and User Facility Device Experience (MAUDE) database.
  
  
  METHODS
  We used a subset of 570 272 incidents including 1534 HIT incidents reported to MAUDE between 1 January 2008 and 1 July 2010. Text classifiers using regularized logistic regression were evaluated with both 'balanced' (50% HIT) and 'stratified' (0.297% HIT) datasets for training, validation, and testing. Dataset preparation, feature extraction, feature selection, cross-validation, classification, performance evaluation, and error analysis were performed iteratively to further improve the classifiers. Feature-selection techniques such as removing short words and stop words, stemming, lemmatization, and principal component analysis were examined.
  
  
  METHODS
  κ statistic, F1 score, precision and recall.
  
  
  RESULTS
  Classification performance was similar on both the stratified (0.954 F1 score) and balanced (0.995 F1 score) datasets. Stemming was the most effective technique, reducing the feature set size to 79% while maintaining comparable performance. Training with balanced datasets improved recall (0.989) but reduced precision (0.165).
  
  
  CONCLUSIONS
  Statistical text classification appears to be a feasible method for identifying HIT reports within large databases of incidents. Automated identification should enable more HIT problems to be detected, analyzed, and addressed in a timely manner. Semi-supervised learning may be necessary when applying machine learning to big data analysis of patient safety incidents and requires further investigation.
authors.affiliation: Centre for Health Informatics, Australian Institute for Health Innovation, The University of New South Wales, Sydney, Australia.
authors.forename: Kevin E K
authors.initials: KE
authors.lastname: Chai
authors.forename: Stephen
authors.initials: S
authors.lastname: Anthony
authors.forename: Enrico
authors.initials: E
authors.lastname: Coiera
authors.forename: Farah
authors.initials: F
authors.lastname: Magrabi
mesh_terms: D057225:Data Mining / Q000379:methods*; D016208:Databases, Factual; D005240:Feasibility Studies; D019300:Medical Errors / Q000706:statistics & numerical data*; D008490:Medical Informatics; D014481:United States; D014486:United States Food and Drug Administration
keywords: FDA; HIT; MAUDE; health information technology; incidents; text classification