title: Patients' involvement in e-health services quality assessment: a system for the automatic interpretation of SMS-based patients' feedback.
abstract: OBJECTIVE
  Effective communication between patients and health services providers is a key aspect for optimizing and maintaining these services. This work describes a system for the automatic evaluation of users' perception of the quality of SmsCup, a reminder system for outpatient visits based on short message service (SMS). The final purpose is the creation of a closed-loop control system for the outpatient service, where patients' complaints and comments represent a feedback that can be used for a better implementation of the service itself.
  
  
  METHODS
  SmsCup was adopted since about eight years by an Italian healthcare organization, with very good results in reducing the no-show (missing visits) phenomenon. During these years, a number of citizens, even if not required, sent a message back, with comments about the service. The automatic interpretation of the content of those SMS may be useful for monitoring and improving service performances.Yet, due to the complex nature of SMS language, their interpretation represents an ongoing challenge. The proposed system uses conditional random fields as the information extraction method for classifying messages into several semantic categories. The categories refer to appreciation of the service or complaints of various types. Then, the system analyzes the extracted content and provides feedback to the service providers, making them learning and acting on this basis.
  
  
  RESULTS
  At each step, the content of the messages reveals the actual state of the service as well as the efficacy of corrective actions previously undertaken. Our evaluations showed that: (i) the SMS classification system has achieved good overall performance with an average F1-measure and an overall accuracy of about 92%; (ii) the notification of the patients' feedbacks to service providers showed a positive impact on service functioning.
  
  
  CONCLUSIONS
  Our study proposed an interactive patient-centered system for continuous monitoring of the service quality. It has demonstrated the feasibility of a tool for the analysis and notification of the patients' feedback on their service experiences, which would support a more regular access to the service.
authors.affiliation: Laboratory for Biomedical Informatics, Deparment of Electrical, Computer and Biomedical Engineering, University of Pavia, Italy. Electronic address: stefania.rubrichi@unipv.it.
authors.forename: Stefania
authors.initials: S
authors.lastname: Rubrichi
authors.affiliation: Azienda Ospedaliera Fatebenefratelli e Oftalmico, Milano, Italy.
authors.forename: Andrea
authors.initials: A
authors.lastname: Battistotti
authors.affiliation: Laboratory for Biomedical Informatics, Deparment of Electrical, Computer and Biomedical Engineering, University of Pavia, Italy.
authors.forename: Silvana
authors.initials: S
authors.lastname: Quaglini
mesh_terms: D000553:Ambulatory Care / Q000145:classification*; D001185:Artificial Intelligence; D001294:Attitude to Health; D009323:Natural Language Processing; D010358:Patient Participation / Q000379:methods*; D017060:Patient Satisfaction; D010363:Pattern Recognition, Automated / Q000379:methods; D011639:Public Opinion; D011785:Quality Assurance, Health Care / Q000379:methods; D017010:Reminder Systems / Q000145:classification*; D017216:Telemedicine / Q000145:classification*; D060145:Text Messaging
keywords: Conditional random fields; Health service assessment; Information extraction; Patients’ feedback; SMS; e-Health