title: User-oriented evaluation of a medical image retrieval system for radiologists.
abstract: OBJECTIVE
  This article reports the user-oriented evaluation of a text- and content-based medical image retrieval system. User tests with radiologists using a search system for images in the medical literature are presented. The goal of the tests is to assess the usability of the system, identify system and interface aspects that need improvement and useful additions. Another objective is to investigate the system's added value to radiology information retrieval. The study provides an insight into required specifications and potential shortcomings of medical image retrieval systems through a concrete methodology for conducting user tests.
  
  
  METHODS
  User tests with a working image retrieval system of images from the biomedical literature were performed in an iterative manner, where each iteration had the participants perform radiology information seeking tasks and then refining the system as well as the user study design itself. During these tasks the interaction of the users with the system was monitored, usability aspects were measured, retrieval success rates recorded and feedback was collected through survey forms.
  
  
  RESULTS
  In total, 16 radiologists participated in the user tests. The success rates in finding relevant information were on average 87% and 78% for image and case retrieval tasks, respectively. The average time for a successful search was below 3 min in both cases. Users felt quickly comfortable with the novel techniques and tools (after 5 to 15 min), such as content-based image retrieval and relevance feedback. User satisfaction measures show a very positive attitude toward the system's functionalities while the user feedback helped identifying the system's weak points. The participants proposed several potentially useful new functionalities, such as filtering by imaging modality and search for articles using image examples.
  
  
  CONCLUSIONS
  The iterative character of the evaluation helped to obtain diverse and detailed feedback on all system aspects. Radiologists are quickly familiar with the functionalities but have several comments on desired functionalities. The analysis of the results can potentially assist system refinement for future medical information retrieval systems. Moreover, the methodology presented as well as the discussion on the limitations and challenges of such studies can be useful for user-oriented medical image retrieval evaluation, as user-oriented evaluation of interactive system is still only rarely performed. Such interactive evaluations can be limited in effort if done iteratively and can give many insights for developing better systems.
authors.affiliation: University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland. Electronic address: dimitrios.markonis@hevs.ch.
authors.forename: Dimitrios
authors.initials: D
authors.lastname: Markonis
authors.affiliation: Medical University of Vienna, Vienna, Austria.
authors.forename: Markus
authors.initials: M
authors.lastname: Holzer
authors.affiliation: Health On the Net Foundation (HON), Geneva, Switzerland.
authors.forename: Frederic
authors.initials: F
authors.lastname: Baroz
authors.affiliation: Health On the Net Foundation (HON), Geneva, Switzerland.
authors.forename: Rafael Luis Ruiz
authors.initials: RL
authors.lastname: De Castaneda
authors.affiliation: Health On the Net Foundation (HON), Geneva, Switzerland.
authors.forename: Célia
authors.initials: C
authors.lastname: Boyer
authors.affiliation: Medical University of Vienna, Vienna, Austria.
authors.forename: Georg
authors.initials: G
authors.lastname: Langs
authors.affiliation: University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland.
authors.forename: Henning
authors.initials: H
authors.lastname: Müller
mesh_terms: D000328:Adult; D001291:Attitude of Health Personnel; D001292:Attitude to Computers; D001317:Austria; D003197:Computer Literacy; D003258:Consumer Behavior / Q000706:statistics & numerical data*; D057225:Data Mining / Q000706:statistics & numerical data; D005260:Female; D006801:Humans; D008297:Male; D008498:Medical Record Linkage / Q000379:methods; D010820:Physicians / Q000706:statistics & numerical data; D011873:Radiology Information Systems / Q000706:statistics & numerical data*; D012984:Software; D013557:Switzerland; D014584:User-Computer Interface; D014600:Utilization Review; D016526:Workload / Q000706:statistics & numerical data; D055815:Young Adult
keywords: Content-based image retrieval; Medical informatics applications; Usability tests; User-centered design