title: Using collective expert judgements to evaluate quality measures of mass spectrometry images.
abstract: BACKGROUND
  Imaging mass spectrometry (IMS) is a maturating technique of molecular imaging. Confidence in the reproducible quality of IMS data is essential for its integration into routine use. However, the predominant method for assessing quality is visual examination, a time consuming, unstandardized and non-scalable approach. So far, the problem of assessing the quality has only been marginally addressed and existing measures do not account for the spatial information of IMS data. Importantly, no approach exists for unbiased evaluation of potential quality measures.
  
  
  RESULTS
  We propose a novel approach for evaluating potential measures by creating a gold-standard set using collective expert judgements upon which we evaluated image-based measures. To produce a gold standard, we engaged 80 IMS experts, each to rate the relative quality between 52 pairs of ion images from MALDI-TOF IMS datasets of rat brain coronal sections. Experts' optional feedback on their expertise, the task and the survey showed that (i) they had diverse backgrounds and sufficient expertise, (ii) the task was properly understood, and (iii) the survey was comprehensible. A moderate inter-rater agreement was achieved with Krippendorff's alpha of 0.5. A gold-standard set of 634 pairs of images with accompanying ratings was constructed and showed a high agreement of 0.85. Eight families of potential measures with a range of parameters and statistical descriptors, giving 143 in total, were evaluated. Both signal-to-noise and spatial chaos-based measures performed highly with a correlation of 0.7 to 0.9 with the gold standard ratings. Moreover, we showed that a composite measure with the linear coefficients (trained on the gold standard with regularized least squares optimization and lasso) showed a strong linear correlation of 0.94 and an accuracy of 0.98 in predicting which image in a pair was of higher quality.
  
  
  METHODS
  The anonymized data collected from the survey and the Matlab source code for data processing can be found at: https://github.com/alexandrovteam/IMS_quality.
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Andrew
authors.initials: A
authors.lastname: Palmer
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Ekaterina
authors.initials: E
authors.lastname: Ovchinnikova
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Mikael
authors.initials: M
authors.lastname: Thuné
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Régis
authors.initials: R
authors.lastname: Lavigne
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Blandine
authors.initials: B
authors.lastname: Guével
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Andrey
authors.initials: A
authors.lastname: Dyatlov
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Olga
authors.initials: O
authors.lastname: Vitek
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Charles
authors.initials: C
authors.lastname: Pineau
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA.
authors.forename: Mats
authors.initials: M
authors.lastname: Borén
authors.affiliation: European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany, Denator, Uppsala, Sweden, Protim, Inserm U1085 - Irset, University of Rennes 1, Rennes, France, SCiLS GmbH, Bremen, Germany, College of Computer and Information Science, Northeastern University, Boston, MA, USA and Skaggs School of Pharmacy and Pharmaceutical Sciences, University of California San Diego, La Jolla, CA, USA European Molecular Biology Laboratory, Heidelberg, Germany, Center for Industrial Mathematics, University of Bremen, Bremen, Germany, High Performance Humanoid Technologies Lab, Institute for Anthropomatics, Karlsruhe Institute of Technolo
authors.forename: Theodore
authors.initials: T
authors.lastname: Alexandrov
mesh_terms: D000818:Animals; D001921:Brain / Q000033:anatomy & histology* / Q000166:cytology*; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000592:standards*; D051379:Mice; D057054:Molecular Imaging / Q000379:methods*; D015588:Observer Variation; D010865:Pilot Projects; D011786:Quality Control; D051381:Rats; D012015:Reference Standards; D019032:Spectrometry, Mass, Matrix-Assisted Laser Desorption-Ionization / Q000379:methods*