Automated Classification of Circulating Tumor Cells and the Impact of Interobsever Variability on Classifier Training and Performance.
Application of personalized medicine requires integration of different data to determine each patient's unique clinical constitution. The automated analysis of medical data is a growing field where different machine learning techniques are used to minimize the time-consuming task of manual analysis. The evaluation, and often training, of automated classifiers requires manually labelled data as ground truth. In many cases such labelling is not perfect, either because of the data being ambiguous even for a trained expert or because of mistakes. Here we investigated the interobserver variability of image data comprising fluorescently stained circulating tumor cells and its effect on the performance of two automated classifiers, a random forest and a support vector machine. We found that uncertainty in annotation between observers limited the performance of the automated classifiers, especially when it was included in the test set on which classifier performance was measured. The random forest classifier turned out to be resilient to uncertainty in the training data while the support vector machine's performance is highly dependent on the amount of uncertainty in the training data. We finally introduced the consensus data set as a possible solution for evaluation of automated classifiers that minimizes the penalty of interobserver variability.
Applied Systems Biology, Leibniz Institute for Natural Product Research and Infection Biology-Hans-Knöll-Institute (HKI), Beutenbergstraße 11a, 07745 Jena, Germany.
Carl-Magnus
CM
Svensson
Applied Systems Biology, Leibniz Institute for Natural Product Research and Infection Biology-Hans-Knöll-Institute (HKI), Beutenbergstraße 11a, 07745 Jena, Germany ; Friedrich Schiller University Jena, Fürstengraben 1, 07743 Jena, Germany.
Ron
R
Hübler
Applied Systems Biology, Leibniz Institute for Natural Product Research and Infection Biology-Hans-Knöll-Institute (HKI), Beutenbergstraße 11a, 07745 Jena, Germany ; Friedrich Schiller University Jena, Fürstengraben 1, 07743 Jena, Germany.
Marc Thilo
MT
Figge
D000465:Algorithms; D000954:Antigens, Surface / Q000378:metabolism; D001499:Bayes Theorem; D015415:Biomarkers; D006801:Humans; D008856:Microscopy, Fluorescence / Q000379:methods / Q000592:standards; D009369:Neoplasms / Q000175:diagnosis* / Q000378:metabolism; D009360:Neoplastic Cells, Circulating / Q000378:metabolism / Q000473:pathology*; D015588:Observer Variation; D015203:Reproducibility of Results; D060388:Support Vector Machine
D000954:Antigens, Surface; D015415:Biomarkers