title: On the evaluation of segmentation editing tools.
abstract: Efficient segmentation editing tools are important components in the segmentation process, as no automatic methods exist that always generate sufficient results. Evaluating segmentation editing algorithms is challenging, because their quality depends on the user's subjective impression. So far, no established methods for an objective, comprehensive evaluation of such tools exist and, particularly, intermediate segmentation results are not taken into account. We discuss the evaluation of editing algorithms in the context of tumor segmentation in computed tomography. We propose a rating scheme to qualitatively measure the accuracy and efficiency of editing tools in user studies. In order to objectively summarize the overall quality, we propose two scores based on the subjective rating and the quantified segmentation quality over time. Finally, a simulation-based evaluation approach is discussed, which allows a more reproducible evaluation without the need for human input. This automated evaluation complements user studies, allowing a more convincing evaluation, particularly during development, where frequent user studies are not possible. The proposed methods have been used to evaluate two dedicated editing algorithms on 131 representative tumor segmentations. We show how the comparison of editing algorithms benefits from the proposed methods. Our results also show the correlation of the suggested quality score with the qualitative ratings.
authors.affiliation: Fraunhofer MEVIS , Universitaetsallee 29, 28357 Bremen, Germany ; University of Leipzig , Innovation Center Computer Assisted Surgery, Semmelweisstraße 14, 04103 Leipzig, Germany.
authors.forename: Frank
authors.initials: F
authors.lastname: Heckel
authors.affiliation: Fraunhofer MEVIS , Universitaetsallee 29, 28357 Bremen, Germany.
authors.forename: Jan H
authors.initials: JH
authors.lastname: Moltz
authors.affiliation: Fraunhofer MEVIS , Universitaetsallee 29, 28357 Bremen, Germany.
authors.forename: Hans
authors.initials: H
authors.lastname: Meine
authors.affiliation: Fraunhofer MEVIS , Universitaetsallee 29, 28357 Bremen, Germany.
authors.forename: Benjamin
authors.initials: B
authors.lastname: Geisler
authors.affiliation: Philipps-University Marburg , Department of Diagnostic Radiology, Baldingerstrasse, 35043 Marburg, Germany.
authors.forename: Andreas
authors.initials: A
authors.lastname: Kießling
authors.affiliation: University Hospital of Munich , Department of Clinical Radiology, Marchioninistrasse 15, 81377 Munich, Germany.
authors.forename: Melvin
authors.initials: M
authors.lastname: D'Anastasi
authors.affiliation: University Hospital Mainz , Department of Diagnostic and Interventional Radiology, Langenbeckstrasse 1, 55131 Mainz, Germany.
authors.forename: Daniel Pinto
authors.initials: DP
authors.lastname: Dos Santos
authors.affiliation: University Hospital Mainz , Department of Diagnostic and Interventional Radiology, Langenbeckstrasse 1, 55131 Mainz, Germany.
authors.forename: Ashok Joseph
authors.initials: AJ
authors.lastname: Theruvath
authors.affiliation: Fraunhofer MEVIS , Universitaetsallee 29, 28357 Bremen, Germany.
authors.forename: Horst K
authors.initials: HK
authors.lastname: Hahn
keywords: automation; evaluation; interactive segmentation; segmentation editing; simulation; validation