title: Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms.
abstract: Although acknowledged to be variable and subjective, manual annotation of cryo-electron tomography data is commonly used to answer structural questions and to create a "ground truth" for evaluation of automated segmentation algorithms. Validation of such annotation is lacking, but is critical for understanding the reproducibility of manual annotations. Here, we used voxel-based similarity scores for a variety of specimens, ranging in complexity and segmented by several annotators, to quantify the variation among their annotations. In addition, we have identified procedures for merging annotations to reduce variability, thereby increasing the reliability of manual annotation. Based on our analyses, we find that it is necessary to combine multiple manual annotations to increase the confidence level for answering structural questions. We also make recommendations to guide algorithm development for automated annotation of features of interest.
authors.affiliation: 1Molecular Virology and Microbiology Department,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Corey W
authors.initials: CW
authors.lastname: Hecksel
authors.affiliation: 2Verna and Marrs McLean Department of Biochemistry and Molecular Biology,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Michele C
authors.initials: MC
authors.lastname: Darrow
authors.affiliation: 3National Center for Macromolecular Imaging,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Wei
authors.initials: W
authors.lastname: Dai
authors.affiliation: 3National Center for Macromolecular Imaging,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Jesús G
authors.initials: JG
authors.lastname: Galaz-Montoya
authors.affiliation: 3National Center for Macromolecular Imaging,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Jessica A
authors.initials: JA
authors.lastname: Chin
authors.affiliation: 3National Center for Macromolecular Imaging,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Patrick G
authors.initials: PG
authors.lastname: Mitchell
authors.affiliation: 3National Center for Macromolecular Imaging,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Shurui
authors.initials: S
authors.lastname: Chen
authors.affiliation: 3National Center for Macromolecular Imaging,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Jemba
authors.initials: J
authors.lastname: Jakana
authors.affiliation: 2Verna and Marrs McLean Department of Biochemistry and Molecular Biology,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Michael F
authors.initials: MF
authors.lastname: Schmid
authors.affiliation: 1Molecular Virology and Microbiology Department,Baylor College of Medicine,Houston,TX 77030,USA.
authors.forename: Wah
authors.initials: W
authors.lastname: Chiu
mesh_terms: D000465:Algorithms; D055032:Electron Microscope Tomography / Q000379:methods* / Q000592:standards*; D015203:Reproducibility of Results
keywords: Dice coefficient; annotation; cryo-electron tomography; segmentation; validation