title: Towards Automated Annotation of Benthic Survey Images: Variability of Human Experts and Operational Modes of Automation.
abstract: Global climate change and other anthropogenic stressors have heightened the need to rapidly characterize ecological changes in marine benthic communities across large scales. Digital photography enables rapid collection of survey images to meet this need, but the subsequent image annotation is typically a time consuming, manual task. We investigated the feasibility of using automated point-annotation to expedite cover estimation of the 17 dominant benthic categories from survey-images captured at four Pacific coral reefs. Inter- and intra- annotator variability among six human experts was quantified and compared to semi- and fully- automated annotation methods, which are made available at coralnet.ucsd.edu. Our results indicate high expert agreement for identification of coral genera, but lower agreement for algal functional groups, in particular between turf algae and crustose coralline algae. This indicates the need for unequivocal definitions of algal groups, careful training of multiple annotators, and enhanced imaging technology. Semi-automated annotation, where 50% of the annotation decisions were performed automatically, yielded cover estimate errors comparable to those of the human experts. Furthermore, fully-automated annotation yielded rapid, unbiased cover estimates but with increased variance. These results show that automated annotation can increase spatial coverage and decrease time and financial outlay for image-based reef surveys.
authors.affiliation: Department of Computer Science and Engineering, University of California, San Diego, La Jolla, CA, United States of America.
authors.forename: Oscar
authors.initials: O
authors.lastname: Beijbom
authors.affiliation: Department of Biology, California State University, Northridge, Northridge, CA, United States of America.
authors.forename: Peter J
authors.initials: PJ
authors.lastname: Edmunds
authors.affiliation: Biophysical Remote Sensing Group, School of Geography, Planning and Environmental Management, University of Queensland, St. Lucia, QLD, Australia.
authors.forename: Chris
authors.initials: C
authors.lastname: Roelfsema
authors.affiliation: Integrative Oceanography Division, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, CA, United States of America.
authors.forename: Jennifer
authors.initials: J
authors.lastname: Smith
authors.affiliation: Integrative Oceanography Division, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, CA, United States of America.
authors.forename: David I
authors.initials: DI
authors.lastname: Kline
authors.affiliation: Integrative Oceanography Division, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, CA, United States of America; Catlin Seaview Survey, Global Change Institute, University of Queensland, St. Lucia, QLD, Australia.
authors.forename: Benjamin P
authors.initials: BP
authors.lastname: Neal
authors.affiliation: Joint Institute for Marine and Atmospheric Research, University of Hawaii at Manoa, Honolulu, HI, United States of America.
authors.forename: Matthew J
authors.initials: MJ
authors.lastname: Dunlap
authors.affiliation: Department of Biology, California State University, Northridge, Northridge, CA, United States of America.
authors.forename: Vincent
authors.initials: V
authors.lastname: Moriarty
authors.affiliation: National Museum of Marine Biology and Aquarium, Checheng, Taiwan, Republic of China.
authors.forename: Tung-Yung
authors.initials: TY
authors.lastname: Fan
authors.affiliation: National Museum of Marine Biology and Aquarium, Checheng, Taiwan, Republic of China.
authors.forename: Chih-Jui
authors.initials: CJ
authors.lastname: Tan
authors.affiliation: Department of Computer Science and Engineering, University of California, San Diego, La Jolla, CA, United States of America.
authors.forename: Stephen
authors.initials: S
authors.lastname: Chan
authors.affiliation: Charney School of Marine Sciences, University of Haifa, Haifa, Israel.
authors.forename: Tali
authors.initials: T
authors.lastname: Treibitz
authors.affiliation: Department of Neuroscience, University of California, San Diego, La Jolla, CA, United States of America.
authors.forename: Anthony
authors.initials: A
authors.lastname: Gamst
authors.affiliation: Integrative Oceanography Division, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, CA, United States of America.
authors.forename: B Greg
authors.initials: BG
authors.lastname: Mitchell
authors.affiliation: Department of Computer Science and Engineering, University of California, San Diego, La Jolla, CA, United States of America.
authors.forename: David
authors.initials: D
authors.lastname: Kriegman
mesh_terms: D000465:Algorithms; D000818:Animals; D037421:Anthozoa; D057231:Climate Change; D057894:Coral Reefs; D017753:Ecosystem; D004784:Environmental Monitoring / Q000379:methods*; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000379:methods*; D015233:Models, Statistical; D015588:Observer Variation; D010363:Pattern Recognition, Automated; D015203:Reproducibility of Results; D012624:Seaweed / Q000502:physiology*