title: Computer vision cracks the leaf code.
abstract: Understanding the extremely variable, complex shape and venation characters of angiosperm leaves is one of the most challenging problems in botany. Machine learning offers opportunities to analyze large numbers of specimens, to discover novel leaf features of angiosperm clades that may have phylogenetic significance, and to use those characters to classify unknowns. Previous computer vision approaches have primarily focused on leaf identification at the species level. It remains an open question whether learning and classification are possible among major evolutionary groups such as families and orders, which usually contain hundreds to thousands of species each and exhibit many times the foliar variation of individual species. Here, we tested whether a computer vision algorithm could use a database of 7,597 leaf images from 2,001 genera to learn features of botanical families and orders, then classify novel images. The images are of cleared leaves, specimens that are chemically bleached, then stained to reveal venation. Machine learning was used to learn a codebook of visual elements representing leaf shape and venation patterns. The resulting automated system learned to classify images into families and orders with a success rate many times greater than chance. Of direct botanical interest, the responses of diagnostic features can be visualized on leaf images as heat maps, which are likely to prompt recognition and evolutionary interpretation of a wealth of novel morphological characters. With assistance from computer vision, leaves are poised to make numerous new contributions to systematic and paleobotanical studies.
authors.affiliation: Department of Geosciences, Pennsylvania State University, University Park, PA 16802; pwilf@psu.edu s.zhang@hit.edu.cn thomas_serre@brown.edu.
authors.forename: Peter
authors.initials: P
authors.lastname: Wilf
authors.affiliation: Department of Cognitive, Linguistic and Psychological Sciences, Brown Institute for Brain Science, Brown University, Providence, RI 02912; School of Computer Science and Technology, Harbin Institute of Technology, Weihai 264209, Shandong, People's Republic of China; pwilf@psu.edu s.zhang@hit.edu.cn thomas_serre@brown.edu.
authors.forename: Shengping
authors.initials: S
authors.lastname: Zhang
authors.affiliation: Azure Machine Learning, Microsoft, Cambridge, MA 02142;
authors.forename: Sharat
authors.initials: S
authors.lastname: Chikkerur
authors.affiliation: Department of Geosciences, Pennsylvania State University, University Park, PA 16802; Laboratoire Ecologie, Systématique et Evolution, Université Paris-Sud, 91405 Orsay Cedex, France;
authors.forename: Stefan A
authors.initials: SA
authors.lastname: Little
authors.affiliation: Department of Paleobiology, National Museum of Natural History, Smithsonian Institution, Washington, DC 20013.
authors.forename: Scott L
authors.initials: SL
authors.lastname: Wing
authors.affiliation: Department of Cognitive, Linguistic and Psychological Sciences, Brown Institute for Brain Science, Brown University, Providence, RI 02912; pwilf@psu.edu s.zhang@hit.edu.cn thomas_serre@brown.edu.
authors.forename: Thomas
authors.initials: T
authors.lastname: Serre
mesh_terms: D000069550:Machine Learning; D018515:Plant Leaves
keywords: computer vision; leaf architecture; leaf venation; paleobotany; sparse coding