title: Analyzing animal behavior via classifying each video frame using convolutional neural networks.
abstract: High-throughput analysis of animal behavior requires software to analyze videos. Such software analyzes each frame individually, detecting animals' body parts. But the image analysis rarely attempts to recognize "behavioral states"-e.g., actions or facial expressions-directly from the image instead of using the detected body parts. Here, we show that convolutional neural networks (CNNs)-a machine learning approach that recently became the leading technique for object recognition, human pose estimation, and human action recognition-were able to recognize directly from images whether Drosophila were "on" (standing or walking) or "off" (not in physical contact with) egg-laying substrates for each frame of our videos. We used multiple nets and image transformations to optimize accuracy for our classification task, achieving a surprisingly low error rate of just 0.072%. Classifying one of our 8 h videos took less than 3 h using a fast GPU. The approach enabled uncovering a novel egg-laying-induced behavior modification in Drosophila. Furthermore, it should be readily applicable to other behavior analysis tasks.
authors.affiliation: Independent researcher, Durham, NC 27705.
authors.forename: Ulrich
authors.initials: U
authors.lastname: Stern
authors.affiliation: Dept. of Neurobiology, Duke University, Durham, NC 27710.
authors.forename: Ruo
authors.initials: R
authors.lastname: He
authors.affiliation: Dept. of Neurobiology, Duke University, Durham, NC 27710.
authors.forename: Chung-Hui
authors.initials: CH
authors.lastname: Yang
mesh_terms: D000465:Algorithms; D000818:Animals; D000067289:Behavior Observation Techniques / Q000379:methods; D004330:Drosophila / Q000502:physiology*; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D000069550:Machine Learning; D016571:Neural Networks, Computer; D010363:Pattern Recognition, Automated / Q000379:methods*; D012726:Sexual Behavior, Animal / Q000502:physiology*; D012984:Software