An expanded evaluation of protein function prediction methods shows an improvement in accuracy.
A major bottleneck in our understanding of the molecular underpinnings of life is the assignment of function to proteins. While molecular experiments provide the most reliable annotation of proteins, their relatively low throughput and restricted purview have led to an increasing role for computational function prediction. However, assessing methods for protein function prediction and tracking progress in the field remain challenging.



We conducted the second critical assessment of functional annotation (CAFA), a timed challenge to assess computational methods that automatically assign protein function. We evaluated 126 methods from 56 research groups for their ability to predict biological functions using Gene Ontology and gene-disease associations using Human Phenotype Ontology on a set of 3681 proteins from 18 species. CAFA2 featured expanded analysis compared with CAFA1, with regards to data set size, variety, and assessment metrics. To review progress in the field, the analysis compared the best methods from CAFA1 to those of CAFA2.



The top-performing methods in CAFA2 outperformed those from CAFA1. This increased accuracy can be attributed to a combination of the growing number of experimental annotations and improved methods for function prediction. The assessment also revealed that the definition of top-performing algorithms is ontology specific, that different performance metrics can be used to probe the nature of accurate predictions, and the relative diversity of predictions in the biological process and human phenotype ontologies. While there was methodological improvement between CAFA1 and CAFA2, the interpretation of results and usefulness of individual methods remain context-dependent.
Department of Computer Science and Informatics, Indiana University, Bloomington, IN, USA.
Yuxiang
Y
Jiang
Buck Institute for Research on Aging, Novato, CA, USA.
Tal Ronnen
TR
Oron
Department of Molecular Biophysics and Biochemistry, Yale University, New Haven, CT, USA.
Wyatt T
WT
Clark
Department of Microbiology, Miami University, Oxford, OH, USA.
Asma R
AR
Bankapur
University of Rome, La Sapienza, Rome, Italy.
Daniel
D
D'Andrea
University of Rome, La Sapienza, Rome, Italy.
Rosalba
R
Lepore
Computational Bioscience Program, University of Colorado School of Medicine, Aurora, CO, USA.
Christopher S
CS
Funk
Department of Computer Science, Colorado State University, Fort Collins, CO, USA.
Indika
I
Kahanda
Department of Computing and Information Systems, University of Melbourne, Parkville, Victoria, Australia.|Health and Biomedical Informatics Centre, University of Melbourne, Parkville, Victoria, Australia.
Karin M
KM
Verspoor
Department of Computer Science, Colorado State University, Fort Collins, CO, USA.
Asa
A
Ben-Hur
Department of Biology, New York University, New York, NY, USA.
Da Chen Emily
da CE
Koo
Social Media and Political Participation Lab, New York University, New York, NY, USA.|CY Data Science, New York, NY, USA.
Duncan
D
Penfold-Brown
Department of Computer Science, New York University, New York, NY, USA.
Dennis
D
Shasha
CY Data Science, New York, NY, USA.|Department of Computer Science, New York University, New York, NY, USA.|Simons Center for Data Analysis, New York, NY, USA.
Noah
N
Youngs
Department of Computer Science, New York University, New York, NY, USA.|Simons Center for Data Analysis, New York, NY, USA.|Center for Genomics and Systems Biology, Department of Biology, New York University, New York, NY, USA.
Richard
R
Bonneau
Department of Electrical Engineering and Computer Sciences, University of California Berkeley, Berkeley, CA, USA.
Alexandra
A
Lin
Department of Plant and Microbial Biology, University of California Berkeley, Berkeley, CA, USA.
Sayed M E
SM
Sahraeian
Biocomputing Group, BiGeA, University of Bologna, Bologna, Italy.
Pier Luigi
PL
Martelli
Biocomputing Group, BiGeA, University of Bologna, Bologna, Italy.
Giuseppe
G
Profiti
Biocomputing Group, BiGeA, University of Bologna, Bologna, Italy.
Rita
R
Casadio
Computer Science Department, University of Missouri, Columbia, MO, USA.
Renzhi
R
Cao
Computer Science Department, University of Missouri, Columbia, MO, USA.
Zhaolong
Z
Zhong
Computer Science Department, University of Missouri, Columbia, MO, USA.
Jianlin
J
Cheng
ETH Zurich, Zurich, Switzerland.|Swiss Institute of Bioinformatics, Zurich, Switzerland.
Adrian
A
Altenhoff
ETH Zurich, Zurich, Switzerland.|Swiss Institute of Bioinformatics, Zurich, Switzerland.
Nives
N
Skunca
Bioinformatics Group, Department of Computer Science, University College London, London, UK.|University of Lausanne, Lausanne, Switzerland.|Swiss Institute of Bioinformatics, Lausanne, Switzerland.
Christophe
C
Dessimoz
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Tunca
T
Dogan
Department of Information Technology, University of Turku, Turku, Finland.|University of Turku Graduate School, University of Turku, Turku, Finland.
Kai
K
Hakala
Department of Information Technology, University of Turku, Turku, Finland.|University of Turku Graduate School, University of Turku, Turku, Finland.|Turku Centre for Computer Science, Turku, Finland.
Suwisa
S
Kaewphan
Department of Information Technology, University of Turku, Turku, Finland.|University of Turku Graduate School, University of Turku, Turku, Finland.
Farrokh
F
Mehryary
Department of Information Technology, University of Turku, Turku, Finland.|Turku Centre for Computer Science, Turku, Finland.
Tapio
T
Salakoski
Department of Information Technology, University of Turku, Turku, Finland.
Filip
F
Ginter
University of Bristol, Bristol, UK.
Hai
H
Fang
University of Bristol, Bristol, UK.
Ben
B
Smithers
University of Bristol, Bristol, UK.
Matt
M
Oates
University of Bristol, Bristol, UK.
Julian
J
Gough
Institute of Biotechnology, University of Helsinki, Helsinki, Finland.
Petri
P
Törönen
Institute of Biotechnology, University of Helsinki, Helsinki, Finland.
Patrik
P
Koskinen
Institute of Biotechnology, University of Helsinki, Helsinki, Finland.|Department of Biological and Environmental Sciences, Universitity of Helsinki, Helsinki, Finland.
Liisa
L
Holm
Institute of Information Science, Academia Sinica, Taipei, Taiwan.
Ching-Tai
CT
Chen
Institute of Information Science, Academia Sinica, Taipei, Taiwan.
Wen-Lian
WL
Hsu
Bioinformatics Group, Department of Computer Science, University College London, London, UK.
Kevin
K
Bryson
Bioinformatics Group, Department of Computer Science, University College London, London, UK.
Domenico
D
Cozzetto
Bioinformatics Group, Department of Computer Science, University College London, London, UK.
Federico
F
Minneci
Bioinformatics Group, Department of Computer Science, University College London, London, UK.
David T
DT
Jones
Department of Computational Science and Engineering, North Carolina A&T State University, Greensboro, NC, USA.
Samuel
S
Chapman
Department of Computational Science and Engineering, North Carolina A&T State University, Greensboro, NC, USA.
Dukka
D
Bkc
Department of Computer Science, Purdue University, West Lafayette, IN, USA.
Ishita K
IK
Khan
Department of Computer Science, Purdue University, West Lafayette, IN, USA.|Department of Biological Sciences, Purdue University, West Lafayette, IN, USA.
Daisuke
D
Kihara
Department of Biological Chemistry, Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem, Israel.
Dan
D
Ofer
Department of Biological Chemistry, Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem, Israel.|School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel.
Nadav
N
Rappoport
Department of Biological Chemistry, Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem, Israel.|School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel.
Amos
A
Stern
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Elena
E
Cibrian-Uhalte
Centre for Cardiovascular Genetics, Institute of Cardiovascular Science, University College London, London, UK.
Paul
P
Denny
Centre for Cardiovascular Genetics, Institute of Cardiovascular Science, University College London, London, UK.
Rebecca E
RE
Foulger
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Reija
R
Hieta
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Duncan
D
Legge
Centre for Cardiovascular Genetics, Institute of Cardiovascular Science, University College London, London, UK.
Ruth C
RC
Lovering
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Michele
M
Magrane
Centre for Cardiovascular Genetics, Institute of Cardiovascular Science, University College London, London, UK.
Anna N
AN
Melidoni
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Prudence
P
Mutowo-Meullenet
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Klemens
K
Pichler
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Aleksandra
A
Shypitsyna
Buck Institute for Research on Aging, Novato, CA, USA.
Biao
B
Li
Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, KU Leuven, Leuven, Belgium.|iMinds Department Medical Information Technologies, Leuven, Belgium.
Pooya
P
Zakeri
Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, KU Leuven, Leuven, Belgium.|iMinds Department Medical Information Technologies, Leuven, Belgium.
Sarah
S
ElShal
Inserm UMR-S1052, CNRS UMR5286, Cancer Research Centre of Lyon, Lyon, France.|Université de Lyon 1, Villeurbanne, France.|Centre Léon Bérard, Lyon, France.
Léon-Charles
LC
Tranchevent
Institute of Structural and Molecular Biology, University College London, London, UK.
Sayoni
S
Das
Institute of Structural and Molecular Biology, University College London, London, UK.
Natalie L
NL
Dawson
Institute of Structural and Molecular Biology, University College London, London, UK.
David
D
Lee
Institute of Structural and Molecular Biology, University College London, London, UK.
Jonathan G
JG
Lees
Institute of Structural and Molecular Biology, University College London, London, UK.
Ian
I
Sillitoe
Cerenode Inc., Boston, MA, USA.
Prajwal
P
Bhat
Molde University College, Molde, Norway.
Tamás
T
Nepusz
Department of Computer Science, Centre for Systems and Synthetic Biology, Royal Holloway University of London, Egham, UK.
Alfonso E
AE
Romero
Department of Molecular, Cell and Developmental Biology, University of California at Los Angeles, Los Angeles, CA, USA.
Rajkumar
R
Sasidharan
School of Mathematics, Statistics and Applied Mathematics, National University of Ireland, Galway, Ireland.
Haixuan
H
Yang
Department of Computer Science, Centre for Systems and Synthetic Biology, Royal Holloway University of London, Egham, UK.
Alberto
A
Paccanaro
Stanley Institute for Cognitive Genomics Cold Spring Harbor Laboratory, New York, NY, USA.
Jesse
J
Gillis
Graduate Program in Bioinformatics, University of British Columbia, Vancouver, Canada.
Adriana E
AE
Sedeño-Cortés
Department of Psychiatry and Michael Smith Laboratories, University of British Columbia, Vancouver, Canada.
Paul
P
Pavlidis
Department of Computer Science and Informatics, Indiana University, Bloomington, IN, USA.
Shou
S
Feng
Department for Bioinformatics and Computational Biology-I12, Technische Universität München, Garching, Germany.
Juan M
JM
Cejuela
Department for Bioinformatics and Computational Biology-I12, Technische Universität München, Garching, Germany.
Tatyana
T
Goldberg
Department for Bioinformatics and Computational Biology-I12, Technische Universität München, Garching, Germany.
Tobias
T
Hamp
Department for Bioinformatics and Computational Biology-I12, Technische Universität München, Garching, Germany.
Lothar
L
Richter
DOE Joint Genome Institute, Walnut Creek, CA, USA.
Asaf
A
Salamov
Bioinformatics and Genomics, Centre for Genomic Regulation, Barcelona, Spain.|Universitat Pompeu Fabra, Barcelona, Spain.|Institució Catalana de Recerca i Estudis Avançats, Barcelona, Spain.
Toni
T
Gabaldon
Bioinformatics and Genomics, Centre for Genomic Regulation, Barcelona, Spain.|Universitat Pompeu Fabra, Barcelona, Spain.
Marina
M
Marcet-Houben
Universitat Pompeu Fabra, Barcelona, Spain.|Division of Electronics, Rudjer Boskovic Institute, Zagreb, Croatia.|EMBL/CRG Systems Biology Research Unit, Centre for Genomic Regulation, Barcelona, Spain.
Fran
F
Supek
State Key Laboratory of Genetic Engineering, Collaborative Innovation Center of Genetics and Development, Department of Biostatistics and Computational Biology, School of Life Science, Fudan University, Shanghai, China.|Children's Hospital of Fudan University, Shanghai, China.
Qingtian
Q
Gong
State Key Laboratory of Genetic Engineering, Collaborative Innovation Center of Genetics and Development, Department of Biostatistics and Computational Biology, School of Life Science, Fudan University, Shanghai, China.|Children's Hospital of Fudan University, Shanghai, China.
Wei
W
Ning
State Key Laboratory of Genetic Engineering, Collaborative Innovation Center of Genetics and Development, Department of Biostatistics and Computational Biology, School of Life Science, Fudan University, Shanghai, China.|Children's Hospital of Fudan University, Shanghai, China.
Yuanpeng
Y
Zhou
State Key Laboratory of Genetic Engineering, Collaborative Innovation Center of Genetics and Development, Department of Biostatistics and Computational Biology, School of Life Science, Fudan University, Shanghai, China.|Children's Hospital of Fudan University, Shanghai, China.
Weidong
W
Tian
Department of Molecular Medicine, University of Padua, Padua, Italy.
Marco
M
Falda
Research and Innovation Center, Edmund Mach Foundation, San Michele all'Adige, Italy.
Paolo
P
Fontana
Department of Molecular Medicine, University of Padua, Padua, Italy.
Enrico
E
Lavezzo
Department of Molecular Medicine, University of Padua, Padua, Italy.
Stefano
S
Toppo
Department of Information Engineering, University of Padua, Padova, Italy.
Carlo
C
Ferrari
Department of Information Engineering, University of Padua, Padova, Italy.|Department of Biomedical Sciences, University of Padua, Padova, Italy.
Manuel
M
Giollo
Department of Information Engineering, University of Padua, Padova, Italy.
Damiano
D
Piovesan
Department of Information Engineering, University of Padua, Padova, Italy.
Silvio C E
SC
Tosatto
Instituto De Genetica Medica y Molecular, Hospital Universitario de La Paz, Madrid, Spain.
Angela
A
Del Pozo
Spanish National Bioinformatics Institute, Spanish National Cancer Research Institute, Madrid, Spain.
José M
JM
Fernández
Structural and Computational Biology Programme, Spanish National Cancer Research Institute, Madrid, Spain.
Paolo
P
Maietta
Structural and Computational Biology Programme, Spanish National Cancer Research Institute, Madrid, Spain.
Alfonso
A
Valencia
Structural and Computational Biology Programme, Spanish National Cancer Research Institute, Madrid, Spain.
Michael L
ML
Tress
Control and Computer Engineering Department, Politecnico di Torino, Torino, Italy.
Alfredo
A
Benso
Control and Computer Engineering Department, Politecnico di Torino, Torino, Italy.
Stefano
S
Di Carlo
Control and Computer Engineering Department, Politecnico di Torino, Torino, Italy.
Gianfranco
G
Politano
Control and Computer Engineering Department, Politecnico di Torino, Torino, Italy.
Alessandro
A
Savino
National University of Computer & Emerging Sciences, Islamabad, Pakistan.
Hafeez Ur
HU
Rehman
Anacleto Lab, Dipartimento di informatica, Università degli Studi di Milano, Milan, Italy.
Matteo
M
Re
Anacleto Lab, Dipartimento di informatica, Università degli Studi di Milano, Milan, Italy.
Marco
M
Mesiti
Anacleto Lab, Dipartimento di informatica, Università degli Studi di Milano, Milan, Italy.
Giorgio
G
Valentini
Applied Bioinformatics, Bioscience, Wageningen University and Research Centre, Wageningen, Netherlands.
Joachim W
JW
Bargsten
Applied Bioinformatics, Bioscience, Wageningen University and Research Centre, Wageningen, Netherlands.|Biometris, Wageningen University, Wageningen, Netherlands.
Aalt D J
AD
van Dijk
Center for Multidisciplinary Research, Institute of Nuclear Sciences Vinca, University of Belgrade, Belgrade, Serbia.
Branislava
B
Gemovic
Center for Multidisciplinary Research, Institute of Nuclear Sciences Vinca, University of Belgrade, Belgrade, Serbia.
Sanja
S
Glisic
Center for Multidisciplinary Research, Institute of Nuclear Sciences Vinca, University of Belgrade, Belgrade, Serbia.
Vladmir
V
Perovic
Center for Multidisciplinary Research, Institute of Nuclear Sciences Vinca, University of Belgrade, Belgrade, Serbia.
Veljko
V
Veljkovic
Center for Multidisciplinary Research, Institute of Nuclear Sciences Vinca, University of Belgrade, Belgrade, Serbia.
Nevena
N
Veljkovic
Department of Computing and Mathematics FFCLRP-USP, University of Sao Paulo, Ribeirao Preto, Brazil.
Danillo C
DC
Almeida-E-Silva
Department of Computing and Mathematics FFCLRP-USP, University of Sao Paulo, Ribeirao Preto, Brazil.
Ricardo Z N
RZ
Vencio
Institute for Molecular Infection Biology, University of Würzburg, Würzburg, Germany.
Malvika
M
Sharan
Institute for Molecular Infection Biology, University of Würzburg, Würzburg, Germany.
Jörg
J
Vogel
Department of Computer and Information Sciences, Temple University, Philadelphia, PA, USA.
Lakesh
L
Kansakar
Department of Computer and Information Sciences, Temple University, Philadelphia, PA, USA.
Shanshan
S
Zhang
Department of Computer and Information Sciences, Temple University, Philadelphia, PA, USA.
Slobodan
S
Vucetic
University of Southern Mississippi, Hattiesburg, MS, USA.
Zheng
Z
Wang
Centre for Integrative Systems Biology and Bioinformatics, Department of Life Sciences, Imperial College London, London, UK.
Michael J E
MJ
Sternberg
School of Biosciences, University of Kent, Canterbury, Kent, UK.
Mark N
MN
Wass
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Rachael P
RP
Huntley
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Maria J
MJ
Martin
European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK.
Claire
C
O'Donovan
Institut für Medizinische Genetik und Humangenetik, Charité - Universitätsmedizin Berlin, Berlin, Germany.
Peter N
PN
Robinson
Department of Electrical Engineering ESAT-SCD and IBBT-KU Leuven Future Health Department, Katholieke Universiteit Leuven, Leuven, Belgium.
Yves
Y
Moreau
University of Rome, La Sapienza, Rome, Italy.
Anna
A
Tramontano
California Institute for Quantitative Biosciences, University of California San Francisco, San Francisco, CA, USA.
Patricia C
PC
Babbitt
Department of Plant and Microbial Biology, University of California Berkeley, Berkeley, CA, USA.
Steven E
SE
Brenner
Department of Chemical Biology, The Hebrew University of Jerusalem, Jerusalem, Israel.
Michal
M
Linial
Institute of Structural and Molecular Biology, University College London, London, UK.
Christine A
CA
Orengo
Department for Bioinformatics and Computational Biology-I12, Technische Universität München, Garching, Germany.
Burkhard
B
Rost
Department of Systems Pharmacology and Translational Therapeutics, University of Pennsylvania, Philadelphia, PA, USA.
Casey S
CS
Greene
Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, USA.
Sean D
SD
Mooney
Department of Microbiology, Miami University, Oxford, OH, USA. idoerg@iastate.edu.|Department of Computer Science, Miami University, Oxford, OH, USA. idoerg@iastate.edu.
Iddo
0000-0002-1789-8000
I
Friedberg
Department of Computer Science and Informatics, Indiana University, Bloomington, IN, USA. predrag@indiana.edu.
Predrag
P
Radivojac
D000465:Algorithms; D019295:Computational Biology; D030562:Databases, Protein; D063990:Gene Ontology; D006801:Humans; D058977:Molecular Sequence Annotation; D011506:Proteins / Q000737:chemistry* / Q000235:genetics; D012984:Software; D013329:Structure-Activity Relationship
D011506:Proteins
Disease gene prioritization; Protein function prediction