An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition.
BACKGROUND
This article provides an overview of the first BIOASQ challenge, a competition on large-scale biomedical semantic indexing and question answering (QA), which took place between March and September 2013. BIOASQ assesses the ability of systems to semantically index very large numbers of biomedical scientific articles, and to return concise and user-understandable answers to given natural language questions by combining information from biomedical articles and ontologies.


RESULTS
The 2013 BIOASQ competition comprised two tasks, Task 1a and Task 1b. In Task 1a participants were asked to automatically annotate new PUBMED documents with MESH headings. Twelve teams participated in Task 1a, with a total of 46 system runs submitted, and one of the teams performing consistently better than the MTI indexer used by NLM to suggest MESH headings to curators. Task 1b used benchmark datasets containing 29 development and 282 test English questions, along with gold standard (reference) answers, prepared by a team of biomedical experts from around Europe and participants had to automatically produce answers. Three teams participated in Task 1b, with 11 system runs. The BIOASQ infrastructure, including benchmark datasets, evaluation mechanisms, and the results of the participants and baseline methods, is publicly available.


CONCLUSIONS
A publicly available evaluation infrastructure for biomedical semantic indexing and QA has been developed, which includes benchmark datasets, and can be used to evaluate systems that: assign MESH headings to published articles or to English questions; retrieve relevant RDF triples from ontologies, relevant articles and snippets from PUBMED Central; produce "exact" and paragraph-sized "ideal" answers (summaries). The results of the systems that participated in the 2013 BIOASQ competition are promising. In Task 1a one of the systems performed consistently better from the NLM's MTI indexer. In Task 1b the systems received high scores in the manual evaluation of the "ideal" answers; hence, they produced high quality summaries as answers. Overall, BIOASQ helped obtain a unified view of how techniques from text classification, semantic indexing, document and passage retrieval, question answering, and text summarization can be combined to allow biomedical experts to obtain concise, user-understandable answers to questions reflecting their real information needs.
Biotechnology Center, TU Dresden, 01307, Tatzberg 47-49, Dresden, Germany. george.tsatsaronis@biotec.tu-dresden.de.
George
G
Tsatsaronis
Université Pierre et Marie Curie-Paris 6, 4 Place Jussieu, Paris, 75005, France. georgios.balikas@imag.fr.
Georgios
G
Balikas
Athens University of Economics and Business, Patission 76, Athens, 10434, Greece. rulller@gmail.com.
Prodromos
P
Malakasiotis
Université Joseph Fourier, 621 Avenue Centrale, Saint-Martin-d'Héres, 38041, France. ioannis.partalas@imag.fr.
Ioannis
I
Partalas
Transinsight GmbH, Tatzberg 47-49, Dresden, 01307, Germany. mzschunke@transinsight.com.
Matthias
M
Zschunke
Transinsight GmbH, Tatzberg 47-49, Dresden, 01307, Germany. malvers@transinsight.com.
Michael R
MR
Alvers
Biotechnology Center, TU Dresden, 01307, Tatzberg 47-49, Dresden, Germany. dirk.weissenborn@gmail.com.
Dirk
D
Weissenborn
NCSR Demokritos, Ag. Paraskevi, Athens, 60228, Greece. akrithara@iit.demokritos.gr.
Anastasia
A
Krithara
NCSR Demokritos, Ag. Paraskevi, Athens, 60228, Greece. eserxio@gmail.com.
Sergios
S
Petridis
NCSR Demokritos, Ag. Paraskevi, Athens, 60228, Greece. dpolychr@gmail.com.
Dimitris
D
Polychronopoulos
NCSR Demokritos, Ag. Paraskevi, Athens, 60228, Greece. yalmir@bio.demokritos.gr.
Yannis
Y
Almirantis
Athens University of Economics and Business, Patission 76, Athens, 10434, Greece. annis@aueb.gr.
John
J
Pavlopoulos
Université Pierre et Marie Curie-Paris 6, 4 Place Jussieu, Paris, 75005, France. nicolas.baskiotis@lip6.fr.
Nicolas
N
Baskiotis
Université Pierre et Marie Curie-Paris 6, 4 Place Jussieu, Paris, 75005, France. Patrick.Gallinari@lip6.fr.
Patrick
P
Gallinari
Université Pierre et Marie Curie-Paris 6, 4 Place Jussieu, Paris, 75005, France. thierry.artieres@lip6.fr.
Thierry
T
Artiéres
Universität Leipzig, Augustusplatz 10, 04109, Leipzig, Germany. axel.ngonga@gmail.com.
Axel-Cyrille Ngonga
AC
Ngomo
Universität Leipzig, Augustusplatz 10, 04109, Leipzig, Germany. heino@informatik.uni-leipzig.de.
Norman
N
Heino
Université Joseph Fourier, 621 Avenue Centrale, Saint-Martin-d'Héres, 38041, France. eric.gaussier@imag.fr.
Eric
E
Gaussier
Transinsight GmbH, Tatzberg 47-49, Dresden, 01307, Germany. lalvers@transinsight.com.
Liliana
L
Barrio-Alvers
Biotechnology Center, TU Dresden, 01307, Tatzberg 47-49, Dresden, Germany. ms@biotec.tu-dresden.de.
Michael
M
Schroeder
Athens University of Economics and Business, Patission 76, Athens, 10434, Greece. ion@aueb.gr.
Ion
I
Androutsopoulos
NCSR Demokritos, Ag. Paraskevi, Athens, 60228, Greece. paliourg@iit.demokritos.gr.
Georgios
G
Paliouras
D000043:Abstracting and Indexing / Q000379:methods*; D000465:Algorithms; D006801:Humans; D046650:Medical Subject Headings; D009317:National Library of Medicine (U.S.); D009323:Natural Language Processing; D039781:PubMed; D012660:Semantics; D012984:Software; D014481:United States