title: Assessment of NER solutions against the first and second CALBC Silver Standard Corpus.
abstract: BACKGROUND
  Competitions in text mining have been used to measure the performance of automatic text processing solutions against a manually annotated gold standard corpus (GSC). The preparation of the GSC is time-consuming and costly and the final corpus consists at the most of a few thousand documents annotated with a limited set of semantic groups. To overcome these shortcomings, the CALBC project partners (PPs) have produced a large-scale annotated biomedical corpus with four different semantic groups through the harmonisation of annotations from automatic text mining solutions, the first version of the Silver Standard Corpus (SSC-I). The four semantic groups are chemical entities and drugs (CHED), genes and proteins (PRGE), diseases and disorders (DISO) and species (SPE). This corpus has been used for the First CALBC Challenge asking the participants to annotate the corpus with their text processing solutions.
  
  
  RESULTS
  All four PPs from the CALBC project and in addition, 12 challenge participants (CPs) contributed annotated data sets for an evaluation against the SSC-I. CPs could ignore the training data and deliver the annotations from their genuine annotation system, or could train a machine-learning approach on the provided pre-annotated data. In general, the performances of the annotation solutions were lower for entities from the categories CHED and PRGE in comparison to the identification of entities categorized as DISO and SPE. The best performance over all semantic groups were achieved from two annotation solutions that have been trained on the SSC-I.The data sets from participants were used to generate the harmonised Silver Standard Corpus II (SSC-II), if the participant did not make use of the annotated data set from the SSC-I for training purposes. The performances of the participants' solutions were again measured against the SSC-II. The performances of the annotation solutions showed again better results for DISO and SPE in comparison to CHED and PRGE.
  
  
  CONCLUSIONS
  The SSC-I delivers a large set of annotations (1,121,705) for a large number of documents (100,000 Medline abstracts). The annotations cover four different semantic groups and are sufficiently homogeneous to be reproduced with a trained classifier leading to an average F-measure of 85%. Benchmarking the annotation solutions against the SSC-II leads to better performance for the CPs' annotation solutions in comparison to the SSC-I.
authors.affiliation: EMBL Outstation, European Bioinformatics Institute, Hinxton, Cambridge, CB10 1SD, U,K.
authors.forename: Dietrich
authors.initials: D
authors.lastname: Rebholz-Schuhmann
authors.forename: Antonio
authors.initials: A
authors.lastname: Jimeno Yepes
authors.forename: Chen
authors.initials: C
authors.lastname: Li
authors.forename: Senay
authors.initials: S
authors.lastname: Kafkas
authors.forename: Ian
authors.initials: I
authors.lastname: Lewin
authors.forename: Ning
authors.initials: N
authors.lastname: Kang
authors.forename: Peter
authors.initials: P
authors.lastname: Corbett
authors.forename: David
authors.initials: D
authors.lastname: Milward
authors.forename: Ekaterina
authors.initials: E
authors.lastname: Buyko
authors.forename: Elena
authors.initials: E
authors.lastname: Beisswanger
authors.forename: Kerstin
authors.initials: K
authors.lastname: Hornbostel
authors.forename: Alexandre
authors.initials: A
authors.lastname: Kouznetsov
authors.forename: René
authors.initials: R
authors.lastname: Witte
authors.forename: Jonas B
authors.initials: JB
authors.lastname: Laurila
authors.forename: Christopher Jo
authors.initials: CJ
authors.lastname: Baker
authors.forename: Cheng-Ju
authors.initials: CJ
authors.lastname: Kuo
authors.forename: Simone
authors.initials: S
authors.lastname: Clematide
authors.forename: Fabio
authors.initials: F
authors.lastname: Rinaldi
authors.forename: Richárd
authors.initials: R
authors.lastname: Farkas
authors.forename: György
authors.initials: G
authors.lastname: Móra
authors.forename: Kazuo
authors.initials: K
authors.lastname: Hara
authors.forename: Laura I
authors.initials: LI
authors.lastname: Furlong
authors.forename: Michael
authors.initials: M
authors.lastname: Rautschka
authors.forename: Mariana Lara
authors.initials: ML
authors.lastname: Neves
authors.forename: Alberto
authors.initials: A
authors.lastname: Pascual-Montano
authors.forename: Qi
authors.initials: Q
authors.lastname: Wei
authors.forename: Nigel
authors.initials: N
authors.lastname: Collier
authors.forename: Md Faisal Mahbub
authors.initials: MF
authors.lastname: Chowdhury
authors.forename: Alberto
authors.initials: A
authors.lastname: Lavelli
authors.forename: Rafael
authors.initials: R
authors.lastname: Berlanga
authors.forename: Roser
authors.initials: R
authors.lastname: Morante
authors.forename: Vincent
authors.initials: V
authors.lastname: Van Asch
authors.forename: Walter
authors.initials: W
authors.lastname: Daelemans
authors.forename: José Luís
authors.initials: JL
authors.lastname: Marina
authors.forename: Erik
authors.initials: E
authors.lastname: van Mulligen
authors.forename: Jan
authors.initials: J
authors.lastname: Kors
authors.forename: Udo
authors.initials: U
authors.lastname: Hahn