title: Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects.
abstract: OBJECTIVE
  Document annotation is a key task in the development of Text Mining methods and applications. High quality annotated corpora are invaluable, but their preparation requires a considerable amount of resources and time. Although the existing annotation tools offer good user interaction interfaces to domain experts, project management and quality control abilities are still limited. Therefore, the current work introduces Marky, a new Web-based document annotation tool equipped to manage multi-user and iterative projects, and to evaluate annotation quality throughout the project life cycle.
  
  
  METHODS
  At the core, Marky is a Web application based on the open source CakePHP framework. User interface relies on HTML5 and CSS3 technologies. Rangy library assists in browser-independent implementation of common DOM range and selection tasks, and Ajax and JQuery technologies are used to enhance user-system interaction.
  
  
  RESULTS
  Marky grants solid management of inter- and intra-annotator work. Most notably, its annotation tracking system supports systematic and on-demand agreement analysis and annotation amendment. Each annotator may work over documents as usual, but all the annotations made are saved by the tracking system and may be further compared. So, the project administrator is able to evaluate annotation consistency among annotators and across rounds of annotation, while annotators are able to reject or amend subsets of annotations made in previous rounds. As a side effect, the tracking system minimises resource and time consumption.
  
  
  CONCLUSIONS
  Marky is a novel environment for managing multi-user and iterative document annotation projects. Compared to other tools, Marky offers a similar visually intuitive annotation experience while providing unique means to minimise annotation effort and enforce annotation quality, and therefore corpus consistency. Marky is freely available for non-commercial use at http://sing.ei.uvigo.es/marky.
authors.affiliation: ESEI - Escuela Superior de Ingeniería Informática, Edificio Politécnico, Campus Universitario As Lagoas s/n, Universidad de Vigo, 32004 Ourense, Spain(1).
authors.forename: Martín
authors.initials: M
authors.lastname: Pérez-Pérez
authors.affiliation: ESEI - Escuela Superior de Ingeniería Informática, Edificio Politécnico, Campus Universitario As Lagoas s/n, Universidad de Vigo, 32004 Ourense, Spain(1).
authors.forename: Daniel
authors.initials: D
authors.lastname: Glez-Peña
authors.affiliation: ESEI - Escuela Superior de Ingeniería Informática, Edificio Politécnico, Campus Universitario As Lagoas s/n, Universidad de Vigo, 32004 Ourense, Spain(1).
authors.forename: Florentino
authors.initials: F
authors.lastname: Fdez-Riverola
authors.affiliation: ESEI - Escuela Superior de Ingeniería Informática, Edificio Politécnico, Campus Universitario As Lagoas s/n, Universidad de Vigo, 32004 Ourense, Spain(1); Centre of Biological Engineering, University of Minho, Campus de Gualtar, 4710-057 Braga, Portugal. Electronic address: analia@uvigo.es.
authors.forename: Anália
authors.initials: A
authors.lastname: Lourenço
mesh_terms: D020407:Internet; D008962:Models, Theoretical
keywords: Collaborative annotation; Document annotation; Inter-annotator agreement; Iterative annotation; Tracking system