title: Measuring the outcome of biomedical research: a systematic literature review.
abstract: BACKGROUND
  There is an increasing need to evaluate the production and impact of medical research produced by institutions. Many indicators exist, yet we do not have enough information about their relevance. The objective of this systematic review was (1) to identify all the indicators that could be used to measure the output and outcome of medical research carried out in institutions and (2) enlist their methodology, use, positive and negative points.
  
  
  METHODS
  We have searched 3 databases (Pubmed, Scopus, Web of Science) using the following keywords: [Research outcome* OR research output* OR bibliometric* OR scientometric* OR scientific production] AND [indicator* OR index* OR evaluation OR metrics]. We included articles presenting, discussing or evaluating indicators measuring the scientific production of an institution. The search was conducted by two independent authors using a standardised data extraction form. For each indicator we extracted its definition, calculation, its rationale and its positive and negative points. In order to reduce bias, data extraction and analysis was performed by two independent authors.
  
  
  RESULTS
  We included 76 articles. A total of 57 indicators were identified. We have classified those indicators into 6 categories: 9 indicators of research activity, 24 indicators of scientific production and impact, 5 indicators of collaboration, 7 indicators of industrial production, 4 indicators of dissemination, 8 indicators of health service impact. The most widely discussed and described is the h-index with 31 articles discussing it.
  
  
  CONCLUSIONS
  The majority of indicators found are bibliometric indicators of scientific production and impact. Several indicators have been developed to improve the h-index. This indicator has also inspired the creation of two indicators to measure industrial production and collaboration. Several articles propose indicators measuring research impact without detailing a methodology for calculating them. Many bibliometric indicators identified have been created but have not been used or further discussed.
authors.affiliation: European and International Affairs Unit, Gustave Roussy, Villejuif, France; AP-HP, Hôpital Robert Debré, Unité d'épidémiologie clinique, Paris, France; Université Paris Diderot, Sorbonne Paris Cité, UMR-S 1123 and CIC-EC 1426, ECEVE, Paris, France.
authors.forename: Frédérique
authors.initials: F
authors.lastname: Thonon
authors.affiliation: AP-HP, Hôpital Robert Debré, Unité d'épidémiologie clinique, Paris, France; INSERM, U 1123 and CIC-EC 1426, ECEVE, Paris, France.
authors.forename: Rym
authors.initials: R
authors.lastname: Boulkedid
authors.affiliation: AP-HP, Hôpital Bichat, Département d'Epidémiologie et de recherche clinique, Paris, France.
authors.forename: Tristan
authors.initials: T
authors.lastname: Delory
authors.affiliation: Direction de la Recherche Clinique, Gustave Roussy, Villejuif, France; Centre Hygée, Department of Public Health, Lucien Neuwirth Cancer Institute, CIC-EC 3 Inserm, IFR 143, Saint-Etienne, France.
authors.forename: Sophie
authors.initials: S
authors.lastname: Rousseau
authors.affiliation: European and International Affairs Unit, Gustave Roussy, Villejuif, France.
authors.forename: Mahasti
authors.initials: M
authors.lastname: Saghatchian
authors.affiliation: The Netherlands Cancer Institute, Amsterdam, the Netherlands.
authors.forename: Wim
authors.initials: W
authors.lastname: van Harten
authors.affiliation: AP-HP, Hôpital Robert Debré, Unité d'épidémiologie clinique, Paris, France.
authors.forename: Claire
authors.initials: C
authors.lastname: O'Neill
authors.affiliation: AP-HP, Hôpital Robert Debré, Unité d'épidémiologie clinique, Paris, France; Université Paris Diderot, Sorbonne Paris Cité, UMR-S 1123 and CIC-EC 1426, ECEVE, Paris, France; INSERM, U 1123 and CIC-EC 1426, ECEVE, Paris, France.
authors.forename: Corinne
authors.initials: C
authors.lastname: Alberti
mesh_terms: D015706:Bibliometrics; D035843:Biomedical Research; D003299:Cooperative Behavior; D016206:Databases, Bibliographic; D004526:Efficiency; D062489:Health Impact Assessment; D006801:Humans; D033181:Information Dissemination; D055811:Journal Impact Factor; D039781:PubMed
references.citation: BMJ. 2001 Sep 8;323(7312):528
references.pmid: 11546684
references.citation: BMC Health Serv Res. 2012 Dec 23;12:478
references.pmid: 23259467
references.citation: Open Med. 2009;3(3):e123-30
references.pmid: 21603045
references.citation: PLoS One. 2011 Apr 06;6(4):e18521
references.pmid: 21494691
references.citation: PLoS One. 2009;4(3):e4831
references.pmid: 19293924
references.citation: Am J Psychiatry. 2007 Feb;164(2):206-14
references.pmid: 17267781
references.citation: J Contemp Dent Pract. 2014 Mar 01;15(2):258-62
references.pmid: 25095854
references.citation: Sci Transl Med. 2011 Aug 3;3(94):94cm20
references.pmid: 21813751
references.citation: Nord J Psychiatry. 2008;62(2):136-43
references.pmid: 18569777
references.citation: Med Sci (Paris). 2004 Oct;20(10):909-15
references.pmid: 15461970
references.citation: Arch Environ Occup Health. 2013;68(2):123-7
references.pmid: 23428063
references.citation: Basic Clin Pharmacol Toxicol. 2005 Nov;97(5):261-75
references.pmid: 16236137
references.citation: Arch Immunol Ther Exp (Warsz). 2009 Jan-Feb;57(1):19-32
references.pmid: 19219531
references.citation: J R Soc Med. 2011 Jun;104(6):251-61
references.pmid: 21659400
references.citation: Surgery. 2013 Apr;153(4):493-501
references.pmid: 23465942
references.citation: Radiology. 2010 May;255(2):342-51
references.pmid: 20413749
references.citation: PLoS One. 2009;4(5):e5429
references.pmid: 19415119
references.citation: Cell Cycle. 2012 Nov 15;11(22):4118-21
references.pmid: 22983124
references.citation: Aust New Zealand Health Policy. 2007 Jun 28;4:14
references.pmid: 17597545
references.citation: Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72
references.pmid: 16275915
references.citation: Health Res Policy Syst. 2010 Oct 12;8:30
references.pmid: 20939915
references.citation: Rheumatology (Oxford). 2005 Sep;44(9):1145-56
references.pmid: 16049052
references.citation: J Health Serv Res Policy. 2003 Jul;8(3):165-70
references.pmid: 12869343
references.citation: Psychol Med. 2012 Feb;42(2):435-41
references.pmid: 21676283
references.citation: Proc Natl Acad Sci U S A. 2007 Dec 4;104(49):19193-8
references.pmid: 18040045
references.citation: BJU Int. 2012 Feb;109(3):321-4
references.pmid: 22243665
references.citation: Br J Cancer. 2008 Aug 19;99(4):569-76
references.pmid: 18665166