title: Nine criteria for a measure of scientific output.
abstract: Scientific research produces new knowledge, technologies, and clinical treatments that can lead to enormous returns. Often, the path from basic research to new paradigms and direct impact on society takes time. Precise quantification of scientific output in the short-term is not an easy task but is critical for evaluating scientists, laboratories, departments, and institutions. While there have been attempts to quantifying scientific output, we argue that current methods are not ideal and suffer from solvable difficulties. Here we propose criteria that a metric should have to be considered a good index of scientific output. Specifically, we argue that such an index should be quantitative, based on robust data, rapidly updated and retrospective, presented with confidence intervals, normalized by number of contributors, career stage and discipline, impractical to manipulate, and focused on quality over quantity. Such an index should be validated through empirical testing. The purpose of quantitatively evaluating scientific output is not to replace careful, rigorous review by experts but rather to complement those efforts. Because it has the potential to greatly influence the efficiency of scientific research, we have a duty to reflect upon and implement novel and rigorous ways of evaluating scientific output. The criteria proposed here provide initial steps toward the systematic development and validation of a metric to evaluate scientific output.
authors.affiliation: Department of Ophthalmology, Children's Hospital, Harvard Medical School Boston, MA, USA.
authors.forename: Gabriel
authors.initials: G
authors.lastname: Kreiman
authors.forename: John H R
authors.initials: JH
authors.lastname: Maunsell
references.citation: Nature. 2003 Oct 16;425(6959):661
references.pmid: 14562076
references.citation: Science. 2008 Oct 10;322(5899):165
references.pmid: 18845714
references.citation: JAMA. 2006 Jan 4;295(1):90-3
references.pmid: 16391221
references.citation: Nature. 2011 Jan 20;469(7330):286-7
references.pmid: 21248816
references.citation: J Am Soc Inf Sci. 1986 Jul;37(4):261-70
references.pmid: 10300954
references.citation: Nature. 2010 Jul 8;466(7303):179
references.pmid: 20613817
references.citation: Lancet. 2001 Oct 20;358(9290):1373
references.pmid: 11684260
references.citation: Cortex. 2006 Jan;42(1):1-2
references.pmid: 16509101
references.citation: Science. 2008 Jul 4;321(5885):15
references.pmid: 18599743
references.citation: Science. 2011 Jan 7;331(6013):29
references.pmid: 21212337
references.citation: PLoS Med. 2006 Jun;3(6):e291
references.pmid: 16749869
references.citation: BMJ. 2008 Apr 12;336(7648):789
references.pmid: 18403509
references.citation: Sci Transl Med. 2010 Oct 13;2(53):53ps49
references.pmid: 20944088
references.citation: Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72
references.pmid: 16275915
references.citation: Proc Natl Acad Sci U S A. 2009 Apr 28;106(17):6883-4
references.pmid: 19380731
references.citation: Cancer Genet Cytogenet. 1998 Jul 15;104(2):77-81
references.pmid: 9666797
references.citation: PLoS One. 2009 Jun 29;4(6):e6022
references.pmid: 19562078
references.citation: Cortex. 2008 Oct;44(9):1139-45
references.pmid: 18708185
references.citation: Genome Biol. 2008;9(7):107
references.pmid: 18671833
references.citation: Science. 2010 Sep 17;329(5998):1466
references.pmid: 20847251
references.citation: Science. 2009 Mar 27;323(5922):1662-4
references.pmid: 19325094