title: Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute.
abstract: There are conflicting data about the ability of peer review percentile rankings to predict grant productivity, as measured through publications and citations. To understand the nature of these apparent conflicting findings, we analyzed bibliometric outcomes of 6873 de novo cardiovascular R01 grants funded by the National Heart, Lung, and Blood Institute (NHLBI) between 1980 and 2011. Our outcomes focus on top-10% articles, meaning articles that were cited more often than 90% of other articles on the same topic, of the same type (eg, article, editorial), and published in the same year. The 6873 grants yielded 62 468 articles, of which 13 507 (or 22%) were top-10% articles. There was a modest association between better grant percentile ranking and number of top-10% articles. However, discrimination was poor (area under receiver operating characteristic curve [ROC], 0.52; 95% confidence interval, 0.51-0.53). Furthermore, better percentile ranking was also associated with higher annual and total inflation-adjusted grant budgets. There was no association between grant percentile ranking and grant outcome as assessed by number of top-10% articles per $million spent. Hence, the seemingly conflicting findings on peer review percentile ranking of grants and subsequent productivity largely reflect differing questions and outcomes. Taken together, these findings raise questions about how best National Institutes of Health (NIH) should use peer review assessments to make complex funding decisions.
authors.affiliation: From the Office of the Director (M.S.L.), Advanced Technologies and Surgery Branch (N.S.D.), Heart Development and Structural Diseases Branch (J.K.), and Office of Biostatistics Research (C.W.), Division of Cardiovascular Sciences, NHLBI, Bethesda, MD. lauerm@nhlbi.nih.gov.
authors.forename: Michael S
authors.initials: MS
authors.lastname: Lauer
authors.affiliation: From the Office of the Director (M.S.L.), Advanced Technologies and Surgery Branch (N.S.D.), Heart Development and Structural Diseases Branch (J.K.), and Office of Biostatistics Research (C.W.), Division of Cardiovascular Sciences, NHLBI, Bethesda, MD.
authors.forename: Narasimhan S
authors.initials: NS
authors.lastname: Danthi
authors.affiliation: From the Office of the Director (M.S.L.), Advanced Technologies and Surgery Branch (N.S.D.), Heart Development and Structural Diseases Branch (J.K.), and Office of Biostatistics Research (C.W.), Division of Cardiovascular Sciences, NHLBI, Bethesda, MD.
authors.forename: Jonathan
authors.initials: J
authors.lastname: Kaltman
authors.affiliation: From the Office of the Director (M.S.L.), Advanced Technologies and Surgery Branch (N.S.D.), Heart Development and Structural Diseases Branch (J.K.), and Office of Biostatistics Research (C.W.), Division of Cardiovascular Sciences, NHLBI, Bethesda, MD.
authors.forename: Colin
authors.initials: C
authors.lastname: Wu
mesh_terms: D019540:Area Under Curve; D015706:Bibliometrics; D035843:Biomedical Research / Q000191:economics; D002017:Budgets / Q000706:statistics & numerical data; D003657:Decision Making; D017598:Efficiency, Organizational / Q000706:statistics & numerical data / Q000639:trends; D005380:Financing, Government / Q000706:statistics & numerical data*; D005381:Financing, Organized / Q000191:economics; D005544:Forecasting; D006079:Government Publications as Topic; D007250:Inflation, Economic; D007449:Investments / Q000191:economics* / Q000639:trends; D054555:National Heart, Lung, and Blood Institute (U.S.) / Q000191:economics / Q000458:organization & administration* / Q000639:trends; D010380:Peer Review / Q000639:trends; D012372:ROC Curve; D012109:Research Support as Topic / Q000191:economics / Q000706:statistics & numerical data / Q000639:trends; D014481:United States
keywords: National Heart, Lung, and Blood Institute (US); National Institutes of Health (US); ROC curve; bibliometrics; peer review