Analysis of National Institutes of Health R01 Application Critiques, Impact, and Criteria Scores: Does the Sex of the Principal Investigator Make a Difference?
Prior text analysis of R01 critiques suggested that female applicants may be disadvantaged in National Institutes of Health (NIH) peer review, particularly for renewals. NIH altered its review format in 2009. The authors examined R01 critiques and scoring in the new format for differences due to principal investigator (PI) sex.



The authors analyzed 739 critiques-268 from 88 unfunded and 471 from 153 funded applications for grants awarded to 125 PIs (76 males, 49 females) at the University of Wisconsin-Madison between 2010 and 2014. The authors used seven word categories for text analysis: ability, achievement, agentic, negative evaluation, positive evaluation, research, and standout adjectives. The authors used regression models to compare priority and criteria scores, and results from text analysis for differences due to PI sex and whether the application was for a new (Type 1) or renewal (Type 2) R01.



Approach scores predicted priority scores for all PIs' applications (P < .001), but scores and critiques differed significantly for male and female PIs' Type 2 applications. Reviewers assigned significantly worse priority, approach, and significance scores to female than male PIs' Type 2 applications, despite using standout adjectives (e.g., "outstanding," "excellent") and making references to ability in more critiques (P < .05 for all comparisons).



The authors' analyses suggest that subtle gender bias may continue to operate in the post-2009 NIH review format in ways that could lead reviewers to implicitly hold male and female applicants to different standards of evaluation, particularly for R01 renewals.
A. Kaatz is director of computational sciences, Center for Women's Health Research, University of Wisconsin-Madison, Madison, Wisconsin. Y.G. Lee is associate researcher, Wisconsin Center for Education Research, University of Wisconsin-Madison, Madison, Wisconsin. A. Potvien is a doctoral candidate, Department of Statistics, and researcher, Health Innovation Program, University of Wisconsin-Madison, Madison, Wisconsin. W. Magua is postdoctoral research associate, Center for Women's Health Research, University of Wisconsin-Madison, Madison, Wisconsin. A. Filut is research assistant, Center for Women's Health Research, University of Wisconsin-Madison, Madison, Wisconsin. A. Bhattacharya is an undergraduate student and data science scholar, Center for Women's Health Research, University of Wisconsin-Madison, Madison, Wisconsin. R. Leatherberry is staff researcher, Center for Women's Health Research, University of Wisconsin-Madison, Madison, Wisconsin. X. Zhu is associate professor, Department of Computer Science, University of Wisconsin-Madison, Madison, Wisconsin. M. Carnes is director, Center for Women's Health Research, professor in the Departments of Medicine, Psychiatry, and Industrial and Systems Engineering, University of Wisconsin-Madison, and part-time physician, William S. Middleton Veterans Hospital, Madison, Wisconsin.
Anna
A
Kaatz
You-Geon
YG
Lee
Aaron
A
Potvien
Wairimu
W
Magua
Amarette
A
Filut
Anupama
A
Bhattacharya
Renee
R
Leatherberry
Xiaojin
X
Zhu
Molly
M
Carnes
D035843:Biomedical Research; D005260:Female; D006801:Humans; D008297:Male; D009316:National Institutes of Health (U.S.); D017712:Peer Review, Research; D012108:Research Personnel / Q000706:statistics & numerical data*; D012737:Sex Factors; D063507:Sexism / Q000706:statistics & numerical data*; D014481:United States