title: A randomized trial of fellowships for early career researchers finds a high reliability in funding decisions.
abstract: OBJECTIVE
  Funding for early career researchers in Australia's largest medical research funding scheme is determined by a competitive peer-review process using a panel of four reviewers. The purpose of this experiment was to appraise the reliability of funding by duplicating applications that were considered by separate grant review panels.
  
  
  METHODS
  Sixty duplicate applications were considered by two independent grant review panels that were awarding funding for Australia's National Health and Medical Research Council. Panel members were blinded to which applications were included in the experiment and to whether it was the original or duplicate application. Scores were compared across panels using Bland-Altman plots to determine measures of agreement, including whether agreement would have impacted on actual funding.
  
  
  RESULTS
  Twenty-three percent of the applicants were funded by both panels and 60 percent were not funded by both, giving an overall agreement of 83 percent [95% confidence interval (CI): 73%, 92%]. The chance-adjusted agreement was 0.75 (95% CI: 0.58, 0.92).
  
  
  CONCLUSIONS
  There was a comparatively high level of agreement when compared with other types of funding schemes. Further experimental research could be used to determine if this higher agreement is due to nature of the application, the composition of the assessment panel, or the characteristics of the applicants.
authors.affiliation: Melbourne School of Population and Global Health, The University of Melbourne, 207 Bouverie Street, Melbourne, Victoria, 3010, Australia.
authors.forename: Philip
authors.initials: P
authors.lastname: Clarke
authors.affiliation: School of Public Health and Social Work, Queensland University of Technology, Victoria Park Road, Brisbane, Queensland, 4059, Australia; Institute of Health and Biomedical Innovation, Queensland University of Technology, 60 Musk Avenue, Kelvin Grove, Brisbane, Queensland 4059, Australia.
authors.forename: Danielle
authors.initials: D
authors.lastname: Herbert
authors.affiliation: School of Public Health and Social Work, Queensland University of Technology, Victoria Park Road, Brisbane, Queensland, 4059, Australia; Institute of Health and Biomedical Innovation, Queensland University of Technology, 60 Musk Avenue, Kelvin Grove, Brisbane, Queensland 4059, Australia.
authors.forename: Nick
authors.initials: N
authors.lastname: Graves
authors.affiliation: School of Public Health and Social Work, Queensland University of Technology, Victoria Park Road, Brisbane, Queensland, 4059, Australia; Institute of Health and Biomedical Innovation, Queensland University of Technology, 60 Musk Avenue, Kelvin Grove, Brisbane, Queensland 4059, Australia. Electronic address: a.barnett@qut.edu.au.
authors.forename: Adrian G
authors.initials: AG
authors.lastname: Barnett
mesh_terms: D001315:Australia; D003657:Decision Making; D005257:Fellowships and Scholarships; D015203:Reproducibility of Results; D012106:Research / Q000191:economics*; D016037:Single-Blind Method
keywords: Career; Fellowship; Funding; Peer review; Reliability; Research