title: Funding Medical Research Projects: Taking into Account Referees' Severity and Consistency through Many-Faceted Rasch Modeling of Projects' Scores.
abstract: The funding policy of research projects often relies on scores assigned by a panel of experts (referees). The non-linear nature of raw scores and the severity and inconsistency of individual raters may generate unfair numeric project rankings. Rasch measurement (many-facets version, MFRM) provides a valid alternative to scoring. MFRM was applied to the scores achieved by 75 research projects on multiple sclerosis sent in response to a previous annual call by FISM-Italian Foundation for Multiple Sclerosis. This allowed to simulate, a posteriori, the impact of MFRM on the funding scenario. The applications were each scored by 2 to 4 independent referees (total = 131) on a 10-item, 0-3 rating scale called FISM-ProQual-P. The rotation plan assured "connection" of all pairs of projects through at least 1 shared referee.The questionnaire fulfilled satisfactorily the stringent criteria of Rasch measurement for psychometric quality (unidimensionality, reliability and data-model fit). Arbitrarily, 2 acceptability thresholds were set at a raw score of 21/30 and at the equivalent Rasch measure of 61.5/100, respectively. When the cut-off was switched from score to measure 8 out of 18 acceptable projects had to be rejected, while 15 rejected projects became eligible for funding. Some referees, of various severity, were grossly inconsistent (z-std fit indexes less than -1.9 or greater than 1.9). The FISM-ProQual-P questionnaire seems a valid and reliable scale. MFRM may help the decision-making process for allocating funds to MS research projects but also in other fields. In repeated assessment exercises it can help the selection of reliable referees. Their severity can be steadily calibrated, thus obviating the need to connect them with other referees assessing the same projects.
authors.affiliation: Luigi Tesio, Department of Neurorehabilitation Sciences, Istituto Auxologico Italiano, IRCCS, Via Giuseppe Mercalli 32, 20122 Milano, Italy, luigi.tesio@unimi.it.
authors.forename: Luigi
authors.initials: L
authors.lastname: Tesio
authors.forename: Anna
authors.initials: A
authors.lastname: Simone
authors.forename: Mariuzs T
authors.initials: MT
authors.lastname: Grzeda
authors.forename: Michela
authors.initials: M
authors.lastname: Ponzio
authors.forename: Gabriele
authors.initials: G
authors.lastname: Dati
authors.forename: Paola
authors.initials: P
authors.lastname: Zaratin
authors.forename: Laura
authors.initials: L
authors.lastname: Perucca
authors.forename: Mario A
authors.initials: MA
authors.lastname: Battaglia
mesh_terms: D026683:Advisory Committees; D035843:Biomedical Research / Q000191:economics*; D006801:Humans; D008962:Models, Theoretical; D009103:Multiple Sclerosis; D012109:Research Support as Topic; D011795:Surveys and Questionnaires