How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations.
CONSTRUCT: The study compares paper and online ratings of instructional units and analyses, with the G-study using the symmetry principle, the response rates needed to ensure acceptable precision of the measure when compliance is low.


BACKGROUND
Students' ratings of teaching contribute to the quality of medical training programs. To date, many schools have replaced pen-and-paper questionnaires with electronic forms, despite the lower response rates consistently reported with the latter. Few available studies have examined the effects of low response rates on the reliability and precision of the evaluation measure. Moreover, the minimum number of raters to target when response rates are low remains unclear.


METHODS
Descriptive data were derived from 799 students' paper and online ratings of 11 preclinical instructional units (PIUs). Reliability was assessed by Cronbach's alpha coefficients. The generalizability method applying the symmetry principle approach was used to analyze the precision of the measure with a reference standard error of mean (SEM) set at 0.10; optimization models were built to estimate minimum response rates.


RESULTS
Overall, response rates were 74% and 30% (p < .001) and PIUs ratings were 3.8 ± 0.5 and 3.6 ± 0.5 (p = .02), respectively in paper and online questionnaires. Higher SEM levels and significantly larger 95% confidence intervals of PIUs rating scores were observed with online evaluations. To keep the SEM within preset limits of precision, a minimum of 48% response rate was estimated for online formats.


CONCLUSIONS
The proposed generalizability analysis allowed estimating the minimum response needed to maintain acceptable precision in online evaluations. The effects of response rates on accuracy are discussed.
a Unit of Development and Research in Medical Education, University of Geneva Faculty of Medicine , Geneva , Switzerland.
Margaret W
MW
Gerbase
a Unit of Development and Research in Medical Education, University of Geneva Faculty of Medicine , Geneva , Switzerland.
Michèle
M
Germond
a Unit of Development and Research in Medical Education, University of Geneva Faculty of Medicine , Geneva , Switzerland.
Bernard
B
Cerutti
a Unit of Development and Research in Medical Education, University of Geneva Faculty of Medicine , Geneva , Switzerland.
Nu V
NV
Vu
a Unit of Development and Research in Medical Education, University of Geneva Faculty of Medicine , Geneva , Switzerland.
Anne
A
Baroffio
D004504:Education, Medical, Undergraduate; D006801:Humans; D020407:Internet; D009862:Online Systems; D010209:Paper; D015397:Program Evaluation / Q000706:statistics & numerical data*; D018401:Sample Size; D013337:Students, Medical; D011795:Surveys and Questionnaires
compliance; evaluation of teaching and learning; online evaluations; response rates; undergraduate medical education