Developing a scoring rubric for resident research presentations: a pilot study.
BACKGROUND
A requirement for all Accreditation Council for Graduate Medical Education (ACGME) approved residencies is the provision of "an opportunity for residents to participate in research." To comply with this requirement, most training programs encourage their residents to conduct research and to report their results. Few guidelines exist, however, for assessing the efficacy of the presentations. The goal of this pilot study was to develop a valid, one-page scoring rubric to be used during oral resident research presentations. Such a scoring rubric will facilitate acceptable agreement among faculty raters.


METHODS
Content validity was addressed by adhering to the Standards for Educational and Psychological Testing. A one-page, five-domain, behaviorally worded scoring rubric was developed. Inter-rater reliability was derived and three ACGME General Competencies were also addressed within the rubric.


RESULTS
The initial scoring rubric was tested with 11 resident oral presentations. The inter-rater reliability was 0.56 using Cronbach's alpha. The rubric was modified and the scale restricted to a 3-point scale. It was then tested with 17 additional presentations, which were independently rated by two general surgery faculty members. Cronbach's Alpha increased to 0.61.


CONCLUSIONS
An objective method to evaluate a resident's oral research presentation has been successfully piloted. This content valid rubric possesses good inter-rater reliability according to established guidelines. Clearly defined behaviors have been outlined within the rubric. Program directors will have psychometrically sound evidence for the ACGME. Future research will address generalizability and concurrent validity using other types of resident assessment data.
Musial
Joseph L
JL
Department of Surgery, Henry Ford Hospital, Detroit, Michigan 48202, USA. JMUSIAL1@hfhs.org
Rubinfeld
Ilan S
IS
Parker
Alton O
AO
Reickert
Craig A
CA
Adams
Sarah A
SA
Rao
Sishir
S
Shepard
Alexander D
AD
D000068:Accreditation / Q000379:methods / Q000592:standards*; D035843:Biomedical Research / Q000592:standards*; D004503:Education, Medical, Graduate / Q000592:standards*; D004521:Educational Measurement / Q000379:methods / Q000592:standards*; D017408:Guidelines as Topic; D007396:Internship and Residency / Q000592:standards*; D010865:Pilot Projects; D015203:Reproducibility of Results