Incorporation of core competency questions into an annual national self-assessment examination for residents in physical medicine and rehabilitation: results and implications.
OBJECTIVE
To determine the performance and change over time when incorporating questions in the core competency domains of practice-based learning and improvement (PBLI), systems-based practice (SBP), and professionalism (PROF) into the national PM&R Self-Assessment Examination for Residents (SAER).


METHODS
Prospective, longitudinal analysis.


METHODS
The national Self-Assessment Examination for Residents (SAER) in Physical Medicine and Rehabilitation, which is administered annually.


METHODS
Approximately 1100 PM&R residents who take the examination annually.


METHODS
Inclusion of progressively more challenging questions in the core competency domains of PBLI, SBP, and PROF.


METHODS
Individual test item level of difficulty (P value) and discrimination (point biserial index).


RESULTS
Compared with the overall test, questions in the subtopic areas of PBLI, SBP, and PROF were relatively easier and less discriminating (correlation of resident performance on these domains compared with that on the total test). These differences became smaller during the 3-year time period. The difficulty level of the questions in each of the subtopic domains was raised during the 3 year period to a level close to the overall exam. Discrimination of the test items improved or remained stable.


CONCLUSIONS
This study demonstrates that, with careful item writing and review, multiple-choice items in the PBLI, SBP, and PROF domains can be successfully incorporated into an annual, national self-assessment examination for residents. The addition of these questions had value in assessing competency while not compromising the overall validity and reliability of the exam. It is yet to be determined if resident performance on these questions corresponds to performance on other measures of competency in the areas of PBLI, SBP, and PROF.
Webster
Joseph B
JB
American Academy of PM&R Self-Assessment Committee, Division of PM&R, University of Utah, Salt Lake City, UT, USA. joseph.webster@hsc.utah.edu
D003162:Competency-Based Education; D032962:Consent Forms; D004521:Educational Measurement; D004992:Ethics, Medical / Q000193:education; D006801:Humans; D007396:Internship and Residency; D008137:Longitudinal Studies; D010810:Physical and Rehabilitation Medicine / Q000193:education*; D011361:Professional Competence; D011785:Quality Assurance, Health Care; D012647:Self-Assessment