Republished paper: Assuring validity of multisource feedback in a national programme.
OBJECTIVE
To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Tool (SPRAT) with paediatric Specialist Registrars (SpRs) across the UK as part of Royal College of Paediatrics and Child Health workplace based assessment programme.


METHODS
Quality assurance analysis, including generalisability, of a multisource feedback questionnaire study.


METHODS
All UK Deaneries between August 2005 and May 2006.


METHODS
577 year 2 and 4 Paediatric SpRs.


METHODS
Trainees were evaluated using SPRAT sent to clinical colleagues of their choosing. Data were analysed reporting totals, means and SD, and year groups were compared using independent t tests. A factor analysis was undertaken. Reliability was estimated using generalisability theory. Trainee and assessor demographic details were explored to try to explain variability in scores.


METHODS
4770 SPRAT assessments were provided about 577 paediatric SpRs. The mean scores between years were significantly different (Year 2 mean=5.08, SD=0.34, Year 4 mean=5.18, SD=0.34). A factor analysis returned a two-factor solution, clinical care and psychosocial skills. The 95% CI showed that trainees scoring ≥4.3 with nine assessors can be seen as achieving satisfactory performance with statistical confidence. Consultants marked trainees significantly lower (t=-4.52) whereas Senior House Officers and Foundation doctors scored their SpRs significantly higher (SHO t=2.06, Foundation t=2.77).


CONCLUSIONS
There is increasing evidence that multisource feedback (MSF) assesses two generic traits, clinical care and psychosocial skills. The validity of MSF is threatened by systematic bias, namely leniency bias and the seniority of assessors. Unregulated self-selection of assessors needs to end.
Archer
Julian
J
Peninsula College of Medicine and Dentistry, University of Plymouth, UK. julian.archer@pms.ac.uk
McGraw
Mary
M
Davies
Helena
H
D002983:Clinical Competence; D004503:Education, Medical, Graduate / Q000379:methods*; D004521:Educational Measurement / Q000379:methods*; D004650:Employee Performance Appraisal / Q000379:methods; D030141:Feedback, Psychological; D005260:Female; D006801:Humans; D008297:Male; D008505:Medical Staff, Hospital / Q000193:education / Q000592:standards; D010372:Pediatrics / Q000193:education* / Q000592:standards; D010380:Peer Review / Q000379:methods; D015203:Reproducibility of Results; D006113:United Kingdom