title: Assessing the inter-rater reliability and agreement of a hybrid assessment framework in the Emergency Department.
abstract: OBJECTIVE
  There is often little consensus on the levels of trainees' performances, given the varied expertise and experience of the assessors, different tasks to be assessed and dynamic circumstances. Consequently, assessors are considered to be a major source of measurement error in workplace assessment. The aim of this pilot study is to assess the inter-rater reliability and agreement of a hybrid framework for assessment of nonspecialist doctors in our Emergency Department.
  
  
  METHODS
  A hybrid framework was used to evaluate two core competencies of medical knowledge and patient care of nonspecialist doctors in our Emergency Department. Pre and post scores of these doctors were obtained at the end of their posting.
  
  
  RESULTS
  The inter-rater agreement and reliability was markedly higher for the group of nonspecialist doctors with previous Emergency Medicine experience, with fair-to-good reliability of the ratings among the assessors after application of the framework.
  
  
  CONCLUSIONS
  The use of our hybrid framework improved the inter-rater reliability and agreement in our Emergency Department for a specific group of learners who have had previous Emergency Medicine experience. To improve the reliability of ratings, faculty development has to address how observations and assessments should be synthesized into an overall rating.
authors.affiliation: Changi General Hospital, Singapore, Singapore.
authors.forename: Han Nee
authors.initials: HN
authors.lastname: Gan
authors.forename: Ling
authors.initials: L
authors.lastname: Tiah
authors.forename: Shu Woan
authors.initials: SW
authors.lastname: Lee
authors.forename: Wee Yee
authors.initials: WY
authors.lastname: Lee
mesh_terms: D000328:Adult; D002983:Clinical Competence / Q000592:standards*; D004521:Educational Measurement / Q000592:standards; D004635:Emergency Medicine / Q000193:education / Q000592:standards; D004636:Emergency Service, Hospital / Q000592:standards*; D006786:Hospitals, Urban; D006801:Humans; D008875:Middle Aged; D015588:Observer Variation; D015203:Reproducibility of Results; D012846:Singapore