title: Assessing the validity evidence of an objective structured assessment tool of technical skills for neonatal lumbar punctures.
abstract: BACKGROUND
  The lumbar puncture (LP) is a procedural competency deemed necessary by the Accreditation Council for Graduate Medical Education and the Emergency Medicine and Pediatric Residency Review Committees. The emergency department (ED) is a primary site for residents to be evaluated performing neonatal LPs. Current evaluation methods lack validity evidence as assessment tools.
  
  
  OBJECTIVE
  This was a pilot study to develop an objective structured assessment of technical skills for neonatal LP (OSATS-LP) and to document validity evidence for the instrument in regard to five sources of test validity: content, response process, relation to other variables, inter-rater reliability, and consequences of testing.
  
  
  METHODS
  Pediatric residents were videotaped in the fall of 2011 for comparison of faculty evaluation of resident performance during a neonatal LP using a video-delayed format. Residents completed a demographic experience survey evaluating relations to other variables. Content and response process validity was obtained through expert panel meetings and resulted in the following seven domains of performance for the OSATS-LP: preparation, positioning, analgesia, needle insertion, cerebrospinal fluid (CSF) collection, management of laboratory studies, and sterility. t-tests assessed significance between level of training, previous intensive care unit experience, and residents' self-assessed confidence in comparison with their total performance score. The inter-rater agreement of the OSATS-LP was obtained using the Fleiss' kappa for each domain.
  
  
  RESULTS
  Sixteen pediatric residents completed the simulation with six raters evaluating each resident (96 ratings). The domains of sterility and CSF collection had moderate statistical reliability (κ = 0.41 and 0.51, respectively). The domains of preparation, analgesia, and management of laboratories had substantial reliability (κ = 0.60, 0.62, and 0.62, respectively). The domains of positioning and needle insertion were less reliable (κ = 0.16 and 0.16, respectively). Individuals who had completed one or more rotations in the neonatal intensive care unit (NICU) had a higher total score (12.5 vs. 16.9; p < 0.01). The residents' own perception of ability to perform an LP unsupervised did not result in a higher total score.
  
  
  CONCLUSIONS
  The OSATS-LP has reasonable evidence in four of the five sources for test validity. This study serves as a launching point for using this tool in clinical environments such as the ED and, therefore, has the potential to provide real-time formative and summative feedback to improve resident skills and ultimately lead to improvements in patient care.
authors.affiliation: Department of Pediatrics, Division of Emergency Medicine, Children's Hospital of Pittsburgh, Pittsburgh, PA, USA. maya.iyer@chp.edu
authors.forename: Maya S
authors.initials: MS
authors.lastname: Iyer
authors.forename: Sally A
authors.initials: SA
authors.lastname: Santen
authors.forename: Michele
authors.initials: M
authors.lastname: Nypaver
authors.forename: Kavita
authors.initials: K
authors.lastname: Warrier
authors.forename: Stuart
authors.initials: S
authors.lastname: Bradin
authors.forename: Rachel
authors.initials: R
authors.lastname: Chapman
authors.forename: Jennifer
authors.initials: J
authors.lastname: McAllister
authors.forename: Jennifer
authors.initials: J
authors.lastname: Vredeveld
authors.forename: Joseph B
authors.initials: JB
authors.lastname: House
authors.lastname: Accreditation Council for Graduate Medical Education Committee
authors.lastname: Emergency Medicine and Pediatric Residency Review Committee
mesh_terms: D000328:Adult; D002983:Clinical Competence / Q000592:standards*; D004635:Emergency Medicine / Q000379:methods / Q000592:standards*; D005260:Female; D006801:Humans; D007231:Infant, Newborn; D015931:Intensive Care, Neonatal / Q000379:methods* / Q000592:standards*; D007396:Internship and Residency / Q000592:standards*; D008297:Male; D010865:Pilot Projects; D011237:Predictive Value of Tests; D011446:Prospective Studies; D015203:Reproducibility of Results; D013129:Spinal Puncture / Q000379:methods* / Q000592:standards*; D013337:Students, Medical; D014481:United States