title: Assessment of competency in clinical measurement: comparison of two forms of sequential test and sensitivity of test error rates to parameter choice.
abstract: OBJECTIVE
  To assess clinical measurement competency by two sequential test formulations [resetting sequential probability ratio test (R-SPRT) and learning curve cumulative summation (LC-CUSUM)].
  
  
  METHODS
  Numerical simulation and retrospective observational study.
  
  
  METHODS
  Obstetric ultrasound department.
  
  
  METHODS
  Cohorts of 10,000 simulated trainees and 62 obstetric sonographers training in nuchal translucency (NT) measurement at the 11-14-week pregnancy scan with limited case availability.
  
  
  METHODS
  Application of LC-CUSUM and R-SPRT to clinical measurement training.
  
  
  METHODS
  Proportions of real trainees achieving competency by LC-CUSUM and R-SPRT, proportions of simulated competent trainees not achieving competency (Type I error), proportions of simulated incompetent trainees achieving competency (Type II error), distribution of case number required to achieve competency (run length) and frequency of resets.
  
  
  RESULTS
  For simulated cohorts, significant differences in run-length distribution and true test error rates were found between the R-SPRT and LC-CUSUM tests with equivalent parameters. Increasing the cases available to each trainee reduced the Type I error rate but increased the Type II error rate for both sequential tests for all choices of unacceptable failure rate. Discontinuities in the proportion of trainees expected to be test competent were found at critical values of unacceptable failure rate.
  
  
  CONCLUSIONS
  With equivalent parameters, the R-SPRT and LC-CUSUM formulations of sequential tests produced different outcomes, demonstrating that the choice of test method, as well as the choice of parameters, is important in designing a training scheme. The R-SPRT detects incompetence as well as competence and may indicate need for further training. Simulations are valuable in estimating the proportions of trainees expected to be assessed as competent.
authors.affiliation: Regional Medical Physics Department, Newcastle upon Tyne Hospitals NHS Foundation Trust, Freeman Hospital, Newcastle upon Tyne NE7 7DN, UK. andrew.sims@nuth.nhs.uk
authors.forename: Andrew J
authors.initials: AJ
authors.lastname: Sims
authors.forename: Kim
authors.initials: K
authors.lastname: Keltie
authors.forename: Julie
authors.initials: J
authors.lastname: Burn
authors.forename: Stephen C
authors.initials: SC
authors.lastname: Robson
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D004521:Educational Measurement; D005260:Female; D006801:Humans; D011247:Pregnancy; D012189:Retrospective Studies; D016216:Ultrasonography, Prenatal / Q000379:methods / Q000592:standards
keywords: computer simulation; outcome assessment; quality control; statistical; training