title: Automated determination of bubble grades from Doppler ultrasound recordings.
abstract: BACKGROUND
  One of the difficulties in the development of automated algorithms for the detection of bubbles in Doppler ultrasound recordings is that expert labels are only available on an aggregate basis, i.e., the expert provides a single label for a recording which may contain many bubbles. It is thus very difficult to determine whether an algorithm is correctly identifying the actual bubbles or simply identifying the correct number of events, but mislabeling some events that are not due to bubbles.
  
  
  METHODS
  The analysis presented here shows that the classification probabilities for the detection of bubble events and other artifacts can be determined if a large number of recordings are available.
  
  
  CONCLUSIONS
  Using a half-integer scoring system from 0-4 gives a bias of approximately 1-2% and a standard deviation that varies with the number of event sequences available, dropping from approximately 7.5% for 100 60-s recordings to 3% for 1000 60-s recordings. These values are larger if an integer scoring system is used, but using a scale finer than half-integers confers no extra benefit due to the fact that the expert labels the whole recording rather than individual bubbles.
  
  
  CONCLUSIONS
  It is thus possible to estimate the classification probabilities with a reasonably high degree of accuracy, but difficult to show that one bubble detection algorithm is superior to another to any degree of statistical significance. Expert labels can be used to validate, but not to compare the performance of bubble detection algorithms.
authors.affiliation: Department of Engineering Science, University of Oxford, Parks Rd., Oxford OX1 3PJ, UK. stephen.payne@eng.ox.ac.uk
authors.forename: Stephen J
authors.initials: SJ
authors.lastname: Payne
authors.forename: Michael A
authors.initials: MA
authors.lastname: Chappell
mesh_terms: D000465:Algorithms; D016477:Artifacts; D001331:Automation; D003665:Decompression Sickness / Q000000981:diagnostic imaging*; D006339:Heart Rate; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D018608:Ultrasonography, Doppler / Q000379:methods*