title: Clinical assessment of transthoracic echocardiography skills: a generalizability study.
abstract: BACKGROUND
  Transthoracic echocardiography (TTE) is a widely used cardiac imaging technique that all cardiologists should be able to perform competently. Traditionally, TTE competence has been assessed by unstructured observation or in test situations separated from daily clinical practice. An instrument for assessment of clinical TTE technical proficiency including a global rating score and a checklist score has previously shown reliability and validity in a standardised setting. As clinical test situations typically have several sources of error giving rise to variance in scores, a more thorough examination of the generalizability of the assessment instrument is needed.
  
  
  METHODS
  Nine physicians performed a TTE scan on the same three patients. Then, two raters rated all 27 TTE scans using the TTE technical assessment instrument in a fully crossed, all random generalizability study. Estimated variance components were calculated for both the global rating and checklist scores. Finally, dependability (phi) coefficients were also calculated for both outcomes in a decision study.
  
  
  RESULTS
  For global rating scores, 66.6% of score variance can be ascribed to true differences in performance. For checklist scores this was 88.8%. The difference was primarily due to physician-rater interaction. Four random cases rated by one random rater resulted in a phi value of 0.81 for global ratings and two random cases rated by one random rater showed a phi value of 0.92 for checklist scores.
  
  
  CONCLUSIONS
  Using the TTE checklist as opposed to the TTE global rating score had the effect of minimising the largest source of error variance in test scores. Two cases rated by one rater using the TTE checklist are sufficiently reliable for high stakes examinations. As global rating is less time consuming it could be considered performing four global rating assessments in addition to the checklist assessments to account for both reliability and content validity of the assessment.
authors: Department of Cardiology, Aarhus University Hospital, Aarhus, Denmark. dornis@rm.dk.
authors: Dorte
authors: D
authors: Guldbrand Nielsen
authors: Department of Cardiology, Aarhus University Hospital, Aarhus, Denmark. signelj4@hotmail.com.
authors: Signe Lichtenstein
authors: SL
authors: Jensen
authors: Center for Medical Education, Aarhus University, Aarhus, Denmark. loni@medu.au.dk.
authors: Lotte
authors: L
authors: O'Neill
mesh_terms: D000704:Analysis of Variance; D001024:Aortic Valve Stenosis / Q000000981:diagnostic imaging*; D057189:Checklist; D018592:Cross-Over Studies; D004452:Echocardiography; D004521:Educational Measurement / Q000379:methods*; D005260:Female; D005793:Generalization, Psychological; D006801:Humans; D008297:Male; D008944:Mitral Valve Insufficiency / Q000000981:diagnostic imaging*; D015588:Observer Variation; D015203:Reproducibility of Results; D012107:Research Design
references: BMC Med Educ. 2012 Dec 07;12:121
references: 23216816
references: Med Educ. 2001 Apr;35(4):321-5
references: 11318993
references: Med Teach. 2012;34(8):635-42
references: 22830320
references: Adv Health Sci Educ Theory Pract. 2012 Dec;17(5):703-16
references: 22234383
references: BMC Med Educ. 2013 Mar 28;13:47
references: 23537204
references: J Am Coll Cardiol. 2003 Feb 19;41(4):687-708
references: 12598084
references: Eur J Echocardiogr. 2007 Jan;8(1):74-9
references: 17175201
references: Med Teach. 2013 Aug;35(8):684-91
references: 23782043
references: Med Educ. 2002 Oct;36(10):901-9
references: 12390456
references: Acad Med. 2012 Aug;87(8):1077-82
references: 22722349
references: Med Educ. 2007 Oct;41(10):926-34
references: 17908111
references: Adv Med Educ Pract. 2010 Nov 19;1:67-73
references: 23745065
references: Adv Health Sci Educ Theory Pract. 2014 Mar;19(1):71-84
references: 23709188
references: Eur Heart J. 2013 Aug;34(30):2381-411
references: 23847131
references: Br J Anaesth. 2009 May;102(5):633-41
references: 19336537
references: Med Educ. 2006 Jul;40(7):618-23
references: 16836533
references: Adv Health Sci Educ Theory Pract. 2013 Mar;18(1):5-13
references: 22193944
references: Med Educ. 2004 Sep;38(9):1006-12
references: 15327684
references: Acad Med. 2011 Aug;86(8):1032-9
references: 21694560
references: Eval Health Prof. 2008 Dec;31(4):419-36
references: 18842619
references: Med Educ. 2001 Apr;35(4):326-30
references: 11318994
references: Adv Health Sci Educ Theory Pract. 2008 Nov;13(4):521-33
references: 17476579
references: Acad Med. 2012 Oct;87(10):1330-4
references: 22914517
references: J Am Coll Cardiol. 2008 Jan 22;51(3):361-7
references: 18206753
references: Eur J Echocardiogr. 2009 Dec;10(8):893-905
references: 19889658
references: Patient Educ Couns. 2012 Jun;87(3):293-9
references: 22169634