title: An evaluation of automatic coronary artery calcium scoring methods with cardiac CT using the orCaScore framework.
abstract: OBJECTIVE
  The amount of coronary artery calcification (CAC) is a strong and independent predictor of cardiovascular disease (CVD) events. In clinical practice, CAC is manually identified and automatically quantified in cardiac CT using commercially available software. This is a tedious and time-consuming process in large-scale studies. Therefore, a number of automatic methods that require no interaction and semiautomatic methods that require very limited interaction for the identification of CAC in cardiac CT have been proposed. Thus far, a comparison of their performance has been lacking. The objective of this study was to perform an independent evaluation of (semi)automatic methods for CAC scoring in cardiac CT using a publicly available standardized framework.
  
  
  METHODS
  Cardiac CT exams of 72 patients distributed over four CVD risk categories were provided for (semi)automatic CAC scoring. Each exam consisted of a noncontrast-enhanced calcium scoring CT (CSCT) and a corresponding coronary CT angiography (CCTA) scan. The exams were acquired in four different hospitals using state-of-the-art equipment from four major CT scanner vendors. The data were divided into 32 training exams and 40 test exams. A reference standard for CAC in CSCT was defined by consensus of two experts following a clinical protocol. The framework organizers evaluated the performance of (semi)automatic methods on test CSCT scans, per lesion, artery, and patient.
  
  
  RESULTS
  Five (semi)automatic methods were evaluated. Four methods used both CSCT and CCTA to identify CAC, and one method used only CSCT. The evaluated methods correctly detected between 52% and 94% of CAC lesions with positive predictive values between 65% and 96%. Lesions in distal coronary arteries were most commonly missed and aortic calcifications close to the coronary ostia were the most common false positive errors. The majority (between 88% and 98%) of correctly identified CAC lesions were assigned to the correct artery. Linearly weighted Cohen's kappa for patient CVD risk categorization by the evaluated methods ranged from 0.80 to 1.00.
  
  
  CONCLUSIONS
  A publicly available standardized framework for the evaluation of (semi)automatic methods for CAC identification in cardiac CT is described. An evaluation of five (semi)automatic methods within this framework shows that automatic per patient CVD risk categorization is feasible. CAC lesions at ambiguous locations such as the coronary ostia remain challenging, but their detection had limited impact on CVD risk determination.
authors.affiliation: Image Sciences Institute, University Medical Center Utrecht, Utrecht 3508 GA, The Netherlands.
authors.forename: Jelmer M
authors.initials: JM
authors.lastname: Wolterink
authors.affiliation: Department of Radiology, University Medical Center Utrecht, Utrecht 3508 GA, The Netherlands.
authors.forename: Tim
authors.initials: T
authors.lastname: Leiner
authors.affiliation: Image Sciences Institute, University Medical Center Utrecht, Utrecht 3508 GA, The Netherlands.
authors.forename: Bob D
authors.initials: BD
authors.lastname: de Vos
authors.affiliation: INSERM, U1099, Rennes F-35000, France; LTSI, Université de Rennes 1, Rennes F-35000, France; and Centre de Recherche en Information Biomédicale Sino-Français (LIA CRIBs), Nanjing 210096, China.
authors.forename: Jean-Louis
authors.initials: JL
authors.lastname: Coatrieux
authors.affiliation: Imaging and Computer Vision, Corporate Technology, Siemens AG, Erlangen 91051, Germany.
authors.forename: B Michael
authors.initials: BM
authors.lastname: Kelm
authors.affiliation: Konica Minolta, Inc., Osaka 569-8503, Japan.
authors.forename: Satoshi
authors.initials: S
authors.lastname: Kondo
authors.affiliation: Department of Radiology, University Hospital Antwerpen, Edegem 2650, Belgium.
authors.forename: Rodrigo A
authors.initials: RA
authors.lastname: Salgado
authors.affiliation: Division of Image Processing, Department of Radiology, Leiden University Medical Center, Leiden 2300 RC, The Netherlands; Biomedical Imaging Group Rotterdam, Departments of Radiology and Medical Informatics, Erasmus MC, Rotterdam 3000 CA, The Netherlands; and Quantitative Imaging Group, Department of Imaging Physics, Faculty of Applied Sciences, Delft University of Technology, Delft 2600 GA, The Netherlands.
authors.forename: Rahil
authors.initials: R
authors.lastname: Shahzad
authors.affiliation: Centre de Recherche en Information Biomédicale Sino-Français (LIA CRIBs), Nanjing 210096, China and Lab of Image Science and Technology, School of Computer Science and Technology, Nanjing 210096, China.
authors.forename: Huazhong
authors.initials: H
authors.lastname: Shu
authors.affiliation: Department of Radiology, Radboud University Medical Center, Nijmegen 6500 HB, The Netherlands.
authors.forename: Miranda
authors.initials: M
authors.lastname: Snoeren
authors.affiliation: Department of Radiology, University Medical Center Utrecht, Utrecht 3508 GA, The Netherlands.
authors.forename: Richard A P
authors.initials: RA
authors.lastname: Takx
authors.affiliation: Quantitative Imaging Group, Department of Imaging Physics, Faculty of Applied Sciences, Delft University of Technology, Delft 2600 GA, The Netherlands.
authors.forename: Lucas J
authors.initials: LJ
authors.lastname: van Vliet
authors.affiliation: Biomedical Imaging Group Rotterdam, Departments of Radiology and Medical Informatics, Erasmus MC, Rotterdam 3000 CA, The Netherlands.
authors.forename: Theo
authors.initials: T
authors.lastname: van Walsum
authors.affiliation: Department of Radiology, University Medical Center Groningen, Groningen 9700 RB, The Netherlands.
authors.forename: Tineke P
authors.initials: TP
authors.lastname: Willems
authors.affiliation: Lab of Image Science and Technology, School of Computer Science and Technology, Nanjing 210096, China and Centre de Recherche en Information Biomédicale Sino-Français (LIA CRIBs), Nanjing 210096, China.
authors.forename: Guanyu
authors.initials: G
authors.lastname: Yang
authors.affiliation: Imaging and Computer Vision, Corporate Technology, Siemens Corporation, Princeton, New Jersey 08540-6632.
authors.forename: Yefeng
authors.initials: Y
authors.lastname: Zheng
authors.affiliation: Image Sciences Institute, University Medical Center Utrecht, Utrecht 3508 GA, The Netherlands.
authors.forename: Max A
authors.initials: MA
authors.lastname: Viergever
authors.affiliation: Image Sciences Institute, University Medical Center Utrecht, Utrecht 3508 GA, The Netherlands.
authors.forename: Ivana
authors.initials: I
authors.lastname: Išgum
mesh_terms: D002118:Calcium / Q000378:metabolism; D017023:Coronary Angiography / Q000295:instrumentation / Q000379:methods*; D003324:Coronary Artery Disease / Q000000981:diagnostic imaging / Q000378:metabolism; D003331:Coronary Vessels / Q000000981:diagnostic imaging* / Q000378:metabolism; D005260:Female; D006801:Humans; D008297:Male; D010363:Pattern Recognition, Automated / Q000379:methods*; D012307:Risk Factors; D012680:Sensitivity and Specificity; D014057:Tomography, X-Ray Computed / Q000295:instrumentation / Q000379:methods*; D061205:Vascular Calcification / Q000000981:diagnostic imaging / Q000378:metabolism
chemical_list: D002118:Calcium