title: How to assess intra- and inter-observer agreement with quantitative PET using variance component analysis: a proposal for standardisation.
abstract: BACKGROUND
  Quantitative measurement procedures need to be accurate and precise to justify their clinical use. Precision reflects deviation of groups of measurement from another, often expressed as proportions of agreement, standard errors of measurement, coefficients of variation, or the Bland-Altman plot. We suggest variance component analysis (VCA) to estimate the influence of errors due to single elements of a PET scan (scanner, time point, observer, etc.) to express the composite uncertainty of repeated measurements and obtain relevant repeatability coefficients (RCs) which have a unique relation to Bland-Altman plots. Here, we present this approach for assessment of intra- and inter-observer variation with PET/CT exemplified with data from two clinical studies.
  
  
  METHODS
  In study 1, 30 patients were scanned pre-operatively for the assessment of ovarian cancer, and their scans were assessed twice by the same observer to study intra-observer agreement. In study 2, 14 patients with glioma were scanned up to five times. Resulting 49 scans were assessed by three observers to examine inter-observer agreement. Outcome variables were SUVmax in study 1 and cerebral total hemispheric glycolysis (THG) in study 2.
  
  
  RESULTS
  In study 1, we found a RC of 2.46 equalling half the width of the Bland-Altman limits of agreement. In study 2, the RC for identical conditions (same scanner, patient, time point, and observer) was 2392; allowing for different scanners increased the RC to 2543. Inter-observer differences were negligible compared to differences owing to other factors; between observer 1 and 2: -10 (95 % CI: -352 to 332) and between observer 1 vs 3: 28 (95 % CI: -313 to 370).
  
  
  CONCLUSIONS
  VCA is an appealing approach for weighing different sources of variation against each other, summarised as RCs. The involved linear mixed effects models require carefully considered sample sizes to account for the challenge of sufficiently accurately estimating variance components.
authors.affiliation: Department of Nuclear Medicine, Odense University Hospital, Sdr. Boulevard 29, 5000, Odense C, Denmark. oke.gerke@rsyd.dk.|Centre of Health Economics Research, University of Southern Denmark, Campusvej 55, 5230, Odense M, Denmark. oke.gerke@rsyd.dk.
authors.forename: Oke
authors.identifier: http://orcid.org/0000-0001-6335-3303
authors.initials: O
authors.lastname: Gerke
authors.affiliation: Department of Nuclear Medicine, Odense University Hospital, Sdr. Boulevard 29, 5000, Odense C, Denmark.
authors.forename: Mie Holm
authors.initials: MH
authors.lastname: Vilstrup
authors.affiliation: Department of Nuclear Medicine, Odense University Hospital, Sdr. Boulevard 29, 5000, Odense C, Denmark.
authors.forename: Eivind Antonsen
authors.initials: EA
authors.lastname: Segtnan
authors.affiliation: Epidemiology, Biostatistics and Biodemography, University of Southern Denmark, J. B. Winsløws Vej 9b, 5000, Odense C, Denmark.
authors.forename: Ulrich
authors.initials: U
authors.lastname: Halekoh
authors.affiliation: Department of Nuclear Medicine, Odense University Hospital, Sdr. Boulevard 29, 5000, Odense C, Denmark.|Department of Clinical Research, University of Southern Denmark, Winsløwparken 19, 5000, Odense C, Denmark.
authors.forename: Poul Flemming
authors.initials: PF
authors.lastname: Høilund-Carlsen
keywords: Bland-Altman limits of agreement; Inter-observer; Inter-rater; Interobserver; Intra-observer; Intra-rater; Intraobserver; Repeatability coefficient; Sample size; Standardised uptake value