title: Assessing discriminative ability of risk models in clustered data.
abstract: BACKGROUND
  The discriminative ability of a risk model is often measured by Harrell's concordance-index (c-index). The c-index estimates for two randomly chosen subjects the probability that the model predicts a higher risk for the subject with poorer outcome (concordance probability). When data are clustered, as in multicenter data, two types of concordance are distinguished: concordance in subjects from the same cluster (within-cluster concordance probability) and concordance in subjects from different clusters (between-cluster concordance probability). We argue that the within-cluster concordance probability is most relevant when a risk model supports decisions within clusters (e.g. who should be treated in a particular center). We aimed to explore different approaches to estimate the within-cluster concordance probability in clustered data.
  
  
  METHODS
  We used data of the CRASH trial (2,081 patients clustered in 35 centers) to develop a risk model for mortality after traumatic brain injury. To assess the discriminative ability of the risk model within centers we first calculated cluster-specific c-indexes. We then pooled the cluster-specific c-indexes into a summary estimate with different meta-analytical techniques. We considered fixed effect meta-analysis with different weights (equal; inverse variance; number of subjects, events or pairs) and random effects meta-analysis. We reflected on pooling the estimates on the log-odds scale rather than the probability scale.
  
  
  RESULTS
  The cluster-specific c-index varied substantially across centers (IQR = 0.70-0.81; I2 = 0.76 with 95% confidence interval 0.66 to 0.82). Summary estimates resulting from fixed effect meta-analysis ranged from 0.75 (equal weights) to 0.84 (inverse variance weights). With random effects meta-analysis - accounting for the observed heterogeneity in c-indexes across clusters - we estimated a mean of 0.77, a between-cluster variance of 0.0072 and a 95% prediction interval of 0.60 to 0.95. The normality assumptions for derivation of a prediction interval were better met on the probability than on the log-odds scale.
  
  
  CONCLUSIONS
  When assessing the discriminative ability of risk models used to support decisions at cluster level we recommend meta-analysis of cluster-specific c-indexes. Particularly, random effects meta-analysis should be considered.
authors.affiliation: Department of Public Health, Erasmus MC, Dr, Molewaterplein 50, Rotterdam 3015 GE, The Netherlands. d.vanklaveren.1@erasmusmc.nl.
authors.forename: David
authors.initials: D
authors.lastname: van Klaveren
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
authors.forename: Pablo
authors.initials: P
authors.lastname: Perel
authors.forename: Yvonne
authors.initials: Y
authors.lastname: Vergouwe
mesh_terms: D000328:Adult; D001930:Brain Injuries / Q000401:mortality*; D016000:Cluster Analysis; D053483:Eye Movement Measurements; D006801:Humans; D008875:Middle Aged; D015233:Models, Statistical; D008962:Models, Theoretical; D018570:Risk Assessment; D055815:Young Adult