title: Using the bootstrap to establish statistical significance for relative validity comparisons among patient-reported outcome measures.
abstract: BACKGROUND
  Relative validity (RV), a ratio of ANOVA F-statistics, is often used to compare the validity of patient-reported outcome (PRO) measures. We used the bootstrap to establish the statistical significance of the RV and to identify key factors affecting its significance.
  
  
  METHODS
  Based on responses from 453 chronic kidney disease (CKD) patients to 16 CKD-specific and generic PRO measures, RVs were computed to determine how well each measure discriminated across clinically-defined groups of patients compared to the most discriminating (reference) measure. Statistical significance of RV was quantified by the 95% bootstrap confidence interval. Simulations examined the effects of sample size, denominator F-statistic, correlation between comparator and reference measures, and number of bootstrap replicates.
  
  
  RESULTS
  The statistical significance of the RV increased as the magnitude of denominator F-statistic increased or as the correlation between comparator and reference measures increased. A denominator F-statistic of 57 conveyed sufficient power (80%) to detect an RV of 0.6 for two measures correlated at r = 0.7. Larger denominator F-statistics or higher correlations provided greater power. Larger sample size with a fixed denominator F-statistic or more bootstrap replicates (beyond 500) had minimal impact.
  
  
  CONCLUSIONS
  The bootstrap is valuable for establishing the statistical significance of RV estimates. A reasonably large denominator F-statistic (F > 57) is required for adequate power when using the RV to compare the validity of measures with small or moderate correlations (r < 0.7). Substantially greater power can be achieved when comparing measures of a very high correlation (r > 0.9).
authors.affiliation: Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA 01655, USA. Nina.Deng@umassmed.edu
authors.forename: Nina
authors.initials: N
authors.lastname: Deng
authors.forename: Jeroan J
authors.initials: JJ
authors.lastname: Allison
authors.forename: Hua Julia
authors.initials: HJ
authors.lastname: Fang
authors.forename: Arlene S
authors.initials: AS
authors.lastname: Ash
authors.forename: John E
authors.initials: JE
authors.lastname: Ware
mesh_terms: D000328:Adult; D000704:Analysis of Variance; D016001:Confidence Intervals; D005260:Female; D006304:Health Status; D006801:Humans; D008297:Male; D015233:Models, Statistical; D017063:Outcome Assessment, Health Care; D011788:Quality of Life; D051436:Renal Insufficiency, Chronic / Q000175:diagnosis / Q000523:psychology*; D057566:Self Report