title: The design of observer agreement studies with binary assessments.
abstract: We discuss the design of observer agreement studies with binary assessments, with particular emphasis on the need for adequate sample size and the use of replicate observations. First, we present a method and tables for determining the sample size required for ensuring a desired precision for the estimate of the probability of disagreement between two observers. Second, for studies including replicate observations, we present a statistical model that allows estimation of the magnitude of within- and between-observer variation. We then derive sample sizes guaranteeing a specified precision for these estimates, present tables of these sample sizes and give examples of their use.
authors.affiliation: Biometry Branch, National Cancer Institute, Bethesda, Maryland 20892.
authors.forename: L S
authors.initials: LS
authors.lastname: Freedman
authors.forename: M K
authors.initials: MK
authors.lastname: Parmar
authors.forename: S G
authors.initials: SG
authors.lastname: Baker
mesh_terms: D016001:Confidence Intervals; D006801:Humans; D015233:Models, Statistical; D015588:Observer Variation; D012107:Research Design