title: Is consensus reproducible? A study of an algorithmic guidelines development process.
abstract: The authors evaluated the reproducibility of a clinical algorithm consensus development process across three different physician panels at a health maintenance organization. Physician groups were composed of primary care internists, who were provided with identical selections from the medical literature and first-draft "seed" algorithms on the management of two common clinical problems: acute sinusitis and dyspepsia. Each panel used nominal group process and a modified Delphi method to create final algorithm drafts. To compare the clinical logic in the final algorithms, the authors applied a new qualitative and quantitative comparison method, the Clinical Algorithm Patient Abstraction (CAPA). Dyspepsia algorithms from all physician groups recommended empiric anti-acid therapy for most patients, favored endoscopy over barium swallow, and had very similar indications for endoscopy. The average CAPA comparison score among final physician algorithms was 6.1 on a scale of 0 (different) to 10 (identical). Sinusitis algorithms from all groups proposed empiric antibiotic therapy for most patients. Indications for sinus radiographs were similar between two algorithms (CAPA = 4.9), but differed significantly in the third, resulting in lower CAPA scores (average CAPA = 1.9, P < 0.03). The clinical similarity of the algorithms produced by these physician panels suggests a high level of reproducibility in this consensus-driven algorithm development process. However, the difference among the sinusitis algorithms suggests that physician consensus groups using a consensus process that a health maintenance organization can do with limited resources will produce some guidelines that vary due to differences in interpretation of evidence and physician experience.
authors.affiliation: Department of Ambulatory Care and Prevention, Harvard Community Health Plan, Boston, MA, USA.
authors.forename: S D
authors.initials: SD
authors.lastname: Pearson
authors.forename: C Z
authors.initials: CZ
authors.lastname: Margolis
authors.forename: S
authors.initials: S
authors.lastname: Davis
authors.forename: L K
authors.initials: LK
authors.lastname: Schreier
authors.forename: H N
authors.initials: HN
authors.lastname: Sokol
authors.forename: L K
authors.initials: LK
authors.lastname: Gottlieb
mesh_terms: D000208:Acute Disease; D000465:Algorithms; D003245:Consensus Statements as Topic; D003697:Delphi Technique; D004415:Dyspepsia / Q000175:diagnosis / Q000628:therapy; D006126:Group Processes; D006801:Humans; D017410:Practice Guidelines as Topic; D015203:Reproducibility of Results; D012107:Research Design; D012852:Sinusitis / Q000175:diagnosis / Q000628:therapy