title: Assessing the sources of unreliability (rater, subject, time-point) in a failed clinical trial using items of the Positive and Negative Syndrome Scale (PANSS).
abstract: BACKGROUND
  Considering the increasing attention to the study of failed clinical trials, the goal of this study was to identify the sources of unreliability in a failed clinical trial by assessing scores on the Positive and Negative Syndrome Scale (PANSS).
  
  
  METHODS
  This study is a substudy from a failed phase 2 double-blind, placebo-controlled trial of schizophrenia. Using the generalizability theory, this substudy assesses reliability on 3 conditions: raters, time points (PANSS evaluations, 1 week apart), subjects for 3 groups (placebo responders, placebo nonresponders, and treatment group).
  
  
  RESULTS
  The placebo response rate was 40.07% (32/71). For all PANSS positive symptom items, the most variability was for raters (range, 33%-72%) for the placebo responders, 31% to 68% for the placebo nonresponders, and 29% to 60% for the treatment group. The variability of the interaction of rater and time point was the second source of unreliability, with an average of 12.28% compared to 12.00% for the placebo nonresponders and 10.00% for the treatment group. All items of the negative symptom subscale showed the most percent variability for raters, for all groups. For general psychopathology items (except preoccupation), raters accounted for the most variability in the scores for placebo responders with an average of 51.00% across items. A similar pattern was observed for the placebo nonresponders and for the treatment group; for the treatment group, the interaction between rater and time point accounted for the most variability for somatic concern and anxiety.
  
  
  CONCLUSIONS
  Results confirm the efficacy of applying the generalizability theory to the estimation of reliability to identify a source of unreliability and provide evidence for the relationship between low reliability and failed trials. Findings can be used to guide data monitoring, rater training, and identification of PANSS items, which may require supplementary training.
authors.affiliation: Nathan S. Kline Institute for Psychiatric Research, Orangeburg, NY 10016, USA. AKhan@nki.rfmh.org
authors.forename: Anzalee
authors.initials: A
authors.lastname: Khan
authors.forename: William Christian
authors.initials: WC
authors.lastname: Yavorsky
authors.forename: Stacy
authors.initials: S
authors.lastname: Liechti
authors.forename: Guillermo
authors.initials: G
authors.lastname: DiClemente
authors.forename: Brian
authors.initials: B
authors.lastname: Rothman
authors.forename: Mark
authors.initials: M
authors.lastname: Opler
authors.forename: Ashleigh
authors.initials: A
authors.lastname: DeFries
authors.forename: Sofija
authors.initials: S
authors.lastname: Jovic
mesh_terms: D000293:Adolescent; D000328:Adult; D000368:Aged; D014150:Antipsychotic Agents / Q000627:therapeutic use*; D002983:Clinical Competence; D004311:Double-Blind Method; D023381:Endpoint Determination; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D015233:Models, Statistical; D015588:Observer Variation; D018579:Patient Selection; D011237:Predictive Value of Tests; D011569:Psychiatric Status Rating Scales; D015203:Reproducibility of Results; D012107:Research Design; D012559:Schizophrenia / Q000175:diagnosis / Q000188:drug therapy*; D012565:Schizophrenic Psychology; D013997:Time Factors; D017211:Treatment Failure; D014481:United States; D055815:Young Adult
chemical_list: D014150:Antipsychotic Agents