title: A novel approach to assess inter-rater reliability in the use of the Overt Aggression Scale-Modified.
abstract: The ability of raters to apply measures of efficacy accurately and reproducibly in clinical trials of psychotropic medications is vital to the validity of study data. In a large, multi-center trial using the Overt Aggression Scale-Modified (OAS-M), inter-rater reliability was evaluated using standardized patients (SP) in a live 'mock' interview. Thirty raters experienced in the OAS-M each interviewed two actors trained to portray patients with intermittent explosive disorder, borderline and antisocial personality disorder, and/or post-traumatic stress disorder. Inter-rater reliability of OAS-M scores was evaluated using the intra-class correlation coefficient (ICC). The joint reliability of the raters in the study with an expert rater was also assessed by an ICC. The ICC was 0.96 between the raters, and 0.98 between the raters and expert rater. Results suggest that SP can be utilized in rater assessment and that raters administering the OAS-M in this model of rater reliability assessment demonstrate a high level of consistency and reliability. The method described here may be useful in future assessments of inter-rater reliability.
authors.affiliation: New York State Psychiatric Institute, Unit 123, 1051 Riverside Drive, New York 10032, USA. je10@columbia.edu
authors.forename: Jean
authors.initials: J
authors.lastname: Endicott
authors.forename: Katherine
authors.initials: K
authors.lastname: Tracy
authors.forename: David
authors.initials: D
authors.lastname: Burt
authors.forename: Evelyn
authors.initials: E
authors.lastname: Olson
authors.forename: Emil F
authors.initials: EF
authors.lastname: Coccaro
mesh_terms: D000328:Adult; D000374:Aggression / Q000523:psychology*; D000987:Antisocial Personality Disorder / Q000175:diagnosis / Q000523:psychology; D001883:Borderline Personality Disorder / Q000175:diagnosis / Q000523:psychology; D007174:Disruptive, Impulse Control, and Conduct Disorders / Q000175:diagnosis / Q000523:psychology; D005260:Female; D006801:Humans; D008297:Male; D015588:Observer Variation; D016544:Patient Simulation; D010552:Personality Assessment / Q000706:statistics & numerical data*; D010554:Personality Disorders / Q000175:diagnosis* / Q000523:psychology; D011594:Psychometrics; D015203:Reproducibility of Results; D013313:Stress Disorders, Post-Traumatic / Q000175:diagnosis / Q000523:psychology