Clinical indicators for routine use in the evaluation of early psychosis intervention: development, training support and inter-rater reliability.
OBJECTIVE
Clinical practice improvement carried out in a quality assurance framework relies on routinely collected data using clinical indicators. Herein we describe the development, minimum training requirements, and inter-rater agreement of indicators that were used in an Australian multi-site evaluation of the effectiveness of early psychosis (EP) teams.


METHODS
Surveys of clinician opinion and face-to-face consensus-building meetings were used to select and conceptually define indicators. Operationalization of definitions was achieved by iterative refinement until clinicians could be quickly trained to code indicators reliably. Calculation of percentage agreement with expert consensus coding was based on ratings of paper-based clinical vignettes embedded in a 2-h clinician training package.


RESULTS
Consensually agreed upon conceptual definitions for seven clinical indicators judged most relevant to evaluating EP teams were operationalized for ease-of-training. Brief training enabled typical clinicians to code indicators with acceptable percentage agreement (60% to 86%). For indicators of suicide risk, psychosocial function, and family functioning this level of agreement was only possible with less precise 'broad range' expert consensus scores. Estimated kappa values indicated fair to good inter-rater reliability (kappa > 0.65). Inspection of contingency tables (coding category by health service) and modal scores across services suggested consistent, unbiased coding across services.


CONCLUSIONS
Clinicians are able to agree upon what information is essential to routinely evaluate clinical practice. Simple indicators of this information can be designed and coding rules can be reliably applied to written vignettes after brief training. The real world feasibility of the indicators remains to be tested in field trials.
University of Queensland, Mental Health Centre, Royal Brisbane and Women's Hospital, Herston, Australia. s.catts@.uq.edu.au
Stanley V
SV
Catts
Aaron D J
AD
Frost
Brian I
BI
O'Toole
Vaughan J
VJ
Carr
Terry
T
Lewin
Amanda L
AL
Neil
Meredith G
MG
Harris
Russell W
RW
Evans
Belinda R
BR
Crissman
Kathy
K
Eadie
D042241:Early Diagnosis; D006801:Humans; D011618:Psychotic Disorders / Q000175:diagnosis* / Q000628:therapy*; D011785:Quality Assurance, Health Care; D058996:Quality Improvement; D019984:Quality Indicators, Health Care