Inter-rater reliability of the Illinois Structured Decision Support Protocol.
OBJECTIVE
The purpose of this study was to determine the level of inter-rater reliability of the Illinois Structured Decision Support Protocol by examining the level of Child Protective Services (CPS) caseworkers' agreement regarding state interventions. The Protocol was designed to guide CPS workers to consistent decisions related to the level of state intervention in child maltreatment cases.


METHODS
Forty-one workers recruited from three CPS field offices in Illinois participated in this study by assessing three cases using the Illinois Structured Decision Support Protocol. The three cases were developed based on actual cases investigated by the agency. The kappa statistic was used to calculate the level of workers' agreement.


RESULTS
The result showed a weak reliability (kappa=.29). Further analysis showed that workers identified a wide range of risk factors and differing levels of intervention for each identified risk factor even among the workers who made the same final decisions.


CONCLUSIONS
Decision support protocols are developed to provide systematic structures for workers' decision-making. However, this study suggested that workers demonstrated a low level of agreement on such a protocol. Several suggestions for developing more reliable computerized protocols include: more specified inquiries with criteria to assess the range of risk for a factor, consideration of interactions among risk factors, assessment of family strengths, and having the final decision made by a computerized program.
Kang
Hyun-Ah
HA
Department of Child Welfare, SookMyung Women's University, Chungpa-Dong 2-Ka, Yongsan-Ku, Seoul 140-742, South Korea.
Poertner
John
J
D019090:Case Management; D002648:Child; D002649:Child Abuse / Q000331:legislation & jurisprudence / Q000517:prevention & control / Q000706:statistics & numerical data; D002651:Child Advocacy / Q000331:legislation & jurisprudence*; D003657:Decision Making; D005260:Female; D006801:Humans; D007087:Illinois; D008297:Male; D015233:Models, Statistical; D015588:Observer Variation; D018570:Risk Assessment; D012944:Social Support