Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory.
OBJECTIVE
Few research designs look at the deep structure of complex social systems. We report the design and implementation of a multimethod evaluation model to assess the impact of computerized order entry systems on both the technical and social systems within a health care organization.


METHODS
We designed a multimethod evaluation model informed by sociotechnical theory and an appreciation of the nature of wicked problems. We mobilized this model to assess the impact of an electronic medication management system via a three-year program of research at a major academic hospital.


METHODS
Model components include measurements relating to three dimensions of system impact: safety and quality, organizational culture, and work and communication patterns.


RESULTS
Application of the evaluation model required the development and testing of purpose-built measurement tools such as software to collect multidimensional work measurement data. The model applied established research methods including medication error audits and social network analysis. Design features of these tools and techniques are described, along with the practical challenges of their implementation. The distinctiveness of doing research within a unique paradigm of complex systems, explicating the wickedness and the dimensionality of sociotechnical theory, is articulated.


CONCLUSIONS
Designing an effective evaluation model requires a deep understanding of the nature and complexity of the problems that information technology interventions in health care are trying to address. Adopting a sociotechnical perspective for model generation improves our ability to develop evaluation models that are adaptive and sensitive to the characteristics of wicked problems and provides a strong theoretical basis from which to analyze and interpret findings.
Westbrook
Johanna I
JI
Health Informatics Research and Evaluation Unit, Faculty of Health Sciences, University of Sydney, Sydney, Australia. J.Westbrook@usyd.edu.au
Braithwaite
Jeffrey
J
Georgiou
Andrew
A
Ampt
Amanda
A
Creswick
Nerida
N
Coiera
Enrico
E
Iedema
Rick
R
D000046:Academic Medical Centers / Q000458:organization & administration*; D003142:Communication; D034201:Computers, Handheld; D005069:Evaluation Studies as Topic; D058016:Health Facility Administration; D006801:Humans; D007257:Information Theory; D050316:Medical Order Entry Systems / Q000458:organization & administration; D008510:Medication Systems, Hospital / Q000458:organization & administration*; D017146:Models, Organizational; D008962:Models, Theoretical; D009517:New South Wales; D015279:Organizational Culture; D012944:Social Support; D012961:Sociology
Stud Health Technol Inform. 2001;84(Pt 2):1207-11
11604922
Methods Inf Med. 1998 Sep;37(3):294-301
9787631
Jt Comm J Qual Improv. 2000 Oct;26(10):563-75
11042820
Proc AMIA Symp. 2000;:833-7
11080001
Acad Emerg Med. 2000 Nov;7(11):1239-43
11073472
BMJ. 2004 May 15;328(7449):1174
15142924
Int J Med Inform. 2007 Jun;76 Suppl 1:S98-103
16807084
Br J Cancer. 2003 Jul 7;89(1):15-22
12838294
J Am Med Dir Assoc. 2006 Nov;7(9):541-4
17095417
Can J Hosp Pharm. 1992 Aug;45(4):145-50
10123057
JAMA. 1995 Jul 5;274(1):29-34
7791255
J Gen Intern Med. 1995 Apr;10(4):199-205
7790981
Yearb Med Inform. 2006;:20-5
17051290
J Am Med Inform Assoc. 1999 Jul-Aug;6(4):313-21
10428004
J Health Organ Manag. 2005;19(1):16-31
15938600
Stud Health Technol Inform. 2004;107(Pt 1):701-5
15360903
Stud Health Technol Inform. 2004;107(Pt 2):1323-7
15361029
J Am Med Inform Assoc. 2005 Sep-Oct;12(5):505-16
15905487
Acad Med. 1993 Jan;68(1):25-30
8447888
Soc Sci Med. 1999 Mar;48(5):633-46
10080364
Int J Nurs Pract. 1998 Jun;4(2):134-41
9748943
J Appl Behav Sci. 1986;22(3):329-39
10279777
Int J Med Inform. 2003 Mar;69(2-3):235-50
12810127
BMJ. 2000 Mar 18;320(7237):788-91
10720369
Stud Health Technol Inform. 2007;130:157-67
17917190
J Am Med Inform Assoc. 2001 Jul-Aug;8(4):299-308
11418536
Lancet. 2002 Apr 20;359(9315):1373-8
11978334
Int J Med Inform. 1999 Aug;55(2):87-101
10530825
Qual Saf Health Care. 2005 Oct;14(5):364-6
16195571
Stud Health Technol Inform. 2001;84(Pt 2):1107-11
11604900
Am J Health Syst Pharm. 2001 Jan 1;58(1):54-9
11194136
Stud Health Technol Inform. 2007;130:217-31
17917195
Int J Med Inform. 2005 Aug;74(7-8):629-41
16043087
Ann Intern Med. 2006 May 16;144(10):742-52
16702590
J Am Med Inform Assoc. 1997 Mar-Apr;4(2):94-101
9067875
BMJ. 2000 Mar 18;320(7237):745-9
10720356
Am J Hosp Pharm. 1980 Sep;37(9):1235-43
7416180
Drug Saf. 1996 Nov;15(5):303-10
8941492
JAMA. 1998 Oct 21;280(15):1311-6
9794308
Acad Med. 1993 Jan;68(1):20-5
8447887
Int J Med Inform. 2001 Nov;64(1):39-56
11673101
J Health Serv Res Policy. 2007 Jan;12(1):18-24
17244393
Health Care Manag Sci. 1998 Sep;1(1):61-74
10916585
J Qual Clin Pract. 1999 Mar;19(1):13-7
10096718
Med J Aust. 2002 May 6;176(9):415-8
12056992
Pharm Pract Manag Q. 1997 Apr;17(1):17-29
10166241
Stud Health Technol Inform. 2007;129(Pt 2):1083-7
17911882