title: Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory.
abstract: OBJECTIVE
  Few research designs look at the deep structure of complex social systems. We report the design and implementation of a multimethod evaluation model to assess the impact of computerized order entry systems on both the technical and social systems within a health care organization.
  
  
  METHODS
  We designed a multimethod evaluation model informed by sociotechnical theory and an appreciation of the nature of wicked problems. We mobilized this model to assess the impact of an electronic medication management system via a three-year program of research at a major academic hospital.
  
  
  METHODS
  Model components include measurements relating to three dimensions of system impact: safety and quality, organizational culture, and work and communication patterns.
  
  
  RESULTS
  Application of the evaluation model required the development and testing of purpose-built measurement tools such as software to collect multidimensional work measurement data. The model applied established research methods including medication error audits and social network analysis. Design features of these tools and techniques are described, along with the practical challenges of their implementation. The distinctiveness of doing research within a unique paradigm of complex systems, explicating the wickedness and the dimensionality of sociotechnical theory, is articulated.
  
  
  CONCLUSIONS
  Designing an effective evaluation model requires a deep understanding of the nature and complexity of the problems that information technology interventions in health care are trying to address. Adopting a sociotechnical perspective for model generation improves our ability to develop evaluation models that are adaptive and sensitive to the characteristics of wicked problems and provides a strong theoretical basis from which to analyze and interpret findings.
authors: Health Informatics Research and Evaluation Unit, Faculty of Health Sciences, University of Sydney, Sydney, Australia. J.Westbrook@usyd.edu.au
authors: Johanna I
authors: JI
authors: Westbrook
authors: Jeffrey
authors: J
authors: Braithwaite
authors: Andrew
authors: A
authors: Georgiou
authors: Amanda
authors: A
authors: Ampt
authors: Nerida
authors: N
authors: Creswick
authors: Enrico
authors: E
authors: Coiera
authors: Rick
authors: R
authors: Iedema
mesh_terms: D000046:Academic Medical Centers / Q000458:organization & administration*; D003142:Communication; D034201:Computers, Handheld; D005069:Evaluation Studies as Topic; D058016:Health Facility Administration; D006801:Humans; D007257:Information Theory; D050316:Medical Order Entry Systems / Q000458:organization & administration; D008510:Medication Systems, Hospital / Q000458:organization & administration*; D017146:Models, Organizational; D008962:Models, Theoretical; D009517:New South Wales; D015279:Organizational Culture; D012944:Social Support; D012961:Sociology
references: Stud Health Technol Inform. 2001;84(Pt 2):1207-11
references: 11604922
references: Methods Inf Med. 1998 Sep;37(3):294-301
references: 9787631
references: Jt Comm J Qual Improv. 2000 Oct;26(10):563-75
references: 11042820
references: Proc AMIA Symp. 2000;:833-7
references: 11080001
references: Acad Emerg Med. 2000 Nov;7(11):1239-43
references: 11073472
references: BMJ. 2004 May 15;328(7449):1174
references: 15142924
references: Int J Med Inform. 2007 Jun;76 Suppl 1:S98-103
references: 16807084
references: Br J Cancer. 2003 Jul 7;89(1):15-22
references: 12838294
references: J Am Med Dir Assoc. 2006 Nov;7(9):541-4
references: 17095417
references: Can J Hosp Pharm. 1992 Aug;45(4):145-50
references: 10123057
references: JAMA. 1995 Jul 5;274(1):29-34
references: 7791255
references: J Gen Intern Med. 1995 Apr;10(4):199-205
references: 7790981
references: Yearb Med Inform. 2006;:20-5
references: 17051290
references: J Am Med Inform Assoc. 1999 Jul-Aug;6(4):313-21
references: 10428004
references: J Health Organ Manag. 2005;19(1):16-31
references: 15938600
references: Stud Health Technol Inform. 2004;107(Pt 1):701-5
references: 15360903
references: Stud Health Technol Inform. 2004;107(Pt 2):1323-7
references: 15361029
references: J Am Med Inform Assoc. 2005 Sep-Oct;12(5):505-16
references: 15905487
references: Acad Med. 1993 Jan;68(1):25-30
references: 8447888
references: Soc Sci Med. 1999 Mar;48(5):633-46
references: 10080364
references: Int J Nurs Pract. 1998 Jun;4(2):134-41
references: 9748943
references: J Appl Behav Sci. 1986;22(3):329-39
references: 10279777
references: Int J Med Inform. 2003 Mar;69(2-3):235-50
references: 12810127
references: BMJ. 2000 Mar 18;320(7237):788-91
references: 10720369
references: Stud Health Technol Inform. 2007;130:157-67
references: 17917190
references: J Am Med Inform Assoc. 2001 Jul-Aug;8(4):299-308
references: 11418536
references: Lancet. 2002 Apr 20;359(9315):1373-8
references: 11978334
references: Int J Med Inform. 1999 Aug;55(2):87-101
references: 10530825
references: Qual Saf Health Care. 2005 Oct;14(5):364-6
references: 16195571
references: Stud Health Technol Inform. 2001;84(Pt 2):1107-11
references: 11604900
references: Am J Health Syst Pharm. 2001 Jan 1;58(1):54-9
references: 11194136
references: Stud Health Technol Inform. 2007;130:217-31
references: 17917195
references: Int J Med Inform. 2005 Aug;74(7-8):629-41
references: 16043087
references: Ann Intern Med. 2006 May 16;144(10):742-52
references: 16702590
references: J Am Med Inform Assoc. 1997 Mar-Apr;4(2):94-101
references: 9067875
references: BMJ. 2000 Mar 18;320(7237):745-9
references: 10720356
references: Am J Hosp Pharm. 1980 Sep;37(9):1235-43
references: 7416180
references: Drug Saf. 1996 Nov;15(5):303-10
references: 8941492
references: JAMA. 1998 Oct 21;280(15):1311-6
references: 9794308
references: Acad Med. 1993 Jan;68(1):20-5
references: 8447887
references: Int J Med Inform. 2001 Nov;64(1):39-56
references: 11673101
references: J Health Serv Res Policy. 2007 Jan;12(1):18-24
references: 17244393
references: Health Care Manag Sci. 1998 Sep;1(1):61-74
references: 10916585
references: J Qual Clin Pract. 1999 Mar;19(1):13-7
references: 10096718
references: Med J Aust. 2002 May 6;176(9):415-8
references: 12056992
references: Pharm Pract Manag Q. 1997 Apr;17(1):17-29
references: 10166241
references: Stud Health Technol Inform. 2007;129(Pt 2):1083-7
references: 17911882