title: On mixed reality environments for minimally invasive therapy guidance: systems architecture, successes and challenges in their implementation from laboratory to clinic.
abstract: Mixed reality environments for medical applications have been explored and developed over the past three decades in an effort to enhance the clinician's view of anatomy and facilitate the performance of minimally invasive procedures. These environments must faithfully represent the real surgical field and require seamless integration of pre- and intra-operative imaging, surgical instrument tracking, and display technology into a common framework centered around and registered to the patient. However, in spite of their reported benefits, few mixed reality environments have been successfully translated into clinical use. Several challenges that contribute to the difficulty in integrating such environments into clinical practice are presented here and discussed in terms of both technical and clinical limitations. This article should raise awareness among both developers and end-users toward facilitating a greater application of such environments in the surgical practice of the future.
authors: Biomedical Imaging Resource, Mayo Clinic, Rochester, MN, USA.
authors: Cristian A
authors: CA
authors: Linte
authors: Katherine P
authors: KP
authors: Davenport
authors: Kevin
authors: K
authors: Cleary
authors: Craig
authors: C
authors: Peters
authors: Kirby G
authors: KG
authors: Vosburgh
authors: Nassir
authors: N
authors: Navab
authors: Philip Eddie
authors: PE
authors: Edwards
authors: Pierre
authors: P
authors: Jannin
authors: Terry M
authors: TM
authors: Peters
authors: David R
authors: DR
authors: Holmes
authors: Richard A
authors: RA
authors: Robb
mesh_terms: D003196:Computer Graphics; D004777:Environment; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D019060:Minimally Invasive Surgical Procedures / Q000379:methods*; D012984:Software; D025321:Surgery, Computer-Assisted / Q000379:methods*; D057170:Translational Research, Biomedical / Q000379:methods*; D014584:User-Computer Interface
references: Comput Med Imaging Graph. 2009 Sep;33(6):461-76
references: 19467840
references: Comput Methods Biomech Biomed Engin. 2000;4(1):27-57
references: 11264860
references: Comput Methods Programs Biomed. 2009 Aug;95(2):95-104
references: 19285747
references: Acad Radiol. 2007 Aug;14(8):901-9
references: 17659235
references: IEEE Trans Med Imaging. 1998 Oct;17(5):694-702
references: 9874293
references: Stud Health Technol Inform. 2006;119:455-60
references: 16404098
references: Stud Health Technol Inform. 2001;81:534-40
references: 11317804
references: IEEE Trans Med Imaging. 2005 Nov;24(11):1428-40
references: 16279080
references: Med Phys. 1998 Nov;25(11):2242-8
references: 9829253
references: Neurosurg Rev. 2010 Jan;33(1):83-90
references: 19823884
references: Arch Surg. 2004 Feb;139(2):170-4
references: 14769575
references: Med Phys. 2006 Sep;33(9):3348-58
references: 17022231
references: Sci Am. 1999 Jun;280(6):62-9
references: 10349732
references: Comput Methods Programs Biomed. 2009 Jun;94(3):250-66
references: 19249113
references: Med Biol Eng Comput. 2007 Oct;45(10):957-67
references: 17653784
references: Annu Rev Biomed Eng. 1999;1:377-99
references: 11701494
references: Comput Aided Surg. 1997;2(5):292-7
references: 9484590
references: J Craniomaxillofac Surg. 1999 Apr;27(2):77-81
references: 10342142
references: Magn Reson Med. 2006 Nov;56(5):958-64
references: 17036300
references: IEEE Trans Med Imaging. 2008 Feb;27(2):255-70
references: 18334447
references: Stud Health Technol Inform. 2004;98:397-403
references: 15544314
references: Neuroimage. 2007 Sep 1;37(3):783-91
references: 17613249
references: Med Image Comput Comput Assist Interv. 2005;8(Pt 2):65-72
references: 16685944
references: Phys Med Biol. 2006 Jul 21;51(14):R505-40
references: 16825730
references: IEEE Trans Med Imaging. 2010 Feb;29(2):260-72
references: 20129843
references: Med Phys. 2004 Jun;31(6):1348-56
references: 15259638
references: Med Image Anal. 1998 Jun;2(2):121-32
references: 10646758
references: IEEE Trans Med Imaging. 2002 Aug;21(8):991-7
references: 12472271
references: J Ultrasound Med. 2001 Mar;20(3):235-40
references: 11270527
references: IEEE Trans Med Imaging. 2008 Sep;27(9):1189-201
references: 18753041
references: Phys Med Biol. 2010 Jan 7;55(1):207-19
references: 20009197
references: J Laparoendosc Adv Surg Tech A. 2008 Oct;18(5):697-706
references: 18803512
references: Comput Aided Surg. 2007 Nov;12(6):325-34
references: 18066948
references: Comput Med Imaging Graph. 1989 Nov-Dec;13(6):433-54
references: 2688869
references: Med Image Comput Comput Assist Interv. 2006;9(Pt 1):604-11
references: 17354940
references: IEEE Trans Med Imaging. 2005 Nov;24(11):1500-11
references: 16279086
references: J Vasc Interv Radiol. 2005 Apr;16(4):493-505
references: 15802449
references: Med Image Comput Comput Assist Interv. 2005;8(Pt 2):402-10
references: 16685985
references: Comput Aided Surg. 2001;6(1):1-13
references: 11335954
references: Clin Anat. 2006 Apr;19(3):240-53
references: 16565946
references: Med Phys. 2000 Apr;27(4):787-800
references: 10798702
references: Med Image Anal. 1998 Jun;2(2):133-42
references: 10646759
references: Comput Aided Surg. 2000;5(2):98-107
references: 10862132
references: Med Image Anal. 2004 Mar;8(1):81-91
references: 14644148
references: AJR Am J Roentgenol. 1992 Jun;158(6):1387-9
references: 1590163
references: J Thorac Cardiovasc Surg. 2004 Oct;128(4):579-87
references: 15457159
references: Med Image Anal. 2000 Mar;4(1):67-72
references: 10972322
references: J Am Med Inform Assoc. 2009 Jan-Feb;16(1):72-80
references: 18952942
references: Heart. 2007 Nov;93(11):1492
references: 17934013
references: J Image Guid Surg. 1995;1(3):172-8
references: 9079443
references: Cardiovasc Intervent Radiol. 2010 Dec;33(6):1230-4
references: 20502897
references: IEEE Trans Med Imaging. 2002 Dec;21(12):1445-9
references: 12588028
references: Clin Biomech (Bristol, Avon). 1995 Sep;10(6):293-303
references: 11415570
references: IEEE Trans Med Imaging. 1998 Oct;17(5):737-42
references: 9874297
references: Med Image Comput Comput Assist Interv. 2007;10(Pt 1):302-10
references: 18051072
references: IEEE Trans Biomed Eng. 2013 Apr;60(4):1034-40
references: 23047858
references: Med Image Comput Comput Assist Interv. 2008;11(Pt 2):425-33
references: 18982633
references: Inf Process Med Imaging. 2009;21:589-601
references: 19694296
references: Med Image Comput Comput Assist Interv. 2007;10(Pt 1):60-7
references: 18051044
references: Stud Health Technol Inform. 1997;39:337-48
references: 10168929
references: Comput Aided Surg. 2008 Mar;13(2):82-94
references: 18317957
references: Med Image Comput Comput Assist Interv. 2008;11(Pt 2):880-7
references: 18982688
references: Neurosurgery. 2007 Apr;60(4 Suppl 2):366-71; discussion 371-2
references: 17415176
references: Ultrasonics. 1998 Feb;36(1-5):629-33
references: 9651592
references: J Urol. 2008 Oct;180(4):1455-9
references: 18710760
references: Brachytherapy. 2004;3(4):222-30
references: 15607154
references: IEEE Trans Med Imaging. 1998 Oct;17(5):779-85
references: 9874302
references: Int J Radiat Oncol Biol Phys. 2002 Aug 1;53(5):1337-49
references: 12128137
references: Med Image Comput Comput Assist Interv. 2008;11(Pt 2):61-9
references: 18982590
references: Neurosurgery. 1998 Oct;43(4):739-47; discussion 747-8
references: 9766299
references: Med Image Anal. 2005 Apr;9(2):163-75
references: 15721231
references: Neurosurg Clin N Am. 1996 Apr;7(2):323-31
references: 8726445
references: Comput Med Imaging Graph. 2000 May-Jun;24(3):133-51
references: 10838008
references: Eur Urol. 2009 Aug;56(2):332-8
references: 19477580
references: Med Phys. 2005 Jul;32(7Part1):2371-2379
references: 28493577
references: Conf Proc IEEE Eng Med Biol Soc. 2006;2006:3086-9
references: 17947007
references: Acta Neurochir (Wien). 1990;105(1-2):5-13
references: 2239380
references: Ultrasound Med Biol. 1998 Jul;24(6):841-54
references: 9740386
references: J Magn Reson Imaging. 2001 Jan;13(1):78-86
references: 11169807
references: IEEE Comput Graph Appl. 2006 Nov-Dec;26(6):36-47
references: 17120912
references: Stud Health Technol Inform. 1997;39:131-40
references: 10168910
references: Radiology. 2002 Oct;225(1):277-82
references: 12355016
references: Med Image Comput Comput Assist Interv. 2006;9(Pt 1):420-8
references: 17354918
references: Radiology. 2006 Jul;240(1):230-5
references: 16720866
references: Phys Med Biol. 2008 Oct 7;53(19):5295-312
references: 18757999
references: Med Image Comput Comput Assist Interv. 2009;12(Pt 1):516-23
references: 20426027
references: IEEE Trans Med Imaging. 2009 Jun;28(6):951-67
references: 19211352
references: Int J Cardiovasc Imaging. 2008 Feb;24(2):201-10
references: 17564812
references: Surg Endosc. 2009 Aug;23(8):1693-700
references: 18813987
references: Stud Health Technol Inform. 2007;125:485-90
references: 17377333
references: IEEE Trans Med Imaging. 2009 Aug;28(8):1179-89
references: 19131293
references: J Neurosurg. 1986 Oct;65(4):545-9
references: 3531430
references: J Image Guid Surg. 1995;1(1):30-4
references: 9079424
references: Neurosurgery. 1982 Mar;10(3):324-31
references: 7041004
references: Annu Rev Biomed Eng. 2010 Aug 15;12:119-42
references: 20415592
references: Med Image Anal. 2010 Feb;14(1):39-49
references: 19897403
references: IEEE Trans Biomed Eng. 1995 Sep;42(9):867-78
references: 7558061
references: IEEE Trans Biomed Eng. 1989 Jun;36(6):608-17
references: 2659493
references: Med Phys. 2000 Jan;27(1):30-8
references: 10659735
references: Med Image Anal. 2007 Oct;11(5):458-64
references: 17681483
references: Med Image Anal. 2004 Sep;8(3):387-401
references: 15450231
references: Neurosurgery. 1997 Oct;41(4):831-42; discussion 842-5
references: 9316044
references: Comput Aided Surg. 2001;6(6):370-83
references: 11954068
references: Proc SPIE Int Soc Opt Eng. 2012 Feb 4;8316:
references: 26401065
references: Neuroimage. 2003 Nov;20(3):1425-37
references: 14642457
references: Conf Proc IEEE Eng Med Biol Soc. 2005;2005:7242-5
references: 17281951
references: IEEE Trans Med Imaging. 2000 Nov;19(11):1082-93
references: 11204846
references: Radiol Med. 2005 Jan-Feb;109(1-2):98-107
references: 15729190
references: Phys Med Biol. 2003 Jul 21;48(14):2241-51
references: 12894982
references: Radiology. 1995 Jun;195(3):805-14
references: 7754014
references: J Comput Assist Tomogr. 1989 Jan-Feb;13(1):20-6
references: 2492038
references: IEEE Trans Med Imaging. 2002 Dec;21(12):1524-35
references: 12588036
references: IEEE Trans Med Imaging. 1996;15(2):129-40
references: 18215896
references: Br J Radiol. 1973 Dec;46(552):1016-22
references: 4757352
references: Med Phys. 2005 Jan;32(1):241-54
references: 15719975
references: Comput Med Imaging Graph. 2010 Jan;34(1):61-8
references: 19773147
references: Med Phys. 2010 Feb;37(2):802-13
references: 20229890
references: IEEE Trans Med Imaging. 2006 Nov;25(11):1492-501
references: 17117778
references: AMIA Annu Symp Proc. 2003;:888
references: 14728393
references: Int J Cardiovasc Imaging. 2010 Aug;26(6):711-20
references: 20339922
references: Med Image Comput Comput Assist Interv. 2009;12(Pt 1):361-8
references: 20426008
references: Laryngoscope. 1998 Jul;108(7):949-61
references: 9665238
references: IEEE Trans Med Imaging. 2002 Sep;21(9):1151-66
references: 12564883