Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface

Cristina Marie Morales Mojica; Nikolaos V. Tsekos; Jose Daniel Velazco-Garcia; Haoran Zhao; Ioannis Seimenis; Ernst L. Leiss; Dipan J. Shah; Andrew Webb; Aaron T. Becker; Panagiotis Tsiamyrtzis

doi:10.1109/BIBE.2019.00186

Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface

Cristina Marie Morales Mojica, Nikolaos V. Tsekos, Jose Daniel Velazco-Garcia, Haoran Zhao, Ioannis Seimenis, Ernst L. Leiss, Dipan J. Shah, Andrew Webb, Aaron T. Becker, Panagiotis Tsiamyrtzis

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Scopus citations

Abstract

The emerging potential of augmented reality (AR) to improve 3D medical image visualization for diagnosis, by immersing the user into 3D morphology is further enhanced with the advent of wireless head-mounted displays (HMD). Such information-immersive capabilities may also enhance planning and visualization of interventional procedures. To this end, we introduce a computational platform to generate an augmented reality holographic scene that fuses pre-operative magnetic resonance imaging (MRI) sets, segmented anatomical structures, and an actuated model of an interventional robot for performing MRI-guided and robot-assisted interventions. The interface enables the operator to manipulate the presented images and rendered structures using voice and gestures, as well as to robot control. The software uses forbidden-region virtual fixtures that alerts the operator of collisions with vital structures. The platform was tested with a HoloLens HMD in silico. To address the limited computational power of the HMD, we deployed the platform on a desktop PC with two-way communication to the HMD. Operation studies demonstrated the functionality and underscored the importance of interface customization to fit a particular operator and/or procedure, as well as the need for on-site studies to assess its merit in the clinical realm.

Original language	English (US)
Title of host publication	Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1002-1005
Number of pages	4
ISBN (Electronic)	9781728146171
DOIs	https://doi.org/10.1109/BIBE.2019.00186
State	Published - Oct 1 2019
Event	19th International Conference on Bioinformatics and Bioengineering, BIBE 2019 - Athens, Greece Duration: Oct 28 2019 → Oct 30 2019

Other

Other	19th International Conference on Bioinformatics and Bioengineering, BIBE 2019
Country/Territory	Greece
City	Athens
Period	10/28/19 → 10/30/19

Keywords

Augmented reality
Image guided interventions
Robot-assistance

ASJC Scopus subject areas

Information Systems
Biomedical Engineering
Health Informatics

Access to Document

10.1109/BIBE.2019.00186

Cite this

Mojica, C. M. M., Tsekos, N. V., Velazco-Garcia, J. D., Zhao, H., Seimenis, I., Leiss, E. L., Shah, D. J., Webb, A., Becker, A. T., & Tsiamyrtzis, P. (2019). Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface. In Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019 (pp. 1002-1005). Article 8942031 Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BIBE.2019.00186

Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface. / Mojica, Cristina Marie Morales; Tsekos, Nikolaos V.; Velazco-Garcia, Jose Daniel et al.
Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1002-1005 8942031.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Mojica, CMM, Tsekos, NV, Velazco-Garcia, JD, Zhao, H, Seimenis, I, Leiss, EL, Shah, DJ, Webb, A, Becker, AT & Tsiamyrtzis, P 2019, Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface. in Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019., 8942031, Institute of Electrical and Electronics Engineers Inc., pp. 1002-1005, 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019, Athens, Greece, 10/28/19. https://doi.org/10.1109/BIBE.2019.00186

Mojica CMM, Tsekos NV, Velazco-Garcia JD, Zhao H, Seimenis I, Leiss EL et al. Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface. In Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1002-1005. 8942031 doi: 10.1109/BIBE.2019.00186

Mojica, Cristina Marie Morales ; Tsekos, Nikolaos V. ; Velazco-Garcia, Jose Daniel et al. / Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface. Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 1002-1005

@inproceedings{1ef5736e55d34b709b662c572f6621a3,

title = "Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface",

abstract = "The emerging potential of augmented reality (AR) to improve 3D medical image visualization for diagnosis, by immersing the user into 3D morphology is further enhanced with the advent of wireless head-mounted displays (HMD). Such information-immersive capabilities may also enhance planning and visualization of interventional procedures. To this end, we introduce a computational platform to generate an augmented reality holographic scene that fuses pre-operative magnetic resonance imaging (MRI) sets, segmented anatomical structures, and an actuated model of an interventional robot for performing MRI-guided and robot-assisted interventions. The interface enables the operator to manipulate the presented images and rendered structures using voice and gestures, as well as to robot control. The software uses forbidden-region virtual fixtures that alerts the operator of collisions with vital structures. The platform was tested with a HoloLens HMD in silico. To address the limited computational power of the HMD, we deployed the platform on a desktop PC with two-way communication to the HMD. Operation studies demonstrated the functionality and underscored the importance of interface customization to fit a particular operator and/or procedure, as well as the need for on-site studies to assess its merit in the clinical realm.",

keywords = "Augmented reality, Image guided interventions, Robot-assistance",

author = "Mojica, {Cristina Marie Morales} and Tsekos, {Nikolaos V.} and Velazco-Garcia, {Jose Daniel} and Haoran Zhao and Ioannis Seimenis and Leiss, {Ernst L.} and Shah, {Dipan J.} and Andrew Webb and Becker, {Aaron T.} and Panagiotis Tsiamyrtzis",

year = "2019",

month = oct,

day = "1",

doi = "10.1109/BIBE.2019.00186",

language = "English (US)",

pages = "1002--1005",

booktitle = "Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

note = "19th International Conference on Bioinformatics and Bioengineering, BIBE 2019 ; Conference date: 28-10-2019 Through 30-10-2019",

}

TY - GEN

T1 - Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface

AU - Mojica, Cristina Marie Morales

AU - Tsekos, Nikolaos V.

AU - Velazco-Garcia, Jose Daniel

AU - Zhao, Haoran

AU - Seimenis, Ioannis

AU - Leiss, Ernst L.

AU - Shah, Dipan J.

AU - Webb, Andrew

AU - Becker, Aaron T.

AU - Tsiamyrtzis, Panagiotis

PY - 2019/10/1

Y1 - 2019/10/1

N2 - The emerging potential of augmented reality (AR) to improve 3D medical image visualization for diagnosis, by immersing the user into 3D morphology is further enhanced with the advent of wireless head-mounted displays (HMD). Such information-immersive capabilities may also enhance planning and visualization of interventional procedures. To this end, we introduce a computational platform to generate an augmented reality holographic scene that fuses pre-operative magnetic resonance imaging (MRI) sets, segmented anatomical structures, and an actuated model of an interventional robot for performing MRI-guided and robot-assisted interventions. The interface enables the operator to manipulate the presented images and rendered structures using voice and gestures, as well as to robot control. The software uses forbidden-region virtual fixtures that alerts the operator of collisions with vital structures. The platform was tested with a HoloLens HMD in silico. To address the limited computational power of the HMD, we deployed the platform on a desktop PC with two-way communication to the HMD. Operation studies demonstrated the functionality and underscored the importance of interface customization to fit a particular operator and/or procedure, as well as the need for on-site studies to assess its merit in the clinical realm.

AB - The emerging potential of augmented reality (AR) to improve 3D medical image visualization for diagnosis, by immersing the user into 3D morphology is further enhanced with the advent of wireless head-mounted displays (HMD). Such information-immersive capabilities may also enhance planning and visualization of interventional procedures. To this end, we introduce a computational platform to generate an augmented reality holographic scene that fuses pre-operative magnetic resonance imaging (MRI) sets, segmented anatomical structures, and an actuated model of an interventional robot for performing MRI-guided and robot-assisted interventions. The interface enables the operator to manipulate the presented images and rendered structures using voice and gestures, as well as to robot control. The software uses forbidden-region virtual fixtures that alerts the operator of collisions with vital structures. The platform was tested with a HoloLens HMD in silico. To address the limited computational power of the HMD, we deployed the platform on a desktop PC with two-way communication to the HMD. Operation studies demonstrated the functionality and underscored the importance of interface customization to fit a particular operator and/or procedure, as well as the need for on-site studies to assess its merit in the clinical realm.

KW - Augmented reality

KW - Image guided interventions

KW - Robot-assistance

UR - http://www.scopus.com/inward/record.url?scp=85078573719&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85078573719&partnerID=8YFLogxK

U2 - 10.1109/BIBE.2019.00186

DO - 10.1109/BIBE.2019.00186

M3 - Conference contribution

SP - 1002

EP - 1005

BT - Proceedings - 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019

Y2 - 28 October 2019 through 30 October 2019

ER -

Interactive and immersive image-guided control of interventional manipulators with a prototype holographic interface

Abstract

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this