A modular and scalable computational framework for interactive immersion into imaging data with a holographic augmented reality interface

Jose D. Velazco-Garcia, Dipan J. Shah, Ernst L. Leiss, Nikolaos V. Tsekos

Research output: Contribution to journalArticle

Abstract

Background and objective: Modern imaging scanners produce an ever-growing body of 3D/4D multimodal data requiring image analytics and visualization of fused images, segmentations, and information. For the latter, augmented reality (AR) with head-mounted displays (HMDs) has shown potential. This work describes a framework (FI3D) for interactive immersion with data, integration of image processing and analytics, and rendering and fusion with an AR interface. Methods: The FI3D was designed and endowed with modules to communicate with peripherals, including imaging scanners and HMDs, and to provide computational power for data acquisition and processing. The core of FI3D is deployed to a dedicated computational unit that performs the computationally demanding processes in real-time, and the HMD is used as a display output peripheral and an input peripheral through gestures and voice commands. FI3D offers user-made processing and analysis dedicated modules. Users can customize and optimize these for a particular workflow while incorporating current or future libraries. Results: The FI3D framework was used to develop a workflow for processing, rendering, and visualization of CINE MRI cardiac sets. In this version, the data were loaded from a remote database, and the endocardium and epicardium of the left ventricle (LV) were segmented using a machine learning model and transmitted to a HoloLens HMD to be visualized in 4D. Performance results show that the system is capable of maintaining an image stream of one image per second with a resolution of 512 × 512. Also, it can modify visual properties of the holograms at 1 update per 16 milliseconds (62.5 Hz) while providing enough resources for the segmentation and surface reconstruction tasks without hindering the HMD. Conclusions: We provide a system design and framework to be used as a foundation for medical applications that benefit from AR visualization, removing several technical challenges from the developmental pipeline.

Original languageEnglish (US)
Article number105779
JournalComputer Methods and Programs in Biomedicine
Volume198
DOIs
StatePublished - Jan 2021

Keywords

  • Augmented reality
  • Computational framework
  • Human-cyber Interface
  • Interactive Immersion
  • Medical imaging
  • Rendering and visualization

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Health Informatics

Fingerprint Dive into the research topics of 'A modular and scalable computational framework for interactive immersion into imaging data with a holographic augmented reality interface'. Together they form a unique fingerprint.

Cite this