The State of the Technology
Recent advances in the minimally invasive fields of interventional cardiology (IC) and electrophysiology (EP) have leveraged advances in imaging modalities and novel technologies allowing for improved visualization and control of data. Historically, catheter manipulation and interventions have relied on the clinician’s ability to integrate 2D imaging in conjunction with pre-procedural acquired images to composite a 3D map in his or her head. The advent of imaging modalities such as CT scans and CT angiography (CTA), MRI, 3D printing, and 3D rotational angiography (3DRA), have enabled interventionalists and electrophysiologists to expand the reach of these respective fields.1 Despite each of these imaging modalities having their own limitations, the opportunity to couple them with the medical extended realities (MXR) to present the physician with true 3D visualization is intriguing.
MXR represents a type of spatial computing spanning a spectrum of technologies, including virtual reality (VR), augmented reality (AR) and mixed reality (MR). VR technology is a completely immersive experience for the user, where visual and auditory inputs are all digital and completely replace the natural physical environment. AR allows the user to view both 2D and 3D images within their physical environment. MR allows the user to visualize and interact with 2D and 3D projected images through a semitransparent display allowing them to interact and maintain awareness of their physical environment (Figure 1).2 Many of the MXR technologies use head-mounted displays to create the user environment and display these digital images. Examples of some of these head-mounted displays include Oculus Rift (VR), Google Glass (AR), Microsoft HoloLens (MR), and Magic Leap (MR).2
Many of the MXR head-mounted displays are stereoscopic displays that generate 3D digital images that many refer to as ‘holograms’.3 While these head-mounted displays are able to simulate 3D images with depth perception, they are not true holograms, as defined as images created from refracted light wave fronts.4 For the purposes of this review, we will be referring to these images as 3D digital images when discussing the MXR spectrum of technologies and use cases.
Uniquely, minimally invasive procedures represent an important early set of use cases for MXR, as the 3D data do not need to be co-registered to the patient’s anatomy to be clinically useful. In fact, MXR procedural navigation involves the display of 3D digital images in a user’s physical environment that can aid or guide tool manipulation, often catheters. There is a limited but emerging data set regarding the use of MXR displays for planning and guidance/navigation in both IC and EP procedures, as both fields are relatively young. In this review, we will discuss the current states of the field, current evidence and limitations of the technologies, and potential future considerations (Figures 2 and 3, Table 1).
Pre-procedural Planning
MXR and 3D digital imaging for pre-procedural planning in IC and EP have emerged as common use cases for these technologies, demonstrating the clinical need for improved anatomic and 3D visualization while planning for these minimally invasive procedures. The growth in this space ranges from being an educational platform in training medical students and cardiologists, to developing custom interactive 3D renderings of complex anatomy easily accessible to proceduralists prior to performing innovative procedures.5,6 Imaging remains the cornerstone modality of pre-procedural planning.
Given the unique complexities for patients with congenital heart disease, this population may derive exponential benefit from pre-procedural 3D digital image reconstruction. An example of this was published by a team in Oslo, Norway, who created a tool using a Microsoft HoloLens (Microsoft) to facilitate the interpretation and understanding of a 3D reconstruction in a case of double outlet right ventricle/transposition of the great arteries.7 This early feasibility study evaluated the experience of 36 users with the single patient’s 3D MR digital image prior to congenital heart disease surgery, assessing user diagnostic and anatomic understanding as well as usability of the 3D experience.7 Not surprisingly, the identification of anatomic landmarks and diagnostic output were high across the user group as was the rating of the 3D experience, concluding that MR-displayed 3D images for pre-surgical planning in congenital heart disease may have a high diagnostic value and contribute to improved understanding of complex anatomies.
Minimally invasive procedures within IC and EP have the unique distinction of relying more heavily on imaging for procedural guidance and navigation compared with cardiac surgery. Additionally, real time intraprocedural images remain the cornerstone for procedural guidance, though pre-procedural imaging can further refine, understand and guide intraprocedural navigation. Butera et al. described a case in which AR 3D digital imaging was used in the pre-procedural planning of an elderly patient with a sinus venosus atrial septal defect (SVASD) with partial anomalous pulmonary venous return and pulmonary hypertension.8 Pre-operative 3D images were created using raw data from CT images (Mimics Medical, Materialise), then were displayed using a Microsoft HoloLens. This 3D reconstruction and AR system allowed the clinicians to “investigate the feasibility of the procedure on the instrumented model, predicting the post-operative anatomic relationship between the superior vena cava (SVC) and the anomalous pulmonary veins.”8 Based on this pre-procedural evaluation and testing, the investigative team was able to create a custom stent for use during the procedure to exclude the SVASD while directing pulmonary venous blood flow preferentially to the left atrium and allowing for closure of the atrial septal defect (ASD). In this case, the pre-procedural imaging had a direct bearing on the intraprocedural techniques and tools used for an optimal patient outcome.
Additionally, a cohort of 28 patients with SVASDs had their anatomic substrate displayed in VR using either CT or cardiac MRI as the input imaging data set.9 The goal of the study was to evaluate patient’s candidacy for transcatheter closure. Based on the VR imaging of patient specific anatomy, users were able to insert covered stents to evaluate anatomic suitability for transcatheter closure, including potential obstruction of adjacent anatomy such as anomalous pulmonary veins. The authors suggest that 3D VR imaging provided valuable additional information that was not obvious on 2D imaging.9
The generation of these case reports demonstrates important potential use cases for MXR in medicine. However, measuring objective data to prove a patient outcome benefit for pre-procedural 3D digital imaging has yet to be studied. As described above, there are non-quantifiable, subjective measures of physician comfort/preference, patient candidacy, etc. that can begin building the evidence base for the utility of XR for these use cases. Next, we will explore use cases where MXR and 3D digital images have aided in intraprocedural navigation.
Intraprocedural Navigation Using Medical Extended Realities and 3D Digital Images
Interventional Cardiac Catheterization Laboratory
Intraprocedural guidance in the cardiac catheterization laboratory currently relies on imaging modalities, such as fluoroscopy, 3DRA, ultrasound, cineangiography, etc., to provide a roadmap for the operating physician. Advances in 3D digital imaging and accessibility may improve the efficiency and safety of commonly performed procedures soon. We will review several case examples of the use of MXR, and 3D digital imaging used intraprocedurally, and the impact these examples may have within the field of IC (Table 2).
Diagnostic Cardiac Catheterization
The emergence of 3D digital imaging has the potential to improve procedural efficiency and patient outcomes in IC procedures by creating accurate 3D anatomical models that can be used to increase spatial awareness and depth perception. The RealView Holographic Display (RealView Imaging), described in 2016 by Bruckheimer et al., is an early study evaluating the feasibility of real time, interactive, 3D digital images used during interventional catheterization procedures.10 In this single-center study, eight patients undergoing clinical cardiac catheterization procedures were enrolled. The patient cohort included: five patients undergoing ASD closure using 3D transesophageal echo (TEE), three patients using 3DRA, two patients undergoing pulmonary valve implantation, and one patient with a bidirectional Glenn shunt undergoing interstage catheterization. Patient-specific 3D digital images were created intraprocedurally during all eight cases from 3D TEE and 3DRA source imaging and projected through the Holoscope, a free-standing device that displayed the images as floating within the catheterization lab. The primary outcome measured was correct identification of all clinically relevant anatomical landmarks during each procedure compared with 3D TEE and 3DRA. Clinical operators were able to identify anatomical landmarks on both the 3DTEE/3DRA as well as the reconstructed 3D digital images. Despite a small sample size, this study represents the clinical feasibility study of using 3D digital images in the interventional cardiac catheterization laboratory.10
Percutaneous Coronary Intervention
Percutaneous coronary intervention (PCI) is a minimally invasive procedure to recanalize stenotic, partially or totally occluded coronary arteries, and is a commonly performed procedure in adult interventional cardiac catheterization laboratories.11 In cases of chronic total occlusion, it is difficult to delineate the post occluded vessel to safely advance a guidewire intraprocedurally. Opolski et al. developed a platform to project CT images using the head-mounted display Google Glass during a procedure, to assist with guidewire placement within the true lumen of the obstructed vessel. In their study of 15 patients who were prospectively enrolled and had pre-procedural coronary artery CTA reconstructed and displayed on a head-mounted display worn by the interventional cardiologist during the PCI procedure (AR-guided group).12
There was no significant difference in patient outcomes in the AR-guided group compared to the traditional angiography-based group, as defined by guidewire crossing of the occluded vessel. The AR-guided group achieved a 67% success rate, compared with 75% in the angiography guided group (p=0.583). AR-guided PCI required longer fluoroscopy time compared to the conventional angiography group (26.8 ± 12.1 versus 19.8 ± 0 minutes, p=0.032), though the contrast load was decreased in the AR-guided group (134 ± 43 versus 166 ± 52 ml; p=0.03). There was no significant difference between the groups in radiation dose between groups (AR-guided 40.6 ± 24.5 mSv versus angiography group 38.9 ± 25.8 mSv, p=0.816).12 Though representing a small sample size in a single institution, this study was able to generate quantifiable, clinically relevant measures by which to compare AR-guided navigation to traditional methods and was able to show a clinically important potential patient benefit.
Valve Interventions
Advances in catheter-based valve technologies have expanded considerably in the past decade, allowing for both valve implantation and repair through minimally invasive procedures. Minimally invasive transcatheter valve implantation is now performed routinely in the aortic position (transcatheter aortic valve implantation [TAVI]) and pulmonary position (transcatheter pulmonary valve implantation), allowing for patients to undergo valve replacement procedures without the need for cardiothoracic surgery. Transcatheter valve replacement procedures require careful positioning of the valve apparatus within the landing zone without impinging on nearby critical structures, such as coronary arteries. A report from Rymuza et al. describes the use of 3D digital imaging during a TAVI on a patient with symptomatic aortic stenosis.13 Based on pre-procedural CTA imaging, a patient-specific 3D digital image was created using both CarnaLife Holo (MedApp SA) and Microsoft HoloLens. The 3D and 4D images (includes imaging during the cardiac cycle) allowed the physicians to accurately view the anatomy of the patient’s aortic root throughout the procedure in addition to their fluoroscopic imaging.13
Another case report from Kasprzak et al. describes the use of MR and 3D TEE during balloon mitral commissurotomy, a procedure in which a balloon is dilated across a stenotic mitral valve to relieve the stenosis.14 Also using CarnaLife Holo, the users were able to view 3D TEE images as 3D digital images while wearing a HoloLens head-mounted display, allowing the user to control the image using voice and gesture commands. The image was used to “assess preprocedural anatomy, support transseptal puncture, positioning of wires, and balloon positioning for inflation.”14 The authors state that this example represents the first feasibility report of “real-time intraprocedural use of holographic mixed reality… to support percutaneous mitral valve structural intervention.”14 This application represents a true early feasibility test with more objective data required to support widespread use of MR 3D digital images to guide mitral valve procedures.
Generally, 3D digital image-based navigation during valvar interventions represents a promising area for future development and investigation as it requires careful pre-procedural and intraprocedural imaging and analysis to best understand the positioning of the valve. This use case may be of interest to both adult and pediatric interventional cardiologists given the expanding use of transcatheter valves in pulmonary, mitral, and aortic positions.
Pulmonary Artery Intervention
Catheter-based intervention on pulmonary arteries is a relatively safe and effective therapeutic option for an array of clinical diseases in both adult and pediatric medicine. The complex architecture of the pulmonary arterial tree requires advanced imaging techniques for both pre-procedural, and intraprocedural planning. The use of 3DRA during interventions on the branch pulmonary arteries has enabled physicians to have a clearer understanding of complex anatomy during a procedure, however exposed the patient to additional radiation and contrast loads.15
In 2019, Witowski et al. evaluated the use of 3D printed CT images and an AR 3D digital image platform used during pre- and intraprocedural planning for balloon pulmonary angioplasty for the treatment of chronic thromboembolic pulmonary hypertension.16 This early feasibility study included two patients who met criteria for balloon pulmonary angioplasty and stent implantation. Both patients had CT imaging of their branch pulmonary arteries submitted to 3D modeling software. For pre-procedural planning, the pulmonary arterial architecture was 3D printed for the physician to interact with. Intraprocedurally, operators wearing HoloLens head-mounted displays were able to look at and interact with the 3D rendered digital images (developed from the pre-procedural imaging dataset) throughout the duration of the procedure using voice commands and hand gestures (Figure 4).16 Despite the small sample size, there is potential for the widespread use of AR in pulmonary angioplasty and stenting given the complexities of the pulmonary arterial anatomy, particularly in patients with congenital heart disease where serial procedures are often required. As shown by the PCI early data, the use of AR navigational guidance during these lengthy procedures may lead to a reduction in patient and provider radiation exposure and provide accurate anatomical maps for the interventionalist throughout the procedure.12
Left Atrial Appendage Occlusion
In 2018, Zbronski et al. published two cases of left atrial appendage (LAA) occlusion performed using the assistance of 3D digital images and AR.17 In some patients with chronic atrial arrhythmias (such as AF) at risk for blood clot formation and stroke and a contraindication to oral anticoagulation, the LAA occlusion procedure may be recommended to reduce the risk of stroke. In these cases, both patients underwent pre-procedural TEE and cardiac CT for better anatomic definition. From the CTs, patient-specific 3D digital images were generated and displayed on the Microsoft HoloLens, allowing for intraprocedural use of the pre-procedural dataset to quantify landing zone parameters and select optimal size of the Amplatzer Amulet (Abbott) occlusion device.17 The authors concluded that having the users perform a thorough assessment of 3D anatomy and nearby structures was essential prior to performing the procedure, and that accessing the 3D reconstructions during the procedure proved to be a useful enhancement ensuring safe and accurate delivery of the device.17
Recently, an IC team in Minnesota has completed LAA occlusion with a Watchman LAA occlusion device (Boston Scientific) using a 4D digital image developed using the True3D Software package (EchoPixel), allowing for 3D digital image assistance during the procedure. The EchoPixel technology uses glasses and a stylus to allow the operator to visualize complex cardiac anatomy in 3D. The physician leading this procedure states that, “EchoPixel pre-planning True3D software helped us reduce our procedure times by more than 27% and increase optimal procedure outcome by 20%.”18
Interventional Electrophysiology
In interventional cardiac EP, there is limited experience with the use of intraprocedural 3D digital images and AR though it may be closer to clinical implementation than in IC. Broadly, EP procedures can be categorized as catheter ablation procedures versus device implantation procedures, and both have entered the space of MXR (Table 3).
Electrophysiology Study and Transcatheter Ablation
Ventricular tachycardia (VT) ablations, where myocardial scar visualization and location guide where radiofrequency therapy lesions are applied may be an important use case for MXR technologies. In 2018, Jang et al. published their animal work in which five swine underwent controlled infarction and subsequently had electroanatomic maps created to identify scar and VT substrate.19 The data were then processed and displayed in a Microsoft HoloLens which allowed for a fully endoscopic view of the scar. While this study did not study outcomes, they did ask about perceived usefulness of the 3D visualization, with both operators and mapping system specialists finding this 3D contextualization of the data useful (rates 5.8/7 and 5/5.7 respectively). The authors concluded a true 3D visualization of myocardial scar using MXR may better guide VT ablations.
In 2020, electrophysiologists at Washington University in St. Louis published the CARE Study, the first prospective study evaluating the use of the CommandEP system using SentEP software (SentiAR), which interfaces with an electroanatomic mapping system (EAMS) to provide a real-time 3D digital image of the patient’s geometry with electroanatomic data and real time catheter locations. This system is intended for intraprocedural guidance and navigation (Figure 5). In the CARE study, patients who were undergoing a clinically indicated EP study underwent a series of ‘study tasks’ during the post-ablation waiting phase to assess navigational accuracy, procedural communication efficiency and physician usability.3 Navigational accuracy was measured by calculation of the Euclidean distance between a navigational target, and the point at which the physician believed they had reached the target. These x, y, z coordinate data were obtained from the EAMS.
The authors hypothesized that providing physicians with real-time 3D digital image with hands-free control and interaction improve navigational accuracy and improve workflow/communication. Using a Microsoft HoloLens head-mounted display, physicians were able to view and control the 3D digital data, and visualize and control the electroanatomic 3D map of the patient, using a gaze-dwell mode of control (hands-free).20 The results showed that users were significantly more accurate with navigational accuracy tasks using the CommandEP MR system (accuracy for CommandEP 2.9 ± 1.9 mm versus standard of care 4.5 ± 3.7 mm; p<0.001).3 Additionally, there was a significant reduction in physician-mapping technician interactions during navigation tasks (CommandEP 0.17 versus standard of care 1.64; p<0.05) demonstrating that the ability to control and manipulate the maps reduced interactions.20 Lastly, the users found the system highly usable with 100% of respondents being comfortable using the system, 93% finding the 3D digital images easier to interpret than standard of care, and 87% discovering something new about the anatomy when using CommandEP.20 This improvement in accuracy was then simulated in a virtual heart model showing more terminations of simulated ventricular tachycardia circuits than compared to standard of care lesions (p<0.01).21 These benefits, the authors speculate, will lead to improved patient outcomes in a wide variety of ablation procedures, including ventricular tachycardia and pulmonary vein isolation/AF ablations where accuracy of lesion placement is critical to patient outcomes.21
This novel imaging system proved there is potential for the use of mixed reality imaging during EP procedures, specifically cardiac mapping, and catheter ablations. Although the study was limited by a small sample size of users, further testing with a larger physician user group and a wider variety of study tasks and chambers mapped will assess usability and clinical applications.
Device Implants
Transvenous pacemaker implantation is a relatively safe and common procedure performed on patients with advanced degrees of heart block, sinus node dysfunction, and chronotropic incompetence. Here again, patients with complex cardiac anatomy – and specifically repaired congenital heart disease – present a multitude of challenges. A 2018 case report from Poland describes the use of AR for assistance with leadless transvenous pacemaker implantation in a patient with dextrocardia, congenitally corrected transposition of the great arteries status post ventricular septal defect repair who developed complete heart block.22 3D reconstructed images of a CTA were displayed using a Google Glass head-mounted display and oriented in a similar angle as the fluoroscopic views from the C-arm in the catheterization lab. This allowed the clinical operator to identify the most ‘optimal’ orientation for visualizing the trajectory of the leadless pacemaker as it was positioned into the ventricle. Although only one patient was included in this case report, it showed that AR may be useful in the planning of leadless pacemaker implantation procedures when anatomy is ill defined, complex, or surgically altered.22 A larger, comprehensive randomized study assessing AR assisted pacemaker implantation (both lead implant and leadless pacemakers) versus fluoroscopic guidance alone would be an appropriate next step in understanding how XR technologies can improve device implantation cases.
Limitations
Despite MXR advances in imaging and control, there are limitations to widespread translation to clinical practice. In addition to the complexities innate to both the hardware and software, the human system integration between hardware, software, user, and interface must also be studied. Understanding how the complexities of medical decision making can be positively or negatively influenced by MXR, and how the technology impacts the cognitive workload of the physician should also be discerned.
Focusing on generalizable clinical endpoints for studies that can be used across use cases will also be important as the fields grow and expand. To date, the literature is saturated with examples of AR technology impacting procedures – we now know that it can be done. However, to get past the technical question of “can it be done?” to the critical clinical question of “should it be done?,” will require objective clinical measurements that can impact outcomes for patients and/or providers.
While there are many outcome measures by which any medical device technology is measured, there are three major categories that must be considered: patient-facing consequences (including outcomes and potential complications), physician (user) and user group facing consequences, and consequences to the health care system. Patient-facing consequences are the most often considered as to how a medical device may impact patient outcomes, morbidity, and mortality. In the cases outlined above, the patient is often the beneficiary of the improved understanding of the physician performing the procedure. Future directions should consider other patient facing outcome measures. Physician- (and other user-) facing consequences and outcomes must also be considered, both positive and negative. For instance, for improved 3D visual-spatial understanding, a clear benefit to these systems, should be weighed against potential limitations of the visual field or cyber sickness. Currently, this is most often measured through qualitative means but deserve consideration for quantitative metrics. Future additions to advance 3D digital images to 4D digital images represent a technical barrier that will likely be addressed in the coming years. Assessment of cognitive load on the physician user should be considered, perhaps with a modified task load index.20,23–25 Finally, the impact on the health care system must be considered. These technologies offer an important possibility to better educate patients and families, although this has not been rigorously studied. Additionally, there may be an added cost to the health care system for MXR guided procedures. Understanding the relative cost/benefit ratio will be an important aspect to widespread use of these technologies.
Future Directions
In this review, we have described a comprehensive review of available case reports and case series describing the use of MXR technologies in IC and EP procedures. Just as CT, MRI, and ultrasound have become standard imaging modalities in every day clinical cardiology, we predict this will also happen for MXR and 3D digital images. The fields of IC and EP are primed to incorporate these advanced imaging modalities with even more possibilities in the future including 4D digital images.
As these MXR technologies become more available, accessible, and efficient, we anticipate clinicians in the field of IC and EP will be able to use these 3D digital images in real-time during procedures, as we anticipate this is where there will be greatest benefit to patient. The use of these images would allow physicians to visualize accurate anatomic models to enhance a broad scope of procedures, from an ASD device closure to transcatheter heart valve implants, and catheter ablations, to device implantation in the EP laboratory. In addition to enhanced visual-spatial understanding, there is the potential for reduction in ionizing fluoroscopy (benefit to patient, provider, and care team) and other improved intraprocedural efficiencies (such as communication, personnel use, etc.).
Future research and development will likely focus on accurate real-time imaging conversion and rendering, and addition of cardiac motion into these models. This has already started emerging in the EP laboratory and there are ongoing studies evaluating real time catheter tracing in high fidelity cardiac models for IC.3,26 Another barrier will be the co-registration of these images to currently available images or to the patient’s body, a hybrid software and hardware issue that will require further investigation. There are early technologies investigating co-registered images between MXR and the patient.27,28
Accessibility of MXR technologies will be important to adoption in the catheterization lab. Understanding the economic incentives for using these technologies will be an important factor. For instance, if procedures that use MXR are performed with greater accuracy, this may limit the need for future repeat procedures creating an economic argument for using MXR. Also, budgets of cardiac catheterization laboratories, including the purchase of capital equipment, may impact the accessibility of these technologies. Input from on-site information technology and biomedical engineering teams within the hospital will also be needed for the adoption of this equipment into the lab.
Lastly, future studies will need to evaluate MXR technologies for user friendliness and overall usability, including cognitive load. While the primary goal for these technologies is improvement in patient outcomes, technologies that integrate well into physician workflows and provide early demonstrable benefit are more likely to be adopted. Additionally, careful assessment of potential risks and hazards, including cyber sickness will need to be performed to understand the risk/benefit ratio. Studies will need to show data describing how they may reduce procedural duration, improve procedural efficiency and accuracy, and are comfortable and intuitive to use. The vast majority of MXR publications have relied on user feedback surveys to assess operator comfort and ease of use. Involving large user groups, with detailed surveys will help evidence generation around usability and intuitiveness of these technologies.
Conclusion
As the fields of IC and EP continue to advance, the development of novel imaging modalities remains key to rapid acceleration and broad progress. 3D digital imaging and MXR technologies will propel both fields forward and ideally allow for safer, more efficient and accurate procedural outcomes. Where surgery was once required, patients are now undergoing minimally invasive procedures for valve replacements, stenting, vascular occlusion, device closures, ablation, and cardiac implantable electronic devices. Patients with congenital heart disease are living longer, and most will require frequent diagnostic and interventional procedures. Similarly, indication for transcatheter ablations and device implants are expanding, broadening the pool of patients eligible for these procedures. 3D digital imaging allows for high quality, accurate imaging, which is vital for a successful outcome. In addition, the ability to control the data and maintain the sterile field, may be an additive benefit only now being realized and studied.
Clinicians in the fields of IC and EP developing cutting-edge, innovative procedures should not lose sight of efforts to improve patient outcomes and safety. As demonstrated in many of the case reports and studies presented, there is potential to reduce patient exposure to radiation and contrast, limit patient exposure to anesthesia, and – maybe most importantly – improve procedural accuracy when using MXR navigation. Despite limited data, users are organically expanding use cases and environments, allowing for XR to have far-reaching impact. To move the current state of XR from potentially applicable to widely adopted, there needs to be an emphasis on quantifiable data with larger study subjects and patient-facing and physician-facing benefits.