Select Page

Research digital skills training 2021

A collaborative extended reality tool to examine tumour evolution (Phase II)

 

Rose McColl, Sina Masoud-Ansari, Technical Specialists, Centre for eResearch

  1. Home
  2.  • 
  3. Project
  4.  • A collaborative extended reality tool to examine tumour evolution (Phase II)

Background and objectives

The cognitive challenge of conceptualising complex molecular tumour changes in 3 dimensional space and across a 4th dimension of time pushes the capability of the human brain. This project aims to develop an immersive extended reality (XR) tool to interface human problem solving ability with genomic datasets, to transform the way we think about cancer evolution and therefore cancer treatment. The XR application will be used to facilitate generation of new hypotheses of how cancer evolves, by combining detailed genomic, pathological, spatial and temporal data from a single patient with cancer. This project is centred around a patient with a primary lung neuroendocrine tumour and 88 metastases, who requested and consented to a rapid autopsy following comprehensive ethical consultation with the patient and their extended family. Importantly, we work in an ongoing partnership with the patient’s whānau who are wholly supportive of the project.

The process and collaboration

The project brings together four diverse disciplines; Architecture (spatial resolution, human interface, social interaction, visual design), Medicine (pathology, radiology, clinical context), Molecular sciences (genomics, bioinformatics and cancer biology) and eResearch (coding, data manipulation, solution architecture). This is a unique collaboration with each discipline bringing their own perspective and set of skills to the project. The project recognises the value of collaboration in science, putting interaction at the centre of finding solutions to complex problems.

The project commenced in August 2020 and will run for two years. The project team¹ has been meeting fortnightly throughout the project to date, to work together towards designing and building the XR application and presentation space. In the first quarter of Year 1, we completed a formal qualitative review of the Phase I model. Participants were grouped into multidisciplinary pairs consisting of a cancer clinician and a scientist, to undertake evaluation of the augmented reality model under multi-user conditions. Feedback from this review contributed to the redesign of the application for Phase II.

Following on from the qualitative review, Yinan Liu (Architecture and Planning arc/sec Lab), Rose McColl and Sina Ansari-Masoud (Centre for eResearch) have developed a prototype XR application for Microsoft

In addition to the digital application, a portable physical presentation stand (Figure 1) has been developed which acts as a functional work environment and demonstration space for the application. The stand is a circular structure consisting of a pop-up frame for transportability. Fabric has been custom cut to fit the frame, and printed with relevant project information. Custom shelving fits into the walls of the frame for storage of items including the HoloLens 2 headsets, chargers, and tablets. The physical objects within the space have been carefully considered, including an oval table, folding chairs, Samsung tablets linked to the application, and TV monitors. There is a close relationship between the holograms in the digital application space and the physical stand layout. For example, the patient model is positioned in the center of the physical stand as the central point of interest, and the walls of the stand act as spaces for positioning of other holographic objects.

For presentation of detailed genomic data, Dr Daniel Hurley (Medical and Health Sciences/NETwork!²) has developed an RShiny web application which provides a way to view specific genomic information for each of the patient tumour samples, and provides hyperlinks to traditional online genomics resources and databases for quick reference. The web application is intended to be viewed on a tablet device at the same time that users are also immersed in the XR application. The web application is linked with the XR application so users can, for example, select a mutation they are interested in on the tablet, and it will then highlight the samples with that mutation in XR, enabling linkage between traditional 2D data and the virtual 3D space.

Why Hololens 2?

Hololens 2 has improved technical, visual, and functional aspects than the Microsoft HoloLens 1 headsets which were used in Phase I of the project. Specific challenges that participants raised in the qualitative review of the Phase I prototype included complaints of headset pain and narrow field of view, both of which are improved with the new HoloLens 2 hardware. The HoloLens 2 headsets are significantly more comfortable to wear than the original HoloLens, they have an improved field of view, improved computational power and graphics, and much richer and naturalistic gestures and interactions. Resizing, scaling, grabbing, moving and holding the digital objects is done in a way that is very close to interactions with physical objects. The Hololens 2 achieves this natural interaction by using hand tracking. It also tracks your eye movements so you can interact with objects just by looking at them, and it has voice recognition so voice commands can also be used.

Following the success of the 2018 Phase-I project, the Health Research Council of New Zealand provided funding for Phase-II (HRC Explorer Grant). The primary goal for Phase-II is to improve the prototype XR application using the latest platform and devices, adding richer genomic information, more naturalistic user interactions, and integration with traditional 2D devices, as well as developing a physical presentation space that will allow this novel XR tool to be evaluated by members of the medical research community in a formal qualitative research exercise, and exhibited to the wider public in an educational context. While this is a research tool in the first instance, it may form a blueprint for transformative clinical tools of the future for use by health professionals, patients and the public.

Hololens 2 headsets using Unity 3D game engine and Microsoft Mixed Reality Toolkit (MRTK) (see how it works in YouTube). The application uses Photon Unity Networking (PUN) to enable real time multiplayer capabilities, and Azure Spatial Anchors for locating and aligning virtual objects in physical space, so that all users see the objects in the same position. The combination of these technologies is key to achieving the goal of creating a multi-user collaborative experience.

The application enables users to visualise a life-size model of the patient’s skeleton in three dimensions and the location of tumours in relation to the skeleton and other anatomical features (e.g. organs). Locations of sampling sites are indicated and a genomic evolutionary tree shows the development of mutations in the tumours over time. Change and spread of the tumours in the patient’s body is shown using a timescale which users can interact with. The models of the patient’s bone structure, tumours and organs were constructed from CT scans (read more about the method used here) which were annotated by consultant radiologist Dr Jane Reeve (Auckland City Hospital). Most of this work was done in Phase I of the project, but Dr Reeve has continued to provide her expertise and time in Phase II to add organ annotations and refine the positioning of the 3D models.

User experience tests

We have executed several demonstrations of the prototype in the physical presentation stand. Firstly, a demonstration to the family of the patient whom this project is centred around was performed. Our application is built using the clinical information of a single patient, and the engagement and support of the patient’s family is of utmost importance. Secondly, a structured session with collaborating colleagues from both scientific and medical backgrounds was performed to gather observations and feedback around the user experience and effectiveness of the application. The method used was Moderated User Testing with a semi-structured follow-up interview. Participants made their way through a series of tasks and guided discussion points, and observations were recorded by project team members. This demonstrated aspects of the design that were effective, such as the ability to see tumours appearing across time, with overlaid tumour phylogenetic information. In addition, other areas were highlighted where the user experience could be improved such as the placement of information and the types of gestures integrated. This was useful feedback as we look to design a cohesive application/physical space that promotes collaboration and deep investigation of cancer evolution.

Exhibitions

The XR application was exhibited at the ARS Electronica Festival Aotearoa 2021, an Art, technology and society conference from 8th-12th September 2021 at Victoria University Wellington. Due to COVID-19, the exhibition was moved to an online format, and guests could experience the exhibits in a virtual 3D gallery and through a series of online events. Further events are planned including a special showcase event planned for the first quarter of 2022, which will include international invited experts, and an international exhibition in the second half of 2022 if COVID-19 travel restrictions allow.

Reference

  1. The project team: Dr Ben Lawrence (Principal Investigator), Prof Cristin Print, Tamsin Robb, Braden Woodhouse, Dr Daniel Hurley, Faculty of Medical and Health Sciences; Dr Michael Davis, Assoc Prof Uwe Rieger, Yinan Liu, Denice Belsten, School of Architecture and Planning; Dr Jane Reeve, Auckland City Hospital; Rose McColl, Sina Masoud-Ansari, Centre for eResearch.
  2. NETwork! is a New Zealand wide alliance of cancer clinicians and scientists working together in a national framework to manage and study neuroendocrine cancers. See https://www.network.ac.nz/ neuroendocrine cancers. See https://www.network.ac.nz.

See more case study projects

Developing a genomics-specific Data Management Plan (DMP) using the  Data Stewardship Wizard

Developing a genomics-specific Data Management Plan (DMP) using the Data Stewardship Wizard

Understanding the effects of Airbnb on land use, land value and regulation

Understanding the effects of Airbnb on land use, land value and regulation

Calibrating gravitational wave signal parameters of Extreme Mass Ratio Inspirals (EMRIs)

Calibrating gravitational wave signal parameters of Extreme Mass Ratio Inspirals (EMRIs)

Automated stone artefacts classification using machine learning

Automated stone artefacts classification using machine learning

Hands-on DNA: exploring the impact of virtual reality on teaching DNA structure and function

Hands-on DNA: exploring the impact of virtual reality on teaching DNA structure and function

Re-assess urban spaces under COVID-19 impact: sensing Auckland social ‘hotspots’ with mobile location data

Re-assess urban spaces under COVID-19 impact: sensing Auckland social ‘hotspots’ with mobile location data

Aotearoa New Zealand’s changing coastline – Resilience to Nature’s Challenges (National Science Challenge)

Aotearoa New Zealand’s changing coastline – Resilience to Nature’s Challenges (National Science Challenge)

Auckland housing and land use geo-data

Auckland housing and land use geo-data

Rapid monitoring of infrastructural health using remote sensing

Rapid monitoring of infrastructural health using remote sensing

Enhancing Spontaneous Recovery after Stroke Study (ESPRESSo)

Enhancing Spontaneous Recovery after Stroke Study (ESPRESSo)

Data analytics and visualisation for improving  public health and transport planning

Data analytics and visualisation for improving public health and transport planning

Data maturity project in High Value Nutrition (Phase 2) – National Science Challenge

Data maturity project in High Value Nutrition (Phase 2) – National Science Challenge

Supporting the airborne remote sensing mission – Rongowai

Supporting the airborne remote sensing mission – Rongowai

A collaborative extended reality tool to examine tumour evolution (Phase II)

A collaborative extended reality tool to examine tumour evolution (Phase II)

Data maturity project in High Value Nutrition, National Science Challenge

Data maturity project in High Value Nutrition, National Science Challenge

Haka on the move: sport circuits and cultural performance 

Haka on the move: sport circuits and cultural performance 

Proteins under a computational microscope: designing in-silico strategies to understand and develop molecular functionalities in Life Sciences and Engineering

Proteins under a computational microscope: designing in-silico strategies to understand and develop molecular functionalities in Life Sciences and Engineering

Remote temperature monitoring to reduce the spread of COVID-19

Remote temperature monitoring to reduce the spread of COVID-19

COVID-19 exponential growth visualisation

COVID-19 exponential growth visualisation

Developing virtual capabilities for the Science Payload Operations Centre

Developing virtual capabilities for the Science Payload Operations Centre

Hosting visualisation and analytics tools for COVID-19 studies

Hosting visualisation and analytics tools for COVID-19 studies

Exploring perceptions towards climate change over time on Twitter

Exploring perceptions towards climate change over time on Twitter

Coastal image classification and nalysis based on convolutional neural betworks and pattern recognition

Coastal image classification and nalysis based on convolutional neural betworks and pattern recognition

Calcium signalling in salivary gland acinar cells

Calcium signalling in salivary gland acinar cells

Anti-corruption regulations for promoting socially responsible practices

Anti-corruption regulations for promoting socially responsible practices

Determinants of translation efficiency in the evolutionarily-divergent protist Trichomonas vaginalis

Determinants of translation efficiency in the evolutionarily-divergent protist Trichomonas vaginalis

Analysing text data by time-series feature engineering

Analysing text data by time-series feature engineering

An investigation into Leap Motion device for “gesture-as-sign”

An investigation into Leap Motion device for “gesture-as-sign”

Antibiotic resistance and the “end of modern medicine ”

Antibiotic resistance and the “end of modern medicine ”

Develop short-term eruption warning systems for Whakaari and other volcanoes

Develop short-term eruption warning systems for Whakaari and other volcanoes

Evenly spaced observation fields from irregularly sampled data in the Southern Ocean

Evenly spaced observation fields from irregularly sampled data in the Southern Ocean

Measuring impact of entrepreneurship activities on students’ mindset, capabilities and entrepreneurial intentions

Measuring impact of entrepreneurship activities on students’ mindset, capabilities and entrepreneurial intentions

Using Zebra Finch data and deep learning classification to identify individual bird calls from audio recordings

Using Zebra Finch data and deep learning classification to identify individual bird calls from audio recordings

NETwork! analysis in cancer – managing genomics research data and building a repository workflow

NETwork! analysis in cancer – managing genomics research data and building a repository workflow

The Coronary Atlas – data processing workflow optimisation

The Coronary Atlas – data processing workflow optimisation

3D visualisation of indigenous burial site in Roonka

3D visualisation of indigenous burial site in Roonka

Automated measurement of intracranial cerebrospinal fluid volume and outcome after endovascular thrombectomy for ischemic stroke

Automated measurement of intracranial cerebrospinal fluid volume and outcome after endovascular thrombectomy for ischemic stroke

A new ‘stratigraphy’: interpreting object relationships with 3D point densities

A new ‘stratigraphy’: interpreting object relationships with 3D point densities

Towards the use of deep learning techniques for storm surge prediction

Towards the use of deep learning techniques for storm surge prediction

Using simple models to explore complex dynamics: A case study of macomona liliana (wedge-shell) and nutrient variations

Using simple models to explore complex dynamics: A case study of macomona liliana (wedge-shell) and nutrient variations

Development of Machine Learning methodology for genomic research

Development of Machine Learning methodology for genomic research

An Archaeological database for threatened North Island rock art in New Zealand

An Archaeological database for threatened North Island rock art in New Zealand

Presence: distributed mixed reality learning environment

Presence: distributed mixed reality learning environment

Digital video and the early learning lab

Digital video and the early learning lab

Publishing the Bay of Island Bottlenose dolphin catalogue

Publishing the Bay of Island Bottlenose dolphin catalogue

Modelling the diurnal cycle* of winds and clouds

Modelling the diurnal cycle* of winds and clouds

Presence: distributed mixed reality learning environment

Presence: distributed mixed reality learning environment

Using research virtual machines to analyse fMRI datasets

Using research virtual machines to analyse fMRI datasets

Genomic Virtual Lab (GVL) as a bioinformatics training platform

Genomic Virtual Lab (GVL) as a bioinformatics training platform

SwiftLaTeX- Exploring web-based true WYSIWYG editing for digital publishing

SwiftLaTeX- Exploring web-based true WYSIWYG editing for digital publishing

Climate change impacts on weather-related hazards

Climate change impacts on weather-related hazards

Understanding tumour evolution through augmented reality

Understanding tumour evolution through augmented reality

Myocardial motion tracking and strain calculation using Deep Learning networks

Myocardial motion tracking and strain calculation using Deep Learning networks

OnTask pilot at the Centre for Learning and Research in Higher Education

OnTask pilot at the Centre for Learning and Research in Higher Education

Visualising the University campus in 3D

Visualising the University campus in 3D

Visualising protein interaction

Visualising protein interaction

Biological heritage National Science Challenge eDNA virtual hub

Biological heritage National Science Challenge eDNA virtual hub

Interactive AR art – Project Gordon

Interactive AR art – Project Gordon

1-D numerical models of post-glacial river evolution

1-D numerical models of post-glacial river evolution

Mathematically modelling gastrointestinal electrical activity

Mathematically modelling gastrointestinal electrical activity

3D Cryo-EM reconstructions of macromolecular complexes

3D Cryo-EM reconstructions of macromolecular complexes

Engine knock in a spark-ignition engine with hydrogen supplementation

Engine knock in a spark-ignition engine with hydrogen supplementation

The complex unsteady flow within a fluid-filled annulus and its transition to turbulence

The complex unsteady flow within a fluid-filled annulus and its transition to turbulence

Using data mining for digital ink recognition

Using data mining for digital ink recognition

The landscape costs of brushtail possum dispersal

The landscape costs of brushtail possum dispersal

Accelerating the discovery of natural products made by orphan megasynthases

Accelerating the discovery of natural products made by orphan megasynthases

Improving the short term precipitation forecasts for New Zealand

Improving the short term precipitation forecasts for New Zealand

Finding genetic variants responsible  for human disease hiding in the universe of benign variants

Finding genetic variants responsible for human disease hiding in the universe of benign variants

Revealing key processes in enzyme efficiency through high performance computing

Revealing key processes in enzyme efficiency through high performance computing

3D Electromagnetic modeling and simulation using heterogeneous computing

3D Electromagnetic modeling and simulation using heterogeneous computing

Hemodynamics in the microcirculation

Hemodynamics in the microcirculation

Putting turbulence to work

Putting turbulence to work

Why are some molecules drugs?

Why are some molecules drugs?

Bayesian additive regression trees  vs logistic regression – estimation of propensity scores

Bayesian additive regression trees vs logistic regression – estimation of propensity scores

Fully coupled thermo-hydro-mechanical modelling of permeability enhancement by the finite element method

Fully coupled thermo-hydro-mechanical modelling of permeability enhancement by the finite element method

Modelling dispersal and ecological competition in a statistical phylogeographic framework

Modelling dispersal and ecological competition in a statistical phylogeographic framework

Studying the shape and the size of the universe

Studying the shape and the size of the universe

Planet hunting

Planet hunting

Simulating quantum mechanics on high performance computing cluster

Simulating quantum mechanics on high performance computing cluster

Multiscale modelling of saliva secretion

Multiscale modelling of saliva secretion

Modelling dual reflux pressure swing adsorption (DR-PSA) units for gas separation in natural gas processing

Modelling dual reflux pressure swing adsorption (DR-PSA) units for gas separation in natural gas processing

Improving the treatment of heart disease

Improving the treatment of heart disease

Estimating migration rates in the budding yeast Saccharomyces cerevisiae

Estimating migration rates in the budding yeast Saccharomyces cerevisiae

Number theoretic algorithms in cryptography

Number theoretic algorithms in cryptography

Molecular phylogenetics uses genetic data to reconstruct the evolutionary history of individuals, populations or species

Molecular phylogenetics uses genetic data to reconstruct the evolutionary history of individuals, populations or species

Phylogeny and phylogeography of the family kyphosidae (Perciformes: teleostei)

Phylogeny and phylogeography of the family kyphosidae (Perciformes: teleostei)

Testing what cosmic inflation really predicts

Testing what cosmic inflation really predicts

Multigene environmental DNA data analysis for New Zealand genomic observatory

Multigene environmental DNA data analysis for New Zealand genomic observatory

Finding genetic variants responsible for human disease hiding in universe of benign variants

Finding genetic variants responsible for human disease hiding in universe of benign variants

BEAST, Bayesian evolutionary analysis sampling trees

BEAST, Bayesian evolutionary analysis sampling trees

The formation of surface archaeological deposits in arid Australia

The formation of surface archaeological deposits in arid Australia

Statistical modelling of carryover effects after cessation of treatments

Statistical modelling of carryover effects after cessation of treatments

High-resolution cryo-electron microscopy of protein complexes and machines

High-resolution cryo-electron microscopy of protein complexes and machines

ARCI, archaeology eResearch collaboration initiative

ARCI, archaeology eResearch collaboration initiative

Optimisation of blades on large wind turbines with individual pitch control and trailing edge flaps

Optimisation of blades on large wind turbines with individual pitch control and trailing edge flaps

Quality of care and outcomes in children with cleft lip and/or palate

Quality of care and outcomes in children with cleft lip and/or palate

Geographic and temporal information retrieval on massive document collections

Geographic and temporal information retrieval on massive document collections

Homodynamics in the microcirculation

Homodynamics in the microcirculation

Processing structure-from-motion photogrammetry on the cluster

Processing structure-from-motion photogrammetry on the cluster

Computational investigation of catalysis mechanisms for polyurethane synthesis

Computational investigation of catalysis mechanisms for polyurethane synthesis

Virtual childhood obesity prevention laboratory

Virtual childhood obesity prevention laboratory

Giving Pacific research greater reach

Giving Pacific research greater reach

Development of novel waveguides  in the terahertz (THz) region

Development of novel waveguides in the terahertz (THz) region

Modelling of costs of diets  by INFORMAS

Modelling of costs of diets by INFORMAS

Foodback

Foodback

Finite element method code for  modelling biological cells

Finite element method code for modelling biological cells

The future of memory: Neuroimaging memory and imagination with functional MRI

The future of memory: Neuroimaging memory and imagination with functional MRI

Modelling and visualisation of calcium waves in parotid acinar cells

Modelling and visualisation of calcium waves in parotid acinar cells

Mapping donor contributions in the Pacific

Mapping donor contributions in the Pacific

Visualising humpback whale migration

Visualising humpback whale migration

Visualising the 2010 and 2011  Canterbury earthquakes

Visualising the 2010 and 2011 Canterbury earthquakes

Data management planning for MOA*

Data management planning for MOA*

Research data publishing  and preservation at COMPASS

Research data publishing and preservation at COMPASS

Centre for eResearch machine learning service

Centre for eResearch machine learning service

Building a discrete global  grid gazetteer service

Building a discrete global grid gazetteer service

The new Wanhal catalogue

The new Wanhal catalogue

Passive acoustic modelling

Passive acoustic modelling

Using GPUs to expand our understanding of the Solar System

Using GPUs to expand our understanding of the Solar System

Shedding new light on dark matter

Shedding new light on dark matter

Aerodynamics modelling paves the way for improved yacht designs

Aerodynamics modelling paves the way for improved yacht designs

Modernising models to help diagnose or treat disease and injury

Modernising models to help diagnose or treat disease and injury

Wandering around the molecular landscape: embracing virtual reality as a research showcasing outreach and teaching tool

Wandering around the molecular landscape: embracing virtual reality as a research showcasing outreach and teaching tool

ALTER: Between human and nonhuman – a VR art exhibition

ALTER: Between human and nonhuman – a VR art exhibition

Disposition of Microsoft HoloLenses for a Pop-Up Reality Shop to demonstrate the progress of a research project

Disposition of Microsoft HoloLenses for a Pop-Up Reality Shop to demonstrate the progress of a research project

Improving diagnosis for schistosomiasis by using the ‘metabolic footprint’ of urine samples from an animal model of Schistosoma infection to identify possible biomarkers

Improving diagnosis for schistosomiasis by using the ‘metabolic footprint’ of urine samples from an animal model of Schistosoma infection to identify possible biomarkers

Making stroke recovery prediction tools freely available

Making stroke recovery prediction tools freely available

MFT-ICR mass spectrometry data management and analysis workflow

MFT-ICR mass spectrometry data management and analysis workflow

Taking a ‘Big Data’ approach to find new clinical-omic associations in cancer

Taking a ‘Big Data’ approach to find new clinical-omic associations in cancer

Growing Up in New Zealand

Growing Up in New Zealand

Improving arrival time predictions for vehicles in a public transport network

Improving arrival time predictions for vehicles in a public transport network

Distributed and cloud-based control at field-level for systems interacting with soft bodies

Distributed and cloud-based control at field-level for systems interacting with soft bodies

Mobile Click Fraud Attack (MCFA)

Mobile Click Fraud Attack (MCFA)

Skin-omics: exploring the volatile organic compounds on human skin

Skin-omics: exploring the volatile organic compounds on human skin

New analytics tools for workload planning for the 2018 New Zealand Census

New analytics tools for workload planning for the 2018 New Zealand Census

Visualising the New Zealand Index of Multiple Deprivation

Visualising the New Zealand Index of Multiple Deprivation