Select Page

Research digital skills training 2021

Understanding tumour evolution through augmented reality

Ben Lawrence, Tamsin Robb, Kate Parker, Braden Woodhouse, Faculty of Medical and Health Sciences, Mike Davis, Uwe Rieger, Jack Guo, School of Architecture & Planning, Jane Reeve, Auckland District Health Board

Home • Projects • Understanding tumour evolution through augmented reality
The image is credited to Jack Guo, Architecture & Planning

Introduction

The project presented a rare opportunity to study tumour evolution and heterogeneity in a single patient. The patient had a primary lung neuroendocrine tumour and 88 metastases, and requested and consented to rapid autopsy following a comprehensive ethical consultation with the patient and their extended family. We embarked on a multi-layered genomic investigation, utilising clinical notes and imaging in order to build a personalised evolutionary model of the disease progression. Initially 30 tumour samples, representing a broad spatial distribution of the patient’s total tumour burden, were analysed via DNA whole exome sequencing (WES), transcriptome mRNA sequencing and RNA microarray analysis, providing a bank of genomic information to guide evolutionary investigations. We have since built statistical phylogenetic models from differences observed across the genomic data to produce a broad evolutionary picture encompassing single nucleotide variants (SNVs), indels, copy number variants (CNVs), and RNA expression.

Augmented Reality (AR)

Understanding the complexities of genomic data coming from a multitude of spatially distinct sites, that further change across time, is a complex task. A novel approach was required to link spatial orientation to mutational patterns. A three-dimensional approach was preferred because of the regional complexities of sampling multiple sites within a single tumour, as well as from multiple tumour sites in a single patient. By using AR technology, the team has developed a 3D model of the patient’s tumours, the sampling sites, and representations of corresponding genomic data, generating a tool to better understand tumour evolution in a single patient.

Project collaboration

The project is a collaboration between the Centre for eResearch (CeR), School of Architecture & Planning (Open Media Lab) and the NETwork! project team from the Faculty of Medical and Health Sciences (FMHS). The team comprises of multidisciplinary expertise in different areas including the fields of clinical cancer care, cell biology, genomics (DNA / RNA data), histology (microscope slides), radiology (scans) and architecture and visualisation specialists.

The benefits of the model include pushing for conceptual change. This pilot will allow us to visualise tumours on a scale previously unachievable. It will enhance the understanding of tumour progression for a wider audience including patients and their families, as well as scientists, the research community and clinicians, and many more. We hope to break open the way scientists, clinicians and patients think about cancer, as well as enabling new conceptual connections.

Annotating sampling sites: Once the tumour meshes were constructed, Bianca worked with Tamsin Robb to translate information from the tumour collection process into accurate 3D sampling sites for visualisation in the HoloLens. At this stage, only genomic data collected from 30 sampling sites (of over 300 sites sampled) were included. We navigated to the point(s) in the tumours where incisions were made in order to best approximate sampling sites. Sampling sites were named and saved alongside the relative tumour mesh to be integrated later.
Assembling a skeletal framework: Fittingly, the patient’s own skeleton, produced from the most recent CT scans, was used as a frame of reference to place tumours and sampling sites in 3D space. Semi-automatic segmentation techniques were used to separate the bony components from soft organ tissue, completed by Bianca Haux. Manual editing was required to remove segmentation artefacts such as superimposed structures caused by the injection of contrast.

Proof of concept (6 months from March to October 2018)

Over this period, the team met weekly or fortnightly to work towards presenting these tumours in AR. The final product features a representation of the patient’s skeleton as the framework for all tumours (extracted from CT data). The model includes all tumours overlaid for multiple time-points following diagnosis, sampling sites for which we have genomic data indicated in space, and genomic groupings indicated with dynamic colouring options.

The process

In-HoloLens CT-annotation system: Nick Young from the CeR produced pseudo coloured 3D volumetric renders of CT scans that can be opened in the HoloLens, and a system for adding sphere annotations to the scans to allow an experienced radiologist or clinician to interpret CT scans in 3D and indicate tumour locations with 3D markers. We then proceeded to construct tumour meshes manually as following:

Constructing tumour meshes: In collaboration with radiologist Dr Jane Reeve, the team identified tumours on CT scans, and outlines were traced manually on the computer. These traces were used by Bianca Haux from the CeR to construct 3D models of each identified tumour from the scans. The term tumour “mesh” is used to describe a 3D model of the external shape of the tumour as taken from the clinical data.

In order to ensure users interacting with the model were left with the impression of a complete augmented reality patient rather than part-patient, Jack Guo attached default female skeletal limbs to the patient’s skeleton.

CT imaging: Skeletal, tumour and sampling site elements were combined in Unity, along with menus and controls needed to navigate the space, in order to produce an interactive 3D representation of the patient. Together with colour coding based on genomic data this allows users to study the possible progression of the patient’s disease over time and evolutionary space.

Developing “multiplayer”: One of the key reasons why AR technology was chosen for this model was the possibility to interact with the model in 3D space and, most importantly, with other users. Multiplayer was developed to enable this collaborative approach and encourage conversation and exchange of knowledge while looking at the 3 dimensional data. In a group setting this allows for users to point out, show and connect spatially relevant information with a tested group size of up to 6 people. The ability for users with different backgrounds and expertise to discuss and interact with the model in the same space and time allows for truly collaborative and novel thinking.

Other areas of concerns:

  • Ethics: This work was conducted in close consultation with the patient’s family. We need to be vigilant of the high risk of patient identification. The team will continue to take all reasonable precautions to protect the patient’s privacy. The patient’s family are regularly updated on progress and have had a chance to see an early version of the AR model.
  • System design: The team will look into a design that can support more than one brand of AR headset, so that it will be able to move with new technology when they become more widely available.
  • Other ongoing questions include consideration of data flow, the usability test, the generation of IP, and building a one-off model or constructing a platform.

 

Next steps

After completion of Proof of Concept phase (over 6 months), we will then make future decisions based on outcomes of this pilot. We expect a considerable amount of circular thinking and regrouping. We are hoping to secure additional funding for the expansion of the project into further stages and determine a specific time point to make this decision.

See more case study projects

Data maturity project in High Value Nutrition, National Science Challenge

Data maturity project in High Value Nutrition, National Science Challenge

Haka on the move: sport circuits and cultural performance 

Haka on the move: sport circuits and cultural performance 

Proteins under a computational microscope: designing in-silico strategies to understand and develop molecular functionalities in Life Sciences and Engineering

Proteins under a computational microscope: designing in-silico strategies to understand and develop molecular functionalities in Life Sciences and Engineering

Remote temperature monitoring to reduce the spread of COVID-19

Remote temperature monitoring to reduce the spread of COVID-19

COVID-19 exponential growth visualisation

COVID-19 exponential growth visualisation

Developing virtual capabilities for the Science Payload Operations Centre

Developing virtual capabilities for the Science Payload Operations Centre

Hosting visualisation and analytics tools for COVID-19 studies

Hosting visualisation and analytics tools for COVID-19 studies

Exploring perceptions towards climate change over time on Twitter

Exploring perceptions towards climate change over time on Twitter

Coastal image classification and nalysis based on convolutional neural betworks and pattern recognition

Coastal image classification and nalysis based on convolutional neural betworks and pattern recognition

Calcium signalling in salivary gland acinar cells

Calcium signalling in salivary gland acinar cells

Anti-corruption regulations for promoting socially responsible practices

Anti-corruption regulations for promoting socially responsible practices

Determinants of translation efficiency in the evolutionarily-divergent protist Trichomonas vaginalis

Determinants of translation efficiency in the evolutionarily-divergent protist Trichomonas vaginalis

Analysing text data by time-series feature engineering

Analysing text data by time-series feature engineering

An investigation into Leap Motion device for “gesture-as-sign”

An investigation into Leap Motion device for “gesture-as-sign”

Antibiotic resistance and the “end of modern medicine ”

Antibiotic resistance and the “end of modern medicine ”

Develop short-term eruption warning systems for Whakaari and other volcanoes

Develop short-term eruption warning systems for Whakaari and other volcanoes

Evenly spaced observation fields from irregularly sampled data in the Southern Ocean

Evenly spaced observation fields from irregularly sampled data in the Southern Ocean

Measuring impact of entrepreneurship activities on students’ mindset, capabilities and entrepreneurial intentions

Measuring impact of entrepreneurship activities on students’ mindset, capabilities and entrepreneurial intentions

Using Zebra Finch data and deep learning classification to identify individual bird calls from audio recordings

Using Zebra Finch data and deep learning classification to identify individual bird calls from audio recordings

NETwork! analysis in cancer – managing genomics research data and building a repository workflow

NETwork! analysis in cancer – managing genomics research data and building a repository workflow

The Coronary Atlas – data processing workflow optimisation

The Coronary Atlas – data processing workflow optimisation

3D visualisation of indigenous burial site in Roonka

3D visualisation of indigenous burial site in Roonka

Automated measurement of intracranial cerebrospinal fluid volume and outcome after endovascular thrombectomy for ischemic stroke

Automated measurement of intracranial cerebrospinal fluid volume and outcome after endovascular thrombectomy for ischemic stroke

A new ‘stratigraphy’: interpreting object relationships with 3D point densities

A new ‘stratigraphy’: interpreting object relationships with 3D point densities

Towards the use of deep learning techniques for storm surge prediction

Towards the use of deep learning techniques for storm surge prediction

Using simple models to explore complex dynamics: A case study of macomona liliana (wedge-shell) and nutrient variations

Using simple models to explore complex dynamics: A case study of macomona liliana (wedge-shell) and nutrient variations

Development of Machine Learning methodology for genomic research

Development of Machine Learning methodology for genomic research

An Archaeological database for threatened North Island rock art in New Zealand

An Archaeological database for threatened North Island rock art in New Zealand

Presence: distributed mixed reality learning environment

Presence: distributed mixed reality learning environment

Digital video and the early learning lab

Digital video and the early learning lab

Publishing the Bay of Island Bottlenose dolphin catalogue

Publishing the Bay of Island Bottlenose dolphin catalogue

Modelling the diurnal cycle* of winds and clouds

Modelling the diurnal cycle* of winds and clouds

Presence: distributed mixed reality learning environment

Presence: distributed mixed reality learning environment

Using research virtual machines to analyse fMRI datasets

Using research virtual machines to analyse fMRI datasets

Genomic Virtual Lab (GVL) as a bioinformatics training platform

Genomic Virtual Lab (GVL) as a bioinformatics training platform

SwiftLaTeX- Exploring web-based true WYSIWYG editing for digital publishing

SwiftLaTeX- Exploring web-based true WYSIWYG editing for digital publishing

Climate change impacts on weather-related hazards

Climate change impacts on weather-related hazards

Understanding tumour evolution through augmented reality

Understanding tumour evolution through augmented reality

Myocardial motion tracking and strain calculation using Deep Learning networks

Myocardial motion tracking and strain calculation using Deep Learning networks

OnTask pilot at the Centre for Learning and Research in Higher Education

OnTask pilot at the Centre for Learning and Research in Higher Education

Visualising the University campus in 3D

Visualising the University campus in 3D

Visualising protein interaction

Visualising protein interaction

Biological heritage National Science Challenge eDNA virtual hub

Biological heritage National Science Challenge eDNA virtual hub

Interactive AR art – Project Gordon

Interactive AR art – Project Gordon

1-D numerical models of post-glacial river evolution

1-D numerical models of post-glacial river evolution

Mathematically modelling gastrointestinal electrical activity

Mathematically modelling gastrointestinal electrical activity

3D Cryo-EM reconstructions of macromolecular complexes

3D Cryo-EM reconstructions of macromolecular complexes

Engine knock in a spark-ignition engine with hydrogen supplementation

Engine knock in a spark-ignition engine with hydrogen supplementation

The complex unsteady flow within a fluid-filled annulus and its transition to turbulence

The complex unsteady flow within a fluid-filled annulus and its transition to turbulence

Using data mining for digital ink recognition

Using data mining for digital ink recognition

The landscape costs of brushtail possum dispersal

The landscape costs of brushtail possum dispersal

Accelerating the discovery of natural products made by orphan megasynthases

Accelerating the discovery of natural products made by orphan megasynthases

Improving the short term precipitation forecasts for New Zealand

Improving the short term precipitation forecasts for New Zealand

Finding genetic variants responsible  for human disease hiding in the universe of benign variants

Finding genetic variants responsible for human disease hiding in the universe of benign variants

Revealing key processes in enzyme efficiency through high performance computing

Revealing key processes in enzyme efficiency through high performance computing

3D Electromagnetic modeling and simulation using heterogeneous computing

3D Electromagnetic modeling and simulation using heterogeneous computing

Hemodynamics in the microcirculation

Hemodynamics in the microcirculation

Putting turbulence to work

Putting turbulence to work

Why are some molecules drugs?

Why are some molecules drugs?

Bayesian additive regression trees  vs logistic regression – estimation of propensity scores

Bayesian additive regression trees vs logistic regression – estimation of propensity scores

Fully coupled thermo-hydro-mechanical modelling of permeability enhancement by the finite element method

Fully coupled thermo-hydro-mechanical modelling of permeability enhancement by the finite element method

Modelling dispersal and ecological competition in a statistical phylogeographic framework

Modelling dispersal and ecological competition in a statistical phylogeographic framework

Studying the shape and the size of the universe

Studying the shape and the size of the universe

Planet hunting

Planet hunting

Simulating quantum mechanics on high performance computing cluster

Simulating quantum mechanics on high performance computing cluster

Multiscale modelling of saliva secretion

Multiscale modelling of saliva secretion

Modelling dual reflux pressure swing adsorption (DR-PSA) units for gas separation in natural gas processing

Modelling dual reflux pressure swing adsorption (DR-PSA) units for gas separation in natural gas processing

Improving the treatment of heart disease

Improving the treatment of heart disease

Estimating migration rates in the budding yeast Saccharomyces cerevisiae

Estimating migration rates in the budding yeast Saccharomyces cerevisiae

Number theoretic algorithms in cryptography

Number theoretic algorithms in cryptography

Molecular phylogenetics uses genetic data to reconstruct the evolutionary history of individuals, populations or species

Molecular phylogenetics uses genetic data to reconstruct the evolutionary history of individuals, populations or species

Phylogeny and phylogeography of the family kyphosidae (Perciformes: teleostei)

Phylogeny and phylogeography of the family kyphosidae (Perciformes: teleostei)

Testing what cosmic inflation really predicts

Testing what cosmic inflation really predicts

Multigene environmental DNA data analysis for New Zealand genomic observatory

Multigene environmental DNA data analysis for New Zealand genomic observatory

Finding genetic variants responsible for human disease hiding in universe of benign variants

Finding genetic variants responsible for human disease hiding in universe of benign variants

BEAST, Bayesian evolutionary analysis sampling trees

BEAST, Bayesian evolutionary analysis sampling trees

The formation of surface archaeological deposits in arid Australia

The formation of surface archaeological deposits in arid Australia

Statistical modelling of carryover effects after cessation of treatments

Statistical modelling of carryover effects after cessation of treatments

High-resolution cryo-electron microscopy of protein complexes and machines

High-resolution cryo-electron microscopy of protein complexes and machines

ARCI, archaeology eResearch collaboration initiative

ARCI, archaeology eResearch collaboration initiative

Optimisation of blades on large wind turbines with individual pitch control and trailing edge flaps

Optimisation of blades on large wind turbines with individual pitch control and trailing edge flaps

Quality of care and outcomes in children with cleft lip and/or palate

Quality of care and outcomes in children with cleft lip and/or palate

Geographic and temporal information retrieval on massive document collections

Geographic and temporal information retrieval on massive document collections

Homodynamics in the microcirculation

Homodynamics in the microcirculation

Processing structure-from-motion photogrammetry on the cluster

Processing structure-from-motion photogrammetry on the cluster

Computational investigation of catalysis mechanisms for polyurethane synthesis

Computational investigation of catalysis mechanisms for polyurethane synthesis

Virtual childhood obesity prevention laboratory

Virtual childhood obesity prevention laboratory

Giving Pacific research greater reach

Giving Pacific research greater reach

Development of novel waveguides  in the terahertz (THz) region

Development of novel waveguides in the terahertz (THz) region

Modelling of costs of diets  by INFORMAS

Modelling of costs of diets by INFORMAS

Foodback

Foodback

Finite element method code for  modelling biological cells

Finite element method code for modelling biological cells

The future of memory: Neuroimaging memory and imagination with functional MRI

The future of memory: Neuroimaging memory and imagination with functional MRI

Modelling and visualisation of calcium waves in parotid acinar cells

Modelling and visualisation of calcium waves in parotid acinar cells

Mapping donor contributions in the Pacific

Mapping donor contributions in the Pacific

Visualising humpback whale migration

Visualising humpback whale migration

Visualising the 2010 and 2011  Canterbury earthquakes

Visualising the 2010 and 2011 Canterbury earthquakes

Data management planning for MOA*

Data management planning for MOA*

Research data publishing  and preservation at COMPASS

Research data publishing and preservation at COMPASS

Centre for eResearch machine learning service

Centre for eResearch machine learning service

Building a discrete global  grid gazetteer service

Building a discrete global grid gazetteer service

The new Wanhal catalogue

The new Wanhal catalogue

Passive acoustic modelling

Passive acoustic modelling

Using GPUs to expand our understanding of the Solar System

Using GPUs to expand our understanding of the Solar System

Shedding new light on dark matter

Shedding new light on dark matter

Aerodynamics modelling paves the way for improved yacht designs

Aerodynamics modelling paves the way for improved yacht designs

Modernising models to help diagnose or treat disease and injury

Modernising models to help diagnose or treat disease and injury

Wandering around the molecular landscape: embracing virtual reality as a research showcasing outreach and teaching tool

Wandering around the molecular landscape: embracing virtual reality as a research showcasing outreach and teaching tool

ALTER: Between human and nonhuman – a VR art exhibition

ALTER: Between human and nonhuman – a VR art exhibition

Disposition of Microsoft HoloLenses for a Pop-Up Reality Shop to demonstrate the progress of a research project

Disposition of Microsoft HoloLenses for a Pop-Up Reality Shop to demonstrate the progress of a research project

Improving diagnosis for schistosomiasis by using the ‘metabolic footprint’ of urine samples from an animal model of Schistosoma infection to identify possible biomarkers

Improving diagnosis for schistosomiasis by using the ‘metabolic footprint’ of urine samples from an animal model of Schistosoma infection to identify possible biomarkers

Making stroke recovery prediction tools freely available

Making stroke recovery prediction tools freely available

MFT-ICR mass spectrometry data management and analysis workflow

MFT-ICR mass spectrometry data management and analysis workflow

Taking a ‘Big Data’ approach to find new clinical-omic associations in cancer

Taking a ‘Big Data’ approach to find new clinical-omic associations in cancer

Growing Up in New Zealand

Growing Up in New Zealand

Improving arrival time predictions for vehicles in a public transport network

Improving arrival time predictions for vehicles in a public transport network

Distributed and cloud-based control at field-level for systems interacting with soft bodies

Distributed and cloud-based control at field-level for systems interacting with soft bodies

Mobile Click Fraud Attack (MCFA)

Mobile Click Fraud Attack (MCFA)

Skin-omics: exploring the volatile organic compounds on human skin

Skin-omics: exploring the volatile organic compounds on human skin

New analytics tools for workload planning for the 2018 New Zealand Census

New analytics tools for workload planning for the 2018 New Zealand Census

Visualising the New Zealand Index of Multiple Deprivation

Visualising the New Zealand Index of Multiple Deprivation