Low to High Dimensional Modality Hallucination Using Aggregated Fields of View

Kausic Gunasekar, Qiang Qiu, Yezhou Yang

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Real-world robotics systems deal with data from a multitude of modalities, especially for tasks such as navigation and recognition. The performance of those systems can drastically degrade when one or more modalities become inaccessible, due to factors such as sensors' malfunctions or adverse environments. Here, we argue modality hallucination as one effective way to ensure consistent modality availability and thereby reduce unfavorable consequences. While hallucinating data from a modality with richer information, e.g., RGB to depth, has been researched extensively, we investigate the more challenging low-to-high modality hallucination with interesting use cases in robotics and autonomous systems. We present a novel hallucination architecture that aggregates information from multiple fields of view of the local neighborhood to recover the lost information from the extant modality. The process is implemented by capturing a non-linear mapping between the data modalities and the learned mapping is used to aid the extant modality to mitigate the risk posed to the system in the adverse scenarios which involve modality loss. We also conduct extensive classification and segmentation experiments on UWRGBD and NYUD datasets and demonstrate that hallucination allays the negative effects of the modality loss. Implementation and models: https://github.com/kausic94/Hallucination.

Original languageEnglish (US)
Article number8977350
Pages (from-to)1983-1990
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number2
StatePublished - Apr 2020


  • Deep learning in robotics and automation
  • computer vision for other robotic applications
  • modality hallucination
  • robot safety

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence


Dive into the research topics of 'Low to High Dimensional Modality Hallucination Using Aggregated Fields of View'. Together they form a unique fingerprint.

Cite this