Rhythmic pixel regions: Multi-resolution visual sensing system towards high-precision visual computing at low power

Venkatesh Kodukula, Alexander Shearer, Van Nguyen, Srinivas Lingutla, Yifei Liu, Robert Likamwa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

High spatiotemporal resolution can offer high precision for vision applications, which is particularly useful to capture the nuances of visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but reduce also the opportunities of high-precision visual sensing. However, our intuition is that not all parts of the scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates. To this end, we develop a visual sensing pipeline architecture that flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different "rhythmic pixel regions"in the scene. We develop a system that ingests pixel streams from commercial image sensors with their standard raster-scan pixel read-out patterns, but only encodes relevant pixels prior to storing them in the memory. We also present streaming hardware to decode the stored rhythmic pixel region stream into traditional frame-based representations to feed into standard computer vision algorithms. We integrate our encoding and decoding hardware modules into existing video pipelines. On top of this, we develop runtime support allowing developers to flexibly specify the region labels. Evaluating our system on a Xilinx FPGA platform over three vision workloads shows 43-64% reduction in interface traffic and memory footprint, while providing controllable task accuracy.

Original languageEnglish (US)
Title of host publicationProceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2021
PublisherAssociation for Computing Machinery
Pages573-586
Number of pages14
ISBN (Electronic)9781450383172
DOIs
StatePublished - Apr 19 2021
Externally publishedYes
Event26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2021 - Virtual, Online, United States
Duration: Apr 19 2021Apr 23 2021

Publication series

NameInternational Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS

Conference

Conference26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2021
Country/TerritoryUnited States
CityVirtual, Online
Period4/19/214/23/21

Keywords

  • augmented reality
  • pixel discard
  • visual computing

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'Rhythmic pixel regions: Multi-resolution visual sensing system towards high-precision visual computing at low power'. Together they form a unique fingerprint.

Cite this