TY - GEN
T1 - Rhythmic pixel regions
T2 - 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2021
AU - Kodukula, Venkatesh
AU - Shearer, Alexander
AU - Nguyen, Van
AU - Lingutla, Srinivas
AU - Liu, Yifei
AU - Likamwa, Robert
N1 - Funding Information:
The authors are grateful for comments made by anonymous reviewers and the paper shepherd Dr. Vijay Janapa Reddi. This material is based upon work supported by the National Science Foundation under grants 1942844 and 1909663.
Publisher Copyright:
© 2021 ACM.
PY - 2021/4/19
Y1 - 2021/4/19
N2 - High spatiotemporal resolution can offer high precision for vision applications, which is particularly useful to capture the nuances of visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but reduce also the opportunities of high-precision visual sensing. However, our intuition is that not all parts of the scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates. To this end, we develop a visual sensing pipeline architecture that flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different "rhythmic pixel regions"in the scene. We develop a system that ingests pixel streams from commercial image sensors with their standard raster-scan pixel read-out patterns, but only encodes relevant pixels prior to storing them in the memory. We also present streaming hardware to decode the stored rhythmic pixel region stream into traditional frame-based representations to feed into standard computer vision algorithms. We integrate our encoding and decoding hardware modules into existing video pipelines. On top of this, we develop runtime support allowing developers to flexibly specify the region labels. Evaluating our system on a Xilinx FPGA platform over three vision workloads shows 43-64% reduction in interface traffic and memory footprint, while providing controllable task accuracy.
AB - High spatiotemporal resolution can offer high precision for vision applications, which is particularly useful to capture the nuances of visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but reduce also the opportunities of high-precision visual sensing. However, our intuition is that not all parts of the scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates. To this end, we develop a visual sensing pipeline architecture that flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different "rhythmic pixel regions"in the scene. We develop a system that ingests pixel streams from commercial image sensors with their standard raster-scan pixel read-out patterns, but only encodes relevant pixels prior to storing them in the memory. We also present streaming hardware to decode the stored rhythmic pixel region stream into traditional frame-based representations to feed into standard computer vision algorithms. We integrate our encoding and decoding hardware modules into existing video pipelines. On top of this, we develop runtime support allowing developers to flexibly specify the region labels. Evaluating our system on a Xilinx FPGA platform over three vision workloads shows 43-64% reduction in interface traffic and memory footprint, while providing controllable task accuracy.
KW - augmented reality
KW - pixel discard
KW - visual computing
UR - http://www.scopus.com/inward/record.url?scp=85104749371&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85104749371&partnerID=8YFLogxK
U2 - 10.1145/3445814.3446737
DO - 10.1145/3445814.3446737
M3 - Conference contribution
AN - SCOPUS:85104749371
T3 - International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS
SP - 573
EP - 586
BT - Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2021
PB - Association for Computing Machinery
Y2 - 19 April 2021 through 23 April 2021
ER -