Video

Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs

Jinhan Hu, Alexander Shearer, Saranya Rajagopalan, Robert LiKamWa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The high energy consumption of visual sensing continues to impede the future of mobile vision in which devices will continuously compute visual information from sensory data, e.g,. for visual personal assistants or for augmented reality (AR). While vision algorithms continue to improve in task accuracy and speed, mobile and wearable vision systems fail to achieve sufficient battery life when vision tasks are continuously running. Continuous video capture drains the battery of Google Glass in 30 minutes [2]. It is well known that a common culprit is the energy-expensive traffic of image data [8, 9]. Transferring high resolutions at high frame rates draws substantial power consumption from the analog-digital conversion, the sensor interface transactions, and the memory usage. Simply capturing 1080p frames at 30 frames per second consumes more than 2.4 W of system power measured on a MOTO Z smartphone. However, capturing and displaying 480p frames only consumes 1.3 W of system power. Image resolution can create an interesting tradeoff for visual tasks: low resolution promotes low energy consumption, while high resolution promotes high imaging fidelity for high visual task accuracy. For example, as we explore with our AR marker-based pose estimation case study, lower resolutions suffice when an AR marker is close, but high resolutions are needed when the AR marker is far away or small. This tradeoff has been explored by several visual computing system works including marker pose estimation, object detection, and face recognition [3–6, 9, 10, 13, 14, 17]. We too advocate that mobile vision systems should be able to benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary. Unfortunately, any change in sensor resolution leads to a substantial pause in frame delivery. This is illustrated in Figure 1a. We measure that reconfiguring sensor resolution in the Android OS prevents the application from receiving frames for about 267 ms, the equivalent of dropping 9 frames (working at 30 FPS) from vision processing pipelines [6]. Consequently, computer vision applications don’t change resolutions at runtime, despite the significant energy savings at lower resolutions. For example, Augmented Reality applications such as "Augment" and "UnifiedAR" constantly work at 1080p, drawing 2.7 W of system power.

Original languageEnglish (US)
Title of host publicationMobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services
PublisherAssociation for Computing Machinery, Inc
Pages705-706
Number of pages2
ISBN (Electronic)9781450366618
DOIs
StatePublished - Jun 12 2019
Event17th ACM International Conference on Mobile Systems, Applications, and Services, MobiSys 2019 - Seoul, Korea, Republic of
Duration: Jun 17 2019Jun 21 2019

Publication series

NameMobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services

Conference

Conference17th ACM International Conference on Mobile Systems, Applications, and Services, MobiSys 2019
CountryKorea, Republic of
CitySeoul
Period6/17/196/21/19

Fingerprint

Augmented reality
Image sensors
Image resolution
Sensors
Energy utilization
Imaging techniques
Smartphones
Analog to digital conversion
Face recognition
Computer vision
Interfaces (computer)
Energy conservation
Electric power utilization
Pipelines
Data storage equipment
Glass
Processing

Keywords

  • Device drivers
  • Efficient visual computing
  • Energy efficiency
  • Operating systems
  • Reconfiguration
  • Resolution-based tradeoff

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications

Cite this

Hu, J., Shearer, A., Rajagopalan, S., & LiKamWa, R. (2019). Video: Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs. In MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services (pp. 705-706). (MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services). Association for Computing Machinery, Inc. https://doi.org/10.1145/3307334.3328594

Video : Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs. / Hu, Jinhan; Shearer, Alexander; Rajagopalan, Saranya; LiKamWa, Robert.

MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. Association for Computing Machinery, Inc, 2019. p. 705-706 (MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hu, J, Shearer, A, Rajagopalan, S & LiKamWa, R 2019, Video: Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs. in MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services, Association for Computing Machinery, Inc, pp. 705-706, 17th ACM International Conference on Mobile Systems, Applications, and Services, MobiSys 2019, Seoul, Korea, Republic of, 6/17/19. https://doi.org/10.1145/3307334.3328594
Hu J, Shearer A, Rajagopalan S, LiKamWa R. Video: Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs. In MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. Association for Computing Machinery, Inc. 2019. p. 705-706. (MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services). https://doi.org/10.1145/3307334.3328594
Hu, Jinhan ; Shearer, Alexander ; Rajagopalan, Saranya ; LiKamWa, Robert. / Video : Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs. MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. Association for Computing Machinery, Inc, 2019. pp. 705-706 (MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services).
@inproceedings{16cab57147404d3cbc373a3721480449,
title = "Video: Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs",
abstract = "The high energy consumption of visual sensing continues to impede the future of mobile vision in which devices will continuously compute visual information from sensory data, e.g,. for visual personal assistants or for augmented reality (AR). While vision algorithms continue to improve in task accuracy and speed, mobile and wearable vision systems fail to achieve sufficient battery life when vision tasks are continuously running. Continuous video capture drains the battery of Google Glass in 30 minutes [2]. It is well known that a common culprit is the energy-expensive traffic of image data [8, 9]. Transferring high resolutions at high frame rates draws substantial power consumption from the analog-digital conversion, the sensor interface transactions, and the memory usage. Simply capturing 1080p frames at 30 frames per second consumes more than 2.4 W of system power measured on a MOTO Z smartphone. However, capturing and displaying 480p frames only consumes 1.3 W of system power. Image resolution can create an interesting tradeoff for visual tasks: low resolution promotes low energy consumption, while high resolution promotes high imaging fidelity for high visual task accuracy. For example, as we explore with our AR marker-based pose estimation case study, lower resolutions suffice when an AR marker is close, but high resolutions are needed when the AR marker is far away or small. This tradeoff has been explored by several visual computing system works including marker pose estimation, object detection, and face recognition [3–6, 9, 10, 13, 14, 17]. We too advocate that mobile vision systems should be able to benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary. Unfortunately, any change in sensor resolution leads to a substantial pause in frame delivery. This is illustrated in Figure 1a. We measure that reconfiguring sensor resolution in the Android OS prevents the application from receiving frames for about 267 ms, the equivalent of dropping 9 frames (working at 30 FPS) from vision processing pipelines [6]. Consequently, computer vision applications don’t change resolutions at runtime, despite the significant energy savings at lower resolutions. For example, Augmented Reality applications such as {"}Augment{"} and {"}UnifiedAR{"} constantly work at 1080p, drawing 2.7 W of system power.",
keywords = "Device drivers, Efficient visual computing, Energy efficiency, Operating systems, Reconfiguration, Resolution-based tradeoff",
author = "Jinhan Hu and Alexander Shearer and Saranya Rajagopalan and Robert LiKamWa",
year = "2019",
month = "6",
day = "12",
doi = "10.1145/3307334.3328594",
language = "English (US)",
series = "MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services",
publisher = "Association for Computing Machinery, Inc",
pages = "705--706",
booktitle = "MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services",

}

TY - GEN

T1 - Video

T2 - Banner – An image sensor reconfiguration framework for seamless resolution-based tradeoffs

AU - Hu, Jinhan

AU - Shearer, Alexander

AU - Rajagopalan, Saranya

AU - LiKamWa, Robert

PY - 2019/6/12

Y1 - 2019/6/12

N2 - The high energy consumption of visual sensing continues to impede the future of mobile vision in which devices will continuously compute visual information from sensory data, e.g,. for visual personal assistants or for augmented reality (AR). While vision algorithms continue to improve in task accuracy and speed, mobile and wearable vision systems fail to achieve sufficient battery life when vision tasks are continuously running. Continuous video capture drains the battery of Google Glass in 30 minutes [2]. It is well known that a common culprit is the energy-expensive traffic of image data [8, 9]. Transferring high resolutions at high frame rates draws substantial power consumption from the analog-digital conversion, the sensor interface transactions, and the memory usage. Simply capturing 1080p frames at 30 frames per second consumes more than 2.4 W of system power measured on a MOTO Z smartphone. However, capturing and displaying 480p frames only consumes 1.3 W of system power. Image resolution can create an interesting tradeoff for visual tasks: low resolution promotes low energy consumption, while high resolution promotes high imaging fidelity for high visual task accuracy. For example, as we explore with our AR marker-based pose estimation case study, lower resolutions suffice when an AR marker is close, but high resolutions are needed when the AR marker is far away or small. This tradeoff has been explored by several visual computing system works including marker pose estimation, object detection, and face recognition [3–6, 9, 10, 13, 14, 17]. We too advocate that mobile vision systems should be able to benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary. Unfortunately, any change in sensor resolution leads to a substantial pause in frame delivery. This is illustrated in Figure 1a. We measure that reconfiguring sensor resolution in the Android OS prevents the application from receiving frames for about 267 ms, the equivalent of dropping 9 frames (working at 30 FPS) from vision processing pipelines [6]. Consequently, computer vision applications don’t change resolutions at runtime, despite the significant energy savings at lower resolutions. For example, Augmented Reality applications such as "Augment" and "UnifiedAR" constantly work at 1080p, drawing 2.7 W of system power.

AB - The high energy consumption of visual sensing continues to impede the future of mobile vision in which devices will continuously compute visual information from sensory data, e.g,. for visual personal assistants or for augmented reality (AR). While vision algorithms continue to improve in task accuracy and speed, mobile and wearable vision systems fail to achieve sufficient battery life when vision tasks are continuously running. Continuous video capture drains the battery of Google Glass in 30 minutes [2]. It is well known that a common culprit is the energy-expensive traffic of image data [8, 9]. Transferring high resolutions at high frame rates draws substantial power consumption from the analog-digital conversion, the sensor interface transactions, and the memory usage. Simply capturing 1080p frames at 30 frames per second consumes more than 2.4 W of system power measured on a MOTO Z smartphone. However, capturing and displaying 480p frames only consumes 1.3 W of system power. Image resolution can create an interesting tradeoff for visual tasks: low resolution promotes low energy consumption, while high resolution promotes high imaging fidelity for high visual task accuracy. For example, as we explore with our AR marker-based pose estimation case study, lower resolutions suffice when an AR marker is close, but high resolutions are needed when the AR marker is far away or small. This tradeoff has been explored by several visual computing system works including marker pose estimation, object detection, and face recognition [3–6, 9, 10, 13, 14, 17]. We too advocate that mobile vision systems should be able to benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary. Unfortunately, any change in sensor resolution leads to a substantial pause in frame delivery. This is illustrated in Figure 1a. We measure that reconfiguring sensor resolution in the Android OS prevents the application from receiving frames for about 267 ms, the equivalent of dropping 9 frames (working at 30 FPS) from vision processing pipelines [6]. Consequently, computer vision applications don’t change resolutions at runtime, despite the significant energy savings at lower resolutions. For example, Augmented Reality applications such as "Augment" and "UnifiedAR" constantly work at 1080p, drawing 2.7 W of system power.

KW - Device drivers

KW - Efficient visual computing

KW - Energy efficiency

KW - Operating systems

KW - Reconfiguration

KW - Resolution-based tradeoff

UR - http://www.scopus.com/inward/record.url?scp=85069147729&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069147729&partnerID=8YFLogxK

U2 - 10.1145/3307334.3328594

DO - 10.1145/3307334.3328594

M3 - Conference contribution

T3 - MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services

SP - 705

EP - 706

BT - MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services

PB - Association for Computing Machinery, Inc

ER -