Protecting visual information in augmented reality from malicious application developers

Jk Jensen, Jinhan Hu, Amir Rahmati, Robert Likamwa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Visual applications - those that use camera frames as part of the application - allows for a rich, context-aware experience. The continuing development of mixed and augmented reality (MR/AR) on head-mounted displays (HMDs) furthers the richness of this experience by providing users a continuous vision experience, where visual information continuously provides context, and the real world is augmented by the virtual. However, these visual applications raise serious privacy concerns because they can capture private user information. To understand user privacy concerns in continuous vision computing environments, we study three MR/AR applications (augmented markers, augmented faces, and text capture). We show that in modern mobile visual applications, typical users are exposed to potential mass collection of sensitive information. To address such deficiencies, we develop a framework that provides resource isolation between user information contained in camera frames and application access to the network. We implement the design as a proof of concept on the Android operating system and demonstrate its performance and usability with a modern state-of-the-art augmented reality library and several augmented reality applications. By comparing the applications from our case study with modified versions which better protect user privacy, results show that our design efficiently protects users against data collection in MR/AR applications with less than 0.7% performance overhead.

Original languageEnglish (US)
Title of host publicationWearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019
PublisherAssociation for Computing Machinery, Inc
Pages23-28
Number of pages6
ISBN (Electronic)9781450367752
DOIs
StatePublished - Jun 12 2019
Event5th ACM Workshop on Wearable Systems and Applications, WearSys 2019, co-located with MobiSys 2019 - Seoul, Korea, Republic of
Duration: Jun 21 2019 → …

Publication series

NameWearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019

Conference

Conference5th ACM Workshop on Wearable Systems and Applications, WearSys 2019, co-located with MobiSys 2019
CountryKorea, Republic of
CitySeoul
Period6/21/19 → …

Fingerprint

Augmented reality
Cameras
Display devices

Keywords

  • Operating system
  • Resource isolation
  • Split process
  • User privacy
  • Visual information protection

ASJC Scopus subject areas

  • Hardware and Architecture
  • Software
  • Computer Networks and Communications

Cite this

Jensen, J., Hu, J., Rahmati, A., & Likamwa, R. (2019). Protecting visual information in augmented reality from malicious application developers. In WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019 (pp. 23-28). (WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3325424.3329659

Protecting visual information in augmented reality from malicious application developers. / Jensen, Jk; Hu, Jinhan; Rahmati, Amir; Likamwa, Robert.

WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019. Association for Computing Machinery, Inc, 2019. p. 23-28 (WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Jensen, J, Hu, J, Rahmati, A & Likamwa, R 2019, Protecting visual information in augmented reality from malicious application developers. in WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019. WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019, Association for Computing Machinery, Inc, pp. 23-28, 5th ACM Workshop on Wearable Systems and Applications, WearSys 2019, co-located with MobiSys 2019, Seoul, Korea, Republic of, 6/21/19. https://doi.org/10.1145/3325424.3329659
Jensen J, Hu J, Rahmati A, Likamwa R. Protecting visual information in augmented reality from malicious application developers. In WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019. Association for Computing Machinery, Inc. 2019. p. 23-28. (WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019). https://doi.org/10.1145/3325424.3329659
Jensen, Jk ; Hu, Jinhan ; Rahmati, Amir ; Likamwa, Robert. / Protecting visual information in augmented reality from malicious application developers. WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019. Association for Computing Machinery, Inc, 2019. pp. 23-28 (WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019).
@inproceedings{093159581fe84c25ad4fca2194651683,
title = "Protecting visual information in augmented reality from malicious application developers",
abstract = "Visual applications - those that use camera frames as part of the application - allows for a rich, context-aware experience. The continuing development of mixed and augmented reality (MR/AR) on head-mounted displays (HMDs) furthers the richness of this experience by providing users a continuous vision experience, where visual information continuously provides context, and the real world is augmented by the virtual. However, these visual applications raise serious privacy concerns because they can capture private user information. To understand user privacy concerns in continuous vision computing environments, we study three MR/AR applications (augmented markers, augmented faces, and text capture). We show that in modern mobile visual applications, typical users are exposed to potential mass collection of sensitive information. To address such deficiencies, we develop a framework that provides resource isolation between user information contained in camera frames and application access to the network. We implement the design as a proof of concept on the Android operating system and demonstrate its performance and usability with a modern state-of-the-art augmented reality library and several augmented reality applications. By comparing the applications from our case study with modified versions which better protect user privacy, results show that our design efficiently protects users against data collection in MR/AR applications with less than 0.7{\%} performance overhead.",
keywords = "Operating system, Resource isolation, Split process, User privacy, Visual information protection",
author = "Jk Jensen and Jinhan Hu and Amir Rahmati and Robert Likamwa",
year = "2019",
month = "6",
day = "12",
doi = "10.1145/3325424.3329659",
language = "English (US)",
series = "WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019",
publisher = "Association for Computing Machinery, Inc",
pages = "23--28",
booktitle = "WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019",

}

TY - GEN

T1 - Protecting visual information in augmented reality from malicious application developers

AU - Jensen, Jk

AU - Hu, Jinhan

AU - Rahmati, Amir

AU - Likamwa, Robert

PY - 2019/6/12

Y1 - 2019/6/12

N2 - Visual applications - those that use camera frames as part of the application - allows for a rich, context-aware experience. The continuing development of mixed and augmented reality (MR/AR) on head-mounted displays (HMDs) furthers the richness of this experience by providing users a continuous vision experience, where visual information continuously provides context, and the real world is augmented by the virtual. However, these visual applications raise serious privacy concerns because they can capture private user information. To understand user privacy concerns in continuous vision computing environments, we study three MR/AR applications (augmented markers, augmented faces, and text capture). We show that in modern mobile visual applications, typical users are exposed to potential mass collection of sensitive information. To address such deficiencies, we develop a framework that provides resource isolation between user information contained in camera frames and application access to the network. We implement the design as a proof of concept on the Android operating system and demonstrate its performance and usability with a modern state-of-the-art augmented reality library and several augmented reality applications. By comparing the applications from our case study with modified versions which better protect user privacy, results show that our design efficiently protects users against data collection in MR/AR applications with less than 0.7% performance overhead.

AB - Visual applications - those that use camera frames as part of the application - allows for a rich, context-aware experience. The continuing development of mixed and augmented reality (MR/AR) on head-mounted displays (HMDs) furthers the richness of this experience by providing users a continuous vision experience, where visual information continuously provides context, and the real world is augmented by the virtual. However, these visual applications raise serious privacy concerns because they can capture private user information. To understand user privacy concerns in continuous vision computing environments, we study three MR/AR applications (augmented markers, augmented faces, and text capture). We show that in modern mobile visual applications, typical users are exposed to potential mass collection of sensitive information. To address such deficiencies, we develop a framework that provides resource isolation between user information contained in camera frames and application access to the network. We implement the design as a proof of concept on the Android operating system and demonstrate its performance and usability with a modern state-of-the-art augmented reality library and several augmented reality applications. By comparing the applications from our case study with modified versions which better protect user privacy, results show that our design efficiently protects users against data collection in MR/AR applications with less than 0.7% performance overhead.

KW - Operating system

KW - Resource isolation

KW - Split process

KW - User privacy

KW - Visual information protection

UR - http://www.scopus.com/inward/record.url?scp=85074451004&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85074451004&partnerID=8YFLogxK

U2 - 10.1145/3325424.3329659

DO - 10.1145/3325424.3329659

M3 - Conference contribution

AN - SCOPUS:85074451004

T3 - WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019

SP - 23

EP - 28

BT - WearSys 2019 - Proceedings of the 5th ACM Workshop on Wearable Systems and Applications, co-located with MobiSys 2019

PB - Association for Computing Machinery, Inc

ER -