TY - GEN
T1 - GleaM
T2 - 17th ACM International Conference on Mobile Systems, Applications, and Services, MobiSys 2019
AU - Prakash, Siddhant
AU - Bahremand, Alireza
AU - Nguyen, Linda D.
AU - LiKamWa, Robert
N1 - Publisher Copyright:
© 2019 Association for Computing Machinery.
PY - 2019/6/12
Y1 - 2019/6/12
N2 - Mixed reality mobile platforms attempt to co-locate virtual scenes with physical environments, towards creating immersive user experiences. However, to create visual harmony between virtual and physical spaces, the virtual scene must be accurately illuminated with realistic lighting that matches the physical environment. To this end, we design GLEAM, a framework that provides robust illumination estimation in real-time by integrating physical light-probe estimation with current mobile AR systems. GLEAM visually observes reflective objects to compose a realistic estimation of physical lighting. Optionally, GLEAM can network multiple devices to sense illumination from different viewpoints and compose a richer estimation to enhance realism and fidelity. Using GLEAM, AR developers gain the freedom to use a wide range of materials, which is currently limited by the unrealistic appearance of materials that need accurate illumination, such as liquids, glass, and smooth metals. Our controlled environment user studies across 30 participants reveal the effectiveness of GLEAM in providing robust and adaptive illumination estimation over commercial status quo solutions, such as pre-baked directional lighting and ARKit 2.0 illumination estimation. Our benchmarks reveal the need for situation driven tradeoffs to optimize for quality factors in situations requiring freshness over quality and vice-versa. Optimizing for different quality factors in different situations, GLEAM can update scene illumination as fast as 30 ms by sacrificing richness and fidelity in highly dynamic scenes, or prioritize quality by allowing an update interval as high as 400 ms in scenes that require high-fidelity estimation.
AB - Mixed reality mobile platforms attempt to co-locate virtual scenes with physical environments, towards creating immersive user experiences. However, to create visual harmony between virtual and physical spaces, the virtual scene must be accurately illuminated with realistic lighting that matches the physical environment. To this end, we design GLEAM, a framework that provides robust illumination estimation in real-time by integrating physical light-probe estimation with current mobile AR systems. GLEAM visually observes reflective objects to compose a realistic estimation of physical lighting. Optionally, GLEAM can network multiple devices to sense illumination from different viewpoints and compose a richer estimation to enhance realism and fidelity. Using GLEAM, AR developers gain the freedom to use a wide range of materials, which is currently limited by the unrealistic appearance of materials that need accurate illumination, such as liquids, glass, and smooth metals. Our controlled environment user studies across 30 participants reveal the effectiveness of GLEAM in providing robust and adaptive illumination estimation over commercial status quo solutions, such as pre-baked directional lighting and ARKit 2.0 illumination estimation. Our benchmarks reveal the need for situation driven tradeoffs to optimize for quality factors in situations requiring freshness over quality and vice-versa. Optimizing for different quality factors in different situations, GLEAM can update scene illumination as fast as 30 ms by sacrificing richness and fidelity in highly dynamic scenes, or prioritize quality by allowing an update interval as high as 400 ms in scenes that require high-fidelity estimation.
KW - Augmented reality
KW - Geometry
KW - Image processing
KW - Image-based lighting
KW - Light estimation
KW - Light probe
KW - Lighting models
UR - http://www.scopus.com/inward/record.url?scp=85069193152&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85069193152&partnerID=8YFLogxK
U2 - 10.1145/3307334.3326098
DO - 10.1145/3307334.3326098
M3 - Conference contribution
AN - SCOPUS:85069193152
T3 - MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services
SP - 142
EP - 154
BT - MobiSys 2019 - Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services
PB - Association for Computing Machinery, Inc
Y2 - 17 June 2019 through 21 June 2019
ER -