Compressive Light Field Reconstructions Using Deep Learning

Mayank Gupta, Arjun Jauhari, Kuldeep Kulkarni, Suren Jayasuriya, Alyosha Molnar, Pavan Turaga

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Citations (Scopus)

Abstract

Light field imaging is limited in its computational processing demands of high sampling for both spatial and angular dimensions. Single-shot light field cameras sacrifice spatial resolution to sample angular viewpoints, typically by multiplexing incoming rays onto a 2D sensor array. While this resolution can be recovered using compressive sensing, these iterative solutions are slow in processing a light field. We present a deep learning approach using a new, two branch network architecture, consisting jointly of an autoencoder and a 4D CNN, to recover a high resolution 4D light field from a single coded 2D image. This network decreases reconstruction time significantly while achieving average PSNR values of 26-32 dB on a variety of light fields. In particular, reconstruction time is decreased from 35 minutes to 6.7 minutes as compared to the dictionary method for equivalent visual quality. These reconstructions are performed at small sampling/compression ratios as low as 8%, allowing for cheaper coded light field cameras. We test our network reconstructions on synthetic light fields, simulated coded measurements of real light fields captured from a Lytro Illum camera, and real coded images from a custom CMOS diffractive light field camera. The combination of compressive light field capture with deep learning allows the potential for real-time light field video acquisition systems in the future.

Original languageEnglish (US)
Title of host publicationProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017
PublisherIEEE Computer Society
Pages1277-1286
Number of pages10
Volume2017-July
ISBN (Electronic)9781538607336
DOIs
StatePublished - Aug 22 2017
Event30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017 - Honolulu, United States
Duration: Jul 21 2017Jul 26 2017

Other

Other30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017
CountryUnited States
CityHonolulu
Period7/21/177/26/17

Fingerprint

Cameras
Deep learning
Sampling
Sensor arrays
Glossaries
Processing
Network architecture
Multiplexing
Imaging techniques

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering

Cite this

Gupta, M., Jauhari, A., Kulkarni, K., Jayasuriya, S., Molnar, A., & Turaga, P. (2017). Compressive Light Field Reconstructions Using Deep Learning. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017 (Vol. 2017-July, pp. 1277-1286). [8014902] IEEE Computer Society. https://doi.org/10.1109/CVPRW.2017.168

Compressive Light Field Reconstructions Using Deep Learning. / Gupta, Mayank; Jauhari, Arjun; Kulkarni, Kuldeep; Jayasuriya, Suren; Molnar, Alyosha; Turaga, Pavan.

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017. Vol. 2017-July IEEE Computer Society, 2017. p. 1277-1286 8014902.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Gupta, M, Jauhari, A, Kulkarni, K, Jayasuriya, S, Molnar, A & Turaga, P 2017, Compressive Light Field Reconstructions Using Deep Learning. in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017. vol. 2017-July, 8014902, IEEE Computer Society, pp. 1277-1286, 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017, Honolulu, United States, 7/21/17. https://doi.org/10.1109/CVPRW.2017.168
Gupta M, Jauhari A, Kulkarni K, Jayasuriya S, Molnar A, Turaga P. Compressive Light Field Reconstructions Using Deep Learning. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017. Vol. 2017-July. IEEE Computer Society. 2017. p. 1277-1286. 8014902 https://doi.org/10.1109/CVPRW.2017.168
Gupta, Mayank ; Jauhari, Arjun ; Kulkarni, Kuldeep ; Jayasuriya, Suren ; Molnar, Alyosha ; Turaga, Pavan. / Compressive Light Field Reconstructions Using Deep Learning. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017. Vol. 2017-July IEEE Computer Society, 2017. pp. 1277-1286
@inproceedings{d56c2b85704e4c6c8319407108bbc9cf,
title = "Compressive Light Field Reconstructions Using Deep Learning",
abstract = "Light field imaging is limited in its computational processing demands of high sampling for both spatial and angular dimensions. Single-shot light field cameras sacrifice spatial resolution to sample angular viewpoints, typically by multiplexing incoming rays onto a 2D sensor array. While this resolution can be recovered using compressive sensing, these iterative solutions are slow in processing a light field. We present a deep learning approach using a new, two branch network architecture, consisting jointly of an autoencoder and a 4D CNN, to recover a high resolution 4D light field from a single coded 2D image. This network decreases reconstruction time significantly while achieving average PSNR values of 26-32 dB on a variety of light fields. In particular, reconstruction time is decreased from 35 minutes to 6.7 minutes as compared to the dictionary method for equivalent visual quality. These reconstructions are performed at small sampling/compression ratios as low as 8{\%}, allowing for cheaper coded light field cameras. We test our network reconstructions on synthetic light fields, simulated coded measurements of real light fields captured from a Lytro Illum camera, and real coded images from a custom CMOS diffractive light field camera. The combination of compressive light field capture with deep learning allows the potential for real-time light field video acquisition systems in the future.",
author = "Mayank Gupta and Arjun Jauhari and Kuldeep Kulkarni and Suren Jayasuriya and Alyosha Molnar and Pavan Turaga",
year = "2017",
month = "8",
day = "22",
doi = "10.1109/CVPRW.2017.168",
language = "English (US)",
volume = "2017-July",
pages = "1277--1286",
booktitle = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017",
publisher = "IEEE Computer Society",
address = "United States",

}

TY - GEN

T1 - Compressive Light Field Reconstructions Using Deep Learning

AU - Gupta, Mayank

AU - Jauhari, Arjun

AU - Kulkarni, Kuldeep

AU - Jayasuriya, Suren

AU - Molnar, Alyosha

AU - Turaga, Pavan

PY - 2017/8/22

Y1 - 2017/8/22

N2 - Light field imaging is limited in its computational processing demands of high sampling for both spatial and angular dimensions. Single-shot light field cameras sacrifice spatial resolution to sample angular viewpoints, typically by multiplexing incoming rays onto a 2D sensor array. While this resolution can be recovered using compressive sensing, these iterative solutions are slow in processing a light field. We present a deep learning approach using a new, two branch network architecture, consisting jointly of an autoencoder and a 4D CNN, to recover a high resolution 4D light field from a single coded 2D image. This network decreases reconstruction time significantly while achieving average PSNR values of 26-32 dB on a variety of light fields. In particular, reconstruction time is decreased from 35 minutes to 6.7 minutes as compared to the dictionary method for equivalent visual quality. These reconstructions are performed at small sampling/compression ratios as low as 8%, allowing for cheaper coded light field cameras. We test our network reconstructions on synthetic light fields, simulated coded measurements of real light fields captured from a Lytro Illum camera, and real coded images from a custom CMOS diffractive light field camera. The combination of compressive light field capture with deep learning allows the potential for real-time light field video acquisition systems in the future.

AB - Light field imaging is limited in its computational processing demands of high sampling for both spatial and angular dimensions. Single-shot light field cameras sacrifice spatial resolution to sample angular viewpoints, typically by multiplexing incoming rays onto a 2D sensor array. While this resolution can be recovered using compressive sensing, these iterative solutions are slow in processing a light field. We present a deep learning approach using a new, two branch network architecture, consisting jointly of an autoencoder and a 4D CNN, to recover a high resolution 4D light field from a single coded 2D image. This network decreases reconstruction time significantly while achieving average PSNR values of 26-32 dB on a variety of light fields. In particular, reconstruction time is decreased from 35 minutes to 6.7 minutes as compared to the dictionary method for equivalent visual quality. These reconstructions are performed at small sampling/compression ratios as low as 8%, allowing for cheaper coded light field cameras. We test our network reconstructions on synthetic light fields, simulated coded measurements of real light fields captured from a Lytro Illum camera, and real coded images from a custom CMOS diffractive light field camera. The combination of compressive light field capture with deep learning allows the potential for real-time light field video acquisition systems in the future.

UR - http://www.scopus.com/inward/record.url?scp=85030242715&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85030242715&partnerID=8YFLogxK

U2 - 10.1109/CVPRW.2017.168

DO - 10.1109/CVPRW.2017.168

M3 - Conference contribution

VL - 2017-July

SP - 1277

EP - 1286

BT - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2017

PB - IEEE Computer Society

ER -