Automated data processing architecture for the gemini planet imager exoplanet survey

Jason J. Wang, Marshall D. Perrin, Dmitry Savransky, Pauline Arriaga, Jeffrey K. Chilcote, Robert J. De Rosa, Maxwell A. Millar-Blanchaer, Christian Marois, Julien Rameau, Schuyler G. Wolff, Jacob Shapiro, Jean Baptiste Ruffio, Jérôme Maire, Franck Marchis, James R. Graham, Bruce Macintosh, S. Mark Ammons, Vanessa P. Bailey, Travis S. Barman, Sebastian Bruzzone & 32 others Joanna Bulger, Tara Cotton, René Doyon, Gaspard Duchêne, Michael P. Fitzgerald, Katherine B. Follette, Stephen Goodsell, Alexandra Z. Greenbaum, Pascale Hibon, Li Wei Hung, Patrick Ingraham, Paul Kalas, Quinn M. Konopacky, James E. Larkin, Mark S. Marley, Stanimir Metchev, Eric L. Nielsen, Rebecca Oppenheimer, David W. Palmer, Jennifer Patience, Lisa A. Poyneer, Laurent Pueyo, Abhijith Rajan, Fredrik T. Rantakyrö, Adam C. Schneider, Anand Sivaramakrishnan, Inseok Song, Remi Soummer, Sandrine Thomas, J. Kent Wallace, Kimberly Ward-Duong, Sloane J. Wiktorowicz

Research output: Contribution to journalReview article

10 Citations (Scopus)

Abstract

The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

Original languageEnglish (US)
Article number018002
JournalJournal of Astronomical Telescopes, Instruments, and Systems
Volume4
Issue number1
DOIs
StatePublished - Jan 1 2018

Fingerprint

Planets
extrasolar planets
Image sensors
planets
planet
data reduction
Data reduction
data products
files
Pipelines
supercomputers
Supercomputers
Stars
Servers
Display devices
Calibration
occurrences
Imaging techniques
stars
browsing

Keywords

  • circumstellar disks
  • Data Cruncher.
  • data processing
  • exoplanets
  • Gemini planet imager
  • high contrast imaging

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Control and Systems Engineering
  • Instrumentation
  • Astronomy and Astrophysics
  • Mechanical Engineering
  • Space and Planetary Science

Cite this

Wang, J. J., Perrin, M. D., Savransky, D., Arriaga, P., Chilcote, J. K., De Rosa, R. J., ... Wiktorowicz, S. J. (2018). Automated data processing architecture for the gemini planet imager exoplanet survey. Journal of Astronomical Telescopes, Instruments, and Systems, 4(1), [018002]. https://doi.org/10.1117/1.JATIS.4.1.018002

Automated data processing architecture for the gemini planet imager exoplanet survey. / Wang, Jason J.; Perrin, Marshall D.; Savransky, Dmitry; Arriaga, Pauline; Chilcote, Jeffrey K.; De Rosa, Robert J.; Millar-Blanchaer, Maxwell A.; Marois, Christian; Rameau, Julien; Wolff, Schuyler G.; Shapiro, Jacob; Ruffio, Jean Baptiste; Maire, Jérôme; Marchis, Franck; Graham, James R.; Macintosh, Bruce; Mark Ammons, S.; Bailey, Vanessa P.; Barman, Travis S.; Bruzzone, Sebastian; Bulger, Joanna; Cotton, Tara; Doyon, René; Duchêne, Gaspard; Fitzgerald, Michael P.; Follette, Katherine B.; Goodsell, Stephen; Greenbaum, Alexandra Z.; Hibon, Pascale; Hung, Li Wei; Ingraham, Patrick; Kalas, Paul; Konopacky, Quinn M.; Larkin, James E.; Marley, Mark S.; Metchev, Stanimir; Nielsen, Eric L.; Oppenheimer, Rebecca; Palmer, David W.; Patience, Jennifer; Poyneer, Lisa A.; Pueyo, Laurent; Rajan, Abhijith; Rantakyrö, Fredrik T.; Schneider, Adam C.; Sivaramakrishnan, Anand; Song, Inseok; Soummer, Remi; Thomas, Sandrine; Kent Wallace, J.; Ward-Duong, Kimberly; Wiktorowicz, Sloane J.

In: Journal of Astronomical Telescopes, Instruments, and Systems, Vol. 4, No. 1, 018002, 01.01.2018.

Research output: Contribution to journalReview article

Wang, JJ, Perrin, MD, Savransky, D, Arriaga, P, Chilcote, JK, De Rosa, RJ, Millar-Blanchaer, MA, Marois, C, Rameau, J, Wolff, SG, Shapiro, J, Ruffio, JB, Maire, J, Marchis, F, Graham, JR, Macintosh, B, Mark Ammons, S, Bailey, VP, Barman, TS, Bruzzone, S, Bulger, J, Cotton, T, Doyon, R, Duchêne, G, Fitzgerald, MP, Follette, KB, Goodsell, S, Greenbaum, AZ, Hibon, P, Hung, LW, Ingraham, P, Kalas, P, Konopacky, QM, Larkin, JE, Marley, MS, Metchev, S, Nielsen, EL, Oppenheimer, R, Palmer, DW, Patience, J, Poyneer, LA, Pueyo, L, Rajan, A, Rantakyrö, FT, Schneider, AC, Sivaramakrishnan, A, Song, I, Soummer, R, Thomas, S, Kent Wallace, J, Ward-Duong, K & Wiktorowicz, SJ 2018, 'Automated data processing architecture for the gemini planet imager exoplanet survey', Journal of Astronomical Telescopes, Instruments, and Systems, vol. 4, no. 1, 018002. https://doi.org/10.1117/1.JATIS.4.1.018002
Wang, Jason J. ; Perrin, Marshall D. ; Savransky, Dmitry ; Arriaga, Pauline ; Chilcote, Jeffrey K. ; De Rosa, Robert J. ; Millar-Blanchaer, Maxwell A. ; Marois, Christian ; Rameau, Julien ; Wolff, Schuyler G. ; Shapiro, Jacob ; Ruffio, Jean Baptiste ; Maire, Jérôme ; Marchis, Franck ; Graham, James R. ; Macintosh, Bruce ; Mark Ammons, S. ; Bailey, Vanessa P. ; Barman, Travis S. ; Bruzzone, Sebastian ; Bulger, Joanna ; Cotton, Tara ; Doyon, René ; Duchêne, Gaspard ; Fitzgerald, Michael P. ; Follette, Katherine B. ; Goodsell, Stephen ; Greenbaum, Alexandra Z. ; Hibon, Pascale ; Hung, Li Wei ; Ingraham, Patrick ; Kalas, Paul ; Konopacky, Quinn M. ; Larkin, James E. ; Marley, Mark S. ; Metchev, Stanimir ; Nielsen, Eric L. ; Oppenheimer, Rebecca ; Palmer, David W. ; Patience, Jennifer ; Poyneer, Lisa A. ; Pueyo, Laurent ; Rajan, Abhijith ; Rantakyrö, Fredrik T. ; Schneider, Adam C. ; Sivaramakrishnan, Anand ; Song, Inseok ; Soummer, Remi ; Thomas, Sandrine ; Kent Wallace, J. ; Ward-Duong, Kimberly ; Wiktorowicz, Sloane J. / Automated data processing architecture for the gemini planet imager exoplanet survey. In: Journal of Astronomical Telescopes, Instruments, and Systems. 2018 ; Vol. 4, No. 1.
@article{6e84aad3f111415ca8382e6b92963c8f,
title = "Automated data processing architecture for the gemini planet imager exoplanet survey",
abstract = "The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.",
keywords = "circumstellar disks, Data Cruncher., data processing, exoplanets, Gemini planet imager, high contrast imaging",
author = "Wang, {Jason J.} and Perrin, {Marshall D.} and Dmitry Savransky and Pauline Arriaga and Chilcote, {Jeffrey K.} and {De Rosa}, {Robert J.} and Millar-Blanchaer, {Maxwell A.} and Christian Marois and Julien Rameau and Wolff, {Schuyler G.} and Jacob Shapiro and Ruffio, {Jean Baptiste} and J{\'e}r{\^o}me Maire and Franck Marchis and Graham, {James R.} and Bruce Macintosh and {Mark Ammons}, S. and Bailey, {Vanessa P.} and Barman, {Travis S.} and Sebastian Bruzzone and Joanna Bulger and Tara Cotton and Ren{\'e} Doyon and Gaspard Duch{\^e}ne and Fitzgerald, {Michael P.} and Follette, {Katherine B.} and Stephen Goodsell and Greenbaum, {Alexandra Z.} and Pascale Hibon and Hung, {Li Wei} and Patrick Ingraham and Paul Kalas and Konopacky, {Quinn M.} and Larkin, {James E.} and Marley, {Mark S.} and Stanimir Metchev and Nielsen, {Eric L.} and Rebecca Oppenheimer and Palmer, {David W.} and Jennifer Patience and Poyneer, {Lisa A.} and Laurent Pueyo and Abhijith Rajan and Rantakyr{\"o}, {Fredrik T.} and Schneider, {Adam C.} and Anand Sivaramakrishnan and Inseok Song and Remi Soummer and Sandrine Thomas and {Kent Wallace}, J. and Kimberly Ward-Duong and Wiktorowicz, {Sloane J.}",
year = "2018",
month = "1",
day = "1",
doi = "10.1117/1.JATIS.4.1.018002",
language = "English (US)",
volume = "4",
journal = "Journal of Astronomical Telescopes, Instruments, and Systems",
issn = "2329-4124",
publisher = "SPIE",
number = "1",

}

TY - JOUR

T1 - Automated data processing architecture for the gemini planet imager exoplanet survey

AU - Wang, Jason J.

AU - Perrin, Marshall D.

AU - Savransky, Dmitry

AU - Arriaga, Pauline

AU - Chilcote, Jeffrey K.

AU - De Rosa, Robert J.

AU - Millar-Blanchaer, Maxwell A.

AU - Marois, Christian

AU - Rameau, Julien

AU - Wolff, Schuyler G.

AU - Shapiro, Jacob

AU - Ruffio, Jean Baptiste

AU - Maire, Jérôme

AU - Marchis, Franck

AU - Graham, James R.

AU - Macintosh, Bruce

AU - Mark Ammons, S.

AU - Bailey, Vanessa P.

AU - Barman, Travis S.

AU - Bruzzone, Sebastian

AU - Bulger, Joanna

AU - Cotton, Tara

AU - Doyon, René

AU - Duchêne, Gaspard

AU - Fitzgerald, Michael P.

AU - Follette, Katherine B.

AU - Goodsell, Stephen

AU - Greenbaum, Alexandra Z.

AU - Hibon, Pascale

AU - Hung, Li Wei

AU - Ingraham, Patrick

AU - Kalas, Paul

AU - Konopacky, Quinn M.

AU - Larkin, James E.

AU - Marley, Mark S.

AU - Metchev, Stanimir

AU - Nielsen, Eric L.

AU - Oppenheimer, Rebecca

AU - Palmer, David W.

AU - Patience, Jennifer

AU - Poyneer, Lisa A.

AU - Pueyo, Laurent

AU - Rajan, Abhijith

AU - Rantakyrö, Fredrik T.

AU - Schneider, Adam C.

AU - Sivaramakrishnan, Anand

AU - Song, Inseok

AU - Soummer, Remi

AU - Thomas, Sandrine

AU - Kent Wallace, J.

AU - Ward-Duong, Kimberly

AU - Wiktorowicz, Sloane J.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

AB - The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

KW - circumstellar disks

KW - Data Cruncher.

KW - data processing

KW - exoplanets

KW - Gemini planet imager

KW - high contrast imaging

UR - http://www.scopus.com/inward/record.url?scp=85040764970&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85040764970&partnerID=8YFLogxK

U2 - 10.1117/1.JATIS.4.1.018002

DO - 10.1117/1.JATIS.4.1.018002

M3 - Review article

VL - 4

JO - Journal of Astronomical Telescopes, Instruments, and Systems

JF - Journal of Astronomical Telescopes, Instruments, and Systems

SN - 2329-4124

IS - 1

M1 - 018002

ER -