Automated data processing architecture for the gemini planet imager exoplanet survey

Jason J. Wang; Marshall D. Perrin; Dmitry Savransky; Pauline Arriaga; Jeffrey K. Chilcote; Robert J. De Rosa; Maxwell A. Millar-Blanchaer; Christian Marois; Julien Rameau; Schuyler G. Wolff; Jacob Shapiro; Jean Baptiste Ruffio; Jérôme Maire; Franck Marchis; James R. Graham; Bruce Macintosh; S. Mark Ammons; Vanessa P. Bailey; Travis S. Barman; Sebastian Bruzzone; Joanna Bulger; Tara Cotton; René Doyon; Gaspard Duchêne; Michael P. Fitzgerald; Katherine B. Follette; Stephen Goodsell; Alexandra Z. Greenbaum; Pascale Hibon; Li Wei Hung; Patrick Ingraham; Paul Kalas; Quinn M. Konopacky; James E. Larkin; Mark S. Marley; Stanimir Metchev; Eric L. Nielsen; Rebecca Oppenheimer; David W. Palmer; Jennifer Patience; Lisa A. Poyneer; Laurent Pueyo; Abhijith Rajan; Fredrik T. Rantakyrö; Adam C. Schneider; Anand Sivaramakrishnan; Inseok Song; Remi Soummer; Sandrine Thomas; J. Kent Wallace; Kimberly Ward-Duong; Sloane J. Wiktorowicz

doi:10.1117/1.JATIS.4.1.018002

Automated data processing architecture for the gemini planet imager exoplanet survey

Jason J. Wang, Marshall D. Perrin, Dmitry Savransky, Pauline Arriaga, Jeffrey K. Chilcote, Robert J. De Rosa, Maxwell A. Millar-Blanchaer, Christian Marois, Julien Rameau, Schuyler G. Wolff, Jacob Shapiro, Jean Baptiste Ruffio, Jérôme Maire, Franck Marchis, James R. Graham, Bruce Macintosh, S. Mark Ammons, Vanessa P. Bailey, Travis S. Barman, Sebastian BruzzoneJoanna Bulger, Tara Cotton, René Doyon, Gaspard Duchêne, Michael P. Fitzgerald, Katherine B. Follette, Stephen Goodsell, Alexandra Z. Greenbaum, Pascale Hibon, Li Wei Hung, Patrick Ingraham, Paul Kalas, Quinn M. Konopacky, James E. Larkin, Mark S. Marley, Stanimir Metchev, Eric L. Nielsen, Rebecca Oppenheimer, David W. Palmer, Jennifer Patience, Lisa A. Poyneer, Laurent Pueyo, Abhijith Rajan, Fredrik T. Rantakyrö, Adam C. Schneider, Anand Sivaramakrishnan, Inseok Song, Remi Soummer, Sandrine Thomas, J. Kent Wallace, Kimberly Ward-Duong, Sloane J. Wiktorowicz

Earth and Space Exploration, School of (SESE)

Research output: Contribution to journal › Review article › peer-review

21 Scopus citations

Abstract

The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

Original language	English (US)
Article number	018002
Journal	Journal of Astronomical Telescopes, Instruments, and Systems
Volume	4
Issue number	1
DOIs	https://doi.org/10.1117/1.JATIS.4.1.018002
State	Published - Jan 1 2018

Keywords

Data Cruncher.
Gemini planet imager
circumstellar disks
data processing
exoplanets
high contrast imaging

ASJC Scopus subject areas

Electronic, Optical and Magnetic Materials
Control and Systems Engineering
Instrumentation
Astronomy and Astrophysics
Mechanical Engineering
Space and Planetary Science

Access to Document

10.1117/1.JATIS.4.1.018002

Cite this

Wang, J. J., Perrin, M. D., Savransky, D., Arriaga, P., Chilcote, J. K., De Rosa, R. J., Millar-Blanchaer, M. A., Marois, C., Rameau, J., Wolff, S. G., Shapiro, J., Ruffio, J. B., Maire, J., Marchis, F., Graham, J. R., Macintosh, B., Mark Ammons, S., Bailey, V. P., Barman, T. S., ... Wiktorowicz, S. J. (2018). Automated data processing architecture for the gemini planet imager exoplanet survey. Journal of Astronomical Telescopes, Instruments, and Systems, 4(1), Article 018002. https://doi.org/10.1117/1.JATIS.4.1.018002

Wang, JJ, Perrin, MD, Savransky, D, Arriaga, P, Chilcote, JK, De Rosa, RJ, Millar-Blanchaer, MA, Marois, C, Rameau, J, Wolff, SG, Shapiro, J, Ruffio, JB, Maire, J, Marchis, F, Graham, JR, Macintosh, B, Mark Ammons, S, Bailey, VP, Barman, TS, Bruzzone, S, Bulger, J, Cotton, T, Doyon, R, Duchêne, G, Fitzgerald, MP, Follette, KB, Goodsell, S, Greenbaum, AZ, Hibon, P, Hung, LW, Ingraham, P, Kalas, P, Konopacky, QM, Larkin, JE, Marley, MS, Metchev, S, Nielsen, EL, Oppenheimer, R, Palmer, DW, Patience, J, Poyneer, LA, Pueyo, L, Rajan, A, Rantakyrö, FT, Schneider, AC, Sivaramakrishnan, A, Song, I, Soummer, R, Thomas, S, Kent Wallace, J, Ward-Duong, K & Wiktorowicz, SJ 2018, 'Automated data processing architecture for the gemini planet imager exoplanet survey', Journal of Astronomical Telescopes, Instruments, and Systems, vol. 4, no. 1, 018002. https://doi.org/10.1117/1.JATIS.4.1.018002

@article{6e84aad3f111415ca8382e6b92963c8f,

title = "Automated data processing architecture for the gemini planet imager exoplanet survey",

abstract = "The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.",

keywords = "Data Cruncher., Gemini planet imager, circumstellar disks, data processing, exoplanets, high contrast imaging",

author = "Wang, {Jason J.} and Perrin, {Marshall D.} and Dmitry Savransky and Pauline Arriaga and Chilcote, {Jeffrey K.} and {De Rosa}, {Robert J.} and Millar-Blanchaer, {Maxwell A.} and Christian Marois and Julien Rameau and Wolff, {Schuyler G.} and Jacob Shapiro and Ruffio, {Jean Baptiste} and J{\'e}r{\^o}me Maire and Franck Marchis and Graham, {James R.} and Bruce Macintosh and {Mark Ammons}, S. and Bailey, {Vanessa P.} and Barman, {Travis S.} and Sebastian Bruzzone and Joanna Bulger and Tara Cotton and Ren{\'e} Doyon and Gaspard Duch{\^e}ne and Fitzgerald, {Michael P.} and Follette, {Katherine B.} and Stephen Goodsell and Greenbaum, {Alexandra Z.} and Pascale Hibon and Hung, {Li Wei} and Patrick Ingraham and Paul Kalas and Konopacky, {Quinn M.} and Larkin, {James E.} and Marley, {Mark S.} and Stanimir Metchev and Nielsen, {Eric L.} and Rebecca Oppenheimer and Palmer, {David W.} and Jennifer Patience and Poyneer, {Lisa A.} and Laurent Pueyo and Abhijith Rajan and Rantakyr{\"o}, {Fredrik T.} and Schneider, {Adam C.} and Anand Sivaramakrishnan and Inseok Song and Remi Soummer and Sandrine Thomas and {Kent Wallace}, J. and Kimberly Ward-Duong and Wiktorowicz, {Sloane J.}",

note = "Funding Information: The Gemini Observatory is operated by the Association of Universities for Research in Astronomy, Inc., under a co-operative agreement with the NSF on behalf of the Gemini partnership: the National Science Foundation (United States), the National Research Council (Canada), CONICYT (Chile), the Australian Research Council (Australia), Minist{\'e}rio da Ci{\'e}ncia, Tecnologia e Inova{\c c}āo (Brazil), and Ministerio de Ciencia, Tecnolog{\'i}a e Innovaci{\'o}n Productiva (Argentina). This work was supported in part by the NASA{\textquoteright}s NExSS program, Grant No. NNX15AD95G. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation Grant No. ACI-1548562. Support for MMB{\textquoteright}s work was provided by NASA through Hubble Fellowship Grant #51378.01-A awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., for NASA, under Contract No. NAS5-26555. Portions of this work were performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344. We thank the two anonymous referees for their thorough review and suggestions. We thank the NERSC and SDSC staff for their helpful support in providing computational resources and technical help. We also thank Barry Mieny for granting us permission to use the database icon used in Fig. 1. This research made use of astropy, a community-developed core Python package for astronomy.34 Publisher Copyright: {\textcopyright} 2018 Society of Photo-Optical Instrumentation Engineers (SPIE).",

year = "2018",

month = jan,

day = "1",

doi = "10.1117/1.JATIS.4.1.018002",

language = "English (US)",

volume = "4",

journal = "Journal of Astronomical Telescopes, Instruments, and Systems",

issn = "2329-4124",

publisher = "SPIE",

number = "1",

}

TY - JOUR

T1 - Automated data processing architecture for the gemini planet imager exoplanet survey

AU - Wang, Jason J.

AU - Perrin, Marshall D.

AU - Savransky, Dmitry

AU - Arriaga, Pauline

AU - Chilcote, Jeffrey K.

AU - De Rosa, Robert J.

AU - Millar-Blanchaer, Maxwell A.

AU - Marois, Christian

AU - Rameau, Julien

AU - Wolff, Schuyler G.

AU - Shapiro, Jacob

AU - Ruffio, Jean Baptiste

AU - Maire, Jérôme

AU - Marchis, Franck

AU - Graham, James R.

AU - Macintosh, Bruce

AU - Mark Ammons, S.

AU - Bailey, Vanessa P.

AU - Barman, Travis S.

AU - Bruzzone, Sebastian

AU - Bulger, Joanna

AU - Cotton, Tara

AU - Doyon, René

AU - Duchêne, Gaspard

AU - Fitzgerald, Michael P.

AU - Follette, Katherine B.

AU - Goodsell, Stephen

AU - Greenbaum, Alexandra Z.

AU - Hibon, Pascale

AU - Hung, Li Wei

AU - Ingraham, Patrick

AU - Kalas, Paul

AU - Konopacky, Quinn M.

AU - Larkin, James E.

AU - Marley, Mark S.

AU - Metchev, Stanimir

AU - Nielsen, Eric L.

AU - Oppenheimer, Rebecca

AU - Palmer, David W.

AU - Patience, Jennifer

AU - Poyneer, Lisa A.

AU - Pueyo, Laurent

AU - Rajan, Abhijith

AU - Rantakyrö, Fredrik T.

AU - Schneider, Adam C.

AU - Sivaramakrishnan, Anand

AU - Song, Inseok

AU - Soummer, Remi

AU - Thomas, Sandrine

AU - Kent Wallace, J.

AU - Ward-Duong, Kimberly

AU - Wiktorowicz, Sloane J.

N1 - Funding Information: The Gemini Observatory is operated by the Association of Universities for Research in Astronomy, Inc., under a co-operative agreement with the NSF on behalf of the Gemini partnership: the National Science Foundation (United States), the National Research Council (Canada), CONICYT (Chile), the Australian Research Council (Australia), Ministério da Ciéncia, Tecnologia e Inovaçāo (Brazil), and Ministerio de Ciencia, Tecnología e Innovación Productiva (Argentina). This work was supported in part by the NASA’s NExSS program, Grant No. NNX15AD95G. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation Grant No. ACI-1548562. Support for MMB’s work was provided by NASA through Hubble Fellowship Grant #51378.01-A awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., for NASA, under Contract No. NAS5-26555. Portions of this work were performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344. We thank the two anonymous referees for their thorough review and suggestions. We thank the NERSC and SDSC staff for their helpful support in providing computational resources and technical help. We also thank Barry Mieny for granting us permission to use the database icon used in Fig. 1. This research made use of astropy, a community-developed core Python package for astronomy.34 Publisher Copyright: © 2018 Society of Photo-Optical Instrumentation Engineers (SPIE).

PY - 2018/1/1

Y1 - 2018/1/1

N2 - The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

AB - The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-Time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

KW - Data Cruncher.

KW - Gemini planet imager

KW - circumstellar disks

KW - data processing

KW - exoplanets

KW - high contrast imaging

UR - http://www.scopus.com/inward/record.url?scp=85040764970&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85040764970&partnerID=8YFLogxK

U2 - 10.1117/1.JATIS.4.1.018002

DO - 10.1117/1.JATIS.4.1.018002

M3 - Review article

AN - SCOPUS:85040764970

SN - 2329-4124

VL - 4

JO - Journal of Astronomical Telescopes, Instruments, and Systems

JF - Journal of Astronomical Telescopes, Instruments, and Systems

IS - 1

M1 - 018002

ER -

Automated data processing architecture for the gemini planet imager exoplanet survey

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this