Active learning to overcome sample selection bias: Application to photometric variable star classification

Joseph W. Richards, Dan L. Starr, Henrik Brink, Adam A. Miller, Joshua S. Bloom, Nathaniel Butler, J. Berian James, James P. Long, John Rice

Research output: Contribution to journalArticle

32 Citations (Scopus)

Abstract

Despite the great promise of machine-learning algorithms to classify and predict astrophysical parameters for the vast numbers of astrophysical sources and transients observed in large-scale surveys, the peculiarities of the training data often manifest as strongly biased predictions on the data of interest. Typically, training sets are derived from historical surveys of brighter, more nearby objects than those from more extensive, deeper surveys (testing data). This sample selection bias can cause catastrophic errors in predictions on the testing data because (1) standard assumptions for machine-learned model selection procedures break down and (2) dense regions of testing space might be completely devoid of training data. We explore possible remedies to sample selection bias, including importance weighting, co-training, and active learning (AL). We argue that AL - where the data whose inclusion in the training set would most improve predictions on the testing set are queried for manual follow-up - is an effective approach and is appropriate for many astronomical applications. For a variable star classification problem on a well-studied set of stars from Hipparcos and Optical Gravitational Lensing Experiment, AL is the optimal method in terms of error rate on the testing data, beating the off-the-shelf classifier by 3.4% and the other proposed methods by at least 3.0%. To aid with manual labeling of variable stars, we developed a Web interface which allows for easy light curve visualization and querying of external databases. Finally, we apply AL to classify variable stars in the All Sky Automated Survey, finding dramatic improvement in our agreement with the ASAS Catalog of Variable Stars, from 65.5% to 79.5%, and a significant increase in the classifier's average confidence for the testing set, from 14.6% to 42.9%, after a few AL iterations.

Original languageEnglish (US)
Article number192
JournalAstrophysical Journal
Volume744
Issue number2
DOIs
StatePublished - Jan 10 2012
Externally publishedYes

Fingerprint

variable stars
learning
education
classifiers
astrophysics
predictions
prediction
Advanced Solid-State Array Spectroradiometer
machine learning
shelves
light curve
marking
catalogs
iteration
sky
confidence
breakdown
visualization
aid
inclusions

Keywords

  • methods: data analysis
  • methods: statistical
  • stars: variables: general
  • techniques: photometric

ASJC Scopus subject areas

  • Space and Planetary Science
  • Astronomy and Astrophysics

Cite this

Active learning to overcome sample selection bias : Application to photometric variable star classification. / Richards, Joseph W.; Starr, Dan L.; Brink, Henrik; Miller, Adam A.; Bloom, Joshua S.; Butler, Nathaniel; Berian James, J.; Long, James P.; Rice, John.

In: Astrophysical Journal, Vol. 744, No. 2, 192, 10.01.2012.

Research output: Contribution to journalArticle

Richards, JW, Starr, DL, Brink, H, Miller, AA, Bloom, JS, Butler, N, Berian James, J, Long, JP & Rice, J 2012, 'Active learning to overcome sample selection bias: Application to photometric variable star classification', Astrophysical Journal, vol. 744, no. 2, 192. https://doi.org/10.1088/0004-637X/744/2/192
Richards, Joseph W. ; Starr, Dan L. ; Brink, Henrik ; Miller, Adam A. ; Bloom, Joshua S. ; Butler, Nathaniel ; Berian James, J. ; Long, James P. ; Rice, John. / Active learning to overcome sample selection bias : Application to photometric variable star classification. In: Astrophysical Journal. 2012 ; Vol. 744, No. 2.
@article{6911c35b68814437befb557deecedabb,
title = "Active learning to overcome sample selection bias: Application to photometric variable star classification",
abstract = "Despite the great promise of machine-learning algorithms to classify and predict astrophysical parameters for the vast numbers of astrophysical sources and transients observed in large-scale surveys, the peculiarities of the training data often manifest as strongly biased predictions on the data of interest. Typically, training sets are derived from historical surveys of brighter, more nearby objects than those from more extensive, deeper surveys (testing data). This sample selection bias can cause catastrophic errors in predictions on the testing data because (1) standard assumptions for machine-learned model selection procedures break down and (2) dense regions of testing space might be completely devoid of training data. We explore possible remedies to sample selection bias, including importance weighting, co-training, and active learning (AL). We argue that AL - where the data whose inclusion in the training set would most improve predictions on the testing set are queried for manual follow-up - is an effective approach and is appropriate for many astronomical applications. For a variable star classification problem on a well-studied set of stars from Hipparcos and Optical Gravitational Lensing Experiment, AL is the optimal method in terms of error rate on the testing data, beating the off-the-shelf classifier by 3.4{\%} and the other proposed methods by at least 3.0{\%}. To aid with manual labeling of variable stars, we developed a Web interface which allows for easy light curve visualization and querying of external databases. Finally, we apply AL to classify variable stars in the All Sky Automated Survey, finding dramatic improvement in our agreement with the ASAS Catalog of Variable Stars, from 65.5{\%} to 79.5{\%}, and a significant increase in the classifier's average confidence for the testing set, from 14.6{\%} to 42.9{\%}, after a few AL iterations.",
keywords = "methods: data analysis, methods: statistical, stars: variables: general, techniques: photometric",
author = "Richards, {Joseph W.} and Starr, {Dan L.} and Henrik Brink and Miller, {Adam A.} and Bloom, {Joshua S.} and Nathaniel Butler and {Berian James}, J. and Long, {James P.} and John Rice",
year = "2012",
month = "1",
day = "10",
doi = "10.1088/0004-637X/744/2/192",
language = "English (US)",
volume = "744",
journal = "Astrophysical Journal",
issn = "0004-637X",
publisher = "IOP Publishing Ltd.",
number = "2",

}

TY - JOUR

T1 - Active learning to overcome sample selection bias

T2 - Application to photometric variable star classification

AU - Richards, Joseph W.

AU - Starr, Dan L.

AU - Brink, Henrik

AU - Miller, Adam A.

AU - Bloom, Joshua S.

AU - Butler, Nathaniel

AU - Berian James, J.

AU - Long, James P.

AU - Rice, John

PY - 2012/1/10

Y1 - 2012/1/10

N2 - Despite the great promise of machine-learning algorithms to classify and predict astrophysical parameters for the vast numbers of astrophysical sources and transients observed in large-scale surveys, the peculiarities of the training data often manifest as strongly biased predictions on the data of interest. Typically, training sets are derived from historical surveys of brighter, more nearby objects than those from more extensive, deeper surveys (testing data). This sample selection bias can cause catastrophic errors in predictions on the testing data because (1) standard assumptions for machine-learned model selection procedures break down and (2) dense regions of testing space might be completely devoid of training data. We explore possible remedies to sample selection bias, including importance weighting, co-training, and active learning (AL). We argue that AL - where the data whose inclusion in the training set would most improve predictions on the testing set are queried for manual follow-up - is an effective approach and is appropriate for many astronomical applications. For a variable star classification problem on a well-studied set of stars from Hipparcos and Optical Gravitational Lensing Experiment, AL is the optimal method in terms of error rate on the testing data, beating the off-the-shelf classifier by 3.4% and the other proposed methods by at least 3.0%. To aid with manual labeling of variable stars, we developed a Web interface which allows for easy light curve visualization and querying of external databases. Finally, we apply AL to classify variable stars in the All Sky Automated Survey, finding dramatic improvement in our agreement with the ASAS Catalog of Variable Stars, from 65.5% to 79.5%, and a significant increase in the classifier's average confidence for the testing set, from 14.6% to 42.9%, after a few AL iterations.

AB - Despite the great promise of machine-learning algorithms to classify and predict astrophysical parameters for the vast numbers of astrophysical sources and transients observed in large-scale surveys, the peculiarities of the training data often manifest as strongly biased predictions on the data of interest. Typically, training sets are derived from historical surveys of brighter, more nearby objects than those from more extensive, deeper surveys (testing data). This sample selection bias can cause catastrophic errors in predictions on the testing data because (1) standard assumptions for machine-learned model selection procedures break down and (2) dense regions of testing space might be completely devoid of training data. We explore possible remedies to sample selection bias, including importance weighting, co-training, and active learning (AL). We argue that AL - where the data whose inclusion in the training set would most improve predictions on the testing set are queried for manual follow-up - is an effective approach and is appropriate for many astronomical applications. For a variable star classification problem on a well-studied set of stars from Hipparcos and Optical Gravitational Lensing Experiment, AL is the optimal method in terms of error rate on the testing data, beating the off-the-shelf classifier by 3.4% and the other proposed methods by at least 3.0%. To aid with manual labeling of variable stars, we developed a Web interface which allows for easy light curve visualization and querying of external databases. Finally, we apply AL to classify variable stars in the All Sky Automated Survey, finding dramatic improvement in our agreement with the ASAS Catalog of Variable Stars, from 65.5% to 79.5%, and a significant increase in the classifier's average confidence for the testing set, from 14.6% to 42.9%, after a few AL iterations.

KW - methods: data analysis

KW - methods: statistical

KW - stars: variables: general

KW - techniques: photometric

UR - http://www.scopus.com/inward/record.url?scp=84555196171&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84555196171&partnerID=8YFLogxK

U2 - 10.1088/0004-637X/744/2/192

DO - 10.1088/0004-637X/744/2/192

M3 - Article

AN - SCOPUS:84555196171

VL - 744

JO - Astrophysical Journal

JF - Astrophysical Journal

SN - 0004-637X

IS - 2

M1 - 192

ER -