Mission: Ultra large-scale feature selection using count-sketches

Amirali Aghazadeh, Ryan Spring, Daniel Lejeune, Gautam Dasarathy, Anshuniali Shrivastava, Richard G. Baraniuk

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Feature selection is an important challenge in machine learning. Il plays a crucial role in the ex- plainabiliry of machine-driven decisions that are rapidly permeating throughout modem society. Unfortunately, the explosion in the size and dimensionality of real-world datasets poses a severe challenge to standard feature selection algorithms. Today, it is not uncommon for datasets to have billions of dimensions. At such scale, even storing the feature vector is impossible, causing most existing feature selection methods to fail. Workarounds like feature hashing, a standard ap-proach to large-scale machine learning, helps with the computational feasibility, but at the cost of losing the interpretability of features. In this paper, we present MISSION, a novel framework for ultra large-scale feature selection that performs stochastic gradient descent while maintaining an efficient representation of the features in memory using a Count-Sketch data structure. MISSION retains the simplicity of feature hashing without sacrificing the interpretability of the features while using only 0(log2p) working memory. We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.

Original languageEnglish (US)
Title of host publication35th International Conference on Machine Learning, ICML 2018
EditorsAndreas Krause, Jennifer Dy
PublisherInternational Machine Learning Society (IMLS)
Pages143-154
Number of pages12
ISBN (Electronic)9781510867963
StatePublished - Jan 1 2018
Externally publishedYes
Event35th International Conference on Machine Learning, ICML 2018 - Stockholm, Sweden
Duration: Jul 10 2018Jul 15 2018

Publication series

Name35th International Conference on Machine Learning, ICML 2018
Volume1

Conference

Conference35th International Conference on Machine Learning, ICML 2018
CountrySweden
CityStockholm
Period7/10/187/15/18

Fingerprint

Feature extraction
Learning systems
Data storage equipment
Modems
Explosions
Data structures

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Human-Computer Interaction
  • Software

Cite this

Aghazadeh, A., Spring, R., Lejeune, D., Dasarathy, G., Shrivastava, A., & Baraniuk, R. G. (2018). Mission: Ultra large-scale feature selection using count-sketches. In A. Krause, & J. Dy (Eds.), 35th International Conference on Machine Learning, ICML 2018 (pp. 143-154). (35th International Conference on Machine Learning, ICML 2018; Vol. 1). International Machine Learning Society (IMLS).

Mission : Ultra large-scale feature selection using count-sketches. / Aghazadeh, Amirali; Spring, Ryan; Lejeune, Daniel; Dasarathy, Gautam; Shrivastava, Anshuniali; Baraniuk, Richard G.

35th International Conference on Machine Learning, ICML 2018. ed. / Andreas Krause; Jennifer Dy. International Machine Learning Society (IMLS), 2018. p. 143-154 (35th International Conference on Machine Learning, ICML 2018; Vol. 1).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Aghazadeh, A, Spring, R, Lejeune, D, Dasarathy, G, Shrivastava, A & Baraniuk, RG 2018, Mission: Ultra large-scale feature selection using count-sketches. in A Krause & J Dy (eds), 35th International Conference on Machine Learning, ICML 2018. 35th International Conference on Machine Learning, ICML 2018, vol. 1, International Machine Learning Society (IMLS), pp. 143-154, 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, 7/10/18.
Aghazadeh A, Spring R, Lejeune D, Dasarathy G, Shrivastava A, Baraniuk RG. Mission: Ultra large-scale feature selection using count-sketches. In Krause A, Dy J, editors, 35th International Conference on Machine Learning, ICML 2018. International Machine Learning Society (IMLS). 2018. p. 143-154. (35th International Conference on Machine Learning, ICML 2018).
Aghazadeh, Amirali ; Spring, Ryan ; Lejeune, Daniel ; Dasarathy, Gautam ; Shrivastava, Anshuniali ; Baraniuk, Richard G. / Mission : Ultra large-scale feature selection using count-sketches. 35th International Conference on Machine Learning, ICML 2018. editor / Andreas Krause ; Jennifer Dy. International Machine Learning Society (IMLS), 2018. pp. 143-154 (35th International Conference on Machine Learning, ICML 2018).
@inproceedings{96fa06e365ca4cae876db3198529dba3,
title = "Mission: Ultra large-scale feature selection using count-sketches",
abstract = "Feature selection is an important challenge in machine learning. Il plays a crucial role in the ex- plainabiliry of machine-driven decisions that are rapidly permeating throughout modem society. Unfortunately, the explosion in the size and dimensionality of real-world datasets poses a severe challenge to standard feature selection algorithms. Today, it is not uncommon for datasets to have billions of dimensions. At such scale, even storing the feature vector is impossible, causing most existing feature selection methods to fail. Workarounds like feature hashing, a standard ap-proach to large-scale machine learning, helps with the computational feasibility, but at the cost of losing the interpretability of features. In this paper, we present MISSION, a novel framework for ultra large-scale feature selection that performs stochastic gradient descent while maintaining an efficient representation of the features in memory using a Count-Sketch data structure. MISSION retains the simplicity of feature hashing without sacrificing the interpretability of the features while using only 0(log2p) working memory. We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.",
author = "Amirali Aghazadeh and Ryan Spring and Daniel Lejeune and Gautam Dasarathy and Anshuniali Shrivastava and Baraniuk, {Richard G.}",
year = "2018",
month = "1",
day = "1",
language = "English (US)",
series = "35th International Conference on Machine Learning, ICML 2018",
publisher = "International Machine Learning Society (IMLS)",
pages = "143--154",
editor = "Andreas Krause and Jennifer Dy",
booktitle = "35th International Conference on Machine Learning, ICML 2018",

}

TY - GEN

T1 - Mission

T2 - Ultra large-scale feature selection using count-sketches

AU - Aghazadeh, Amirali

AU - Spring, Ryan

AU - Lejeune, Daniel

AU - Dasarathy, Gautam

AU - Shrivastava, Anshuniali

AU - Baraniuk, Richard G.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Feature selection is an important challenge in machine learning. Il plays a crucial role in the ex- plainabiliry of machine-driven decisions that are rapidly permeating throughout modem society. Unfortunately, the explosion in the size and dimensionality of real-world datasets poses a severe challenge to standard feature selection algorithms. Today, it is not uncommon for datasets to have billions of dimensions. At such scale, even storing the feature vector is impossible, causing most existing feature selection methods to fail. Workarounds like feature hashing, a standard ap-proach to large-scale machine learning, helps with the computational feasibility, but at the cost of losing the interpretability of features. In this paper, we present MISSION, a novel framework for ultra large-scale feature selection that performs stochastic gradient descent while maintaining an efficient representation of the features in memory using a Count-Sketch data structure. MISSION retains the simplicity of feature hashing without sacrificing the interpretability of the features while using only 0(log2p) working memory. We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.

AB - Feature selection is an important challenge in machine learning. Il plays a crucial role in the ex- plainabiliry of machine-driven decisions that are rapidly permeating throughout modem society. Unfortunately, the explosion in the size and dimensionality of real-world datasets poses a severe challenge to standard feature selection algorithms. Today, it is not uncommon for datasets to have billions of dimensions. At such scale, even storing the feature vector is impossible, causing most existing feature selection methods to fail. Workarounds like feature hashing, a standard ap-proach to large-scale machine learning, helps with the computational feasibility, but at the cost of losing the interpretability of features. In this paper, we present MISSION, a novel framework for ultra large-scale feature selection that performs stochastic gradient descent while maintaining an efficient representation of the features in memory using a Count-Sketch data structure. MISSION retains the simplicity of feature hashing without sacrificing the interpretability of the features while using only 0(log2p) working memory. We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.

UR - http://www.scopus.com/inward/record.url?scp=85057251294&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057251294&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85057251294

T3 - 35th International Conference on Machine Learning, ICML 2018

SP - 143

EP - 154

BT - 35th International Conference on Machine Learning, ICML 2018

A2 - Krause, Andreas

A2 - Dy, Jennifer

PB - International Machine Learning Society (IMLS)

ER -