A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems

Pinghua Gong, Changshui Zhang, Zhaosong Lu, Jianhua Z. Huang, Jieping Ye

Research output: Chapter in Book/Report/Conference proceedingConference contribution

96 Citations (Scopus)

Abstract

Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-convex penalties remains a big challenge. A commonly used approach is the Multi-Stage (MS) convex relaxation (or DC programming), which relaxes the original non-convex problem to a sequence of convex problems. This approach is usually not very practical for large-scale problems because its computational cost is a multiple of solving a single convex problem. In this paper, we propose a General Iterative Shrinkage and Thresholding (GIST) algorithm to solve the nonconvex optimization problem for a large class of non-convex penalties. The GIST algorithm iteratively solves a proximal operator problem, which in turn has a closed-form solution for many commonly used penalties. At each outer iteration of the algorithm, we use a line search initialized by the Barzilai-Borwein (BB) rule that allows finding an appropriate step size quickly. The paper also presents a detailed convergence analysis of the GIST algorithm. The efficiency of the proposed algorithm is demonstrated by extensive experiments on large-scale data sets.

Original languageEnglish (US)
Title of host publication30th International Conference on Machine Learning, ICML 2013
PublisherInternational Machine Learning Society (IMLS)
Pages696-704
Number of pages9
EditionPART 1
StatePublished - 2013
Event30th International Conference on Machine Learning, ICML 2013 - Atlanta, GA, United States
Duration: Jun 16 2013Jun 21 2013

Other

Other30th International Conference on Machine Learning, ICML 2013
CountryUnited States
CityAtlanta, GA
Period6/16/136/21/13

Fingerprint

penalty
Mathematical operators
learning
programming
efficiency
Costs
experiment
Experiments
costs

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Sociology and Political Science

Cite this

Gong, P., Zhang, C., Lu, Z., Huang, J. Z., & Ye, J. (2013). A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems. In 30th International Conference on Machine Learning, ICML 2013 (PART 1 ed., pp. 696-704). International Machine Learning Society (IMLS).

A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems. / Gong, Pinghua; Zhang, Changshui; Lu, Zhaosong; Huang, Jianhua Z.; Ye, Jieping.

30th International Conference on Machine Learning, ICML 2013. PART 1. ed. International Machine Learning Society (IMLS), 2013. p. 696-704.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Gong, P, Zhang, C, Lu, Z, Huang, JZ & Ye, J 2013, A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems. in 30th International Conference on Machine Learning, ICML 2013. PART 1 edn, International Machine Learning Society (IMLS), pp. 696-704, 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, United States, 6/16/13.
Gong P, Zhang C, Lu Z, Huang JZ, Ye J. A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems. In 30th International Conference on Machine Learning, ICML 2013. PART 1 ed. International Machine Learning Society (IMLS). 2013. p. 696-704
Gong, Pinghua ; Zhang, Changshui ; Lu, Zhaosong ; Huang, Jianhua Z. ; Ye, Jieping. / A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems. 30th International Conference on Machine Learning, ICML 2013. PART 1. ed. International Machine Learning Society (IMLS), 2013. pp. 696-704
@inproceedings{3c6f3cc0f2024e1aad54a62a36e452f1,
title = "A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems",
abstract = "Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-convex penalties remains a big challenge. A commonly used approach is the Multi-Stage (MS) convex relaxation (or DC programming), which relaxes the original non-convex problem to a sequence of convex problems. This approach is usually not very practical for large-scale problems because its computational cost is a multiple of solving a single convex problem. In this paper, we propose a General Iterative Shrinkage and Thresholding (GIST) algorithm to solve the nonconvex optimization problem for a large class of non-convex penalties. The GIST algorithm iteratively solves a proximal operator problem, which in turn has a closed-form solution for many commonly used penalties. At each outer iteration of the algorithm, we use a line search initialized by the Barzilai-Borwein (BB) rule that allows finding an appropriate step size quickly. The paper also presents a detailed convergence analysis of the GIST algorithm. The efficiency of the proposed algorithm is demonstrated by extensive experiments on large-scale data sets.",
author = "Pinghua Gong and Changshui Zhang and Zhaosong Lu and Huang, {Jianhua Z.} and Jieping Ye",
year = "2013",
language = "English (US)",
pages = "696--704",
booktitle = "30th International Conference on Machine Learning, ICML 2013",
publisher = "International Machine Learning Society (IMLS)",
edition = "PART 1",

}

TY - GEN

T1 - A General Iterative Shrinkage and Thresholding algorithm for non-convex regularized optimization problems

AU - Gong, Pinghua

AU - Zhang, Changshui

AU - Lu, Zhaosong

AU - Huang, Jianhua Z.

AU - Ye, Jieping

PY - 2013

Y1 - 2013

N2 - Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-convex penalties remains a big challenge. A commonly used approach is the Multi-Stage (MS) convex relaxation (or DC programming), which relaxes the original non-convex problem to a sequence of convex problems. This approach is usually not very practical for large-scale problems because its computational cost is a multiple of solving a single convex problem. In this paper, we propose a General Iterative Shrinkage and Thresholding (GIST) algorithm to solve the nonconvex optimization problem for a large class of non-convex penalties. The GIST algorithm iteratively solves a proximal operator problem, which in turn has a closed-form solution for many commonly used penalties. At each outer iteration of the algorithm, we use a line search initialized by the Barzilai-Borwein (BB) rule that allows finding an appropriate step size quickly. The paper also presents a detailed convergence analysis of the GIST algorithm. The efficiency of the proposed algorithm is demonstrated by extensive experiments on large-scale data sets.

AB - Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-convex penalties remains a big challenge. A commonly used approach is the Multi-Stage (MS) convex relaxation (or DC programming), which relaxes the original non-convex problem to a sequence of convex problems. This approach is usually not very practical for large-scale problems because its computational cost is a multiple of solving a single convex problem. In this paper, we propose a General Iterative Shrinkage and Thresholding (GIST) algorithm to solve the nonconvex optimization problem for a large class of non-convex penalties. The GIST algorithm iteratively solves a proximal operator problem, which in turn has a closed-form solution for many commonly used penalties. At each outer iteration of the algorithm, we use a line search initialized by the Barzilai-Borwein (BB) rule that allows finding an appropriate step size quickly. The paper also presents a detailed convergence analysis of the GIST algorithm. The efficiency of the proposed algorithm is demonstrated by extensive experiments on large-scale data sets.

UR - http://www.scopus.com/inward/record.url?scp=84897545592&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84897545592&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84897545592

SP - 696

EP - 704

BT - 30th International Conference on Machine Learning, ICML 2013

PB - International Machine Learning Society (IMLS)

ER -