Efficient methods for overlapping group lasso

Lei Yuan, Jun Liu, Jieping Ye

Research output: Contribution to journalArticle

44 Citations (Scopus)

Abstract

The group Lasso is an extension of the Lasso for feature selection on (predefined) nonoverlapping groups of features. The nonoverlapping group structure limits its applicability in practice. There have been several recent attempts to study a more general formulation where groups of features are given, potentially with overlaps between the groups. The resulting optimization is, however, much more challenging to solve due to the group overlaps. In this paper, we consider the efficient optimization of the overlapping group Lasso penalized problem. We reveal several key properties of the proximal operator associated with the overlapping group Lasso, and compute the proximal operator by solving the smooth and convex dual problem, which allows the use of the gradient descent type of algorithms for the optimization. Our methods and theoretical results are then generalized to tackle the general overlapping group Lasso formulation based on the (\ellq) norm. We further extend our algorithm to solve a nonconvex overlapping group Lasso formulation based on the capped norm regularization, which reduces the estimation bias introduced by the convex penalty. We have performed empirical evaluations using both a synthetic and the breast cancer gene expression dataset, which consists of 8,141 genes organized into (overlapping) gene sets. Experimental results show that the proposed algorithm is more efficient than existing state-of-the-art algorithms. Results also demonstrate the effectiveness of the nonconvex formulation for overlapping group Lasso.

Original languageEnglish (US)
Article number6409353
Pages (from-to)2104-2116
Number of pages13
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume35
Issue number9
DOIs
StatePublished - 2013

Fingerprint

Lasso
Overlapping
Genes
Gene expression
Feature extraction
Formulation
Overlapping Genes
Neoplasm Genes
Optimization
Overlap
alachlor
Gene
Norm
Gradient Descent
Breast Neoplasms
Dual Problem
Gene Expression
Operator
Breast Cancer
Feature Selection

Keywords

  • difference of convex programming
  • overlapping group Lasso
  • proximal operator
  • Sparse learning

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Software
  • Computational Theory and Mathematics
  • Applied Mathematics
  • Medicine(all)

Cite this

Efficient methods for overlapping group lasso. / Yuan, Lei; Liu, Jun; Ye, Jieping.

In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, No. 9, 6409353, 2013, p. 2104-2116.

Research output: Contribution to journalArticle

Yuan, Lei ; Liu, Jun ; Ye, Jieping. / Efficient methods for overlapping group lasso. In: IEEE Transactions on Pattern Analysis and Machine Intelligence. 2013 ; Vol. 35, No. 9. pp. 2104-2116.
@article{5fc5fc356cc049ba9dddd2d8979cee34,
title = "Efficient methods for overlapping group lasso",
abstract = "The group Lasso is an extension of the Lasso for feature selection on (predefined) nonoverlapping groups of features. The nonoverlapping group structure limits its applicability in practice. There have been several recent attempts to study a more general formulation where groups of features are given, potentially with overlaps between the groups. The resulting optimization is, however, much more challenging to solve due to the group overlaps. In this paper, we consider the efficient optimization of the overlapping group Lasso penalized problem. We reveal several key properties of the proximal operator associated with the overlapping group Lasso, and compute the proximal operator by solving the smooth and convex dual problem, which allows the use of the gradient descent type of algorithms for the optimization. Our methods and theoretical results are then generalized to tackle the general overlapping group Lasso formulation based on the (\ellq) norm. We further extend our algorithm to solve a nonconvex overlapping group Lasso formulation based on the capped norm regularization, which reduces the estimation bias introduced by the convex penalty. We have performed empirical evaluations using both a synthetic and the breast cancer gene expression dataset, which consists of 8,141 genes organized into (overlapping) gene sets. Experimental results show that the proposed algorithm is more efficient than existing state-of-the-art algorithms. Results also demonstrate the effectiveness of the nonconvex formulation for overlapping group Lasso.",
keywords = "difference of convex programming, overlapping group Lasso, proximal operator, Sparse learning",
author = "Lei Yuan and Jun Liu and Jieping Ye",
year = "2013",
doi = "10.1109/TPAMI.2013.17",
language = "English (US)",
volume = "35",
pages = "2104--2116",
journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence",
issn = "0162-8828",
publisher = "IEEE Computer Society",
number = "9",

}

TY - JOUR

T1 - Efficient methods for overlapping group lasso

AU - Yuan, Lei

AU - Liu, Jun

AU - Ye, Jieping

PY - 2013

Y1 - 2013

N2 - The group Lasso is an extension of the Lasso for feature selection on (predefined) nonoverlapping groups of features. The nonoverlapping group structure limits its applicability in practice. There have been several recent attempts to study a more general formulation where groups of features are given, potentially with overlaps between the groups. The resulting optimization is, however, much more challenging to solve due to the group overlaps. In this paper, we consider the efficient optimization of the overlapping group Lasso penalized problem. We reveal several key properties of the proximal operator associated with the overlapping group Lasso, and compute the proximal operator by solving the smooth and convex dual problem, which allows the use of the gradient descent type of algorithms for the optimization. Our methods and theoretical results are then generalized to tackle the general overlapping group Lasso formulation based on the (\ellq) norm. We further extend our algorithm to solve a nonconvex overlapping group Lasso formulation based on the capped norm regularization, which reduces the estimation bias introduced by the convex penalty. We have performed empirical evaluations using both a synthetic and the breast cancer gene expression dataset, which consists of 8,141 genes organized into (overlapping) gene sets. Experimental results show that the proposed algorithm is more efficient than existing state-of-the-art algorithms. Results also demonstrate the effectiveness of the nonconvex formulation for overlapping group Lasso.

AB - The group Lasso is an extension of the Lasso for feature selection on (predefined) nonoverlapping groups of features. The nonoverlapping group structure limits its applicability in practice. There have been several recent attempts to study a more general formulation where groups of features are given, potentially with overlaps between the groups. The resulting optimization is, however, much more challenging to solve due to the group overlaps. In this paper, we consider the efficient optimization of the overlapping group Lasso penalized problem. We reveal several key properties of the proximal operator associated with the overlapping group Lasso, and compute the proximal operator by solving the smooth and convex dual problem, which allows the use of the gradient descent type of algorithms for the optimization. Our methods and theoretical results are then generalized to tackle the general overlapping group Lasso formulation based on the (\ellq) norm. We further extend our algorithm to solve a nonconvex overlapping group Lasso formulation based on the capped norm regularization, which reduces the estimation bias introduced by the convex penalty. We have performed empirical evaluations using both a synthetic and the breast cancer gene expression dataset, which consists of 8,141 genes organized into (overlapping) gene sets. Experimental results show that the proposed algorithm is more efficient than existing state-of-the-art algorithms. Results also demonstrate the effectiveness of the nonconvex formulation for overlapping group Lasso.

KW - difference of convex programming

KW - overlapping group Lasso

KW - proximal operator

KW - Sparse learning

UR - http://www.scopus.com/inward/record.url?scp=84880854914&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84880854914&partnerID=8YFLogxK

U2 - 10.1109/TPAMI.2013.17

DO - 10.1109/TPAMI.2013.17

M3 - Article

C2 - 23868773

AN - SCOPUS:84880854914

VL - 35

SP - 2104

EP - 2116

JO - IEEE Transactions on Pattern Analysis and Machine Intelligence

JF - IEEE Transactions on Pattern Analysis and Machine Intelligence

SN - 0162-8828

IS - 9

M1 - 6409353

ER -