An accelerated gradient method for trace norm minimization

Shuiwang Ji, Jieping Ye

Research output: Chapter in Book/Report/Conference proceedingConference contribution

315 Citations (Scopus)

Abstract

We consider the minimization of a smooth loss function regularized by the trace norm of the matrix variable. Such formulation finds applications in many machine learning tasks including multi-task learning, matrix classification, and matrix completion. The standard semidefinite programming formulation for this problem is computationally expensive. In addition, due to the non-smooth nature of the trace norm, the optimal first-order black-box method for solving such class of problems converges as O(1/√k), where k is the iteration counter. In this paper, we exploit the special structure of the trace norm, based on which we propose an extended gradient algorithm that converges as O(1/k). We further propose an accelerated gradient algorithm, which achieves the optimal convergence rate of O(1/k2) for smooth problems. Experiments on multi-task learning problems demonstrate the efficiency of the proposed algorithms.

Original languageEnglish (US)
Title of host publicationProceedings of the 26th International Conference On Machine Learning, ICML 2009
Pages457-464
Number of pages8
StatePublished - 2009
Event26th International Conference On Machine Learning, ICML 2009 - Montreal, QC, Canada
Duration: Jun 14 2009Jun 18 2009

Other

Other26th International Conference On Machine Learning, ICML 2009
CountryCanada
CityMontreal, QC
Period6/14/096/18/09

Fingerprint

Gradient methods
Learning systems
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Software

Cite this

Ji, S., & Ye, J. (2009). An accelerated gradient method for trace norm minimization. In Proceedings of the 26th International Conference On Machine Learning, ICML 2009 (pp. 457-464)

An accelerated gradient method for trace norm minimization. / Ji, Shuiwang; Ye, Jieping.

Proceedings of the 26th International Conference On Machine Learning, ICML 2009. 2009. p. 457-464.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ji, S & Ye, J 2009, An accelerated gradient method for trace norm minimization. in Proceedings of the 26th International Conference On Machine Learning, ICML 2009. pp. 457-464, 26th International Conference On Machine Learning, ICML 2009, Montreal, QC, Canada, 6/14/09.
Ji S, Ye J. An accelerated gradient method for trace norm minimization. In Proceedings of the 26th International Conference On Machine Learning, ICML 2009. 2009. p. 457-464
Ji, Shuiwang ; Ye, Jieping. / An accelerated gradient method for trace norm minimization. Proceedings of the 26th International Conference On Machine Learning, ICML 2009. 2009. pp. 457-464
@inproceedings{5a03efc14f3d438dbe4d7b2055d91b7a,
title = "An accelerated gradient method for trace norm minimization",
abstract = "We consider the minimization of a smooth loss function regularized by the trace norm of the matrix variable. Such formulation finds applications in many machine learning tasks including multi-task learning, matrix classification, and matrix completion. The standard semidefinite programming formulation for this problem is computationally expensive. In addition, due to the non-smooth nature of the trace norm, the optimal first-order black-box method for solving such class of problems converges as O(1/√k), where k is the iteration counter. In this paper, we exploit the special structure of the trace norm, based on which we propose an extended gradient algorithm that converges as O(1/k). We further propose an accelerated gradient algorithm, which achieves the optimal convergence rate of O(1/k2) for smooth problems. Experiments on multi-task learning problems demonstrate the efficiency of the proposed algorithms.",
author = "Shuiwang Ji and Jieping Ye",
year = "2009",
language = "English (US)",
isbn = "9781605585161",
pages = "457--464",
booktitle = "Proceedings of the 26th International Conference On Machine Learning, ICML 2009",

}

TY - GEN

T1 - An accelerated gradient method for trace norm minimization

AU - Ji, Shuiwang

AU - Ye, Jieping

PY - 2009

Y1 - 2009

N2 - We consider the minimization of a smooth loss function regularized by the trace norm of the matrix variable. Such formulation finds applications in many machine learning tasks including multi-task learning, matrix classification, and matrix completion. The standard semidefinite programming formulation for this problem is computationally expensive. In addition, due to the non-smooth nature of the trace norm, the optimal first-order black-box method for solving such class of problems converges as O(1/√k), where k is the iteration counter. In this paper, we exploit the special structure of the trace norm, based on which we propose an extended gradient algorithm that converges as O(1/k). We further propose an accelerated gradient algorithm, which achieves the optimal convergence rate of O(1/k2) for smooth problems. Experiments on multi-task learning problems demonstrate the efficiency of the proposed algorithms.

AB - We consider the minimization of a smooth loss function regularized by the trace norm of the matrix variable. Such formulation finds applications in many machine learning tasks including multi-task learning, matrix classification, and matrix completion. The standard semidefinite programming formulation for this problem is computationally expensive. In addition, due to the non-smooth nature of the trace norm, the optimal first-order black-box method for solving such class of problems converges as O(1/√k), where k is the iteration counter. In this paper, we exploit the special structure of the trace norm, based on which we propose an extended gradient algorithm that converges as O(1/k). We further propose an accelerated gradient algorithm, which achieves the optimal convergence rate of O(1/k2) for smooth problems. Experiments on multi-task learning problems demonstrate the efficiency of the proposed algorithms.

UR - http://www.scopus.com/inward/record.url?scp=71149103464&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=71149103464&partnerID=8YFLogxK

M3 - Conference contribution

SN - 9781605585161

SP - 457

EP - 464

BT - Proceedings of the 26th International Conference On Machine Learning, ICML 2009

ER -