Stochastic quasi-Newton methods for non-strongly convex problems: Convergence and rate analysis

Farzad Yousefian, Angelia Nedich, Uday V. Shanbhag

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Motivated by applications in optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving stochastic optimization problems. In the literature, the convergence analysis of these algorithms relies on strong convexity of the objective function. To our knowledge, no rate statements exist in the absence of this assumption. Motivated by this gap, we allow the objective function to be merely convex and develop a regularized SQN method. In this scheme, both the gradient mapping and the Hessian approximation are regularized at each iteration and updated alternatively. Unlike the classical regularization schemes, we allow the regularization parameter to be updated iteratively and decays to zero. Under suitable assumptions on the stepsize and regularization parameters, we show that the function value converges to its optimal value in both an almost sure and an expected-value sense. In each case, a set of regularization and steplength sequences is provided under which convergence may be guaranteed. Moreover, the rate of convergence is derived in terms of function value. Our empirical analysis on a binary classification problem shows that the proposed scheme performs well compared to both classical regularized SQN and stochastic approximation schemes.

Original languageEnglish (US)
Title of host publication2016 IEEE 55th Conference on Decision and Control, CDC 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages4496-4503
Number of pages8
ISBN (Electronic)9781509018376
DOIs
StatePublished - Dec 27 2016
Externally publishedYes
Event55th IEEE Conference on Decision and Control, CDC 2016 - Las Vegas, United States
Duration: Dec 12 2016Dec 14 2016

Other

Other55th IEEE Conference on Decision and Control, CDC 2016
CountryUnited States
CityLas Vegas
Period12/12/1612/14/16

Fingerprint

Quasi-Newton Method
Stochastic Methods
Newton-Raphson method
Regularization Parameter
Value Function
Regularization
Objective function
Analysis of Algorithms
Quasi-Newton
Binary Classification
Stochastic Approximation
Stochastic Optimization
Empirical Analysis
Approximation Scheme
Expected Value
Convergence Analysis
Classification Problems
Convexity
Machine Learning
Rate of Convergence

ASJC Scopus subject areas

  • Artificial Intelligence
  • Decision Sciences (miscellaneous)
  • Control and Optimization

Cite this

Yousefian, F., Nedich, A., & Shanbhag, U. V. (2016). Stochastic quasi-Newton methods for non-strongly convex problems: Convergence and rate analysis. In 2016 IEEE 55th Conference on Decision and Control, CDC 2016 (pp. 4496-4503). [7798953] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CDC.2016.7798953

Stochastic quasi-Newton methods for non-strongly convex problems : Convergence and rate analysis. / Yousefian, Farzad; Nedich, Angelia; Shanbhag, Uday V.

2016 IEEE 55th Conference on Decision and Control, CDC 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 4496-4503 7798953.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Yousefian, F, Nedich, A & Shanbhag, UV 2016, Stochastic quasi-Newton methods for non-strongly convex problems: Convergence and rate analysis. in 2016 IEEE 55th Conference on Decision and Control, CDC 2016., 7798953, Institute of Electrical and Electronics Engineers Inc., pp. 4496-4503, 55th IEEE Conference on Decision and Control, CDC 2016, Las Vegas, United States, 12/12/16. https://doi.org/10.1109/CDC.2016.7798953
Yousefian F, Nedich A, Shanbhag UV. Stochastic quasi-Newton methods for non-strongly convex problems: Convergence and rate analysis. In 2016 IEEE 55th Conference on Decision and Control, CDC 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 4496-4503. 7798953 https://doi.org/10.1109/CDC.2016.7798953
Yousefian, Farzad ; Nedich, Angelia ; Shanbhag, Uday V. / Stochastic quasi-Newton methods for non-strongly convex problems : Convergence and rate analysis. 2016 IEEE 55th Conference on Decision and Control, CDC 2016. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 4496-4503
@inproceedings{c80cc4d9a33f44ab818460fa81c86a8d,
title = "Stochastic quasi-Newton methods for non-strongly convex problems: Convergence and rate analysis",
abstract = "Motivated by applications in optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving stochastic optimization problems. In the literature, the convergence analysis of these algorithms relies on strong convexity of the objective function. To our knowledge, no rate statements exist in the absence of this assumption. Motivated by this gap, we allow the objective function to be merely convex and develop a regularized SQN method. In this scheme, both the gradient mapping and the Hessian approximation are regularized at each iteration and updated alternatively. Unlike the classical regularization schemes, we allow the regularization parameter to be updated iteratively and decays to zero. Under suitable assumptions on the stepsize and regularization parameters, we show that the function value converges to its optimal value in both an almost sure and an expected-value sense. In each case, a set of regularization and steplength sequences is provided under which convergence may be guaranteed. Moreover, the rate of convergence is derived in terms of function value. Our empirical analysis on a binary classification problem shows that the proposed scheme performs well compared to both classical regularized SQN and stochastic approximation schemes.",
author = "Farzad Yousefian and Angelia Nedich and Shanbhag, {Uday V.}",
year = "2016",
month = "12",
day = "27",
doi = "10.1109/CDC.2016.7798953",
language = "English (US)",
pages = "4496--4503",
booktitle = "2016 IEEE 55th Conference on Decision and Control, CDC 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

TY - GEN

T1 - Stochastic quasi-Newton methods for non-strongly convex problems

T2 - Convergence and rate analysis

AU - Yousefian, Farzad

AU - Nedich, Angelia

AU - Shanbhag, Uday V.

PY - 2016/12/27

Y1 - 2016/12/27

N2 - Motivated by applications in optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving stochastic optimization problems. In the literature, the convergence analysis of these algorithms relies on strong convexity of the objective function. To our knowledge, no rate statements exist in the absence of this assumption. Motivated by this gap, we allow the objective function to be merely convex and develop a regularized SQN method. In this scheme, both the gradient mapping and the Hessian approximation are regularized at each iteration and updated alternatively. Unlike the classical regularization schemes, we allow the regularization parameter to be updated iteratively and decays to zero. Under suitable assumptions on the stepsize and regularization parameters, we show that the function value converges to its optimal value in both an almost sure and an expected-value sense. In each case, a set of regularization and steplength sequences is provided under which convergence may be guaranteed. Moreover, the rate of convergence is derived in terms of function value. Our empirical analysis on a binary classification problem shows that the proposed scheme performs well compared to both classical regularized SQN and stochastic approximation schemes.

AB - Motivated by applications in optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving stochastic optimization problems. In the literature, the convergence analysis of these algorithms relies on strong convexity of the objective function. To our knowledge, no rate statements exist in the absence of this assumption. Motivated by this gap, we allow the objective function to be merely convex and develop a regularized SQN method. In this scheme, both the gradient mapping and the Hessian approximation are regularized at each iteration and updated alternatively. Unlike the classical regularization schemes, we allow the regularization parameter to be updated iteratively and decays to zero. Under suitable assumptions on the stepsize and regularization parameters, we show that the function value converges to its optimal value in both an almost sure and an expected-value sense. In each case, a set of regularization and steplength sequences is provided under which convergence may be guaranteed. Moreover, the rate of convergence is derived in terms of function value. Our empirical analysis on a binary classification problem shows that the proposed scheme performs well compared to both classical regularized SQN and stochastic approximation schemes.

UR - http://www.scopus.com/inward/record.url?scp=85010792411&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85010792411&partnerID=8YFLogxK

U2 - 10.1109/CDC.2016.7798953

DO - 10.1109/CDC.2016.7798953

M3 - Conference contribution

AN - SCOPUS:85010792411

SP - 4496

EP - 4503

BT - 2016 IEEE 55th Conference on Decision and Control, CDC 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -