TY - GEN
T1 - Stochastic quasi-Newton methods for non-strongly convex problems
T2 - 55th IEEE Conference on Decision and Control, CDC 2016
AU - Yousefian, Farzad
AU - Nedic, Angelia
AU - Shanbhag, Uday V.
N1 - Publisher Copyright:
© 2016 IEEE.
PY - 2016/12/27
Y1 - 2016/12/27
N2 - Motivated by applications in optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving stochastic optimization problems. In the literature, the convergence analysis of these algorithms relies on strong convexity of the objective function. To our knowledge, no rate statements exist in the absence of this assumption. Motivated by this gap, we allow the objective function to be merely convex and develop a regularized SQN method. In this scheme, both the gradient mapping and the Hessian approximation are regularized at each iteration and updated alternatively. Unlike the classical regularization schemes, we allow the regularization parameter to be updated iteratively and decays to zero. Under suitable assumptions on the stepsize and regularization parameters, we show that the function value converges to its optimal value in both an almost sure and an expected-value sense. In each case, a set of regularization and steplength sequences is provided under which convergence may be guaranteed. Moreover, the rate of convergence is derived in terms of function value. Our empirical analysis on a binary classification problem shows that the proposed scheme performs well compared to both classical regularized SQN and stochastic approximation schemes.
AB - Motivated by applications in optimization and machine learning, we consider stochastic quasi-Newton (SQN) methods for solving stochastic optimization problems. In the literature, the convergence analysis of these algorithms relies on strong convexity of the objective function. To our knowledge, no rate statements exist in the absence of this assumption. Motivated by this gap, we allow the objective function to be merely convex and develop a regularized SQN method. In this scheme, both the gradient mapping and the Hessian approximation are regularized at each iteration and updated alternatively. Unlike the classical regularization schemes, we allow the regularization parameter to be updated iteratively and decays to zero. Under suitable assumptions on the stepsize and regularization parameters, we show that the function value converges to its optimal value in both an almost sure and an expected-value sense. In each case, a set of regularization and steplength sequences is provided under which convergence may be guaranteed. Moreover, the rate of convergence is derived in terms of function value. Our empirical analysis on a binary classification problem shows that the proposed scheme performs well compared to both classical regularized SQN and stochastic approximation schemes.
UR - http://www.scopus.com/inward/record.url?scp=85010792411&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85010792411&partnerID=8YFLogxK
U2 - 10.1109/CDC.2016.7798953
DO - 10.1109/CDC.2016.7798953
M3 - Conference contribution
AN - SCOPUS:85010792411
T3 - 2016 IEEE 55th Conference on Decision and Control, CDC 2016
SP - 4496
EP - 4503
BT - 2016 IEEE 55th Conference on Decision and Control, CDC 2016
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 12 December 2016 through 14 December 2016
ER -