On projected stochastic gradient descent algorithm with weighted averaging for least squares regression

Kobi Cohen, Angelia Nedich, R. Srikant

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

The problem of least squares regression of a ridimensionai unknown parameter is considered. A stochastic gradient descent based algorithm with weighted iterate-averaging that uses a single pass over the data is studied and its convergence rate is analyzed. We first consider a bounded constraint set of the unknown parameter. Under some standard regularity assumptions, we provide an explicit O(1/k) upper bound on the convergence rate, depending on the variance (due to the additive noise in the measurements) and the size of the constraint set. We show that the variance term dominates the error and decreases with rate 1 /k, while the constraint set term decreases with rate log k/k2. We then compare the asymptotic ratio ρ between the convergence rate of the proposed scheme and the empirical risk minimizer (ERM) as the number of iterations approaches infinity. We show that ρ 1. We further improve the upper bound by showing that ρ < 4/3 for the case of d =1 and unbounded parameter set. Simulation results demonstrate strong performance of the algorithm as compared to existing methods, and coincide with ρ < 4/3 even for large d in practice.

Original languageEnglish (US)
Title of host publication2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2314-2318
Number of pages5
Volume2016-May
ISBN (Electronic)9781479999880
DOIs
StatePublished - May 18 2016
Externally publishedYes
Event41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Shanghai, China
Duration: Mar 20 2016Mar 25 2016

Other

Other41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016
CountryChina
CityShanghai
Period3/20/163/25/16

Fingerprint

Additive noise

Keywords

  • Convex optimization
  • empirical risk mini-mizer
  • projected stochastic gradient descent
  • weighted averaging

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering

Cite this

Cohen, K., Nedich, A., & Srikant, R. (2016). On projected stochastic gradient descent algorithm with weighted averaging for least squares regression. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings (Vol. 2016-May, pp. 2314-2318). [7472090] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2016.7472090

On projected stochastic gradient descent algorithm with weighted averaging for least squares regression. / Cohen, Kobi; Nedich, Angelia; Srikant, R.

2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Vol. 2016-May Institute of Electrical and Electronics Engineers Inc., 2016. p. 2314-2318 7472090.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cohen, K, Nedich, A & Srikant, R 2016, On projected stochastic gradient descent algorithm with weighted averaging for least squares regression. in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. vol. 2016-May, 7472090, Institute of Electrical and Electronics Engineers Inc., pp. 2314-2318, 41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016, Shanghai, China, 3/20/16. https://doi.org/10.1109/ICASSP.2016.7472090
Cohen K, Nedich A, Srikant R. On projected stochastic gradient descent algorithm with weighted averaging for least squares regression. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Vol. 2016-May. Institute of Electrical and Electronics Engineers Inc. 2016. p. 2314-2318. 7472090 https://doi.org/10.1109/ICASSP.2016.7472090
Cohen, Kobi ; Nedich, Angelia ; Srikant, R. / On projected stochastic gradient descent algorithm with weighted averaging for least squares regression. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Vol. 2016-May Institute of Electrical and Electronics Engineers Inc., 2016. pp. 2314-2318
@inproceedings{f20cca5b38d64fdaad025e0a3f1b8226,
title = "On projected stochastic gradient descent algorithm with weighted averaging for least squares regression",
abstract = "The problem of least squares regression of a ridimensionai unknown parameter is considered. A stochastic gradient descent based algorithm with weighted iterate-averaging that uses a single pass over the data is studied and its convergence rate is analyzed. We first consider a bounded constraint set of the unknown parameter. Under some standard regularity assumptions, we provide an explicit O(1/k) upper bound on the convergence rate, depending on the variance (due to the additive noise in the measurements) and the size of the constraint set. We show that the variance term dominates the error and decreases with rate 1 /k, while the constraint set term decreases with rate log k/k2. We then compare the asymptotic ratio ρ between the convergence rate of the proposed scheme and the empirical risk minimizer (ERM) as the number of iterations approaches infinity. We show that ρ 1. We further improve the upper bound by showing that ρ < 4/3 for the case of d =1 and unbounded parameter set. Simulation results demonstrate strong performance of the algorithm as compared to existing methods, and coincide with ρ < 4/3 even for large d in practice.",
keywords = "Convex optimization, empirical risk mini-mizer, projected stochastic gradient descent, weighted averaging",
author = "Kobi Cohen and Angelia Nedich and R. Srikant",
year = "2016",
month = "5",
day = "18",
doi = "10.1109/ICASSP.2016.7472090",
language = "English (US)",
volume = "2016-May",
pages = "2314--2318",
booktitle = "2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

TY - GEN

T1 - On projected stochastic gradient descent algorithm with weighted averaging for least squares regression

AU - Cohen, Kobi

AU - Nedich, Angelia

AU - Srikant, R.

PY - 2016/5/18

Y1 - 2016/5/18

N2 - The problem of least squares regression of a ridimensionai unknown parameter is considered. A stochastic gradient descent based algorithm with weighted iterate-averaging that uses a single pass over the data is studied and its convergence rate is analyzed. We first consider a bounded constraint set of the unknown parameter. Under some standard regularity assumptions, we provide an explicit O(1/k) upper bound on the convergence rate, depending on the variance (due to the additive noise in the measurements) and the size of the constraint set. We show that the variance term dominates the error and decreases with rate 1 /k, while the constraint set term decreases with rate log k/k2. We then compare the asymptotic ratio ρ between the convergence rate of the proposed scheme and the empirical risk minimizer (ERM) as the number of iterations approaches infinity. We show that ρ 1. We further improve the upper bound by showing that ρ < 4/3 for the case of d =1 and unbounded parameter set. Simulation results demonstrate strong performance of the algorithm as compared to existing methods, and coincide with ρ < 4/3 even for large d in practice.

AB - The problem of least squares regression of a ridimensionai unknown parameter is considered. A stochastic gradient descent based algorithm with weighted iterate-averaging that uses a single pass over the data is studied and its convergence rate is analyzed. We first consider a bounded constraint set of the unknown parameter. Under some standard regularity assumptions, we provide an explicit O(1/k) upper bound on the convergence rate, depending on the variance (due to the additive noise in the measurements) and the size of the constraint set. We show that the variance term dominates the error and decreases with rate 1 /k, while the constraint set term decreases with rate log k/k2. We then compare the asymptotic ratio ρ between the convergence rate of the proposed scheme and the empirical risk minimizer (ERM) as the number of iterations approaches infinity. We show that ρ 1. We further improve the upper bound by showing that ρ < 4/3 for the case of d =1 and unbounded parameter set. Simulation results demonstrate strong performance of the algorithm as compared to existing methods, and coincide with ρ < 4/3 even for large d in practice.

KW - Convex optimization

KW - empirical risk mini-mizer

KW - projected stochastic gradient descent

KW - weighted averaging

UR - http://www.scopus.com/inward/record.url?scp=84973394947&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84973394947&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2016.7472090

DO - 10.1109/ICASSP.2016.7472090

M3 - Conference contribution

AN - SCOPUS:84973394947

VL - 2016-May

SP - 2314

EP - 2318

BT - 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -