On Projected Stochastic Gradient Descent Algorithm with Weighted Averaging for Least Squares Regression

Kobi Cohen, Angelia Nedich, R. Srikant

Research output: Contribution to journalArticle

10 Scopus citations

Abstract

The problem of least squares regression of a d-dimensional unknown parameter is considered. A stochastic gradient descent based algorithm with weighted iterate-averaging that uses a single pass over the data is studied and its convergence rate is analyzed. We first consider a bounded constraint set of the unknown parameter. Under some standard regularity assumptions, we provide an explicit O(1/k) upper bound on the convergence rate, depending on the variance (due to the additive noise in the measurements) and the size of the constraint set. We show that the variance term dominates the error and decreases with rate 1/k, while the term that is related to the size of the constraint set decreases with rate k/k2. We then compare the asymptotic ratio between the convergence rate of the proposed scheme and the empirical risk minimizer (ERM) as the number of iterations approaches infinity. We show that ≤4 for all ≥1 when the random entries of the sensing vector are uncorrelated and identically distributed. We further improve the upper bound by showing that ≤4/3 for the case of d=1 and unbounded parameter set when the random sensing entries are equal across time. Simulation results demonstrate strong performance of the algorithm as compared to existing methods, and coincide with ≤4/3 even for large d in practice.

Original languageEnglish (US)
Article number7932172
Pages (from-to)5974-5981
Number of pages8
JournalIEEE Transactions on Automatic Control
Volume62
Issue number11
DOIs
StatePublished - Nov 1 2017

Keywords

  • Convex optimization
  • empirical risk minimizer (ERM)
  • projected stochastic gradient descent (SGD)
  • weighted averaging

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Computer Science Applications
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'On Projected Stochastic Gradient Descent Algorithm with Weighted Averaging for Least Squares Regression'. Together they form a unique fingerprint.

  • Cite this