Abstract

Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high-dimensional constrained problems, as the projection step becomes computationally prohibitive. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t2) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1√t) toward a stationary point for smooth but nonconvex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. We demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.

Original languageEnglish (US)
Article number7883821
Pages (from-to)5522-5537
Number of pages16
JournalIEEE Transactions on Automatic Control
Volume62
Issue number11
DOIs
StatePublished - Nov 1 2017

Fingerprint

Communication

Keywords

  • Communication efficient algorithms
  • consensus algorithms
  • decentralized optimization
  • Frank-Wolfe (FW) algorithm
  • high-dimensional optimization
  • least absolute shrinkage and selection operator (LASSO)
  • matrix completion

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

Decentralized Frank-Wolfe Algorithm for Convex and Nonconvex Problems. / Wai, Hoi To; Lafond, Jean; Scaglione, Anna; Moulines, Eric.

In: IEEE Transactions on Automatic Control, Vol. 62, No. 11, 7883821, 01.11.2017, p. 5522-5537.

Research output: Contribution to journalArticle

Wai, Hoi To ; Lafond, Jean ; Scaglione, Anna ; Moulines, Eric. / Decentralized Frank-Wolfe Algorithm for Convex and Nonconvex Problems. In: IEEE Transactions on Automatic Control. 2017 ; Vol. 62, No. 11. pp. 5522-5537.
@article{01cd597181fc4c7484a90bbe06aa51b8,
title = "Decentralized Frank-Wolfe Algorithm for Convex and Nonconvex Problems",
abstract = "Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high-dimensional constrained problems, as the projection step becomes computationally prohibitive. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t2) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1√t) toward a stationary point for smooth but nonconvex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. We demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.",
keywords = "Communication efficient algorithms, consensus algorithms, decentralized optimization, Frank-Wolfe (FW) algorithm, high-dimensional optimization, least absolute shrinkage and selection operator (LASSO), matrix completion",
author = "Wai, {Hoi To} and Jean Lafond and Anna Scaglione and Eric Moulines",
year = "2017",
month = "11",
day = "1",
doi = "10.1109/TAC.2017.2685559",
language = "English (US)",
volume = "62",
pages = "5522--5537",
journal = "IEEE Transactions on Automatic Control",
issn = "0018-9286",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "11",

}

TY - JOUR

T1 - Decentralized Frank-Wolfe Algorithm for Convex and Nonconvex Problems

AU - Wai, Hoi To

AU - Lafond, Jean

AU - Scaglione, Anna

AU - Moulines, Eric

PY - 2017/11/1

Y1 - 2017/11/1

N2 - Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high-dimensional constrained problems, as the projection step becomes computationally prohibitive. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t2) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1√t) toward a stationary point for smooth but nonconvex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. We demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.

AB - Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high-dimensional constrained problems, as the projection step becomes computationally prohibitive. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t2) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1√t) toward a stationary point for smooth but nonconvex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. We demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.

KW - Communication efficient algorithms

KW - consensus algorithms

KW - decentralized optimization

KW - Frank-Wolfe (FW) algorithm

KW - high-dimensional optimization

KW - least absolute shrinkage and selection operator (LASSO)

KW - matrix completion

UR - http://www.scopus.com/inward/record.url?scp=85036465623&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85036465623&partnerID=8YFLogxK

U2 - 10.1109/TAC.2017.2685559

DO - 10.1109/TAC.2017.2685559

M3 - Article

AN - SCOPUS:85036465623

VL - 62

SP - 5522

EP - 5537

JO - IEEE Transactions on Automatic Control

JF - IEEE Transactions on Automatic Control

SN - 0018-9286

IS - 11

M1 - 7883821

ER -