TY - JOUR
T1 - Decentralized Frank-Wolfe Algorithm for Convex and Nonconvex Problems
AU - Wai, Hoi To
AU - Lafond, Jean
AU - Scaglione, Anna
AU - Moulines, Eric
N1 - Funding Information:
Manuscript received December 4, 2016; revised December 5, 2016 and March 1, 2017; accepted March 9, 2017. Date of publication March 21, 2017; date of current version October 25, 2017. The work of H.-T. Wai was supported in part by the National Science Foundation under Grant CCF-1553746, and Grant CCF-1531050, the work of J. Lafond was supported by Direction Générale de l’Armement and the labex LMH (ANR-11-LABX-0056-LMH). This paper was presented in part at IEEE ICASSP, Shanghai, China, March 2016 [1] and IEEE GlobalSIP, Washington, DC, USA, December 2016 [2]. Recommended by Associate Editor G. Pillonetto.
Publisher Copyright:
© 2012 IEEE.
PY - 2017/11
Y1 - 2017/11
N2 - Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high-dimensional constrained problems, as the projection step becomes computationally prohibitive. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t2) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1√t) toward a stationary point for smooth but nonconvex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. We demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.
AB - Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high-dimensional constrained problems, as the projection step becomes computationally prohibitive. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t2) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1√t) toward a stationary point for smooth but nonconvex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. We demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.
KW - Communication efficient algorithms
KW - Frank-Wolfe (FW) algorithm
KW - consensus algorithms
KW - decentralized optimization
KW - high-dimensional optimization
KW - least absolute shrinkage and selection operator (LASSO)
KW - matrix completion
UR - http://www.scopus.com/inward/record.url?scp=85036465623&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85036465623&partnerID=8YFLogxK
U2 - 10.1109/TAC.2017.2685559
DO - 10.1109/TAC.2017.2685559
M3 - Article
AN - SCOPUS:85036465623
VL - 62
SP - 5522
EP - 5537
JO - IRE Transactions on Automatic Control
JF - IRE Transactions on Automatic Control
SN - 0018-9286
IS - 11
M1 - 7883821
ER -