Experience-Driven Congestion Control

When Multi-Path TCP Meets Deep Reinforcement Learning

Zhiyuan Xu, Jian Tang, Chengxiang Yin, Yanzhi Wang, Guoliang Xue

Research output: Contribution to journalArticle

Abstract

In this paper, we aim to study networking problems from a whole new perspective by leveraging emerging deep learning, to develop an experience-driven approach, which enables a network or a protocol to learn the best way to control itself from its own experience (e.g., runtime statistics data), just as a human learns a skill. We present design, implementation and evaluation of a deep reinforcement learning (DRL)-based control framework, DRL-CC (DRL for Congestion Control), which realizes our experience-driven design philosophy on multi-path TCP (MPTCP) congestion control. DRL-CC utilizes a single (instead of multiple independent) agent to dynamically and jointly perform congestion control for all active MPTCP flows on an end host with the objective of maximizing the overall utility. The novelty of our design is to utilize a flexible recurrent neural network, LSTM, under a DRL framework for learning a representation for all active flows and dealing with their dynamics. Moreover, we, for the first time, integrate the above LSTM-based representation network into an actor-critic framework for continuous (congestion) control, which leverages the emerging deterministic policy gradient to train critic, actor, and LSTM networks in an end-to-end manner. We implemented DRL-CC based on the MPTCP implementation in the Linux kernel. The experimental results show that 1) DRL-CC consistently and significantly outperforms a few well-known MPTCP congestion control algorithms in terms of goodput without sacrificing fairness, 2) it is flexible and robust to highly-dynamic network environments with time-varying flows, and 3) it is friendly to regular TCP.

Original languageEnglish (US)
Article number8664598
Pages (from-to)1325-1336
Number of pages12
JournalIEEE Journal on Selected Areas in Communications
Volume37
Issue number6
DOIs
StatePublished - Jun 1 2019

Fingerprint

Reinforcement learning
Recurrent neural networks
Statistics

Keywords

  • AI
  • congestion control
  • deep learning
  • experience-driven control
  • multi-path TCP
  • TCP

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this

Experience-Driven Congestion Control : When Multi-Path TCP Meets Deep Reinforcement Learning. / Xu, Zhiyuan; Tang, Jian; Yin, Chengxiang; Wang, Yanzhi; Xue, Guoliang.

In: IEEE Journal on Selected Areas in Communications, Vol. 37, No. 6, 8664598, 01.06.2019, p. 1325-1336.

Research output: Contribution to journalArticle

Xu, Zhiyuan ; Tang, Jian ; Yin, Chengxiang ; Wang, Yanzhi ; Xue, Guoliang. / Experience-Driven Congestion Control : When Multi-Path TCP Meets Deep Reinforcement Learning. In: IEEE Journal on Selected Areas in Communications. 2019 ; Vol. 37, No. 6. pp. 1325-1336.
@article{a8f3c033b7c24764a5f08586e5a69364,
title = "Experience-Driven Congestion Control: When Multi-Path TCP Meets Deep Reinforcement Learning",
abstract = "In this paper, we aim to study networking problems from a whole new perspective by leveraging emerging deep learning, to develop an experience-driven approach, which enables a network or a protocol to learn the best way to control itself from its own experience (e.g., runtime statistics data), just as a human learns a skill. We present design, implementation and evaluation of a deep reinforcement learning (DRL)-based control framework, DRL-CC (DRL for Congestion Control), which realizes our experience-driven design philosophy on multi-path TCP (MPTCP) congestion control. DRL-CC utilizes a single (instead of multiple independent) agent to dynamically and jointly perform congestion control for all active MPTCP flows on an end host with the objective of maximizing the overall utility. The novelty of our design is to utilize a flexible recurrent neural network, LSTM, under a DRL framework for learning a representation for all active flows and dealing with their dynamics. Moreover, we, for the first time, integrate the above LSTM-based representation network into an actor-critic framework for continuous (congestion) control, which leverages the emerging deterministic policy gradient to train critic, actor, and LSTM networks in an end-to-end manner. We implemented DRL-CC based on the MPTCP implementation in the Linux kernel. The experimental results show that 1) DRL-CC consistently and significantly outperforms a few well-known MPTCP congestion control algorithms in terms of goodput without sacrificing fairness, 2) it is flexible and robust to highly-dynamic network environments with time-varying flows, and 3) it is friendly to regular TCP.",
keywords = "AI, congestion control, deep learning, experience-driven control, multi-path TCP, TCP",
author = "Zhiyuan Xu and Jian Tang and Chengxiang Yin and Yanzhi Wang and Guoliang Xue",
year = "2019",
month = "6",
day = "1",
doi = "10.1109/JSAC.2019.2904358",
language = "English (US)",
volume = "37",
pages = "1325--1336",
journal = "IEEE Journal on Selected Areas in Communications",
issn = "0733-8716",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "6",

}

TY - JOUR

T1 - Experience-Driven Congestion Control

T2 - When Multi-Path TCP Meets Deep Reinforcement Learning

AU - Xu, Zhiyuan

AU - Tang, Jian

AU - Yin, Chengxiang

AU - Wang, Yanzhi

AU - Xue, Guoliang

PY - 2019/6/1

Y1 - 2019/6/1

N2 - In this paper, we aim to study networking problems from a whole new perspective by leveraging emerging deep learning, to develop an experience-driven approach, which enables a network or a protocol to learn the best way to control itself from its own experience (e.g., runtime statistics data), just as a human learns a skill. We present design, implementation and evaluation of a deep reinforcement learning (DRL)-based control framework, DRL-CC (DRL for Congestion Control), which realizes our experience-driven design philosophy on multi-path TCP (MPTCP) congestion control. DRL-CC utilizes a single (instead of multiple independent) agent to dynamically and jointly perform congestion control for all active MPTCP flows on an end host with the objective of maximizing the overall utility. The novelty of our design is to utilize a flexible recurrent neural network, LSTM, under a DRL framework for learning a representation for all active flows and dealing with their dynamics. Moreover, we, for the first time, integrate the above LSTM-based representation network into an actor-critic framework for continuous (congestion) control, which leverages the emerging deterministic policy gradient to train critic, actor, and LSTM networks in an end-to-end manner. We implemented DRL-CC based on the MPTCP implementation in the Linux kernel. The experimental results show that 1) DRL-CC consistently and significantly outperforms a few well-known MPTCP congestion control algorithms in terms of goodput without sacrificing fairness, 2) it is flexible and robust to highly-dynamic network environments with time-varying flows, and 3) it is friendly to regular TCP.

AB - In this paper, we aim to study networking problems from a whole new perspective by leveraging emerging deep learning, to develop an experience-driven approach, which enables a network or a protocol to learn the best way to control itself from its own experience (e.g., runtime statistics data), just as a human learns a skill. We present design, implementation and evaluation of a deep reinforcement learning (DRL)-based control framework, DRL-CC (DRL for Congestion Control), which realizes our experience-driven design philosophy on multi-path TCP (MPTCP) congestion control. DRL-CC utilizes a single (instead of multiple independent) agent to dynamically and jointly perform congestion control for all active MPTCP flows on an end host with the objective of maximizing the overall utility. The novelty of our design is to utilize a flexible recurrent neural network, LSTM, under a DRL framework for learning a representation for all active flows and dealing with their dynamics. Moreover, we, for the first time, integrate the above LSTM-based representation network into an actor-critic framework for continuous (congestion) control, which leverages the emerging deterministic policy gradient to train critic, actor, and LSTM networks in an end-to-end manner. We implemented DRL-CC based on the MPTCP implementation in the Linux kernel. The experimental results show that 1) DRL-CC consistently and significantly outperforms a few well-known MPTCP congestion control algorithms in terms of goodput without sacrificing fairness, 2) it is flexible and robust to highly-dynamic network environments with time-varying flows, and 3) it is friendly to regular TCP.

KW - AI

KW - congestion control

KW - deep learning

KW - experience-driven control

KW - multi-path TCP

KW - TCP

UR - http://www.scopus.com/inward/record.url?scp=85065867897&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065867897&partnerID=8YFLogxK

U2 - 10.1109/JSAC.2019.2904358

DO - 10.1109/JSAC.2019.2904358

M3 - Article

VL - 37

SP - 1325

EP - 1336

JO - IEEE Journal on Selected Areas in Communications

JF - IEEE Journal on Selected Areas in Communications

SN - 0733-8716

IS - 6

M1 - 8664598

ER -