TY - GEN

T1 - Resource allocation for data-parallel computing in networks with data locality

AU - Wang, Weina

AU - Ying, Lei

PY - 2017/2/10

Y1 - 2017/2/10

N2 - This paper studies resource allocation for data-parallel applications in a networked computing system with data locality. Data-parallel computing tasks have two components: data and computation. To support efficient data-processing in the system, the resource allocation algorithm should jointly consider load-balancing, data transmissions and processing scheduling. In this paper, we consider a general model of a computing system where the computing system is a network represented by a graph, with nodes being computing devices or switches and edges being communication links. The data chunks stored at the nodes can be processed locally or be transmitted to other computing nodes in the network to be processed. The throughput of such a system for processing data-parallel applications is determined jointly by the computing capacity of the nodes and the communication capacity of the network, and the resource allocation algorithm should strike a right balance between computing and communication. In this paper, we propose a throughput-optimal resource allocation algorithm, which is also able to control the tradeoff between the expected amount of data transmitted and the expected number of backlogged tasks in steady state through a parameter qth. We show that the gap between the expected data transmission rate under our algorithm and the optimal value is O(1/qth), while the expected number of total backlogged tasks is upper bounded by O(qth).

AB - This paper studies resource allocation for data-parallel applications in a networked computing system with data locality. Data-parallel computing tasks have two components: data and computation. To support efficient data-processing in the system, the resource allocation algorithm should jointly consider load-balancing, data transmissions and processing scheduling. In this paper, we consider a general model of a computing system where the computing system is a network represented by a graph, with nodes being computing devices or switches and edges being communication links. The data chunks stored at the nodes can be processed locally or be transmitted to other computing nodes in the network to be processed. The throughput of such a system for processing data-parallel applications is determined jointly by the computing capacity of the nodes and the communication capacity of the network, and the resource allocation algorithm should strike a right balance between computing and communication. In this paper, we propose a throughput-optimal resource allocation algorithm, which is also able to control the tradeoff between the expected amount of data transmitted and the expected number of backlogged tasks in steady state through a parameter qth. We show that the gap between the expected data transmission rate under our algorithm and the optimal value is O(1/qth), while the expected number of total backlogged tasks is upper bounded by O(qth).

UR - http://www.scopus.com/inward/record.url?scp=85015218617&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85015218617&partnerID=8YFLogxK

U2 - 10.1109/ALLERTON.2016.7852334

DO - 10.1109/ALLERTON.2016.7852334

M3 - Conference contribution

AN - SCOPUS:85015218617

T3 - 54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016

SP - 933

EP - 939

BT - 54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016

Y2 - 27 September 2016 through 30 September 2016

ER -