TY - GEN
T1 - NoSync
T2 - 27th International Conference on Artificial Neural Networks, ICANN 2018
AU - Isakov, Mihailo
AU - Kinsy, Michel A.
N1 - Publisher Copyright:
© Springer Nature Switzerland AG 2018.
PY - 2018
Y1 - 2018
N2 - Training deep neural networks on big datasets remains a computational challenge. It can take hundreds of hours to perform and requires distributed computing systems to accelerate. Common distributed data-parallel approaches share a single model across multiple workers, train on different batches, aggregate gradients, and redistribute the new model. In this work, we propose NoSync, a particle swarm optimization inspired alternative where each worker trains a separate model, and applies pressure forcing models to converge. NoSync explores a greater portion of the parameter space and provides resilience to overfitting. It consistently offers higher accuracy compared to single workers, offers a linear speedup for smaller clusters, and is orthogonal to existing data-parallel approaches.
AB - Training deep neural networks on big datasets remains a computational challenge. It can take hundreds of hours to perform and requires distributed computing systems to accelerate. Common distributed data-parallel approaches share a single model across multiple workers, train on different batches, aggregate gradients, and redistribute the new model. In this work, we propose NoSync, a particle swarm optimization inspired alternative where each worker trains a separate model, and applies pressure forcing models to converge. NoSync explores a greater portion of the parameter space and provides resilience to overfitting. It consistently offers higher accuracy compared to single workers, offers a linear speedup for smaller clusters, and is orthogonal to existing data-parallel approaches.
KW - Artificial neural network
KW - Deep learning
KW - Distributed systems
KW - Evolutionary algorithm
KW - Particle swarm optimization
UR - http://www.scopus.com/inward/record.url?scp=85054881570&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85054881570&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-01421-6_58
DO - 10.1007/978-3-030-01421-6_58
M3 - Conference contribution
AN - SCOPUS:85054881570
SN - 9783030014209
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 607
EP - 619
BT - Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings
A2 - Manolopoulos, Yannis
A2 - Hammer, Barbara
A2 - Maglogiannis, Ilias
A2 - Kurkova, Vera
A2 - Iliadis, Lazaros
PB - Springer Verlag
Y2 - 4 October 2018 through 7 October 2018
ER -