Recent Trends in Distributed Training - Sparsified, Local and Decentralized
Prof Martin Jaggi, EPFL
1-2pm 30th Nov 2018
We discuss three recent techniques to improve distributed training of machine learning models. First, SGD with sparsified or compressed gradients is a key technique offering orders-of-magnitude practical improvements in scalability by improving the communication bottleneck. We prove that the method convergences at the same rate as vanilla SGD .
In the second part of the presentation, we will discuss variants of local SGD, which perform several update steps on a local model before communicating to other nodes, as opposed to classical mini-batch SGD. The scheme and its hierarchical extension offers significantly improved overall performance and communication efficiency, as well as adaptivity to the underlying system resources . Finally, we discuss decentralized machine learning, a promising emerging paradigm in view of global challenges of data ownership and privacy, enabling distributed training with data locality and without a central coordinator. We propose COLA, a new decentralized training algorithm for linear learning with communication efficiency, scalability, elasticity as well as resilience to changes in data and participating devices .
-  Sparsified SGD with Memory, Sebastian U. Stich, Jean-Baptiste Cordonnier, Martin Jaggi, NIPS 2018, https://arxiv.org/abs/1809.07599
-  Don't Use Large Mini-Batches, Use Local SGD, Tao Lin, Sebastian U. Stich, Martin Jaggi, https://arxiv.org/abs/1808.07217
-  COLA: Decentralized Linear Learning, Lie He, An Bian, Martin Jaggi, NIPS 2018, https://arxiv.org/abs/1808.04883
Martin Jaggi is a Tenure Track Assistant Professor at EPFL, heading the Machine Learning and Optimization Laboratory. Before that, he was a post-doctoral researcher at ETH Zurich, at the Simons Institute in Berkeley, US, and at École Polytechnique in Paris, France. He has earned his PhD in Machine Learning and Optimization from ETH Zurich in 2011, and a MSc in Mathematics also from ETH Zurich. He is a co-founder of the text analytics startup SpinningBytes, and also the founder of the Zurich Machine Learning and Data Science Meetup.
Large Conference Room, O'Reilly Institute