Skip to main content

Trinity College Dublin, The University of Dublin

Menu Search



You are here Seminars

Recent Trends in Distributed Training - Sparsified, Local and Decentralized

Prof Martin Jaggi, EPFL
1-2pm  30th Nov 2018

Abstract

We discuss three recent techniques to improve distributed training of machine learning models. First, SGD with sparsified or compressed gradients is a key technique offering orders-of-magnitude practical improvements in scalability by improving the communication bottleneck. We prove that the method convergences at the same rate as vanilla SGD [1].
In the second part of the presentation, we will discuss variants of local SGD, which perform several update steps on a local model before communicating to other nodes, as opposed to classical mini-batch SGD. The scheme and its hierarchical extension offers significantly improved overall performance and communication efficiency, as well as adaptivity to the underlying system resources [2]. Finally, we discuss decentralized machine learning, a promising emerging paradigm in view of global challenges of data ownership and privacy, enabling distributed training with data locality and without a central coordinator. We propose COLA, a new decentralized training algorithm for linear learning with communication efficiency, scalability, elasticity as well as resilience to changes in data and participating devices [3].
References:

Short Bio

Martin Jaggi is a Tenure Track Assistant Professor at EPFL, heading the Machine Learning and Optimization Laboratory. Before that, he was a post-doctoral researcher at ETH Zurich, at the Simons Institute in Berkeley, US, and at École Polytechnique in Paris, France. He has earned his PhD in Machine Learning and Optimization from ETH Zurich in 2011, and a MSc in Mathematics also from ETH Zurich. He is a co-founder of the text analytics startup SpinningBytes, and also the founder of the Zurich Machine Learning and Data Science Meetup.

Venue

Large Conference Room, O'Reilly Institute