Distributed Machine Learning and Gradient Optimization

ISBN-13: 9789811634222 / Angielski / Miękka / 2023 / 169 str.

Jiawei Jiang; Bin Cui; Ce Zhang

Opis

This book presents the state of the art in distributed machine learning algorithms that are based on gradient optimization methods. In the big data era, large-scale datasets pose enormous challenges for the existing machine learning systems. As such, implementing machine learning algorithms in a distributed environment has become a key technology, and recent research has shown gradient-based iterative optimization to be an effective solution. Focusing on methods that can speed up large-scale gradient optimization through both algorithm optimizations and careful system implementations, the book introduces three essential techniques in designing a gradient optimization algorithm to train a distributed machine learning model: parallel strategy, data compression and synchronization protocol.Written in a tutorial style, it covers a range of topics, from fundamental knowledge to a number of carefully designed algorithms and systems of distributed machine learning. It will appeal to a broad audience in the field of machine learning, artificial intelligence, big data and database management.

Spis treści

Chapter 1: Introduction

1.1.  Background

1.2.  Distributed machine learning

1.3.  Gradient optimization

1.4.  Challenges

Chapter 2: The preliminaries

        2.1. Overview

        2.2. Parallel strategy

        2.3. Gradient compression

        2.4. Synchronization protocol

Chapter 3: Parallel strategy

1.1.  Background and problem

1.2.  Data parallelism

1.3.  Model parallelism

1.4.  Hybrid parallelism

        3.5. Benchmark

        3.6. Summary

Chapter 4: Gradient compression

        4.1. Background and problem

        4.2. Lossless gradient compression

        4.3. Lossy gradient compression

        4.4. Sparse gradient compression

        4.5. Benchmark

        4.6. Summary

Chapter 5: Synchronization protocol

        5.1. Background and problem

        5.2. Bulk synchronous protocol

        5.3. Asynchronous protocol

        5.4. Stale synchronous protocol

        5.5. Benchmark

        5.6. Summary

Chapter 6: Conclusion

        6.1. Summary of the book

        6.2. Future work

O autorze

Jiawei Jiang obtained his PhD from Peking University 2018, advised by Prof. Bin Cui. His research interests include distributed machine learning, gradient optimization and automatic machine learning. He has served as a program committee member or reviewer for various international events, including SIGMOD, VLDB, ICDE, KDD, AAAI and TKDE. He was awarded the CCF Outstanding Doctoral Dissertation Award (2019) and ACM China Doctoral Dissertation Award (2018).

Bin Cui is a Professor at the School of EECS and Director of the Institute of Network Computing and Information Systems, at Peking University. His research interests include database system architectures, query and index techniques, and big data management and mining. He has published over 200 refereed papers at international conferences and in journals. Dr. Cui has served on the technical program committee of various international conferences, including SIGMOD, VLDB, ICDE and KDD, and as Vice PC Chair of ICDE 2011, Demo Co-Chair of ICDE 2014, Area Chair of VLDB 2014, PC Co-Chair of APWeb 2015 and WAIM 2016. He is currently a member of the trustee board of VLDB Endowment, is on the editorial board of the VLDB Journal, Distributed and Parallel Databases Journal, and Information Systems, and was formerly an associate editor of IEEE Transactions on Knowledge and Data Engineering (TKDE, 2009-2013). He was selected for a Microsoft Young Professorship award (MSRA 2008), CCF Young Scientist award (2009), Second Prize of Natural Science Award of MOE China (2014), and appointed a Cheung Kong distinguished Professor by the MOE in 2016.