survey-deep-gradient-compression

Gradient Compression (Sparsification)

• Chen, Chia-Yu, et al. "Scalecom: Scalable sparsified gradient compression for communication-efficient distributed training." arXiv preprint arXiv:2104.11125 (2021)
• Han, Pengchao, Shiqiang Wang, and Kin K. Leung. "Adaptive gradient sparsification for efficient federated learning: An online learning approach." arXiv preprint arXiv:2001.04756 (2020).
• Dutta, Aritra, et al. "On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.
• Shi, Shaohuai, et al. "Communication-efficient distributed deep learning with merged gradient sparsification on GPUs." IEEE INFOCOM 2020-IEEE Conference on Computer Communications. IEEE, 2020.
• Mishchenko, Konstantin, Filip Hanzely, and Peter Richtárik. "99% of worker-master communication in distributed optimization is not needed." Conference on Uncertainty in Artificial Intelligence. PMLR, 2020.
• [] Shi, Shaohuai, et al. "A distributed synchronous SGD algorithm with global Top-k sparsification for low bandwidth networks." 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 2019.
• Shi, Shaohuai, et al. "A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification." IJCAI. 2019.
• Sattler, Felix, et al. "Sparse binary compression: Towards distributed deep learning with minimal communication." 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.
• Shi, Shaohuai, et al. "Layer-wise adaptive gradient sparsification for distributed deep learning with convergence guarantees." arXiv preprint arXiv:1911.08727 (2019).
• Alistarh, Dan, et al. "The Convergence of Sparsified Gradient Methods." 32nd Conference on Neural Information Processing Systems (NIPS), DEC 02-08, 2018, Montreal, Canada. Vol. 31. Neural Information Processing Systems (NIPS), 2018.
• Stich, Sebastian U., Jean-Baptiste Cordonnier, and Martin Jaggi. "Sparsified SGD with Memory." Advances in Neural Information Processing Systems 31 (2018): 4447-4458.

Gradient Quantization

• Zhu, Guangxu, et al. "One-bit over-the-air aggregation for communication-efficient federated edge learning: Design and convergence analysis." IEEE Transactions on Wireless Communications (2020).
• Abdi, Afshin, and Faramarz Fekri. "Quantized compressive sampling of stochastic gradients for efficient communication in distributed deep learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.
• Sohn, Jy-yong, et al. "Election coding for distributed learning: Protecting SignSGD against byzantine attacks." Advances in Neural Information Processing Systems 33 (2020).
• Horváth, Samuel, et al. "Stochastic distributed learning with gradient quantization and variance reduction." arXiv preprint arXiv:1904.05115 (2019).
• Zheng, Shuai, Ziyue Huang, and James T. Kwok. "Communication-efficient distributed blockwise momentum SGD with error-feedback." arXiv preprint arXiv:1905.10936 (2019).
• Karimireddy, Sai Praneeth, et al. "Error feedback fixes signsgd and other gradient compression schemes." International Conference on Machine Learning. PMLR, 2019.
• Wu, Jiaxiang, et al. "Error compensated quantized SGD and its applications to large-scale distributed optimization." International Conference on Machine Learning. PMLR, 2018.
• Bernstein, Jeremy, et al. "signSGD with majority vote is communication efficient and fault tolerant." arXiv preprint arXiv:1810.05291 (2018).
• Yu, Mingchao, et al. "Gradiveq: Vector quantization for bandwidth-efficient gradient aggregation in distributed cnn training." arXiv preprint arXiv:1811.03617 (2018).
• Alistarh, Dan, et al. "QSGD: Communication-efficient SGD via gradient quantization and encoding." Advances in Neural Information Processing Systems 30 (2017): 1709-1720.
• Wen, Wei, et al. "TernGrad: ternary gradients to reduce communication in distributed deep learning." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.

Asynchronus Learning

• Grishchenko, Dmitry, et al. "Asynchronous distributed learning with sparse communications and identification." (2018).
• Bogoychev, Nikolay, et al. "Accelerating asynchronous stochastic gradient descent for neural machine translation." arXiv preprint arXiv:1808.08859 (2018).

Federated Learning

• Haddadpour, Farzin, et al. "Federated learning with compression: Unified analysis and sharp guarantees." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.
• Amiri, Mohammad Mohammadi, and Deniz Gündüz. "Federated learning over wireless fading channels." IEEE Transactions on Wireless Communications 19.5 (2020): 3546-3557.
• Abad, M. Salehi Heydar, et al. "Hierarchical federated learning across heterogeneous cellular networks." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.

Others

• Zhu, Ligeng, and Song Han. "Deep leakage from gradients." Federated Learning. Springer, Cham, 2020. 17-31.

• Jiang, Yimin, et al. "A Unified Architecture for Accelerating Distributed {DNN} Training in Heterogeneous GPU/CPU Clusters." 14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20). 2020.

• Singh, Navjot, et al. "SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization." arXiv preprint arXiv:2005.07041 (2020).

• Sapio, Amedeo, et al. "Scaling distributed machine learning with in-network aggregation." arXiv preprint arXiv:1903.06701 (2019).

• Jayarajan, Anand, et al. "Priority-based parameter propagation for distributed DNN training." arXiv preprint arXiv:1905.03960 (2019).

• Renggli, Cèdric, et al. "Sparcml: High-performance sparse communication for machine learning." Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 2019.

• Haddadpour, Farzin, et al. "Local sgd with periodic averaging: Tighter analysis and adaptive synchronization." arXiv preprint arXiv:1910.13598 (2019).

• Coleman, Cody, et al. "Analysis of dawnbench, a time-to-accuracy machine learning performance benchmark." ACM SIGOPS Operating Systems Review 53.1 (2019): 14-25.
  新しいベンチマーク、評価指標の提案

• Shi, Shaohuai, Xiaowen Chu, and Bo Li. "MG-WFBP: Efficient data communication for distributed synchronous SGD algorithms." IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019.

• Luo, Liang, et al. "Parameter hub: a rack-scale parameter server for distributed deep neural network training." Proceedings of the ACM Symposium on Cloud Computing. 2018.

• Li, Youjie, et al. "Pipe-sgd: A decentralized pipelined sgd framework for distributed deep net training." arXiv preprint arXiv:1811.03619 (2018).

• Sridharan, Srinivas, et al. "On scale-out deep learning training for cloud and HPC." arXiv preprint arXiv:1801.08030 (2018).