Discover Shigangli/WAGMA-SGD Open Source project

Stars
6
Rank 2,527,563 (Top 51 %)
Language
Python
License
GNU General Publi...
Created about 3 years ago
Updated about 3 years ago

Shigangli

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

WAGMA-SGD is a decentralized asynchronous SGD based on wait-avoiding group model averaging. The synchronization is relaxed by making the collectives externally-triggerable, namely, a collective can be initiated without requiring that all the processes enter it. It partially reduces the data within non-overlapping groups of process, improving the parallel scalability.

Magicube

Magicube is a high-performance library for quantized sparse matrix operations (SpMM and SDDMM) of deep learning on Tensor Cores.

C++

Chimera

Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines.

Python

Ok-Topk

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

Python

eager-SGD

Eager-SGD is a decentralized asynchronous SGD. It utilizes novel partial collectives operations to accumulate the gradients across all the processes.

Python