Discover Shigangli/Ok-Topk Open Source project

Stars
23
Rank 1,011,361 (Top 21 %)
Language
Python
License
GNU General Publi...
Created almost 3 years ago
Updated almost 2 years ago

Shigangli

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

Magicube

Magicube is a high-performance library for quantized sparse matrix operations (SpMM and SDDMM) of deep learning on Tensor Cores.

C++

Chimera

Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines.

Python

eager-SGD

Eager-SGD is a decentralized asynchronous SGD. It utilizes novel partial collectives operations to accumulate the gradients across all the processes.

Python

SpMV-on-Many-Core

A cross-platform Sparse Matrix Vector Multiplication (SpMV) framework for many-core architectures (GPUs and Xeon Phi).

C++