Shigangli/Magicube

Stars
79
Rank 406,034 (Top 9 %)
Language
C++
License
GNU General Publi...
Created almost 3 years ago
Updated almost 2 years ago

Shigangli/Magicube

Shigangli

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Magicube is a high-performance library for quantized sparse matrix operations (SpMM and SDDMM) of deep learning on Tensor Cores.

Chimera

Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines.

Ok-Topk

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

eager-SGD

Eager-SGD is a decentralized asynchronous SGD. It utilizes novel partial collectives operations to accumulate the gradients across all the processes.

SpMV-on-Many-Core

A cross-platform Sparse Matrix Vector Multiplication (SpMV) framework for many-core architectures (GPUs and Xeon Phi).

WAGMA-SGD

WAGMA-SGD is a decentralized asynchronous SGD based on wait-avoiding group model averaging. The synchronization is relaxed by making the collectives externally-triggerable, namely, a collective can be initiated without requiring that all the processes enter it. It partially reduces the data within non-overlapping groups of process, improving the parallel scalability.

COMPI

Cache-oblivious MPI all-to-all communications based on Morton order

DNN-cpp-proxies

C++/MPI proxies for distributed training of deep neural networks.

cnnARM