Discover lessw2020/sigma_reparam Open Source project

Stars
3
Rank 3,963,521 (Top 79 %)
Language
Python
License
Apache License 2.0
Created over 1 year ago
Updated over 1 year ago

lessw2020

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Sigma Reparam for Transformers (based on Apple's paper)

Ranger-Deep-Learning-Optimizer

Ranger - a synergistic optimizer using RAdam (Rectified Adam), Gradient Centralization and LookAhead in one codebase

Python

1,154

Ranger21

Ranger deep learning optimizer rewrite to use newest components

Python

320

Best-Deep-Learning-Optimizers

Collection of the latest, greatest, deep learning optimizers (for Pytorch) - CNN, NLP suitable

Jupyter Notebook

201

mish

Mish Deep Learning Activation Function for PyTorch / FastAI

Jupyter Notebook

161

res2net-plus

Res2Net architecture with improved stem and Mish activation function

Python

136

Ranger-Mish-ImageWoof-5

Repo to build on / reproduce the record breaking Ranger-Mish-SelfAttention setup on FastAI ImageWoof dataset 5 epochs

Jupyter Notebook

116

training-detr

Unofficial Colab on how to train DETR, the intelligent object detector, with your own dataset. DETR = Detection Transformer

Jupyter Notebook

transformer_central

Various transformers for FSDP research

Jupyter Notebook

FAdam_PyTorch

an implementation of FAdam (Fisher Adam) in PyTorch

Python

Ranger22

Testing various improvements to Ranger21 for 2022

Python

mrnet-fastai

Deep Learning CNN using FastAI for the Stanford MRNet Knee MRI diagnosis challenge

Jupyter Notebook

triton_kernels_for_fun_and_profit

Custom kernels in Triton language for accelerating LLMs

Python

Thunder-Detr

(unofficial) - customized fork of DETR, optimized for intelligent obj detection on 'real world' custom datasets

Jupyter Notebook

fsdp_llm

FSDP optimizations for LLM training

Python

t5_11

housing our model example of fine tuning an 11B t5 with FSDP

Python

transformer_framework

framework for plug and play of various transformers (vision and nlp) with FSDP

Python

FTSwishPlus

FTSwish with mean shifting added to increase performance

Python

LightRelu

Customized PyTorch implementation of LiSHT (linear scaled hyperbolic tangent) activation function for deep learning

Python

hyper_efficient_optimizers

Development of hyper efficient optimizers that can match/exceed AdamW, while using reduced memory

Python

fsdp_review

Some eval and profile routines for fsdp

auto-adaptive-ai

auto adaptive framework for intrinsic hyperparameter selection, adaptive padding, normalized weights

Jupyter Notebook

TRelu

An improved activation function for deep learning - Threshold Relu, or TRelu

Python

EfficientNet-PyTorch

Unofficial port of Google's new EfficientNet to Pytorch and FastAI

Jupyter Notebook

RangerQH-Testing

Repo for running RangerQH + Res2NetPLus with LIP Pooling

Jupyter Notebook

facial-keypoint-detection

Facial keypoint detection CNN - custom architecture using partial convolution padding

Jupyter Notebook

AutoOpt-for-FastAI

Integrate Ebay's AutoOpt Deep Learning Optimizer into the FastAI framework

skycraft2

Minecraft in the sky, written in Python

Python

perception_tools

additional utils for working with Unity perception package

Jupyter Notebook

QuantFour_AdamW_Cuda

Fused 4bit AdamW in Cuda

Python

PolarBearLLM

testing new TransFormer, MoE, and TransNormer features

Python

unet-seg

Jupyter Notebook

FTSwish

Flattened Threshold Swish Activation function - PyTorch implementation

Python

coordinate_clipped_Optimizers

coordinate wise clipped Optimizers in PyTorch

Python

snowfall

helpful image handling utils - abstracts various file and opencv and pil features into result oriented functions

Python

style-transfer-vgg

Artistic Style transfer using VGG19

Jupyter Notebook

cuda-kernel-dev

in progress cuda kernels

Cuda

Curriculum-Learning-Dropout

Implementation of Curriculum Learning Dropout for FastAI framework

Jupyter Notebook

medExam

Training an AI with FSDP to take the US medical exam

5D-Compiler

Auto-Parallelization Compiler using 4D Parallel + Checkpointing (5D)

Python

aot_fsdp

When AOT Autograd meets FSDP = large models train faster

alibi_positional_embeddings

Alibi in PyTorch

Python

optimal-lr-finder

Automated optimal learning rate finder for PyTorch deep learning with FastAI

ft_linen

experiments with flax re-design to interop with pytorch

Python

linear-graph-slam

Linear Graph SLAM

Jupyter Notebook

bfloat_optimizer

Pure bfloat AdamW+ tweaks

Python

snake-id

FastAI deep learning classifier for snakes

Thunder

AI framework for flexible training and results review (pytorch, vision and tabular)

t5_finetuning

T5 and ExT5 fine tuning

Jupyter Notebook

pretrainer

FSDP codebase for pretraining large language models (LLM)

Python

Fusion

Advanced yet low code framework for fully sharded distributed training

Python

hsdp_demo

Tutorial repo for PyTorch FSDP running HSDP on single node.

Python

image-captioning-cnn-lstm

Image captioning system combining CNN + LSTM for caption generation

Jupyter Notebook

self-tuning-ai

implementation of self tuning networks in pytorch, based on https://arxiv.org/pdf/1903.03088v1.pdf

triton_flashv2_alibi

working repo for Triton based Flash2 supporting alibi pos embeddings

Python

Pytorch_train_test_split

Function to randomize and split training data into train/test, from same directory

Python

lessw2020/sigma_reparam

lessw2020

Reviews

Repository Details

More Repositories