catid/dora

Stars
276
Rank 149,319 (Top 3 %)
Language
Python
License
MIT License
Created 9 months ago
Updated 6 months ago

catid/dora

catid

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Implementation of DoRA

DoRA

Implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation" (Liu et al, 2024) https://arxiv.org/pdf/2402.09353.pdf

Demo

Install conda: https://docs.conda.io/projects/miniconda/en/latest/index.html

git clone https://github.com/catid/dora.git
cd dora

conda create -n dora python=3.10 -y && conda activate dora

pip install -U -r requirements.txt

python dora.py

Output

(dora) ➜  dora git:(main) python dora.py
Total Parameters: 11
Trainable Parameters: 11
Final Evaluation Loss: 0.13415579497814178
Total Parameters: 65
Trainable Parameters: 54
Continuing training with DoRA layers...
Final (DoRA) Evaluation Loss: 0.06080308556556702

supercharger

Supercharge Open-Source AI Models

Zpng

Better lossless compression than PNG with a simpler algorithm

wirehair

Wirehair : O(N) Fountain Code for Large Data

self-discover

Implementation of Google's SELF-DISCOVER

WLANOptimizer

Single-header C library that fixes WiFi performance issues for online gaming and other low-latency real-time network traffic.

longhair

Longhair : O(N^2) Cauchy Reed-Solomon Block Erasure Code for Small Data

leopard

Leopard-RS : O(N Log N) MDS Reed-Solomon Block Erasure Code for Large Data

shorthair

Shorthair : Generational Block Streaming Erasure Codes

TimeSync

TimeSync: Time Synchronization Library in Portable C++

cm256

Fast GF(256) Cauchy MDS Block Erasure Codec in C

tonk

Tonk : Reliable UDP (rUDP) Network Library and Infinite Window Erasure Code

Zdepth

Zdepth :: Streaming Depth Compressor in C++ for Azure Kinect DK

xrcap

Azure Kinect multi-camera secure network capture/record/replay

snowshoe

Snowshoe - Portable, Secure, Fast Elliptic Curve Math Library in C

XRmonitors

XRmonitors : User-Friendly Virtual Multi-Monitors for the Workplace

tabby

Tabby - Strong, Fast, and Portable Cryptographic Signatures, Handshakes, and Password Authentication

gf256

GF256 - Fast 8-bit Galois Field Math in C

kvm

Low-Bandwidth IP KVM using Raspberry Pi 4

siamese

Siamese : Infinite-Window Streaming Erasure Code (HARQ)

bitnet_cpu

Experiments with BitNet inference on CPU

ZdepthLossy

Lossy version of Zdepth using video encoders

fecal

FEC-AL : O(N^2) Fountain Code for Small Data

CauchyCaterpillar

Cauchy Caterpillar : O(N^2) Short-Window Streaming Erasure Code

aiwebcam2

Second attempt at AI webcam, this time with OpenAI API

upsampling

Image Upsampling with PyTorch

calico

Calico - Strong, Fast, and Portable Authenticated Encryption

cymric

Cymric - Portable secure random number generator

loraftp

File transfer between two Raspberry Pis using the LoRa Pi HAT from Waveshare

mau

Network simulator for reliable UDP testing in C++

oaillama3

Simple setup to self-host LLaMA3-70B model with an OpenAI API

lllm

Latent Large Language Models

PacketAllocator

C++ Memory allocator for packet queues that free() in roughly the same order that they alloc().

spectral_ssm

Implementation of Spectral State Space Models

libcat

Common code library

AutoAudiobook

Automatically create an audiobook using OpenAI

minigpt4

MiniGPT-4 :: Updated to Torch 2.0, simple setup, easier API, cut out training code

sdxl

SDXL GPU cluster scripts

dataloader

High-performance tokenized language data-loader for Python C++ extension

fp61

Experiment: Fast finite field Fp=2^61-1 in C++

cuda_float_compress

Python package for compressing floating-point PyTorch tensors

phind

Locally hosted: 60% HumanEval

rtmp_receiver

Simple unidirectional RTMP video stream receiver

counter

C++ wrapper for counters that can roll-over (e.g. timestamps/ack-ids)

z16

16-bit monochrome image compressor based on Zstd

AQLM

llamanal.cpp

Static code analysis for C++ projects using llama.cpp and the best LLM you can run offline without an expensive GPU.

unfiltered-diffusers

Simple fork that disables NSFW filter

hloc

boss-balloon

libcatid

Automatically exported from code.google.com/p/libcatid

voron

Voron 3D Printer files

halide-test

Test v14/v15 performance regression

logger

Feature-rich portable C++ logging subsystem in 650 lines of code

cifar10deepspeed

Using DeepSpeed and Nvidia DALI to train various models to solve CIFAR-10

chainlit-anthropic

Chainlit AI UI with Anthropic Backend

fastest_gf_matrix_mult

A fairly hacked together piece of code that can quickly search for the optimal GF polynomials and Cauchy matrices for XOR-based GF matrix multiplication for erasure code encoders. May be useful if the matrices are of fixed size.

recfilter-2020-fail

This is a failed attempt to port Recfilter to latest Halide from 2020. Maybe someone else can figure this out?

bentpipe

Simple UDP rebroadcaster

whisper3

Testing out Whisper 3

sphynx

Sphynx - High Performance Network Transport Layer

textworld_llm_benchmark

TextWorld LLM Benchmark

pixel-perfect-sfm

pixel-perfect-sfm with some minor fixes

train_ticket

Gated PRNGs are all you need? Spoilers: No.

CatsChoice

PRNG Parameter Generation

CRC16Recovery

Optimized CRC16 with error recovery in C

Splane

Archived old code - /ban_ids/ is kind of interesting

rust_webgl_demo

Hello World : Rust Web Assembly

blog2022

swe_agent_playground

swe_agent_playground

Exapunks_Solutions

Exapunks Walkthrough Solutions

audio_prediction

Simple audio prediction example with RNN

never_forget

Implementation of Overcoming Catastrophic Forgetting

DependencyInjected

DependencyInjected : Light-weight and powerful Dependency Injection pattern for C++

quicsend

quicsend :: Super-fast Internet-ready file transfer right from Python