Algomancer/Bayesian-Flow-Networks

Stars
234
Rank 171,630 (Top 4 %)
Language
Jupyter Notebook
License
Apache License 2.0
Created over 1 year ago
Updated 11 months ago

Algomancer/Bayesian-Flow-Networks

Algomancer

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

A simple implimentation of Bayesian Flow Networks (BFN)

Bayesian Flow Networks

A PyTorch implementation of Bayesian Flow Networks

Currently use of a non causal version of LLAMA2. Currently training TinyStories Models matching https://github.com/karpathy/llama2.c/tree/master

I am going to be using this repository to explore training dynamics of this new class of models. I will maintain a minimal implimentation in the Minimal.ipynd for a simple BFN implimentation. Everything else is an early work in progress.

Features

Discrete model with continuous-time loss, training and sampling (completed)
SOTA performance on XOR dataset
Tiny Stories 15m LLAMA2 Initial Code
Tiny Stories weights (training)
Wiki Text8 Dataset
Bayesian Flow GPT-2 Scale
Fancy Visuals

Full-Stack-Scala-Starter

Play 2.5, ScalaJS, Binding.scala starter project.

The-Daily-Train

Training Models Daily

Code-Network-Starter

Seed Projects for Hackathons, Resources for rapid prototyping, tutorials

VCReg

Minimal Implimentation of VCRec (2024) for collapse provention.

Low-Rank-Extension-Token-Linear

This linear layer allows dynamic reallocation of parameters post training (tokenformer). Additionally, it creates a low rank transformation effectively doubling the amount of parameter tokens you can attend to.

Hyperspherical-Discrete-Bayesian-Flow-Like-Vibe-But-also-kinda-tangental-discrete-diffusion

It's a hyperspherical thingy for discrete diffusionishness.

Stablizing-Transformers

Stablizing transformer training by preventing attention entropy collapse

Energy-Sliced-Wasserstein

Computes the Energy Sliced Wasserstein Loss between two distributions. An optimal-transporty-energyish vibe distribution matching loss/regulariser.