huggingface/llm_training_handbook

Stars
437
Rank 99,659 (Top 2 %)
Language
Python
License
Creative Commons ...
Created over 1 year ago
Updated 9 months ago

huggingface/llm_training_handbook

huggingface

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

An open collection of methodologies to help with successful training of large language models.

📖 The Large Language Model Training Handbook

An open collection of methodologies to help with successful training of large language models.

This is technical material suitable for LLM training engineers and operators. That is the content here contains lots of scripts and copy-n-paste commands to enable you to quickly solve your problems.

If you are not interested in technical details but want more of a detailed overview and concepts please refer to the sister The Large Language Model Training Playbook instead.

note: The list of topics will expand over time - at the moment filling in only a subset

Model parallelism

Maximizing throughput

Tensor precision / Data types

Training hyper-parameters and model initializations

Instabilities

Debugging software and hardware failures

SLURM

Resources

License

The content of this site is distributed under Attribution-ShareAlike 4.0 International.

Unless specified otherwise the code in this repo is licensed under Apache License, Version 2.0.

transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

pytorch-image-models

PyTorch image models, scripts, pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNet-V3/V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.

datasets

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools

peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

candle

Minimalist ML framework for Rust

trl

Train transformer language models with reinforcement learning.

text-generation-inference

Large Language Model Text Generation Inference

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

chat-ui

Open source codebase powering the HuggingChat app

lerobot

🤗 LeRobot: Making AI for Robotics more accessible with end-to-end learning

alignment-handbook

Robust recipes to align language models with human and AI preferences

parler-tts

Inference and training library for high-quality TTS models.

autotrain-advanced

🤗 AutoTrain Advanced

deep-rl-class

This repo contains the syllabus of the Hugging Face Deep Reinforcement Learning Course.

diffusion-models-class

Materials for the Hugging Face Diffusion Models Course

Jupyter Notebook

notebooks

Notebooks using the Hugging Face libraries 🤗

Jupyter Notebook

distil-whisper

Distilled variant of Whisper for speech recognition. 6x faster, 50% smaller, within 1% word error rate.

neuralcoref

✨Fast Coreference Resolution in spaCy with Neural Networks

safetensors

Simple, safe way to store and distribute tensors

text-embeddings-inference

A blazing fast inference solution for text embeddings models

knockknock

🚪✊Knock Knock: Get notified when your training ends with only two additional lines of code

speech-to-speech

Speech To Speech: an effort for an open-sourced and modular GPT4-o

swift-coreml-diffusers

Swift app demonstrating Core ML Stable Diffusion

optimum

🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools

blog

Public repo for HF blog posts

Jupyter Notebook

setfit

Efficient few-shot learning with Sentence Transformers

Jupyter Notebook

course

The Hugging Face course on Transformers

awesome-papers

Papers & presentation materials from Hugging Face's internal science day

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

cookbook

Open-source AI cookbook

Jupyter Notebook

transfer-learning-conv-ai

🦄 State-of-the-Art Conversational AI with Transfer Learning

swift-coreml-transformers

Swift Core ML 3 implementations of GPT-2, DistilGPT-2, BERT, and DistilBERT for Question answering. Other Transformers coming soon!

pytorch-openai-transformer-lm

🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI

huggingface.js

Utilities to use the Hugging Face Hub API

Mongoku

🔥The Web-scale GUI for MongoDB

huggingface_hub

All the open source things related to the Hugging Face Hub.

gsplat.js

JavaScript Gaussian Splatting library.

llm-vscode

LLM powered development for VSCode

hmtl

🌊HMTL: Hierarchical Multi-Task Learning - A State-of-the-Art neural network model for several NLP tasks based on PyTorch and AllenNLP

nanotron

Minimalistic large language model 3D-parallelism training

pytorch-pretrained-BigGAN

🦋A PyTorch implementation of BigGAN with pretrained weights and conversion scripts.

optimum-nvidia

torchMoji

😇A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc

awesome-huggingface

🤗 A list of wonderful open-source projects & applications integrated with Hugging Face libraries.

optimum-quanto

A pytorch quantization backend for optimum

llm.nvim

LLM powered development for Neovim

naacl_transfer_learning_tutorial

Repository of code for the tutorial on Transfer Learning in NLP held at NAACL 2019 in Minneapolis, MN, USA

dataset-viewer

Backend that powers the dataset viewer on Hugging Face dataset pages through a public API.

swift-transformers

Swift Package to implement a transformers-like API in Swift

exporters

Export Hugging Face models to Core ML and TensorFlow Lite

llm-ls

LSP server leveraging LLMs for code completion (and more?)

ratchet

A cross-platform browser ML framework.

transformers-bloom-inference

Fast Inference Solutions for BLOOM

lighteval

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

pytorch_block_sparse

Fast Block Sparse Matrices for Pytorch

node-question-answering

Fast and production-ready question answering in Node.js

large_language_model_training_playbook

An open collection of implementation tips, tricks and resources for training large language models

swift-chat

Mac app to demonstrate swift-transformers

text-clustering

Easily embed, cluster and semantically label text datasets

cosmopedia

optimum-intel

🤗 Optimum Intel: Accelerate inference with Intel optimization tools

Jupyter Notebook

controlnet_aux

community-events

Place where folks can contribute to 🤗 community events

Jupyter Notebook

tflite-android-transformers

DistilBERT / GPT-2 for on-device inference thanks to TensorFlow Lite with Android demo apps

nn_pruning

Prune a model while finetuning or training.

Jupyter Notebook

speechbox

100-times-faster-nlp

🚀100 Times Faster Natural Language Processing in Python - iPython notebook

education-toolkit

Educational materials for universities

Jupyter Notebook

transformers.js-examples

A collection of 🤗 Transformers.js demos and example applications

open-muse

Open reproduction of MUSE for fast text2image generation.

local-gemma

Gemma 2 optimized for your local machine.

unity-api

audio-transformers-course

The Hugging Face Course on Transformers for Audio

datablations

Scaling Data-Constrained Language Models

Jupyter Notebook

hf_transfer

dataspeech

huggingface-llama-recipes

Jupyter Notebook

optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

diarizers

hub-docs

Docs of the Hugging Face Hub

llm-swarm

Manage scalable open LLM inference endpoints in Slurm clusters

sam2-studio

optimum-neuron

Easy, fast and very cheap training and inference on AWS Trainium and Inferentia chips.

Jupyter Notebook

data-is-better-together

Let's build better datasets, together!

Jupyter Notebook

instruction-tuned-sd

Code for instruction-tuning Stable Diffusion.

simulate

🎢 Creating and sharing simulation environments for embodied and synthetic data research

OBELICS

Code used for the creation of OBELICS, an open, massive and curated collection of interleaved image-text web documents, containing 141M documents, 115B text tokens and 353M images.

diffusion-fast

Faster generation with text-to-image diffusion models.

olm-datasets

Pipeline for pulling and processing online language model pretraining data from the web

api-inference-community

jat

General multi-task deep RL Agent

workshops

Materials for workshops on the Hugging Face ecosystem

Jupyter Notebook

coreml-examples

Swift Core ML Examples

Jupyter Notebook

optimum-habana

Easy and lightning fast training of 🤗 Transformers on Habana Gaudi processor (HPU)

chug

Minimal sharded dataset loaders, decoders, and utils for multi-modal document, image, and text datasets.

sharp-transformers

A Unity plugin for using Transformers models in Unity.

hf-hub

Rust client for the huggingface hub aiming for minimal subset of features over `huggingface-hub` python package