__ __ __
.---.-.--.--.--.-----.-----.-----.--------.-----.___.----.--.--.-----| |_ ___| | .--------.
| _ | | | | -__|__ --| _ | | -__|___| _| | |__ --| _|___| | | |
|___._|________|_____|_____|_____|__|__|__|_____| |__| |_____|_____|____| |__|__|__|__|__|
https://github.com/jondot/awesome-rust-llm
Awesome Rust LLM is an awesome style list that keeps track and curates the best Rust based LLM frameworks, libraries, tools, tutorials, articles and more. PRs are welcome!
- llm - a Rust library for running inference from a number of supported LLMs, loads ggml based models
- rust-bert - all in one pipelines for for transformer-based models (BERT, DistilBERT, GPT2,...). Good for local embedding, port of
transformers
(python) - llm-chain - chaining LLMs in Rust
- smartgpt (how it works)- use LLMs with the ability to complete complex tasks using plugins
- diffusers - Stable Diffusion using Rust, 45% faster than Pytorch
- postgresml - an amazing Postgres extension to do model fetching, inference all with SQL as part of your Postgres instance
-
aichat - a pure Rust CLI implementing AI chat, with advanced features such as real-time streaming, text highlighting and more
-
browser-agent - a headless browser driven by GPT-4. Sends off a simplified page representation and receives & executes instructions from GPT using a custom message format
-
tenere - TUI interface for LLMs
How it works? here's a prompt snippet:
You must respond with ONLY one of the following commands AND NOTHING ELSE:
- CLICK X - click on a given element. You can only click on links, buttons, and inputs!
- TYPE X \"TEXT\" - type the specified text into the input with id X and press ENTER
- ANSWER \"TEXT\" - Respond to the user with the specified text once you have completed the objective
- ajeto - LLM personal assistant
- shortgpt - Ask shortgpt for instant and concise answers
- autorust - macros that generate AI driven rust code in compile time
- clerk - LLM based file organizer
- gptcommit - prepare commit message with GPT
- indexify - A retrieval and long term memory service for LLMs
- memex - Super-simple, fully Rust powered "memory" (doc store + semantic search) for LLM projects, semantic search, etc.
- motorhead - a memory and information retrieval server for LLMs.
- Uses Redis as vector store for long term memory,
- Works with OpenAI API
- js example, python example
-
tiktoken - tiktoken is a Python library with a Rust core implementing a fast BPE tokeniser for use with OpenAI's models
- BPE is done in Rust
- Made by OpenAI
-
tiktoken-rs - a Rust focused library based on the
tiktoken
core with additional enhancements for use in Rust code. (Pyton parts intiktoken
done in pure Rust)
// Rust
use tiktoken_rs::p50k_base;
let bpe = p50k_base().unwrap();
let tokens = bpe.encode_with_special_tokens(
"This is a sentence with spaces"
);
println!("Token count: {}", tokens.len());
-
polars - a faster, pure Rust pandas alternative
-
rllama - a pure Rust implemenation of LLaMa inference. Great for embedding into other apps or wrapping for a scripting language.
-
whatlang - Rust library using a multiclass logistic regression model to detect languages
-
OpenAI API - a strongly typed Rust client for the OpenAI API
- spider - crawler / spider written in Rust for when you need a whole-website dump. Unlike Scrapy, focuses on dumping data but the post-processing is done later.
- dust - a full service for workflow running with composable blocks. Core is in Rust, various frontends in Typescript.
- pgvecto.rs - Vector database plugin for Postgres, written in Rust, specifically designed for LLM. 20x faster than pgvector
- qdrant - Qdrant - Vector Database for the next generation of AI applications