Langdev libraries for Rust
- Lexers
- Parsers
- Codegen
- String interning
- Just-in-time compilation
- Error reporting
- Language server protocol
- Testing
- Incremental compilation
- Floats/Ints/Bools
- Binary & object file parsing, generating and processing
- Solvers
- CLI
- Repl
- String handling
- Syntax trees
- Pretty printing
- Variable binding
- Caching
- WASM
- Logging
- Storage
- Incremental Analysis
- Fuzzing
- Graphs
- Type Checking
- Peephole Optimization
- Garbage Collection
- Guides & Resources
- Uncategorized
Lexers
logos
Logos has two goals: To make it easy to create a Lexer, so you can focus on more complex problems and to make the generated Lexer faster than anything you'd write by handlrlex
A replacement forlex
/flex
that generates Rust codelexgen
A fully-featured lexer generator, implemented as a proc macro.
Parsers
lalrpop
A convenient LR(1) parser generatornom
A byte-oriented, zero-copy, parser combinators librarycombine
Fast parser combinators on arbitrary streams with zero-copy support.pom
PEG parser combinators using operator overloading without macros.peg
A simple Parsing Expression Grammar (PEG) parser generator.glue
Glue is a parser combinator framework for parsing text based formats, it is easy to use and relatively fast toopratt
A general purpose pratt parser for Rustpest
A general purpose parser written in Rust with a focus on accessibility, correctness, and performance using parsing expression grammars (PEG)lrpar
A Yacc-compatible parser.nimbleparse_lsp
An LSP server for quickly developing lrpar parsers, which parses input buffers on grammar change bypassing code generation.
tree-sitter
A parser generator for building fast editor-friendly parsers with features like streaming, incremental, resumable (re)parsing, with timeouts and cancellations, and advanced syntax node queries.rowan
Generic lossless syntax treesungrammar
A DSL for specifying concrete syntax trees, see this introductory post
cstree
A fork ofrowan
with threadsafe syntax trees and built-in source string interningchomp
Chomp is a fast monadic-style parser combinator library designed to work on stable Rustoak
A typed parser generator embedded in Rust code for Parsing Expression Grammarschumsky
A friendly parser combinator crate that makes writing LL(k) parsers with error recovery easyrust-sitter
A package for defining tree-sitter grammars alongside Rust logic
Regular expressions
regex
An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs.regress
A regular expression engine targeting EcmaScript syntax.onig
Rust-Onig is a set of Rust bindings for the Oniguruma regular expression library.
Codegen
cranelift
Cranelift is a low-level retargetable code generatorllvm-sys
Bindings to LLVM's C APIiced-x86
A blazing fast and correct x86/x64 disassembler, assembler and instruction decoderllama
Friendly LLVM bindingsinkwell
Inkwell aims to help you pen your own programming languages by safely wrapping llvm-sysllvm-ir
LLVM IR in natural Rust data structuresllvmenv
Manage multiple LLVM/Clang buildswalrus
A library for performing WebAssembly transformationscilk
Toy Compiler Infrastructure influenced by LLVM written in Rust.
String interning
lasso
A multithreaded and single threaded string interner with a minimal memory footprint and arena allocationstring-interner
A data structure to cache strings efficientlysimple-interner
A simple append-only internerstring_cache
A string interning library for Rust, developed as part of the Servo project
Just-in-time Compilation
iced-x86
A blazing fast and correct x86/x64 disassembler, assembler and instruction decodermasm-rs
A JSC/SpiderMonkey like macro assemblerb3
Rust port of B3, LLVM-like backendjit
(LibJIT) Just-In-Time Compilation in Rust using LibJIT bindingslightning-sys
GNU lightning bindings for rustgccjit
Higher-level Rust bindings for libgccjitcranelift
Cranelift is a low-level retargetable code generatorcranelift-jit-demo
A JIT compiler and runtime for a toy language, using Cranelift
dynasm
A Dynamic assembler written in Rust for Rust
Error reporting
codespan-reporting
Beautiful diagnostic reporting for text-based programming languagescodespan
Data structures for tracking locations in source codetext-size
A library that provides newtype wrappers foru32
and(u32, u32)
for use as text offsetsariadne
A fancy diagnostics & reporting cratemiette
Fancy diagnostic reporting library and protocol for us mere mortals who aren't compiler hackers
Language server protocol
lsp-types
Types for interaction with a language server, using VSCode's Language Server Protocoltower-lsp
Language Server Protocol implementation based on Towercodespan-lsp
Conversions between codespan types and Language Server Protocol typeslsp-server
A generic LSP server scaffold
Testing
goldentests
A golden file testing library where tests can be configured within the same test filelang_tester
Concise language testing framework for compilers and VMs (Linux only)libtest-mimic
Dynamically construct a test-harness that looks and behaves like Rust's built-in test harnesscompiletest_rs
The compiletest utility from the Rust compiler as a standalone testing harnessinsta
A snapshot testing library for Rustk9
Snapshot testing and better assertionsbulk_examples_generator
A tool created in Rust for create dozens/hundreds/thousands/millions of random examples based on a pest grammar (PEG)
Unicode
unicode-xid
Determine if a char is a valid identifier for a parser and/or lexer according to Unicode Standard Annex #31 rules.unicode-normalisation
Unicode character composition and decomposition utilities as described in Unicode Standard Annex #15unicode-segmentation
Iterators which split strings on Grapheme Cluster or Word boundaries, according to the Unicode Standard Annex #29 rulesunicode-width
Determine displayed width of char and str types according to Unicode Standard Annex #11 ruleslexical-sort
Lexicographical sorting of unicode strings
Incremental compilation
salsa
A generic framework for on-demand, incrementalized computationverde
A refreshingly simple incremental computation library
Floats/Ints/Bools
lexical
Lexical, to- and from-string conversion routines (Fast lexical conversion routines for bothstd
andno_std
environments)lexical-core
Lexical, to- and from-string conversion routines (Low-level, lexical conversion routines for use in ano_std
context)lexical_bool
A bool-like type that can be parsed from a stringryu
A Rust implementation of the PLDI'18 paper Ryลซ: fast float-to-string conversion by Ulf Adamsryu-js
Ryลซ-js is a fork of the ryu crate adjusted to comply to the ECMAScript number-to-string algorithmhexponent
C11 compliant hex float parsingordered-float
Total ordering on floatshalf
A half-precision floating pointf16
type for Rust implementing the IEEE 754-2008 standardf128
Bindings to the gcc quadmath libraryapprox
Approximate floating point equality comparisons and assertions
Binary & object file parsing, generating and processing
goblin
An impish, cross-platform, ELF, Mach-o, and PE binary parsing and loading crategimli
A library for reading and writing the DWARF debugging format.faerie
ELF and Mach-o native binary object file emitterobject
A unified interface for reading and writing object file formats.elf
A pure-rust library for parsing ELF fileselfkit
An elf parser and manipulation library in pure rust
Symbolic Execution
haybale
Symbolic execution of LLVM IR
Solvers
rsmt2
Wrapper for SMT-LIB 2 compliant SMT solvers.z3
A high-level rust bindings for the Z3 SMT solver from Microsoft Researchz3-sys
Low-level bindings for the Z3 SMT solver from Microsoft Researchz3_ref
A high level interface to the Z3 SMT solverz3d
Z3 DSL interface for Rustboolector
Safe high-level bindings for the Boolector SMT solverboolector-sys
Low-level bindings for the Boolector SMT solversmt2utils
Libraries and tools for the SMT-LIB-2 standardsmt2parser
A generic parser for SMT2 commands, as specified by the SMT-LIB-2 standardsmt2proxy
An experimental binary tool to intercept and pre-process SMT2 commands before they are send to an SMT solverz3tracer
An experimental parser for Z3 tracing logs obtained by passingtrace=true proof=true
to Z3
good_lp
Mixed Integer Linear Programming for Rust, with an user-friendly API. This crate allows modeling LP problems, and lets you solve them with various solvers.- Axiom Profiler A tool for visualising, analysing and understanding quantifier instantiations made via E-matching in a run of an SMT solver
- [2017][PDF] Counterexample Guided Inductive Optimization based on Satisfiability Modulo Theories
CLI
structopt
Parse command line arguments by defining a structclap
A simple to use, efficient, and full-featured Command Line Argument Parserpico-args
An ultra simple CLI arguments parserargh
A derive-based argument parser optimized for code size
Configuration
etcetera
A library that aims to allow you to determine the locations of configuration, cache and data files for your application. Existing Rust libraries generally do not give you a choice in terms of which standards/conventions (Etcetera calls these โstrategiesโ) they follow. Etcetera, on the other hand, gives you the choicetoml
A TOML decoder and encoder for Rust (optionally uses serde)envy
Deserialize environment variables into typesafe structs (uses serde)
Repl
rustyline
Rustyline, a readline implementation based on Antirez's Linenoiserepl-rs
Library to generate a REPL for your applicationtermwiz
Terminal Wizardry for Unix and Windows
String handling
beef
Faster, more compact implementation ofCow
.smol_str
Small-string optimized string type with O(1) clonesmallstring
'Small string' optimization: store small strings on the stack using smallvecheck
Heck is a case conversion library that exists to provide case conversion between common cases like CamelCase and snake_case. It is intended to be unicode aware, internally consistent, and reasonably well performingropey
Ropey is a utf8 text rope for Rust, designed to be the backing text-buffer for applications such as text editors. Ropey is fast, robust, and can handle huge texts and memory-incoherent edits with ease.
Syntax trees
rowan
Generic lossless syntax treescstree
A fork ofrowan
with threadsafe syntax trees and built-in source string interning
Pretty printing
pretty
Wadler-style pretty-printing combinators in Rust
Variable binding
moniker
An automagical variable binding library for tracking variables in scopes
Caching
cached
Caching structures and simplified function memoization
WASM
wain
WebAssembly interpreter written in Safe Rust with zero dependencieswasmer
The high-level public API of the Wasmer WebAssembly runtimewasmtime
High-level API to expose the Wasmtime runtimewasmtime-jit
JIT-style execution for WebAsssembly code in Craneliftwasmlite-parser
This crate parses WebAssembly modules and can generate LLVM IR based the data extracted from a moduleparity-wasm-cp
WebAssembly binary format serialization/deserialization/interpretersubstrate-wasm-builder
A utility for building WASM binarieswalrus
A library for performing WebAssembly transformations
Logging
tracing
Application-level tracing for Rusttracing-tree
A Tracing Layer which prints a tree of spans and events.tracing-forest
Preserving contextual coherence among trace data from concurrent taskstracing-timing
Inter-event timing metrics on top oftracing
tracing-coz
Rust-tracing support for the coz Causal Profilertracing-flame
A tracingLayer
for generating a folded stack trace for generating flamegraphs and flamecharts with infernotest-env-log
A crate that takes care of automatically initializing logging and/or tracing for Rust tests.tracing-unwrap
This crate provides.unwrap_or_log()
and.expect_or_log()
methods onResult
andOption
types that log failed unwraps to atracing::Subscriber
log
A Rust library providing a lightweight logging facadeslog
An ecosystem of reusable components for structured, extensible, composable and contextual logging for Rust
Storage
slotmap
Containers with persistent unique keys to access stored values: Insertion, deletion and access all take O(1) time with low overhead.indexmap
A hash table with consistent order and fast iterationvecmap
Vec-based Map and Set data structures
Disjoint-sets
union-find
Struct and methods for union-find operationena
An implementation of union-find in Rust; extracted from (and used by) rustc
Incremental Analysis
timely
A low-latency cyclic dataflow computational model, introduced in the paper Naiad: a timely dataflow systemdifferential-dataflow
A data-parallel programming framework designed to efficiently process large volumes of data and to quickly respond to arbitrary changes in input collections, an implementation of differential dataflow over timely dataflow on Rust.dogsdogsdogs
Worst-case optimal joins and delta queries in differential dataflowdataflow-join
A streaming implementation of Ngo et al'sGenericJoin
in timely dataflowtimely-sort
An cache-aware implementation of radix sort in Rustdiagnostics
Diagnostic tools fortimely-datafflow
anddifferential-dataflow
computations
differential-datalog
A programming language for incremental computation based ontimely-dataflow
anddifferential-dataflow
, similar toIncA
orSouffle
Timely Dataflow Resources
- Naiad: a timely dataflow system
- Timely Dataflow Book
- Differential Dataflow
- Differential Dataflow Book
- Sudoku in Differential Dataflow: blog, video series
- An introduction to Timely Dataflow: part 1, part 2, part 3
- An introduction to Differential Dataflow: part 1, part 2
- Differential Dataflow meets Calculus
Fuzzing
honggfuzz-rs
Honggfuzz is a security oriented fuzzer with powerful analysis options that supports evolutionary, feedback-driven fuzzing based on code coveragecargo-fuzz
A cargo subcommand for using libFuzzerlibfuzzer-sys
Barebones wrapper around LLVM's libFuzzer runtime librarywasm-smith
A WebAssembly test case generatorfuzzcheck
A structure-aware coverage-guided fuzzerlibafl
Fuzzing Resources
Graphs
egg
Egg is a flexible, high-performance e-graph librarypetgraph
A graph data structure library. Provides graph types and graph algorithmstimely
A low-latency cyclic dataflow computational model, introduced in the paper Naiad: a timely dataflow systemdifferential-dataflow
A data-parallel programming framework designed to efficiently process large volumes of data and to quickly respond to arbitrary changes in input collections, an implementation of differential dataflow over timely dataflow on Rust.
Type Checking
rusttyc
An interface that allows for an intuitive translation of inference rules based on a Hindney-Milner-like bounded type meet-semilattice into rust codepolytype
A Hindley-Milner polymorphic typing system
Peephole Optimization
egg
Egg is a flexible, high-performance e-graph librarypeepmatic
A DSL for peephole optimizations and compiler for generating peephole optimizers from them
Garbage Collection
broom
An ergonomic tracing garbage collector that supports mark 'n sweep garbage collectiongc
A simple tracing (mark and sweep) garbage collector for Rustshredder
Garbage collection as a library for Rustcomet
A garbage collection library for implementing VMs in Rustmmtk
Memory Management ToolKit (MMTk) for portable & efficient memory management in language VMs
Guides & Resources
- Making a language in Rust A series about making a programming language called Eldiro using the Rust programming language
- Crafting Interpreters Implementing everything from parsing to garbage collection to bytecode VMs, highly reccomended as an introduction to language development (Homepage)
- Create Your Own Programming Language with Rust
- Where to Start Hand-Writing a Parser (in Rust)
- Writing a simple parser in Rust
- GC and Rust part 0, part 1 and part 2
- Matklad on pratt parsing: Simple but Powerful Pratt Parsing and From Pratt to Dikjstra
- Pure AST based linting sucks
- LLVM's Kaleidoscope in Rust The Kaleidoscope Language tutorial, showing how to implement a simple language using LLVM in Rust
- Where to Start Hand-Writing a Parser (in Rust) An introduction to programming language parsing in which we hand-write a parser and run it on some real input
- x86 and amd64 instruction reference
- X86 Opcode and Instruction Reference
- Linear Scan Register Allocation on SSA Form
- A bibliography of papers related to symbolic execution
- Instruction Latencies, Throughput and Micro-Operation Breakdowns for Intel, AMD and VIA CPUs
- [1995][PDF] Symbolic Range Propagation
- [2001][PDF] Efficient Symbolic Analysis for Optimizing Compilers
- [2021][Video] A New Approach to Removing Redundant Conditions in LLVM
- [2017][Video] Polyhedral Value & Memory Analysis
- Simplify: A Theorem Prover for Program Checking
- Elegant and performant recursion in Rust
- Speculation in JavaScriptCore
- Whippet: A new GC for Guile
Type Checking
- Type Checking in less than 100 lines of Rust (Credit to @zesterer)
- Bidirectional Type Checking in ~200 lines of OCaml (Written in OCaml, credit to @mb64)
- [2020][PDF] Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism
- Implementation of "Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism" (Written in Rust, from @segeljakt)
- Implementation of "Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism" (Written in Rust, from @JDemler)
- A bidirectional type inference system loosely based on Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism. (Written in Haskell)
- Didactic implementation of the type checker described in "Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism" (Written in OCaml)
- An implementation of a predicative polymorphic language with bidirectional type inference and algebraic data types (Written in Haskell)
Inlining
- Inlining Decisions in Visual Studio
- C++ Inliner Improvements: The Zipliner
- John Carmack on Inlined Code
- How RyuJIT inlines a function (heuristics)
- Smarter C/C++ inlining with attribute((flatten))
- Do compilers take inline as a hint?
- cmd/compile: improve inlining cost model
- What does 'inline' mean?
- Zebu VM check_should_inline_func()
- Warp: Improved JS performance in Firefox 83
- When Short Methods Pay Off: JIT Inlining
- cross-module inlining in guile -- wingolog
- Inline In Rust
- Itโs Not Always iCache
- [2003][PDF] Adaptive Online Context-Sensitive Inlining
- [2003][PDF] To Inline or Not to Inline. Enhanced Inlining Decisions
- [2004][PDF] Fast and Effective Procedure Inlining
- [2016][PDF] CSE P 501 - Compilers. Inlining and Devirtualization
- [2017][PDF] TurboFan Inlining Heuristics
- [2019][PDF] Guiding Inlining Decisions Using Post-Inlining Transformations
- [2019][PDF] An Optimization-Driven Incremental Inline Substitution Algorithm for Just-in-Time Compilers
- [2021][PDF] Inlining for Code Size Reduction
RVSDG
- [2020][PDF] RVSDG: An Intermediate Representation for Optimizing Compilers
- [2018][PDF] RVSDG: An Intermediate Representation for the Multi-Core Era
- [2015][PDF] Perfect Reconstructability of Control Flow from Demand Dependence Graphs
- Principles, Techniques, and Tools for Explicit and Automatic Parallelization
- [Slides] Compiling with the Regionalized Value State Dependence Graph
- JLM RVSDG Implementation (C++)
- JIVE RVSDG Graph Implementation (C++)
- RVSDG Viewer (C++)
- Cranial Coitus RVSDG Brainfuck Optimizer (Rust)
- [2018][PDF] Semantic Reasoning About the Sea of Nodes
- [1994][PDF] Value dependence graphs: representation without taxation
- [2016][PDF] Value State Flow Graph: A Dataflow Compiler IR for Accelerating Control-Intensive Code in Spatial Hardware
- Numba Python JIT RVSDG Optimizer
- [Archived] Rain IR A dependently-typed RVSDG with liftetimes, implemented in Rust
Equality Saturation
- [2010][PDF] Equality Saturation: A New Approach to Optimization
- [2020][PDF] egg: Fast and Extensible Equality Saturation
- [2021][PDF] Rewrite Rule Inference Using Equality Saturation
- [2021][PDF] Sketch-Guided Equality Saturation: Scaling Equality Saturation to Complex Optimizations in Languages with Bindings
- [2021][PDF] Caviar: An E-graph Based TRS for Automatic Code Optimization
- [2002][PDF] SPORES: Sum-Product Optimization via Relational Equality Saturation for Large Scale Linear Algebra
- [2021][PDF] Relational E-Matching
- [2021][PDF] Rewrite Rule Inference Using Equality Saturation
Uncategorized
- Turns are Better than Radians An interesting blog post about using turns instead of pi or tau for trigometric calculations, could be inspiration for some interesting code transformations
- Algorithms for Modern Hardware (russian version) Low-level performance tuning for modern hardware