rust-bakery/nom rust-bakery/nom

  • Stars
    star
    9,369
  • Rank 3,816 (Top 0.08 %)
  • Language
    Rust
  • License
    MIT License
  • Created almost 10 years ago
  • Updated 3 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
rust-bakery/nom
github

rust-bakery

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Rust parser combinator framework

nom, eating data byte by byte

LICENSE Join the chat at https://gitter.im/Geal/nom Build Status Coverage Status crates.io Version Minimum rustc version

nom is a parser combinators library written in Rust. Its goal is to provide tools to build safe parsers without compromising the speed or memory consumption. To that end, it uses extensively Rust's strong typing and memory safety to produce fast and correct parsers, and provides functions, macros and traits to abstract most of the error prone plumbing.

nom logo in CC0 license, by Ange Albertini

nom will happily take a byte out of your files :)

Example

Hexadecimal color parser:

use nom::{
  bytes::complete::{tag, take_while_m_n},
  combinator::map_res,
  sequence::Tuple,
  IResult,
  Parser,
};

#[derive(Debug, PartialEq)]
pub struct Color {
  pub red: u8,
  pub green: u8,
  pub blue: u8,
}

fn from_hex(input: &str) -> Result<u8, std::num::ParseIntError> {
  u8::from_str_radix(input, 16)
}

fn is_hex_digit(c: char) -> bool {
  c.is_digit(16)
}

fn hex_primary(input: &str) -> IResult<&str, u8> {
  map_res(
    take_while_m_n(2, 2, is_hex_digit),
    from_hex
  ).parse(input)
}

fn hex_color(input: &str) -> IResult<&str, Color> {
  let (input, _) = tag("#")(input)?;
  let (input, (red, green, blue)) = (hex_primary, hex_primary, hex_primary).parse(input)?;
  Ok((input, Color { red, green, blue }))
}

fn main() {
  println!("{:?}", hex_color("#2F14DF"))
}

#[test]
fn parse_color() {
  assert_eq!(
    hex_color("#2F14DF"),
    Ok((
      "",
      Color {
        red: 47,
        green: 20,
        blue: 223,
      }
    ))
  );
}

Documentation

If you need any help developing your parsers, please ping geal on IRC (Libera, Geeknode, OFTC), go to #nom-parsers on Libera IRC, or on the Gitter chat room.

Why use nom

If you want to write:

Binary format parsers

nom was designed to properly parse binary formats from the beginning. Compared to the usual handwritten C parsers, nom parsers are just as fast, free from buffer overflow vulnerabilities, and handle common patterns for you:

  • TLV
  • Bit level parsing
  • Hexadecimal viewer in the debugging macros for easy data analysis
  • Streaming parsers for network formats and huge files

Example projects:

Text format parsers

While nom was made for binary format at first, it soon grew to work just as well with text formats. From line based formats like CSV, to more complex, nested formats such as JSON, nom can manage it, and provides you with useful tools:

  • Fast case insensitive comparison
  • Recognizers for escaped strings
  • Regular expressions can be embedded in nom parsers to represent complex character patterns succinctly
  • Special care has been given to managing non ASCII characters properly

Example projects:

Programming language parsers

While programming language parsers are usually written manually for more flexibility and performance, nom can be (and has been successfully) used as a prototyping parser for a language.

nom will get you started quickly with powerful custom error types, that you can leverage with nom_locate to pinpoint the exact line and column of the error. No need for separate tokenizing, lexing and parsing phases: nom can automatically handle whitespace parsing, and construct an AST in place.

Example projects:

Streaming formats

While a lot of formats (and the code handling them) assume that they can fit the complete data in memory, there are formats for which we only get a part of the data at once, like network formats, or huge files. nom has been designed for a correct behaviour with partial data: If there is not enough data to decide, nom will tell you it needs more instead of silently returning a wrong result. Whether your data comes entirely or in chunks, the result should be the same.

It allows you to build powerful, deterministic state machines for your protocols.

Example projects:

Parser combinators

Parser combinators are an approach to parsers that is very different from software like lex and yacc. Instead of writing the grammar in a separate file and generating the corresponding code, you use very small functions with very specific purpose, like "take 5 bytes", or "recognize the word 'HTTP'", and assemble them in meaningful patterns like "recognize 'HTTP', then a space, then a version". The resulting code is small, and looks like the grammar you would have written with other parser approaches.

This has a few advantages:

  • The parsers are small and easy to write
  • The parsers components are easy to reuse (if they're general enough, please add them to nom!)
  • The parsers components are easy to test separately (unit tests and property-based tests)
  • The parser combination code looks close to the grammar you would have written
  • You can build partial parsers, specific to the data you need at the moment, and ignore the rest

Technical features

nom parsers are for:

  • byte-oriented: The basic type is &[u8] and parsers will work as much as possible on byte array slices (but are not limited to them)
  • bit-oriented: nom can address a byte slice as a bit stream
  • string-oriented: The same kind of combinators can apply on UTF-8 strings as well
  • zero-copy: If a parser returns a subset of its input data, it will return a slice of that input, without copying
  • streaming: nom can work on partial data and detect when it needs more data to produce a correct result
  • descriptive errors: The parsers can aggregate a list of error codes with pointers to the incriminated input slice. Those error lists can be pattern matched to provide useful messages.
  • custom error types: You can provide a specific type to improve errors returned by parsers
  • safe parsing: nom leverages Rust's safe memory handling and powerful types, and parsers are routinely fuzzed and tested with real world data. So far, the only flaws found by fuzzing were in code written outside of nom
  • speed: Benchmarks have shown that nom parsers often outperform many parser combinators library like Parsec and attoparsec, some regular expression engines and even handwritten C parsers

Some benchmarks are available on GitHub.

Rust version requirements (MSRV)

The 7.0 series of nom supports Rustc version 1.56 or greater.

The current policy is that this will only be updated in the next major nom release.

Installation

nom is available on crates.io and can be included in your Cargo enabled project like this:

[dependencies]
nom = "7"

There are a few compilation features:

  • alloc: (activated by default) if disabled, nom can work in no_std builds without memory allocators. If enabled, combinators that allocate (like many0) will be available
  • std: (activated by default, activates alloc too) if disabled, nom can work in no_std builds

You can configure those features like this:

[dependencies.nom]
version = "7"
default-features = false
features = ["alloc"]

Related projects

Parsers written with nom

Here is a (non exhaustive) list of known projects using nom:

Want to create a new parser using nom? A list of not yet implemented formats is available here.

Want to add your parser here? Create a pull request for it!

Contributors

nom is the fruit of the work of many contributors over the years, many thanks for your help!

More Repositories

1

machine

Type checked state machine library in Rust
Rust
237
star
2

parser_benchmarks

Benchmarks for the nom, the Rust parser combinators library
C
122
star
3

cookie-factory

Rust serializer library
Rust
86
star
4

nom-peg

PEG parser generator built on top of nom
Rust
69
star
5

nom-derive

Declarative parsing for Rust, using a custom derive and nom
Rust
66
star
6

nom-trace

Rust
17
star
7

nom-bitvec

nom input types using bitvec
Rust
12
star
8

generator_nom

example code for using nom parsers with the new Rust generator feature
Rust
9
star
9

nom-bufreader

BufReader adapter for nom parsers
Rust
9
star
10

nom-text

better tools for text parsing
Rust
6
star
11

nom-methods

method combinators for nom
Rust
4
star
12

snack

a small nom
Rust
4
star
13

nom-whitespace

nom combinators to automatically parse whitespace between tokens
Rust
3
star
14

nom-regex

Rust
1
star