• Stars
    star
    2,340
  • Rank 19,661 (Top 0.4 %)
  • Language
    Python
  • License
    BSD 3-Clause "New...
  • Created almost 4 years ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML

msgspec

msgspec is a fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML. It features:

  • 🚀 High performance encoders/decoders for common protocols. The JSON and MessagePack implementations regularly benchmark as the fastest options for Python.

  • 🎉 Support for a wide variety of Python types. Additional types may be supported through extensions.

  • 🔍 Zero-cost schema validation using familiar Python type annotations. In benchmarks msgspec decodes and validates JSON ~2x faster than orjson can decode it alone.

  • A speedy Struct type for representing structured data. If you already use dataclasses or attrs, structs should feel familiar. However, they're 10-100x faster for common operations.

All of this is included in a lightweight library with no required dependencies.


msgspec may be used for serialization alone, as a faster JSON or MessagePack library. For the greatest benefit though, we recommend using msgspec to handle the full serialization & validation workflow:

Define your message schemas using standard Python type annotations.

>>> import msgspec

>>> class User(msgspec.Struct):
...     """A new type describing a User"""
...     name: str
...     groups: set[str] = set()
...     email: str | None = None

Encode messages as JSON, or one of the many other supported protocols.

>>> alice = User("alice", groups={"admin", "engineering"})

>>> alice
User(name='alice', groups={"admin", "engineering"}, email=None)

>>> msg = msgspec.json.encode(alice)

>>> msg
b'{"name":"alice","groups":["admin","engineering"],"email":null}'

Decode messages back into Python objects, with optional schema validation.

>>> msgspec.json.decode(msg, type=User)
User(name='alice', groups={"admin", "engineering"}, email=None)

>>> msgspec.json.decode(b'{"name":"bob","groups":[123]}', type=User)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
msgspec.ValidationError: Expected `str`, got `int` - at `$.groups[0]`

msgspec is designed to be as performant as possible, while retaining some of the nicities of validation libraries like pydantic. For supported types, encoding/decoding a message with msgspec can be ~10-80x faster than alternative libraries.

See the documentation for more information.

LICENSE

New BSD. See the License File.

More Repositories

1

skein

A tool and library for easily deploying applications on Apache YARN
Python
142
star
2

quickle

A quicker pickle 🥒
C
108
star
3

ptime

IPython magic for parallel profiling (like `%time`, but parallel)
Python
71
star
4

venv-pack

Package virtual environments for redistribution
Python
44
star
5

hadoop-test-cluster

Dockerized setup for testing code on realistic hadoop clusters
Python
27
star
6

dask-tutorial-pydata-seattle-2017

Dask Tutorial for PyData Seattle 2017
Jupyter Notebook
21
star
7

codegen_talk

Talk for PyMNtos on code generation in SymPy
18
star
8

pycallgrind

Utilities for profiling python code with callgrind
Python
17
star
9

alpine-dask-docker

Tiny Dask Docker images based on Alpine Linux
Dockerfile
17
star
10

Dask_PyData_NYC

Materials for dask talk at PyData NYC
15
star
11

talks

Various presentations I've given on things.
Jupyter Notebook
14
star
12

hdfscm

An HDFS backed ContentsManager implementation for Jupyter
Python
12
star
13

ibis-datasette

An ibis backend for querying datasette
Python
10
star
14

harvard_cs207_talk

Materials for guest lecture on dask for Harvard CS207
CSS
10
star
15

symcc

Experimental mathematics compilation library for SymPy
Python
9
star
16

Old-Control.jl

Development has been moved to https://github.com/JuliaControl/Control.jl
Julia
8
star
17

automations

Autobots, roll out!
Python
6
star
18

eio

Python
5
star
19

Juniper.jl

A simple computer algebra system, in Julia
Julia
5
star
20

namespace_modules

Example of using namespace modules
Python
2
star
21

prefect-github-example

An example repo for organizing and deploying Prefect flows on GitHub
Python
2
star
22

pyblis

Python interface for BLIS
Python
2
star
23

shmarray

NumPy arrays backed by shared memory
C
2
star
24

Slicot.jl

Julia wrapper for SLICOT Routines
Julia
1
star
25

jupyterhub-jobmanager

Manage user jobs in JupyterHub
Python
1
star
26

pdbokeh

Pandas plotting using bokeh. Do not use.
Python
1
star
27

smolpy

A smol and safe Python VM for executing user-defined functions
Python
1
star