• Stars
    star
    137
  • Rank 266,121 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created about 5 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Drop-in MessagePack support for ASGI applications and frameworks

msgpack-asgi

Build Status Coverage Package version

msgpack-asgi allows you to add automatic MessagePack content negotiation to ASGI applications (Starlette, FastAPI, Quart, etc.), with a single line of code:

app.add_middleware(MessagePackMiddleware)

(You may want to adapt this snippet to your framework-specific middleware API.)

This gives you the bandwitdth usage reduction benefits of MessagePack without having to change existing code.

Note: this comes at a CPU usage cost, since MessagePackMiddleware will perform MsgPack decoding while your application continues to decode and encode JSON data (see also How it works). If your use case is CPU-sensitive, rather than strictly focused on reducing network bandwidth, this package may not be for you.

Installation

Install with pip:

pip install "msgpack-asgi==1.*"

Quickstart

First, you'll need an ASGI application. Let's use this sample application, which exposes an endpoint that returns JSON data:

# For convenience, we use some ASGI components from Starlette.
# Install with: `$ pip install starlette`.
from starlette.requests import Request
from starlette.responses import JSONResponse


async def get_response(request):
    if request.method == "POST":
        data = await request.json()
        return JSONResponse({"data": data}, status_code=201)
    else:
        return JSONResponse({"message": "Hello, msgpack!"})


async def app(scope, receive, send):
    assert scope["type"] == "http"
    request = Request(scope=scope, receive=receive)
    response = await get_response(request)
    await response(scope, receive, send)

Then, wrap your application around MessagePackMiddleware:

from msgpack_asgi import MessagePackMiddleware

app = MessagePackMiddleware(app)

Serve your application using an ASGI server, for example with Uvicorn:

uvicorn app:app

Now, let's make a request that accepts MessagePack data in response:

curl -i http://localhost:8000 -H "Accept: application/x-msgpack"

You should get the following output:

HTTP/1.1 200 OK
date: Fri, 01 Nov 2019 17:40:14 GMT
server: uvicorn
content-length: 25
content-type: application/x-msgpack

��message�Hello, msgpack!

What happened? Since we told the application that we accepted MessagePack-encoded responses, msgpack-asgi automatically converted the JSON data returned by the Starlette application to MessagePack.

We can make sure the response contains valid MessagePack data by making the request again in Python, and decoding the response content:

>>> import requests
>>> import msgpack
>>> url = "http://localhost:8000"
>>> headers = {"accept": "application/x-msgpack"}
>>> r = requests.get(url, headers=headers)
>>> r.content
b'\x81\xa7message\xafHello, msgpack!'
>>> msgpack.unpackb(r.content, raw=False)
{'message': 'Hello, msgpack!'}

msgpack-asgi also works in reverse: it will automatically decode MessagePack-encoded data sent by the client to JSON. We can try this out by making a POST request to our sample application with a MessagePack-encoded body:

>>> import requests
>>> import msgpack
>>> url = "http://localhost:8000"
>>> data = msgpack.packb({"message": "Hi, there!"})
>>> headers = {"content-type": "application/x-msgpack"}
>>> r = requests.post(url, data=data, headers=headers)
>>> r.json()
{'data': {'message': 'Hi, there!'}}

That's all there is to it! You can now go reduce the size of your payloads.

Advanced usage

Custom implementations

msgpack-asgi supports customizing the default encoding/decoding implementation. This is useful for fine-tuning application performance via an alternative msgpack implementation for encoding, decoding, or both.

To do so, use the following arguments:

  • packb - (Optional, type: (obj: Any) -> bytes, default: msgpack.packb) - Used to encode outgoing data.
  • unpackb - (Optional, type: (data: bytes) -> Any, default: msgpack.unpackb) - Used to decode incoming data.

For example, to use the ormsgpack library for encoding:

import ormsgpack  # Installed separately.
from msgpack_asgi import MessagePackMiddleware

def packb(obj):
    option = ...  # See `ormsgpack` options.
    return ormsgpack.packb(obj, option=option)

app = MessagePackMiddleware(..., packb=packb)

Limitations

msgpack-asgi does not support request or response streaming. This is because the full request and response body content has to be loaded in memory before it can be re-encoded.

How it works

An ASGI application wrapped around MessagePackMiddleware will perform automatic content negotiation based on the client's capabilities. More precisely:

  • If the client sends MessagePack-encoded data with the application/x-msgpack content type, msgpack-asgi will automatically re-encode the body to JSON and re-write the request Content-Type to application/json for your application to consume. (Note: this means applications will not be able to distinguish between MessagePack and JSON client requests.)
  • If the client sent the Accept: application/x-msgpack header, msgpack-asgi will automatically re-encode any JSON response data to MessagePack for the client to consume.

(In other cases, msgpack-asgi won't intervene at all.)

License

MIT

More Repositories

1

awesome-asgi

A curated list of awesome ASGI servers, frameworks, apps, libraries, and other resources
Python
1,303
star
2

djangorestframework-api-key

🔐 API key permissions for Django REST Framework
Python
578
star
3

aiometer

A Python concurrency scheduling library, compatible with asyncio and trio.
Python
252
star
4

asgi-lifespan

Programmatic startup/shutdown of ASGI apps.
Python
157
star
5

arel

Lightweight browser hot reload for Python ASGI web apps
Python
116
star
6

kafka-fraud-detector

🚨 Simple, self-contained fraud detection system built with Apache Kafka and Python
Python
77
star
7

httpx-sse

Consume Server-Sent Event (SSE) messages with HTTPX
Python
46
star
8

asgi-htmx

HTMX integration for ASGI applications
Python
39
star
9

starlette-auth-toolkit

Authentication backends and helpers for Starlette-based ASGI apps and frameworks
Python
32
star
10

www

Code for https://florimond.dev
Python
30
star
11

ddtrace-asgi

Unofficial Datadog tracing integration for ASGI apps and frameworks
Python
30
star
12

proxyx

[No maintenance intended] Proof of concept lightweight HTTP/1.1 proxy service built with ASGI and HTTPX.
Python
29
star
13

asgi-sitemaps

Sitemap generation for Python ASGI web apps
Python
20
star
14

asgi-caches

Server-side HTTP caching for ASGI applications, inspired by Django's cache framework
Python
18
star
15

pyboids

A boids flocking behaviour algorithm implementation in Python and Pygame
Python
14
star
16

python-in-browser

🐍🛥🌟 Running Python in the browser with Batavia and Starlette
Python
12
star
17

dataclasses-properties

🐍🤝 Reconciling Python's dataclasses and properties
Python
11
star
18

ng-courses

Fetch and display a list of courses by implementing the Model-Adapter Pattern
TypeScript
10
star
19

httpxprof

A tool for profiling HTTPX using cProfile and SnakeViz
Python
9
star
20

all-my-repos

Apply changes across all my repos using https://github.com/asottile/all-repos.
Python
7
star
21

personal

🖥 Personal blog frontend app
TypeScript
7
star
22

cs-ir

Implementation of an Information Retrieval System (IRS)
Jupyter Notebook
5
star
23

limier

Smart conversion and validation toolkit powered by type annotations
Python
4
star
24

paperiano

Paper + Piano (+ Computer Vision + Deep Learning) = Paperiano
Jupyter Notebook
4
star
25

personal-vue

⛰ Vue remake of my personal blog frontend app. Built for learning purposes.
Vue
4
star
26

subscriptions-transport-ws-python

Pure Python, asynchronous, event-loop-agnostic implementation of the subscriptions-transport-ws protocol
Python
4
star
27

azure-pipelines-templates

Azure Pipelines templates for my repos
3
star
28

azp-python-example

Opinionated example Python package Azure Pipelines setup w/ TestPyPI publish on tags
Shell
2
star
29

roller-coaster-loops

Calculus applied to roller coaster physics
Python
2
star
30

datacenters-drought

Data visualization of datacenters location and water resources vs US West 2021 extreme droughts
Python
2
star
31

personal-api

⚡️ Personal blog API
Python
2
star
32

httpx-unixsocket-poc

POC third-party package for adding Unix Domain Socket support to HTTPX
Python
2
star
33

fetch-metadata-asgi

PoC ASGI middleware implementation of the Fetch Metadata specification
Python
2
star
34

pygame-assets

Lightweight asset management for Pygame
Python
2
star
35

ansible-learn

Learning Ansible by using it to configure a Nginx + Python + Uvicorn + Gunicorn + Supervisor deployment in a Vagrant VM.
Makefile
2
star
36

django-clean-media

Unused media files cleaning plugin for Django
Python
1
star
37

aragon

Rule-based streaming data cleansing application written in Python
Python
1
star
38

dots

Small game made with Love2D
Lua
1
star
39

pytest-unasync

Pytest plugin for async-to-sync test generation
Python
1
star
40

python-package-template

My opinionated cookiecutter template for Python packages.
Python
1
star
41

uptv

📺 TV show alerting web app
Python
1
star
42

solitaire-rs

A Rust terminal UI (TUI) implementation of Solitaire
Rust
1
star
43

ddtrace-examples

Playing around with Datadog APM
Python
1
star
44

simulations

Fun mathematical simulations, using Python
Python
1
star
45

realchat

A web-based real-time chat room application made with Go, Svelte, and SocketIO.
Svelte
1
star
46

tower-defense

A cool Python game about defending a tower. All your base are belong to us.
Python
1
star
47

bocadillo-ws-test

Debugging WebSocket connection closures with Bocadillo and Uvicorn
Python
1
star