• Stars
    star
    495
  • Rank 88,974 (Top 2 %)
  • Language
    C
  • License
    MIT License
  • Created almost 14 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Python wrapper for hiredis

hiredis-py

Build Status License

Python extension that wraps protocol parsing code in hiredis. It primarily speeds up parsing of multi bulk replies.

Install

hiredis-py is available on PyPI, and can be installed via:

pip install hiredis

Building and Testing

Building this repository requires a recursive checkout of submodules, and building hiredis. The following example shows how to clone, compile, and run tests. Please note - you will need the gcc installed.

git clone --recursse-submodules https://github.com/redis/hiredis-py
python setup.py build_ext --inplace
python -m pytest

Requirements

hiredis-py requires Python 3.7+.

Make sure Python development headers are available when installing hiredis-py. On Ubuntu/Debian systems, install them with apt-get install python3-dev.

Usage

The hiredis module contains the Reader class. This class is responsible for parsing replies from the stream of data that is read from a Redis connection. It does not contain functionality to handle I/O.

Reply parser

The Reader class has two methods that are used when parsing replies from a stream of data. Reader.feed takes a string argument that is appended to the internal buffer. Reader.gets reads this buffer and returns a reply when the buffer contains a full reply. If a single call to feed contains multiple replies, gets should be called multiple times to extract all replies.

Example:

>>> reader = hiredis.Reader()
>>> reader.feed("$5\r\nhello\r\n")
>>> reader.gets()
b'hello'

When the buffer does not contain a full reply, gets returns False. This means extra data is needed and feed should be called again before calling gets again. Alternatively you could provide custom sentinel object via parameter, which is useful for RESP3 protocol where native boolean types are supported:

Example:

>>> reader.feed("*2\r\n$5\r\nhello\r\n")
>>> reader.gets()
False
>>> reader.feed("$5\r\nworld\r\n")
>>> reader.gets()
[b'hello', b'world']
>>> reader = hiredis.Reader(notEnoughData=Ellipsis)
>>> reader.gets()
Ellipsis

Unicode

hiredis.Reader is able to decode bulk data to any encoding Python supports. To do so, specify the encoding you want to use for decoding replies when initializing it:

>>> reader = hiredis.Reader(encoding="utf-8", errors="strict")
>>> reader.feed(b"$3\r\n\xe2\x98\x83\r\n")
>>> reader.gets()
'β˜ƒ'

Decoding of bulk data will be attempted using the specified encoding and error handler. If the error handler is 'strict' (the default), a UnicodeDecodeError is raised when data cannot be dedcoded. This is identical to Python's default behavior. Other valid values to errors include 'replace', 'ignore', and 'backslashreplace'. More information on the behavior of these error handlers can be found here.

When the specified encoding cannot be found, a LookupError will be raised when calling gets for the first reply with bulk data.

Error handling

When a protocol error occurs (because of multiple threads using the same socket, or some other condition that causes a corrupt stream), the error hiredis.ProtocolError is raised. Because the buffer is read in a lazy fashion, it will only be raised when gets is called and the first reply in the buffer contains an error. There is no way to recover from a faulty protocol state, so when this happens, the I/O code feeding data to Reader should probably reconnect.

Redis can reply with error replies (-ERR ...). For these replies, the custom error class hiredis.ReplyError is returned, but not raised.

When other error types should be used (so existing code doesn't have to change its except clauses), Reader can be initialized with the protocolError and replyError keywords. These keywords should contain a class that is a subclass of Exception. When not provided, Reader will use the default error types.

Benchmarks

The repository contains a benchmarking script in the benchmark directory, which uses gevent to have non-blocking I/O and redis-py to handle connections. These benchmarks are done with a patched version of redis-py that uses hiredis-py when it is available.

All benchmarks are done with 10 concurrent connections.

  • SET key value + GET key
    • redis-py: 11.76 Kops
    • redis-py with hiredis-py: 13.40 Kops
    • improvement: 1.1x

List entries in the following tests are 5 bytes.

  • LRANGE list 0 9:
    • redis-py: 4.78 Kops
    • redis-py with hiredis-py: 12.94 Kops
    • improvement: 2.7x
  • LRANGE list 0 99:
    • redis-py: 0.73 Kops
    • redis-py with hiredis-py: 11.90 Kops
    • improvement: 16.3x
  • LRANGE list 0 999:
    • redis-py: 0.07 Kops
    • redis-py with hiredis-py: 5.83 Kops
    • improvement: 83.2x

Throughput improvement for simple SET/GET is minimal, but the larger multi bulk replies get, the larger the performance improvement is.

License

This code is released under the BSD license, after the license of hiredis.

More Repositories

1

redis

Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.
C
66,875
star
2

go-redis

Redis Go client
Go
19,891
star
3

node-redis

Redis Node.js client
TypeScript
16,841
star
4

ioredis

πŸš€ A robust, performance-focused, and full-featured Redis client for Node.js.
TypeScript
14,344
star
5

redis-py

Redis Python client
Python
12,506
star
6

jedis

Redis Java client
Java
11,766
star
7

hiredis

Minimalistic C client for Redis >= 1.2
C
6,197
star
8

lettuce

Advanced Java Redis client for thread-safe sync, async, and reactive usage. Supports Cluster, Sentinel, Pipelining, and codecs.
Java
5,352
star
9

redis-rb

A Ruby client library for Redis
Ruby
3,963
star
10

rueidis

A fast Golang Redis client that supports Client Side Caching, Auto Pipelining, Generics OM, RedisJSON, RedisBloom, RediSearch, etc.
Go
2,327
star
11

redis-doc

Redis documentation source code for markdown and metadata files, conversion scripts, and so forth
Shell
2,310
star
12

redis-om-node

Object mapping, and more, for Redis and Node.js. Written in TypeScript.
TypeScript
1,158
star
13

docker-library-redis

Docker Official Image packaging for Redis
Shell
1,117
star
14

redis-om-python

Object mapping, and more, for Redis and Python
Python
1,093
star
15

redis-io

Application running http://redis.io
Ruby
637
star
16

redis-om-spring

Spring Data Redis extensions for better search, documents models, and more
Java
599
star
17

redis-om-dotnet

Object mapping, and more, for Redis and .NET
C#
457
star
18

hiredis-rb

Ruby wrapper for hiredis
Ruby
319
star
19

hiredis-node

Node wrapper for hiredis
JavaScript
305
star
20

riot

🧨 Get data in & out of Redis with RIOT
Java
273
star
21

NRedisStack

Redis Stack .Net client
C#
220
star
22

redis-vl-python

Redis Vector Library (RedisVL) interfaces with Redis' vector database for realtime semantic search, RAG, and recommendation systems.
Python
218
star
23

redis-rcp

Redis Change Proposals
136
star
24

redis-hashes

Redis tarball SHA1 hashes
92
star
25

lettucemod

Java client for Redis Modules
Java
50
star
26

spring-batch-redis

Spring Batch extension for Redis
Java
47
star
27

redis-specifications

A bin for Redis' specs
38
star
28

minipilot

MiniPilot is a GenAI-assisted chatbot backed by Redis. Chat with your documents
HTML
31
star
29

redis-benchmarks-specification

The Redis benchmarks specification describes the cross-language/tools requirements and expectations to foster performance and observability standards around redis related technologies. Members from both industry and academia, including organizations and individuals are encouraged to contribute.
Python
29
star
30

librdb

Redis RDB file parser, with JSON, RESP and RDB-loader extensions
C
23
star
31

docs

Documentation for Redis, Redis Cloud, and Redis Enterprise
Python
23
star
32

redis-debian

Debian packaging
Shell
18
star
33

redis-snap

A repository for snap packaging
10
star
34

redis-website

HTML
9
star
35

Redis-Insight-Guides

Learn modern data models and data processing tools bundled in Redis Stack to build real-time applications with the speed and stability of Redis.
5
star
36

redis-clinterwebz

Python
4
star
37

redis-extra-ci

4
star
38

riot-docker

Dockerfile
2
star
39

redis-rpm

Shell
2
star
40

scoop

1
star
41

homebrew-tap

Homebrew tap for Redis organization
Ruby
1
star