• Stars
    star
    514
  • Rank 82,994 (Top 2 %)
  • Language
    Python
  • License
    BSD 2-Clause "Sim...
  • Created over 10 years ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Lock context manager implemented via redis SET NX EX and BLPOP.

Overview

docs Documentation Status
tests
package

Lock context manager implemented via redis SETNX/BLPOP.

  • Free software: BSD 2-Clause License

Interface targeted to be exactly like threading.Lock.

Usage

Because we don't want to require users to share the lock instance across processes you will have to give them names.

from redis import Redis
conn = Redis()

import redis_lock
lock = redis_lock.Lock(conn, "name-of-the-lock")
if lock.acquire(blocking=False):
    print("Got the lock.")
    lock.release()
else:
    print("Someone else has the lock.")

Locks as Context Managers

conn = StrictRedis()
with redis_lock.Lock(conn, "name-of-the-lock"):
    print("Got the lock. Doing some work ...")
    time.sleep(5)

You can also associate an identifier along with the lock so that it can be retrieved later by the same process, or by a different one. This is useful in cases where the application needs to identify the lock owner (find out who currently owns the lock).

import socket
host_id = "owned-by-%s" % socket.gethostname()
lock = redis_lock.Lock(conn, "name-of-the-lock", id=host_id)
if lock.acquire(blocking=False):
    assert lock.locked() is True
    print("Got the lock.")
    lock.release()
else:
    if lock.get_owner_id() == host_id:
        print("I already acquired this in another process.")
    else:
        print("The lock is held on another machine.")

Avoid dogpile effect in django

The dogpile is also known as the thundering herd effect or cache stampede. Here's a pattern to avoid the problem without serving stale data. The work will be performed a single time and every client will wait for the fresh data.

To use this you will need django-redis, however, python-redis-lock provides you a cache backend that has a cache method for your convenience. Just install python-redis-lock like this:

pip install "python-redis-lock[django]"

Now put something like this in your settings:

CACHES = {
    'default': {
        'BACKEND': 'redis_lock.django_cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient'
        }
    }
}

Note

If using a django-redis < 3.8.x, you'll probably need redis_cache which has been deprecated in favor to django_redis. The redis_cache module is removed in django-redis versions > 3.9.x. See django-redis notes.

This backend just adds a convenient .lock(name, expire=None) function to django-redis's cache backend.

You would write your functions like this:

from django.core.cache import cache

def function():
    val = cache.get(key)
    if not val:
        with cache.lock(key):
            val = cache.get(key)
            if not val:
                # DO EXPENSIVE WORK
                val = ...
                cache.set(key, value)
    return val

Troubleshooting

In some cases, the lock remains in redis forever (like a server blackout / redis or application crash / an unhandled exception). In such cases, the lock is not removed by restarting the application. One solution is to turn on the auto_renewal parameter in combination with expire to set a time-out on the lock, but let Lock() automatically keep resetting the expire time while your application code is executing:

# Get a lock with a 60-second lifetime but keep renewing it automatically
# to ensure the lock is held for as long as the Python process is running.
with redis_lock.Lock(conn, name='my-lock', expire=60, auto_renewal=True):
    # Do work....

Another solution is to use the reset_all() function when the application starts:

# On application start/restart
import redis_lock
redis_lock.reset_all()

Alternatively, you can reset individual locks via the reset method.

Use these carefully, if you understand what you do.

Features

  • based on the standard SETNX recipe
  • optional expiry
  • optional timeout
  • optional lock renewal (use a low expire but keep the lock active)
  • no spinloops at acquire

Implementation

redis_lock will use 2 keys for each lock named <name>:

  • lock:<name> - a string value for the actual lock
  • lock-signal:<name> - a list value for signaling the waiters when the lock is released

This is how it works:

python-redis-lock flow diagram

Documentation

https://python-redis-lock.readthedocs.io/en/latest/

Development

To run the all tests run:

tox

Requirements

OS:Any
Runtime:Python 2.7, 3.3 or later, or PyPy
Services:Redis 2.6.12 or later.

Similar projects

More Repositories

1

cookiecutter-pylibrary

Enhanced cookiecutter template for Python libraries.
Python
1,225
star
2

pytest-benchmark

py.test fixture for benchmarking code
Python
1,159
star
3

python-hunter

Hunter is a flexible code tracing toolkit.
Python
769
star
4

python-manhole

Debugging manhole for python applications.
Python
356
star
5

django-redisboard

Redis monitoring and inspection tool in django admin.
Python
265
star
6

python-remote-pdb

Remote vanilla PDB (over TCP sockets).
Python
252
star
7

python-lazy-object-proxy

A fast and thorough lazy object proxy.
Python
234
star
8

django-prefetch

Generic model related data prefetch framework for Django.
Python
153
star
9

python-tblib

Serialization library for Exceptions and Tracebacks.
Python
149
star
10

python-nameless

Sample project. Use https://github.com/ionelmc/cookiecutter-pylibrary to make your own project. The purpose of this repo is to test the CI configuration.
Python
145
star
11

python-fields

A totally different take on container boilerplate.
Python
137
star
12

jquery-gp-gallery

jQuery gallery plugin (ala google plus photo galeries)
CSS
126
star
13

python-aspectlib

An aspect-oriented programming, monkey-patch and decorators library. It is useful when changing behavior in existing code is desired. It includes tools for debugging and testing: simple mock/record and a complete capture/replay framework.
Python
108
star
14

django-monkey-team

Django middleware and userscript that displays debug tracebacks on production sites (where you would have DEBUG = False) only to developers.
Python
57
star
15

django-uwsgi-cache

uWSGI Django cache backend.
Python
38
star
16

django-admin-customizer

Django admin customizing interface
Python
36
star
17

python-holdup

A tool to wait for services and execute command. Useful in Docker containers.
Python
33
star
18

projectskel

Project skeleton for python 2.7 projects with fabric and virtualenv. It's intended for django projects but can be customized for other types of projects.
Python
28
star
19

sphinx-py3doc-enhanced-theme

A theme based on the theme of https://docs.python.org/3/ with some responsive enhancements.
JavaScript
25
star
20

django-admin-utils

Utility code for easier django admin development
Python
25
star
21

tox-wheel

A Tox plugin that builds and installs wheels instead of sdist. Note that this plugin is obsolte as tox 4.0 already has wheel support.
Python
23
star
22

docker-webdav

NGINX WebDAV container
Shell
23
star
23

python-cookiepatcher

Just a small shim around cookiecutter that alters a bit the CLI to work better when reapplying templates to existing projects.
Python
18
star
24

nose-htmloutput

Python
14
star
25

nose-timelimit

Nose plugin that allows you automatically skip tests that are too slow.
Python
13
star
26

cookiecutter-pylibrary-minimal

This has been merged into https://github.com/ionelmc/cookiecutter-pylibrary - use that instead!
Python
12
star
27

pypi-alias

A small utility to make alias distributions on PyPI.
Python
11
star
28

python-su

Python
9
star
29

python-process-tests

Testcase classes and assertions for testing processes.
Python
9
star
30

python-packaging-blunders

Python
8
star
31

polymer-select-box

Tagging widget implemented as a Polymer webcomponent
HTML
7
star
32

django-badbrowser

Browser detection (including browser upgrade notices) for Django
Python
7
star
33

python-mongoql-conv

Library to convert those MongoDB queries to something else, like a python expresion, a function or a django query (Q) object tree
Python
7
star
34

python-cogen

Automatically exported from https://code.google.com/p/cogen
Python
6
star
35

python-signalfd

CFFI bindings for signalfd.
Python
6
star
36

python-appengine-sdk

Un-official `pip install`-able AppEngine SDK.
Python
6
star
37

python-redis-throttled-queue

WIP
Python
6
star
38

python-unlzw

Python
6
star
39

python-stampede

Event-loop based, miniature job queue and worker that runs the task in a subprocess (via fork).
Python
6
star
40

django-easyfilters

Fork of https://bitbucket.org/evildmp/django-easyfilters/
Python
5
star
41

pytest-cover

Merged into https://github.com/schlamar/pytest-cov - use that instead!
Python
5
star
42

javascript-userscripts

Automatically exported from code.google.com/p/webmonkey-userscripts
JavaScript
4
star
43

python-mongosizeof

Python
4
star
44

pylint-fields

Pylint plugin for python-fields
Python
4
star
45

python-pygaljs

Python package providing assets from https://github.com/Kozea/pygal.js
Python
4
star
46

python-tax

2021 update: use tox-direct instead. This was a variant of Tox that didn't use virtualenvs at all - just installed everything in the current environment.
Python
4
star
47

django-secdownload-storage

Django storage backend that can be used to serve files via lighttpd's mod_secdownload module.
Python
4
star
48

docker-in-docker

An actually usable DIND. Includes a bunch of debug tools and docker-compose.
3
star
49

django-customfields

Couple of custom model fields for django: CachedManyToManyField and InheritedField
Python
3
star
50

python-pth

Simple and brief path traversal and filesystem access library.
Python
3
star
51

python-nameless-minimal

Python
3
star
52

python-ftpd-example

Python
2
star
53

dotfiles

My zsh setup
Shell
2
star
54

docker-manylinux

https://hub.docker.com/r/ionelmc/manylinux
Shell
2
star
55

python-matrix

Python
2
star
56

docker-buildpack-deps

Just buildpack-deps with some extras
Dockerfile
2
star
57

polymer-query-box

Query editor widget implemented as a Polymer webcomponent
JavaScript
2
star
58

polymer-json-box

Simple json edit widget implemented as a Polymer webcomponent
JavaScript
2
star
59

docker-fakebuntu

Ubuntu Xenial image running minimal services: systemd, journald, sshd, dind (docker in docker)
C
2
star
60

python-css-sprite

Python
1
star
61

django-image-editor

Allows to edit images in the browser
JavaScript
1
star
62

ppa-socat

Shell
1
star
63

pytest-benchmark-elasticsearch

Elasticseach storage backend for pytest-benchmark.
Python
1
star
64

t.ionelmc.ro

Google analytics to __utm.gif redirector.
Python
1
star
65

dockerskel

Abandoned. Check out https://github.com/evozon/django-docker
Shell
1
star