• Stars
    star
    347
  • Rank 122,141 (Top 3 %)
  • Language
    Python
  • License
    Other
  • Created about 7 years ago
  • Updated 6 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Django app that integrates with Dramatiq.

django_dramatiq

Build Status PyPI version

django_dramatiq is a Django app that integrates with Dramatiq.

Requirements

Example

You can find an example application built with django_dramatiq here.

Installation

pip install django-dramatiq

Add django_dramatiq to installed apps before any of your custom apps:

import os

INSTALLED_APPS = [
    "django_dramatiq",

    "myprojectapp1",
    "myprojectapp2",
    # etc...
]

Configure your broker in settings.py:

DRAMATIQ_BROKER = {
    "BROKER": "dramatiq.brokers.rabbitmq.RabbitmqBroker",
    "OPTIONS": {
        "url": "amqp://localhost:5672",
    },
    "MIDDLEWARE": [
        "dramatiq.middleware.Prometheus",
        "dramatiq.middleware.AgeLimit",
        "dramatiq.middleware.TimeLimit",
        "dramatiq.middleware.Callbacks",
        "dramatiq.middleware.Retries",
        "django_dramatiq.middleware.DbConnectionsMiddleware",
        "django_dramatiq.middleware.AdminMiddleware",
    ]
}

# Defines which database should be used to persist Task objects when the
# AdminMiddleware is enabled.  The default value is "default".
DRAMATIQ_TASKS_DATABASE = "default"

You may also configure a result backend:

DRAMATIQ_RESULT_BACKEND = {
    "BACKEND": "dramatiq.results.backends.redis.RedisBackend",
    "BACKEND_OPTIONS": {
        "url": "redis://localhost:6379",
    },
    "MIDDLEWARE_OPTIONS": {
        "result_ttl": 1000 * 60 * 10
    }
}

Usage

Declaring tasks

django_dramatiq will auto-discover tasks defined in tasks modules in each of your installed apps. For example, if you have an app named customers, your tasks for that app should live in a module called customers.tasks:

import dramatiq

from django.core.mail import send_mail

from .models import Customer

@dramatiq.actor
def email_customer(customer_id, subject, message):
    customer = Customer.get(pk=customer_id)
    send_mail(subject, message, "[email protected]", [customer.email])

You can override the name of the tasks module by setting one or more names in settings:

DRAMATIQ_AUTODISCOVER_MODULES = ["tasks", "services"]

Running workers

django_dramatiq comes with a management command you can use to auto-discover task modules and run workers:

python manage.py rundramatiq

If your project for some reason has apps with modules named tasks that are not intended for use with Dramatiq, you can ignore them:

DRAMATIQ_IGNORED_MODULES = (
    'app1.tasks',
    'app2.tasks',
    'app3.tasks.utils',
    'app3.tasks.utils.*',
    ...
)

The wildcard detection will ignore all sub modules from that point on. You will need to ignore the module itself if you don't want the __init__.py to be processed.

Testing

You should have a separate settings file for test. In that file, overwrite the broker to use Dramatiq's StubBroker:

DRAMATIQ_BROKER = {
    "BROKER": "dramatiq.brokers.stub.StubBroker",
    "OPTIONS": {},
    "MIDDLEWARE": [
        "dramatiq.middleware.AgeLimit",
        "dramatiq.middleware.TimeLimit",
        "dramatiq.middleware.Callbacks",
        "dramatiq.middleware.Pipelines",
        "dramatiq.middleware.Retries",
        "django_dramatiq.middleware.DbConnectionsMiddleware",
        "django_dramatiq.middleware.AdminMiddleware",
    ]
}

Using pytest-django

In your conftest module set up fixtures for your broker and a worker:

import dramatiq
import pytest

@pytest.fixture
def broker():
    broker = dramatiq.get_broker()
    broker.flush_all()
    return broker

@pytest.fixture
def worker(broker):
    worker = dramatiq.Worker(broker, worker_timeout=100)
    worker.start()
    yield worker
    worker.stop()

In your tests, use those fixtures whenever you want background tasks to be executed:

def test_customers_can_be_emailed(transactional_db, broker, worker, mailoutbox):
    customer = Customer(email="[email protected]")
    # Assuming "send_welcome_email" enqueues an "email_customer" task
    customer.send_welcome_email()

    # Wait for all the tasks to be processed
    broker.join("default")
    worker.join()

    assert len(mailoutbox) == 1
    assert mailoutbox[0].subject == "Welcome Jim!"

Using unittest

A simple test case has been provided that will automatically set up the broker and worker for each test, which are accessible as attributes on the test case. Note that DramatiqTestCase inherits django.test.TransactionTestCase.

from django.core import mail
from django.test import override_settings
from django_dramatiq.test import DramatiqTestCase


class CustomerTestCase(DramatiqTestCase):

    @override_settings(EMAIL_BACKEND='django.core.mail.backends.locmem.EmailBackend')
    def test_customers_can_be_emailed(self):
        customer = Customer(email="[email protected]")
        # Assuming "send_welcome_email" enqueues an "email_customer" task
        customer.send_welcome_email()

        # Wait for all the tasks to be processed
        self.broker.join(customer.queue_name)
        self.worker.join()

        self.assertEqual(len(mail.outbox), 1)
        self.assertEqual(mail.outbox[0].subject, "Welcome Jim!")

Cleaning up old tasks

The AdminMiddleware stores task metadata in a relational DB so it's a good idea to garbage collect that data every once in a while. You can use the delete_old_tasks actor to achieve this on a cron:

from django_dramatiq.tasks import delete_old_tasks

delete_old_tasks.send(max_task_age=60 * 60 * 24)

Middleware

django_dramatiq.middleware.DbConnectionsMiddleware
This middleware is vital in taking care of closing expired connections after each message is processed.
django_dramatiq.middleware.AdminMiddleware
This middleware stores metadata about tasks in flight to a database and exposes them via the Django admin.

Custom keyword arguments to Middleware

Some middleware classes require dynamic arguments. An example of this would be the backend argument to dramatiq.middleware.GroupCallbacks.

To do this, you might add the middleware to your settings.py:

DRAMATIQ_BROKER = {
    ...
    "MIDDLEWARE": [
        ...
        "dramatiq.middleware.GroupCallbacks",
        ...
    ]
    ...
}

Next, you need to extend DjangoDramatiqConfig to provide the arguments for this middleware:

from django_dramatiq.apps import DjangoDramatiqConfig


class CustomDjangoDramatiqConfig(DjangoDramatiqConfig):
    @classmethod
    def middleware_groupcallbacks_kwargs(cls):
        return {"rate_limiter_backend": cls.get_rate_limiter_backend()}


CustomDjangoDramatiqConfig.initialize()

Notice the naming convention, to provide arguments to dramatiq.middleware.GroupCallbacks you need to add a @classmethod with the name middleware_<middleware_name>_kwargs, where <middleware_name> is the lowercase name of the middleware.

Finally, add the custom app config to your settings.py, replacing the existing django_dramatiq app config:

INSTALLED_APPS = [
    ...
    "yourapp.apps.CustomDjangoDramatiqConfig",
    ...
]

Usage with django-configurations

To use django_dramatiq together with django-configurations you need to define your own rundramatiq command as a subclass of the one in this package.

In YOURPACKAGE/management/commands/rundramatiq.py:

from django_dramatiq.management.commands.rundramatiq import Command as RunDramatiqCommand


class Command(RunDramatiqCommand):
    def discover_tasks_modules(self):
        tasks_modules = super().discover_tasks_modules()
        tasks_modules[0] = "YOURPACKAGE.dramatiq_setup"
        return tasks_modules

And in YOURPACKAGE/dramatiq_setup.py:

import django

from configurations.importer import install

install(check_options=True)
django.setup()

Running project tests locally

Install the dev dependencies with pip install -e '.[dev]' and then run tox.

License

django_dramatiq is licensed under Apache 2.0. Please see LICENSE for licensing details.

More Repositories

1

dramatiq

A fast and reliable background task processing library for Python 3.
Python
3,640
star
2

awesome-advent-of-code

A collection of awesome resources related to the yearly Advent of Code challenge.
2,827
star
3

molten

A minimal, extensible, fast and productive framework for building HTTP APIs with Python 3.6 and later.
Python
982
star
4

cursive_re

Readable regular expressions for Python 3.6 and up.
Python
353
star
5

dramatiq_dashboard

A dashboard for dramatiq, specific to its Redis broker.
Python
143
star
6

elm-ast

A parser for Elm in Elm.
Elm
143
star
7

racket-gui-easy

Declarative GUIs in Racket.
Racket
129
star
8

koyo

A web development toolkit for Racket.
Racket
129
star
9

web-app-from-scratch

Supporting material for my blog post series on writing a web application from scratch in Python.
Python
128
star
10

remember

Stash distractions away for later.
Swift
107
star
11

Franz

A desktop client for Apache Kafka.
Racket
106
star
12

browser-connect.vim

Live browser interaction for VIM.
Vim Script
102
star
13

marionette

A Racket library that lets you control Firefox via the Marionette Protocol.
Racket
73
star
14

deta

A database mapper for Racket.
Racket
60
star
15

threadop

Adds a threading operator to Python.
Python
55
star
16

anom-py

An ndb-like object mapper for Google Cloud Datastore.
Python
53
star
17

racket-lua

A #lang implementation of Lua 5.4 for Racket.
Racket
52
star
18

setup-racket

A GH action for installing Racket.
JavaScript
49
star
19

neko

A tiny kitten follows your mouse on macOS.
Swift
48
star
20

dramatiq_sqs

A Dramatiq broker that can be used with Amazon SQS.
Python
47
star
21

racket-http-easy

A high-level HTTP client for Racket.
Racket
43
star
22

rbrepl.vim

VIM plugin that allows you to run a Ruby REPL inside a VIM buffer.
Vim Script
41
star
23

racket-review

A linter for Racket.
Racket
41
star
24

racket-wasm

Wasm tooling in Racket.
Racket
39
star
25

nemea

Privacy focused website analytics.
Racket
35
star
26

racket-redis

Fast, idiomatic Redis bidings for Racket.
Racket
31
star
27

flask_dramatiq_example

An example app demonstrating how you can use Dramatiq with Flask.
Python
29
star
28

django_dramatiq_example

An example app demonstrating django_dramatiq.
Python
28
star
29

rackcheck

A property-based testing library for Racket.
Racket
28
star
30

taskqueues.com

A list of distributed task queueing software.
HTML
26
star
31

racket-chief

Chief runs Procfile-based applications. Like foreman, but written in Racket.
Racket
25
star
32

pyrepl.vim

VIM plugin that provides a way to run a Python REPL inside a VIM buffer.
Vim Script
25
star
33

hebi

A Snake-style game in Racket.
Racket
24
star
34

hugs

Hugs lets you map SQL expressions to Python functions.
Python
24
star
35

Noise

A Swift wrapper around Racket CS.
Racket
23
star
36

quicksilver.vim

Quicksilver is a VIM plugin whose purpose is to quicken the process of opening files from inside VIM.
Vim Script
22
star
37

racket-dbg

A server, client and UI for remotely debugging Racket applications.
Racket
19
star
38

racket-north

A database migration tool written in Racket.
Racket
18
star
39

racket-kafka

A Kafka client for Racket.
Racket
17
star
40

racket-smtp-server

An SMTP server implementation for Racket.
Racket
17
star
41

browser-connect-server

Live browser interaction for <INSERT-EDITOR-NAME-HERE>.
Scala
13
star
42

try-racket

An online playground for Racket.
Racket
13
star
43

racket-crontab

A cron-like scheduler for Racket.
Racket
12
star
44

racksnaps

Daily snapshots of the Racket Package Catalog.
Racket
12
star
45

racket-gui-extra

Platform-specific GUI controls for Racket.
Racket
11
star
46

racket-binfmt

A binary format parser generator DSL with support for limited context-sensitivity.
Racket
11
star
47

defn.io

My personal website.
Racket
10
star
48

.emacs.d

My Emacs configuration.
Emacs Lisp
10
star
49

racket-lz4

A pure-Racket decompressor for LZ4 data.
Racket
10
star
50

apistar_prometheus

Prometheus metrics for API Star apps.
Python
10
star
51

elm-querystring

A library for working with query strings in Elm.
Elm
8
star
52

apistar_dramatiq

Dramatiq integration for API Star apps.
Python
8
star
53

racket-sentry

A Sentry SDK for Racket.
Racket
8
star
54

koyo-shorty

Associated code for the "Racket Web Development with Koyo" screencast.
Racket
8
star
55

racket-protocol-buffers

A Protocol Buffers implementation for Racket.
Racket
7
star
56

h2p

A Python package that converts web pages to PDFs.
Python
7
star
57

falcon_sugar

A little sugar for Falcon applications.
Python
7
star
58

racket-forms

Web form validation for Racket.
Racket
7
star
59

resource_pool

A generic thread-safe resource pool for Python.
Python
7
star
60

elm-generate

Lazy list manipulation for Elm.
Elm
7
star
61

markii

A development-mode error handler for Python web applications.
Python
6
star
62

fargate_scraper

A CLI tool that scrapes AWS Fargate tasks to find Prometheus targets.
Python
6
star
63

dotfiles

Various configuration files from my setup.
Shell
6
star
64

molten_cookiecutter

A cookiecutter for motlen apps.
Python
6
star
65

modviz

Python module dependency visualizer.
Python
6
star
66

ido-clever-match

An alternative matcher for ido-mode.
Emacs Lisp
6
star
67

personal-website

My personal website.
Racket
5
star
68

racketcon2021-talk

Racket
5
star
69

racket-component

Application lifecycle management for Racket.
Racket
5
star
70

aoc2020

My Advent of Code 2020 solutions.
Racket
5
star
71

racket-actions-example

An example of testing Racket code using GitHub actions.
Racket
5
star
72

racketfest2023-talk

A talk I gave at RacketFest 2023.
Racket
5
star
73

apistar_sentry

A Sentry component for API Star.
Python
5
star
74

racket-import-profiler

A command line utility for profiling module dependency import times.
Racket
5
star
75

racket-net-ip

Utilities for working with IP addresses in Racket.
Racket
5
star
76

setup-racket-cache-example

An example repo that demonstrates how setup-racket packages can be cached with GHA.
4
star
77

aoc2021

My solutions to Advent of Code 2021.
Racket
4
star
78

tcopy

Tail call optimization for Python.
Python
4
star
79

racket-json-lexer

This package provides an implementation of a JSON lexer and pretty printer for Racket.
Racket
4
star
80

elm-cookiecutter

A cookiecutter template for Elm projects.
Elm
4
star
81

racket-geoip

Geolocation for Racket based on MaxMind's GeoIP databases.
Racket
4
star
82

ymage

ymage is a simple image slideshow creator written in Python
Python
4
star
83

watchdog_gevent

A gevent-based observer for watchdog.
Python
4
star
84

racket-messagepack

An implementation of the MessagePack serialization format for Racket.
Racket
4
star
85

trio-redis

A Redis client based on Trio.
Python
4
star
86

racket-net-mime-type

Utilities for working with MIME types in Racket.
Racket
4
star
87

apistar_cors

CORS support for API Star.
Python
4
star
88

racket-avro

A Racket implementation of the Apache Avro serialization format.
Racket
4
star
89

racket-bench-ws-logging

Racket
3
star
90

ruse

Scheme
3
star
91

racket-box-extra

Extra utilities for working with boxes in Racket.
Racket
3
star
92

racket-libsass

This package distributes libsass as a Racket package for Linux, MacOS and Windows.
Shell
3
star
93

pirate

A simple TPB CLI magnet link utility.
Python
3
star
94

cedar

A web service definition format and source code generator.
Python
3
star
95

racket-resource-pool

A generic blocking resource pool implementation for Racket.
Racket
3
star
96

py-test.el

A test runner for Python code.
Emacs Lisp
3
star
97

racket-libsqlite3

This package distributes libsqlite3 as a Racket package for Linux and macOS.
Shell
3
star
98

koyo-sentry

Integrates racket-sentry with koyo.
Racket
3
star
99

apistar_request_id

Request id generation and propagation for API Star.
Python
3
star
100

racket-ulid

Universally unique lexicographically sortable identifiers for Racket.
Racket
2
star