• Stars
    star
    650
  • Rank 68,803 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created about 7 years ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Server-Sent Events for Django

Django EventStream

EventStream provides API endpoints for your Django application that can push data to connected clients. Data is sent using the Server-Sent Events protocol (SSE), in which data is streamed over a never-ending HTTP response.

For example, you could create an endpoint, /events/, that a client could connect to with a GET request:

GET /events/ HTTP/1.1
Host: api.example.com
Accept: text/event-stream

The client would receive a streaming HTTP response with content looking like this:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: Transfer-Encoding
Content-Type: text/event-stream

event: message
data: {"foo": "bar"}

event: message
data: {"bar": "baz"}

...

Features:

  • Easy to consume from browsers or native applications.
  • Highly reliable. Events can be persisted to your database, so clients can recover if they get disconnected.
  • Set per-user channel permissions.
  • Clean API contract that could be exposed to third parties if desired.

Requirements

This library requires either:

or

  • A GRIP-compatible proxy such as Pushpin or Fastly Fanout, for delegating the connection handling and keeping the Django app stateless.

Note that it is possible to combine the two. If the app is set up with Channels and a connection arrives through a GRIP proxy, then the handling will be delegated.

Setup

We recommend setting up your project with Channels as this will give you the most flexibility, including being able to run standalone or with runserver.

Otherwise, see Setup without Channels.

Setup with Channels

First, install this module and the channels module:

pip install django-eventstream channels

Add the channels and django_eventstream apps to your settings.py:

INSTALLED_APPS = [
    ...
    'channels',
    'django_eventstream',
]

Add the GripMiddleware:

MIDDLEWARE = [
    'django_grip.GripMiddleware',
    ...
]

The middleware is part of django-grip, which should have been pulled in automatically as a dependency of this module.

Channels introduces an entirely separate routing system for handling async connections. You'll need to declare an ASGI application instead of (or in addition to) a WSGI application.

For example, create an asgi.py file in your Django project dir (next to settings.py) with an endpoint declared:

"""
ASGI entrypoint. Configures Django and then runs the application
defined in the ASGI_APPLICATION setting.
"""

import os
import django
from django.core.asgi import get_asgi_application
from django.urls import path, re_path
from channels.routing import ProtocolTypeRouter, URLRouter
from channels.auth import AuthMiddlewareStack
import django_eventstream

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "server.settings")

application = ProtocolTypeRouter({
    'http': URLRouter([
        path('events/', AuthMiddlewareStack(
            URLRouter(django_eventstream.routing.urlpatterns)
        ), { 'channels': ['test'] }),
        re_path(r'', get_asgi_application()),
    ]),
})

Then set ASGI_APPLICATION in your settings.py file to your project's asgi module:

ASGI_APPLICATION = 'your_project.asgi.application'

For more information about setting up Channels in general, see the Channels Documentation.

That's it! If you run python manage.py runserver, clients will be able to connect to the /events/ endpoint and get a stream.

To send data to clients, call send_event:

from django_eventstream import send_event

send_event('test', 'message', {'text': 'hello world'})

The first argument is the channel to send on, the second is the event type, and the third is the event data. The data will be JSON-encoded using DjangoJSONEncoder.

Deploying with Channels

After following the instructions in the previous section, you'll be able to develop and run locally using runserver. However, you should not use runserver when deploying, and instead launch an ASGI server such as Daphne, e.g.:

daphne your_project.asgi:application

See the Channels Documentation for information about deployment.

Multiple instances and scaling

If you need to run multiple instances of your Django app for high availability or scalability, or need to push data from management commands, then you can introduce a GRIP proxy layer such as Pushpin or Fastly Fanout into your architecture. Otherwise, events originating from an instance will only be delivered to clients connected to that instance.

For example, to use Pushpin with your app, you need to do three things:

  1. In your settings.py, set GRIP_URL to reference Pushpin's private control port:
GRIP_URL = 'http://localhost:5561'
  1. Configure Pushpin to route requests to your app, by adding something like this to Pushpin's routes file (usually /etc/pushpin/routes):
* localhost:8000 # Replace `localhost:8000` with your app's URL and port
  1. Configure your consuming clients to connect to the Pushpin port (by default this is port 7999). Pushpin will forward requests to your app and handle streaming connections on its behalf.

If you would normally use a load balancer in front of your app, it should be configured to forward requests to Pushpin instead of your app. For example, if you are using Nginx you could have configuration similar to:

location /api/ {
    proxy_pass http://localhost:7999
}

The location block above will pass all requests coming on /api/ to Pushpin.

Setup without Channels

It is possible to use this library with a GRIP proxy only, without setting up Channels.

First, install this module:

pip install django-eventstream

A few changes need to be made to settings.py.

Add the django_eventstream app:

INSTALLED_APPS = [
    ...
    'django_eventstream',
]

Add the GripMiddleware:

MIDDLEWARE = [
    'django_grip.GripMiddleware',
    ...
]

The middleware is part of django-grip, which should have been pulled in automatically as a dependency of this module.

Set GRIP_URL with your Pushpin settings:

# pushpin
GRIP_URL = 'http://localhost:5561'

Add an endpoint in urls.py:

from django.urls import path, include
import django_eventstream

urlpatterns = [
    ...
    path('events/', include(django_eventstream.urls), {'channels': ['test']}),
]

That's it! Clients can now connect to the /events/ endpoint through the proxy and get a stream.

To send data to clients, call send_event:

from django_eventstream import send_event

send_event('test', 'message', {'text': 'hello world'})

The first argument is the channel to send on, the second is the event type, and the third is the event data. The data will be JSON-encoded using DjangoJSONEncoder.

Event storage

By default, events aren't persisted anywhere, so if clients get disconnected or if your server fails to send data, then clients can miss messages. For reliable delivery, you'll want to enable event storage.

First, set up the database tables:

python manage.py migrate

Then, set a storage class in settings.py:

EVENTSTREAM_STORAGE_CLASS = 'django_eventstream.storage.DjangoModelStorage'

That's all you need to do. When storage is enabled, events are written to the database before they are published, and they persist for 24 hours. If clients get disconnected, intermediate proxies go down, or your own server goes down or crashes at any time, even mid-publish, the stream will automatically be repaired.

To enable storage selectively by channel, implement a channel manager and override is_channel_reliable.

Receiving in the browser

Include client libraries on the frontend:

<script src="{% static 'django_eventstream/eventsource.min.js' %}"></script>
<script src="{% static 'django_eventstream/reconnecting-eventsource.js' %}"></script>

Listen for data:

var es = new ReconnectingEventSource('/events/');

es.addEventListener('message', function (e) {
    console.log(e.data);
}, false);

es.addEventListener('stream-reset', function (e) {
    // ... client fell behind, reinitialize ...
}, false);

Authorization

Declare a channel manager class with your authorization logic:

from django_eventstream.channelmanager import DefaultChannelManager

class MyChannelManager(DefaultChannelManager):
    def can_read_channel(self, user, channel):
        # require auth for prefixed channels
        if channel.startswith('_') and user is None:
            return False
        return True

Configure settings.py to use it:

EVENTSTREAM_CHANNELMANAGER_CLASS = 'myapp.channelmanager.MyChannelManager'

Whenever permissions change, call channel_permission_changed. This will cause clients to be disconnected if they lost permission to the channel.

from django_eventstream import channel_permission_changed

channel_permission_changed(user, '_mychannel')

Note: OAuth may not work with the AuthMiddlewareStack from Django Channels. See this token middleware.

Routes and channel selection

The channels the client listens to are specified using Django view keyword arguments on the routes. Alternatively, if no keyword arguments are specified, then the client can select the channels on its own by providing one or more channel query parameters in the HTTP request.

Examples:

# specify fixed list of channels
path('foo/events/', include(django_eventstream.urls), {'channels': ['foo']})

# specify a list of dynamic channels using formatting based on view keywords
path('objects/<obj_id>/events/', include(django_eventstream.urls),
    {'format-channels': ['object-{obj_id}']})

# client selects a single channel using a path component
path('events/<channel>/', include(django_eventstream.urls))

# client selects one or more channels using query parameters
path('events/', include(django_eventstream.urls))

Note that if view keywords or a channel path component are used, the client cannot use query parameters to select channels.

If even more advanced channel mapping is needed, implement a channel manager and override get_channels_for_request.

Cross-Origin Resource Sharing (CORS) Headers

There are settings available to set response headers Access-Control-Allow-Origin, Access-Control-Allow-Credentials, and Access-Control-Allow-Headers, which are EVENTSTREAM_ALLOW_ORIGIN, EVENTSTREAM_ALLOW_CREDENTIALS, and EVENTSTREAM_ALLOW_HEADERS, respectively.

Examples:

EVENTSTREAM_ALLOW_ORIGIN = 'your-website.com'
EVENTSTREAM_ALLOW_CREDENTIALS = True
EVENTSTREAM_ALLOW_HEADERS = 'Authorization'

Note that EVENTSTREAM_ALLOW_ORIGIN and EVENTSTREAM_ALLOW_HEADERS only take a single string value and do not process a list.

More Repositories

1

zurl

HTTP and WebSocket client worker with ZeroMQ interface
C++
388
star
2

pollymer

General-purpose AJAX/long-polling library
JavaScript
133
star
3

apollo-serverless-demo

Serverless GraphQL subscriptions
TypeScript
107
star
4

reconnecting-eventsource

A small decorator for the JavaScript EventSource API that automatically reconnects
TypeScript
95
star
5

websockhop

Network resilient WebSocket wrapper
JavaScript
83
star
6

webhookinbox

Receive HTTP callbacks and see the data in realtime
Python
80
star
7

python-faas-grip

FaaS GRIP library for Python
Python
67
star
8

django-grip

Django GRIP library
Python
66
star
9

condure

HTTP/WebSocket connection manager
Rust
63
star
10

js-grip

JavaScript GRIP library
TypeScript
47
star
11

kafka-sse-example

Expose Kafka messages via HTTP streaming API
Python
38
star
12

fanout-graphql-tools

GraphQL subscriptions using Fanout/Pushpin
TypeScript
32
star
13

node-faas-grip

FaaS GRIP library for Node.js
JavaScript
32
star
14

pygripcontrol

Python GRIP library
Python
31
star
15

serverless-websocket

"serverless" WebSocket example
Python
30
star
16

editor

Collaborative text editor using operational transformations
JavaScript
30
star
17

docker-pushpin

Dockerfile
26
star
18

go-gripcontrol

A GRIP library for Go.
Go
23
star
19

php-gripcontrol

A GRIP library for PHP.
PHP
20
star
20

pushpin-c1m

One million push connections on one server
PHP
20
star
21

leaderboard

Leaderboard example
JavaScript
20
star
22

express-grip

Express GRIP library
TypeScript
17
star
23

js-eventstream

Server-Sent Events for JavaScript
TypeScript
12
star
24

flychat

Serverless chat demo
JavaScript
12
star
25

tigase-zmq

Tigase ZeroMQ Gateway
Java
11
star
26

laravel-fanout

Fanout.io library for Laravel.
PHP
11
star
27

js-pubcontrol

JavaScript EPCP library
TypeScript
10
star
28

laravel-grip

A GRIP library for Laravel.
PHP
9
star
29

express-eventstream

Server-Sent Events for Express
JavaScript
8
star
30

pypubcontrol

Python EPCP library
Python
7
star
31

ruby-gripcontrol

A GRIP library for Ruby.
Ruby
7
star
32

qfiber

C++ coroutines library with channels
C++
7
star
33

todomvc-liveresource-react

JavaScript
6
star
34

headline

Python
6
star
35

js-serve-grip

Grip library for Express and Next.js
TypeScript
5
star
36

node-fanoutpub

Node.js FPP library
JavaScript
5
star
37

livecounter

Live counter API
Python
5
star
38

scaledemo

Python
4
star
39

audiostream

Stream live audio
Python
3
star
40

pyfanout

Fanout.io library for Python
Python
3
star
41

engine.io-as-websocket

Wraps Engine.IO to make it usable with the same API as WebSocket
JavaScript
3
star
42

php-pubcontrol

EPCP library for PHP
PHP
3
star
43

php-grip

Grip library for PHP
PHP
3
star
44

go-pubcontrol

Go EPCP library
Go
3
star
45

pysmartfeed

Python Smart Feed library
Python
3
star
46

rails-grip

Ruby on Rails GRIP library
Ruby
2
star
47

java-fanout

Fanout.io library for Java.
Java
2
star
48

xmpp-ftw-fanout

XMPP-FTW Fanout plugin
JavaScript
2
star
49

java-gripcontrol

A GRIP library for Java.
Java
2
star
50

go-fanout

Fanout.io library for Go.
Go
2
star
51

django-fanout

Fanout.io library for Django
Python
2
star
52

chat-demo-android

A chat demo using Fanout's APIs through event source
Java
2
star
53

vercel-websocket

JavaScript
2
star
54

django-liveresource

LiveResource library for Python/Django
Python
1
star
55

chat-demo-ios

An eventsource based chat app built to demo functionalities for Fanout
Swift
1
star
56

reliable-stream

Python
1
star
57

test-api

Minimal realtime API example
Python
1
star
58

demo

JavaScript
1
star