yandex/django_replicated

SUMMARY

Django_replicated is a Django database router designed to support more or less automatic master-slave replication. It keeps an internal state that depends on user intent to read or to write into a database. Depending on this state it automatically uses the right database (master or slave) for all SQL operations.

INSTALLATION

1. Install django_replicated distribution using "python setup.py install".

2. Add import of the default django_replicated settings into your settings.py:

   from django_replicated.settings import *
   

3. In settings.py configure your master and slave databases in a standard way:

   DATABASES {
       'default': {
           # ENGINE, HOST, etc.
       },
       'slave1': {
           # ENGINE, HOST, etc.
       },
       'slave2': {
           # ENGINE, HOST, etc.
       },
   }
   

4. Teach django_replicated which databases are slaves:

   REPLICATED_DATABASE_SLAVES = ['slave1', 'slave2']
   

   The 'default' database is always treated as master.

5. Configure a replication router:

   DATABASE_ROUTERS = ['django_replicated.router.ReplicationRouter']
   

6. Configure timeout to exclude a database from the available list after an unsuccessful ping:

   REPLICATED_DATABASE_DOWNTIME = 20
   

   The default downtime value is 60 seconds.

USAGE

Django_replicated routes SQL queries into different databases based not only on their type (insert/update/delete vs. select) but also on its own current state. This is done to support the situation in which there are both writes and reads in a single logical operation. If the writes and reads used separate databases, the result would be inconsistent because:

• when using transactions, the result of the writes will not be delivered to slaves until committed;
• even in a non-transactional environment, there is always a certain lag before the updates reach slaves.

Django_replicated expects you to define what these logical operations are doing: writing/reading or only reading. Then it will try to use slave databases only for purely reading operations.

There are several methods to define those.

Middleware

If your project is built in accordance with principles of HTTP where GET requests do not cause changes in the system (unless by side effects) then most of the work is done by simply using a middleware:

MIDDLEWARE_CLASSES = [
    ...
    'django_replicated.middleware.ReplicationMiddleware',
    ...
]

The middleware sets replication state to use slaves during handling of GET and HEAD requests and to use a master otherwise.

While this is usually enough there are cases when DB access is not controlled explicitly by your business logic. Good examples are implicit creation of sessions on the first access, writing some bookkeeping info, implicit registration of a user account somewhere inside the system. These things can happen at arbitrary moments of time, including during GET requests.

Generally, django_replicated handles this by always using the master database for write operations. If this is not enough (e.g., if you want to make sure a newly created session is read from the master), you can always instruct Django ORM to use a certain database.

Decorators

If your system does not depend on the method of HTTP request to do writes and reads you can use decorators to wrap individual views into master or slave replication modes:

from django_replicated.decorators import use_master, use_slave

@use_master
def my_view(request, ...):
    # master database used for all db operations during
    # execution of the view (if not explicitly overridden).

@use_slave
def my_view(request, ...):
    # same with slave connection

GET after POST

There is a special case that needs addressing when working with asynchronous replication scheme. Replicas can lag behind a master database on receiving updates. In practice, this means that after submitting a POST form that redirects to a page with updated data this page may be requested from a slave replica that was not updated yet. And the user will have an impression that the submit did not work.

To overcome this problem both ReplicationMiddleware and decorators support special technique where handling of a GET request resulting from a redirect after a POST is explicitly routed to a master database.

Global overrides

In some cases, it might be necessary to override how the middleware chooses a target database based on the HTTP request method. For example, you might want to route certain POST requests to a slave if you know that the request handler does not do any writes. The settings variable REPLICATED_VIEWS_OVERRIDES holds the mapping of view names (urlpatterns names) or view import paths or url path to database names:

REPLICATED_VIEWS_OVERRIDES = {
    'api-store-event': 'slave',
    'app.views.do_smthg': 'master',
    '/admin/*': 'master',
    '/users/': 'slave',
}

CHANGELOG

2.0 Backward incompatible changes

• Default django_replicated.settings file was added.

• Some settings variables were renamed:

    DATABASE_SLAVES -> REPLICATED_DATABASE_SLAVES
    DATABASE_DOWNTIME -> REPLICATED_DATABASE_DOWNTIME
  

• Another setting variable was deleted:

    REPLICATED_SELECT_READ_ONLY
  

• Router import path changed to django_replicated.router.ReplicationRouter.

• Ability to disable state switching with utils.disable_state_change() was removed.

• Database checkers moved to dbchecker.py module.

• db_is_not_read_only check renamed to db_is_writable.

• Added state checking before writes. Enabled by default.

• Now allows relations between objects in same master-slave db set

SIMILAR LIBRARIES

• django-multidb-router