django-minio-backend

The django-minio-backend provides a wrapper around the MinIO Python SDK. See minio/minio-py for the source.

Integration

Get and install the package:

pip install django-minio-backend

Add django_minio_backend to INSTALLED_APPS:

INSTALLED_APPS = [
    # '...'
    'django_minio_backend',  # https://github.com/theriverman/django-minio-backend
]

If you would like to enable on-start consistency check, install via DjangoMinioBackendConfig:

INSTALLED_APPS = [
    # '...'
    'django_minio_backend.apps.DjangoMinioBackendConfig',  # https://github.com/theriverman/django-minio-backend
]

Then add the following parameter to your settings file:

MINIO_CONSISTENCY_CHECK_ON_START = True

Note: The on-start consistency check equals to manually calling python manage.py initialize_buckets.
It is recommended to turn off this feature during development by setting MINIO_CONSISTENCY_CHECK_ON_START to False, because this operation can noticeably slow down Django's boot time when many buckets are configured.

Add the following parameters to your settings.py:

from datetime import timedelta
from typing import List, Tuple

MINIO_ENDPOINT = 'minio.your-company.co.uk'
MINIO_EXTERNAL_ENDPOINT = "external-minio.your-company.co.uk"  # Default is same as MINIO_ENDPOINT
MINIO_EXTERNAL_ENDPOINT_USE_HTTPS = True  # Default is same as MINIO_USE_HTTPS
MINIO_REGION = 'us-east-1'  # Default is set to None
MINIO_ACCESS_KEY = 'yourMinioAccessKey'
MINIO_SECRET_KEY = 'yourVeryS3cr3tP4ssw0rd'
MINIO_USE_HTTPS = True
MINIO_URL_EXPIRY_HOURS = timedelta(days=1)  # Default is 7 days (longest) if not defined
MINIO_CONSISTENCY_CHECK_ON_START = True
MINIO_PRIVATE_BUCKETS = [
    'django-backend-dev-private',
]
MINIO_PUBLIC_BUCKETS = [
    'django-backend-dev-public',
]
MINIO_POLICY_HOOKS: List[Tuple[str, dict]] = []
# MINIO_MEDIA_FILES_BUCKET = 'my-media-files-bucket'  # replacement for MEDIA_ROOT
# MINIO_STATIC_FILES_BUCKET = 'my-static-files-bucket'  # replacement for STATIC_ROOT
MINIO_BUCKET_CHECK_ON_SAVE = True  # Default: True // Creates bucket if missing, then save

# Custom HTTP Client (OPTIONAL)
import os
import certifi
import urllib3
timeout = timedelta(minutes=5).seconds
ca_certs = os.environ.get('SSL_CERT_FILE') or certifi.where()
MINIO_HTTP_CLIENT: urllib3.poolmanager.PoolManager = urllib3.PoolManager(
    timeout=urllib3.util.Timeout(connect=timeout, read=timeout),
    maxsize=10,
    cert_reqs='CERT_REQUIRED',
    ca_certs=ca_certs,
    retries=urllib3.Retry(
        total=5,
        backoff_factor=0.2,
        status_forcelist=[500, 502, 503, 504]
    )
)

Implement your own Attachment handler and integrate django-minio-backend:

from django.db import models
from django_minio_backend import MinioBackend, iso_date_prefix

class PrivateAttachment(models.Model):   
    file = models.FileField(verbose_name="Object Upload",
                            storage=MinioBackend(bucket_name='django-backend-dev-private'),
                            upload_to=iso_date_prefix)

Initialize the buckets & set their public policy (OPTIONAL):
This django-admin command creates both the private and public buckets in case one of them does not exists, and sets the public bucket's privacy policy from private(default) to public.

python manage.py initialize_buckets

Code reference: initialize_buckets.py.

Static Files Support

django-minio-backend allows serving static files from MinIO. To learn more about Django static files, see Managing static files, and STATICFILES_STORAGE.

To enable static files support, update your settings.py:

STATICFILES_STORAGE = 'django_minio_backend.models.MinioBackendStatic'
MINIO_STATIC_FILES_BUCKET = 'my-static-files-bucket'  # replacement for STATIC_ROOT
# Add the value of MINIO_STATIC_FILES_BUCKET to one of the pre-configured bucket lists. eg.:
# MINIO_PRIVATE_BUCKETS.append(MINIO_STATIC_FILES_BUCKET)
# MINIO_PUBLIC_BUCKETS.append(MINIO_STATIC_FILES_BUCKET)

The value of STATIC_URL is ignored, but it must be defined otherwise Django will throw an error.

IMPORTANT
The value set in MINIO_STATIC_FILES_BUCKET must be added either to MINIO_PRIVATE_BUCKETS or MINIO_PUBLIC_BUCKETS, otherwise django-minio-backend will raise an exception. This setting determines the privacy of generated file URLs which can be unsigned public or signed private.

Note: If MINIO_STATIC_FILES_BUCKET is not set, the default value (auto-generated-bucket-static-files) will be used. Policy setting for default buckets is private.

Default File Storage Support

django-minio-backend can be configured as a default file storage. To learn more, see DEFAULT_FILE_STORAGE.

To configure django-minio-backend as the default file storage, update your settings.py:

DEFAULT_FILE_STORAGE = 'django_minio_backend.models.MinioBackend'
MINIO_MEDIA_FILES_BUCKET = 'my-media-files-bucket'  # replacement for MEDIA_ROOT
# Add the value of MINIO_STATIC_FILES_BUCKET to one of the pre-configured bucket lists. eg.:
# MINIO_PRIVATE_BUCKETS.append(MINIO_STATIC_FILES_BUCKET)
# MINIO_PUBLIC_BUCKETS.append(MINIO_STATIC_FILES_BUCKET)

The value of MEDIA_URL is ignored, but it must be defined otherwise Django will throw an error.

IMPORTANT
The value set in MINIO_MEDIA_FILES_BUCKET must be added either to MINIO_PRIVATE_BUCKETS or MINIO_PUBLIC_BUCKETS, otherwise django-minio-backend will raise an exception. This setting determines the privacy of generated file URLs which can be unsigned public or signed private.

Note: If MINIO_MEDIA_FILES_BUCKET is not set, the default value (auto-generated-bucket-media-files) will be used. Policy setting for default buckets is private.

Health Check

To check the connection link between Django and MinIO, use the provided MinioBackend.is_minio_available() method.
It returns a MinioServerStatus instance which can be quickly evaluated as boolean.

Example:

from django_minio_backend import MinioBackend

minio_available = MinioBackend().is_minio_available()  # An empty string is fine this time
if minio_available:
    print("OK")
else:
    print("NOK")
    print(minio_available.details)

Policy Hooks

You can configure django-minio-backend to automatically execute a set of pre-defined policy hooks.
Policy hooks can be defined in settings.py by adding MINIO_POLICY_HOOKS which must be a list of tuples.
Policy hooks are automatically picked up by the initialize_buckets management command.

For an exemplary policy, see the implementation of def set_bucket_to_public(self) in django_minio_backend/models.py or the contents of examples/policy_hook.example.py.

Consistency Check On Start

When enabled, the initialize_buckets management command gets called automatically when Django starts.
This command connects to the configured minIO server and checks if all buckets defined in settings.py.
In case a bucket is missing or its configuration differs, it gets created and corrected.

Reference Implementation

For a reference implementation, see Examples.

Behaviour

The following list summarises the key characteristics of django-minio-backend:

Bucket existence is not checked on a save by default. To enable this guard, set MINIO_BUCKET_CHECK_ON_SAVE = True in your settings.py.
Bucket existences are not checked on Django start by default. To enable this guard, set MINIO_CONSISTENCY_CHECK_ON_START = True in your settings.py.
Many configuration errors are validated through AppConfig but not every error can be captured there.
Files with the same name in the same bucket are not replaced on save by default. Django will store the newer file with an altered file name To allow replacing existing files, pass the replace_existing=True kwarg to MinioBackend. For example: image = models.ImageField(storage=MinioBackend(bucket_name='images-public', replace_existing=True))
Depending on your configuration, django-minio-backend may communicate over two kind of interfaces: internal and external. If your settings.py defines a different value for MINIO_ENDPOINT and MINIO_EXTERNAL_ENDPOINT, then the former will be used for internal communication between Django and MinIO, and the latter for generating URLs for users. This behaviour optimises the network communication. See Networking below for a thorough explanation
The uploaded object's content-type is guessed during save. If mimetypes.guess_type fails to determine the correct content-type, then it falls back to application/octet-stream.

Networking and Docker

If your Django application is running on a shared host with your MinIO instance, you should consider using the MINIO_EXTERNAL_ENDPOINT and MINIO_EXTERNAL_ENDPOINT_USE_HTTPS parameters. This way most traffic will happen internally between Django and MinIO. The external endpoint parameters are required for external pre-signed URL generation.

If your Django application and MinIO instance are running on different hosts, you can omit the MINIO_EXTERNAL_ENDPOINT and MINIO_EXTERNAL_ENDPOINT_USE_HTTPS parameters, and django-minio-backend will default to the value of MINIO_ENDPOINT.

Setting up and configuring custom networks in Docker is not in the scope of this document.
To learn more about Docker networking, see Networking overview and Networking in Compose.

See README.Docker.md for a real-life Docker Compose demonstration.

Compatibility

Django 2.2 or later
Python 3.8.0 or later
MinIO SDK 7.0.2 or later

Contribution

Please find the details in CONTRIBUTE.md

Copyright

theriverman/django-minio-backend licensed under the MIT License
minio/minio-py is licensed under the Apache License 2.0

theriverman/django-minio-backend

theriverman

Reviews

Repository Details