• Stars
    star
    136
  • Rank 267,670 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created over 2 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Python server to on-the-fly extract and serve vector tiles from an mbtiles file on S3

mbtiles-s3-server CircleCI Test Coverage

Python server to on-the-fly extract and serve vector tiles from mbtiles files on S3. Javascript, maps styles, fonts, and sprites are included so you can get setup quickly, especially with OpenMapTiles mbtiles files, but these are not required to be used. Versioning must be enabled on the underlying S3 bucket.

mbtiles-s3-server-example.mov

Video showing expected performance. The server hosting the HTML is running locally, but mbtiles-s3-server is running on AWS, communicating with a real S3 bucket.

Installation

pip install mbtiles-s3-server

The libsqlite3 binary library is also required, but this is typically already installed on most systems. The earliest version of libsqlite3 known to work is 2012-12-12 (3.7.15).

Example usage

  1. Create or obtain an mbtiles file, for example from https://openmaptiles.org/.

  2. (Optional) Create another file with a page size of 65536 bytes (64KiB) using VACUUM or VACUUM INTO

    sqlite3 my-map.mbtiles "PRAGMA page_size=65536; VACUUM INTO 'mytiles-65536.mbtiles';"

    While this step is optional, performance with default mbtiles files that have smaller page sizes can be horrible to the point of being unusable - loading of a single tile can take many seconds. Performance of this server is limited by the latency of calls to S3, and this step effectively reduces the number of calls to S3 per map tile.

    Note both VACUUM and VACUUM INTO need disk space since they create another database which is approximately the size of the original file.

  3. Upload this file to S3, for example to https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles

  4. Ensure to have a IAM user that has s3:GetObject and s3:GetObjectVersion permissions on this S3 object, for example

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:GetObjectVersion"
                ],
                "Resource": [
                    "arn:aws:s3:::my-bucket/mytiles-65536.mbtiles"
                ]
            }
        ]
    }   
  5. Start this server, configured with the location of this object and credentials for this user - it's configured using environment variables. You can assign the tiles file any version you like, in this case, 1.0.0.

    PORT=8080 \
    MBTILES__1__URL=https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles \
    MBTILES__1__MIN_ZOOM=0 \
    MBTILES__1__MAX_ZOOM=14 \
    MBTILES__1__IDENTIFIER=mytiles \
    MBTILES__1__VERSION=1.0.0 \
    AWS_REGION=eu-west-2 \
    AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
    AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
    HTTP_ACCESS_CONTROL_ALLOW_ORIGIN="*" \
        python -m mbtiles_s3_server

    The above is only to show what environment variables are needed - usually credentials should be avoided in shell commands. When developing locally, instead of specifying the AWS_* variables, consider using aws-vault, which populates them for you.

    For example if you have a local AWS profile called tiles, you can run the below.

    PORT=8080 \
    MBTILES__1__URL=https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles \
    MBTILES__1__MIN_ZOOM=0 \
    MBTILES__1__MAX_ZOOM=14 \
    MBTILES__1__IDENTIFIER=mytiles \
    MBTILES__1__VERSION=1.0.0 \
    HTTP_ACCESS_CONTROL_ALLOW_ORIGIN="*" \
        aws-vault exec tiles -- python -m mbtiles_s3_server
  6. On your user-facing site, include HTML that loads these tiles from this server, for example to load maps from a server started as above running locally serving OpenMapTiles

     <!DOCTYPE html>
     <html>
       <head>
         <meta charset="utf-8">
         <title>Example map</title>
         <meta name="viewport" content="initial-scale=1,maximum-scale=1,user-scalable=no">
         <script src="http://localhost:8080/v1/static/[email protected]/maplibre-gl.js"></script>
         <link href="http://localhost:8080/v1/static/[email protected]/maplibre-gl.css" rel="stylesheet">
         <style>
           body, html, #map {margin: 0; padding: 0; height: 100%; width: 100%}
         </style>
       </head>
       <body>
         <div id="map"></div>
         <script>
         var map = new maplibregl.Map({
             container: 'map',
             style: 'http://localhost:8080/v1/styles/[email protected]/[email protected]&[email protected]',
             center: [0, 0],
             zoom: 1,
             attributionControl: false
         });
         map.addControl(new maplibregl.AttributionControl({
             customAttribution: '<a href="https://openmaptiles.org/">© OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright">© OpenStreetMap contributors</a>'
         }), 'bottom-right');
         </script>
       </body>
     </html>

    This HTML is included in this repository in example.html. A simple server can be started to view it by

    python -m http.server 8081 --bind 127.0.0.1

    and going to http://localhost:8081/example.html

Core API

GET /v1/tiles/{identifier}@{version}/{z}/{x}/{y}.mvt

Fetch a tile in Mapbox Vector Tile (MVT) format

  • identifier

    An aribtrary identifier for a tileset configued via environment variables when starting the server, for example MBTILES__1__IDENTIFIER (see Example usage).

  • version

    A version for a tileset configued via environment variables, for example MBTILES__1__VERSION (see Example usage). The version is part of the API to encourage releasing a new version of a tileset rather than replacing an existing one.

    An arbitrary version of the tileset identified by the identifier.

  • z, x, y

    The xyz coordinates of a tile in this tileset

For the curious, advanced, or developers of this server itself

Hosting your own vector map tiles to show them in a browser requires quite a few components:

  1. JavaScript and CSS

    A Javascript and CSS library, such as MapLibre GL, and your own code to run this library, pointing it to a style file

  2. Style file

    A JSON file that defines how the library should visually style the map data, and where it should find the map tiles, glyphs (fonts), and the sprite. This server transforms the built-in Style files on the fly to be able to refer to any map data.

  3. Glyphs (fonts)

    Different fonts can be used for different labels and zoom levels, as defined in the Style file. The fonts must Signed Distance Field (SDF) fonts wrapped in a particular Protocol Buffer format. The Style file can refer to "stacks" of fonts; but unlike CSS, the server combines the fonts on the fly in an API where the resulting "font" has at most one glyph from each source font.

  4. Sprite

    A sprite is actually 4 URLs: a JSON index file and a single PNG file, and a "@2x" JSON index file and PNG files for higher pixel ratio devices (e.g. Retina). The JSON files contains the offsets and sizes of images within corresponding PNG file. The style file refers the common "base" of these. For example, if the style file has "sprite":"https://my.test/sprite" then the 4 files must be at https://my.test/sprite.json, https://my.test/sprite.png, https://my.test/[email protected] and https://my.test/[email protected].

  5. Vector map tiles

    A set of often millions of tiles each covering a different location and different zoom level. These can be distributed as a single mbtiles file, but this is not the format that the Javascript library accepts. This on-the-fly conversion from the mbtiles file to tiles is the main feature of this server.

    The mbtiles file is a SQLite file, containing gzipped Mapbox Vector Tile tiles. This server leaves the un-gzipping to the browser, by sending tiles with a content-encoding: gzip header, which results in browser un-gzipping the tile data before it hits the Javascript.

Licenses

The code of the server itself is released under the MIT license. However, several components included in mbtiles_s3_server/vendor/ are released under different licenses.

More Repositories

1

stream-unzip

Python function to stream unzip all the files in a ZIP archive on the fly
Python
251
star
2

cypress-image-diff

Visual regression test with cypress
Handlebars
233
star
3

sqlite-s3vfs

Python writable virtual filesystem for SQLite on S3
Python
112
star
4

stream-zip

Python function to construct a ZIP archive on the fly
Python
86
star
5

mobius3

Continuously sync folder to S3, using inotify under the hood
Python
42
star
6

data-workspace-frontend

An open source data analysis platform with features for users with a range of technical skills
Python
42
star
7

fargatespawner

Spawns JupyterHub single user servers in Docker containers running in AWS Fargate
Python
38
star
8

pg-bulk-ingest

Python utility function to ingest data into a SQLAlchemy-defined PostgreSQL table
Python
34
star
9

dns-rewrite-proxy

A DNS proxy server that conditionally rewrites and filters A record requests
Python
28
star
10

stream-sqlite

Python function to extract rows from a SQLite file while iterating over its bytes
Python
23
star
11

tidy-json-to-csv

Convert JSON to a set of tidy CSV files
Python
20
star
12

tamato

The Tariff Management Tool (TaMaTo) stores and manages the tariffs and controls that are applied on imports and exports at the UK border. 🍅
Python
18
star
13

jupyters3

Jupyter Notebook Contents Manager for AWS S3
Python
17
star
14

dit-clamav-rest

A minimal ClamD REST interface
Python
15
star
15

great-cms

great.gov.uk CMS
Python
15
star
16

django-workflow-engine

Configurable Django workflow engine
Python
13
star
17

stream-read-xbrl

Python package to parse Companies House accounts data in a streaming way
Python
13
star
18

mock-sso

A mock SSO server to test an applications SSO integration
JavaScript
12
star
19

data-hub-api

Django API for Data Hub frontend
Python
12
star
20

data-hub-frontend

The frontend rendering application for Data Hub
JavaScript
10
star
21

streampq

Python PostgreSQL adapter to stream results of multi-statement queries without a server-side cursor
Python
8
star
22

export-opportunities

UK - Great.gov - Export Opportunities - Find and apply for overseas opportunities from businesses looking for products or services like yours.
HTML
7
star
23

pii-secret-check-hooks

PII (Personal Identifiable Information) and secret check hooks for pre-commit
Python
7
star
24

iterable-subprocess

Python context manager to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed
Python
7
star
25

market-access-python-frontend

Python
7
star
26

to-file-like-obj

Python utility function to convert an iterable of bytes or str to a readable file-like object
Python
7
star
27

directory-cms

Content Management System service for directory apps
Python
6
star
28

django-chunk-s3-av-upload-handlers

Chunking Django file handlers for S3 and ClamAV service uploads
Python
6
star
29

platform-tools

Tooling to assist with setting up and managing AWS copilot infrastructure
Python
6
star
30

tariff-data-manual

Technical documentation about how the UK tariff data works.
Ruby
6
star
31

s3chunkuploader

A multi threaded S3 file chunk uploader which bypasses local file system and pipes the file directly to S3
Python
6
star
32

trade-elements

Front end pattern library for Department of International Trade
HTML
5
star
33

streamlit-gov-uk-components

A collection of Streamlit components that use or are inspired by the GOV.UK Design System
Shell
5
star
34

data-hub-components

Collection of React Components used by DIT's Data Hub.
JavaScript
5
star
35

digital-workspace-v2

DBT's intranet Django/Wagtail
Python
5
star
36

django-log-formatter-ecs

An ECS formatter for Django logs
Python
5
star
37

jwt-postgresql-proxy

Stateless JWT authentication in front of PostgreSQL
Python
5
star
38

trade-tariff-api

Taric file API for Trade Tariff Management application
Python
5
star
39

data-engineering

A collection of information and learning material relating to data engineering at the Department for International Trade
5
star
40

data-hub-prototype

JavaScript
4
star
41

pg-force-execute

Context manager to run PostgreSQL queries with SQLAlchemy, terminating any other clients that block it
Python
4
star
42

terraform-module-eks-base

HCL
4
star
43

directory-tests

Smoke, integration and functional tests for DIT's services
Python
4
star
44

trade-tariff-management

Ruby
4
star
45

countries-of-interest-service

Lightweight API service for querying for companies that have expressed interest in exporting to specific countries
Python
4
star
46

icms

Import Case Management System - Python/Django port on PostgreSQL database
Python
4
star
47

govuk-frontend-django

GOV.UK Frontend Django
Python
4
star
48

directory-ui-buyer

Python
3
star
49

help-desk-service

Python
3
star
50

terraform-prometheus-module

HCL
3
star
51

terraform-modules

HCL
3
star
52

data-hub-api-spec

Swagger files and test server for the Data Hub CRM project
Python
3
star
53

great-international-ui

Python
3
star
54

stream-write-ods

Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk
Python
3
star
55

directory-api

Directory of Exporters API
Python
3
star
56

dnb-service

A Dunn & Bradstreet microservice
Python
3
star
57

export-support

Python
3
star
58

data-workspace-tools

HTML
3
star
59

dit-thumber

A simple django app for soliciting user feedback on django views
Python
3
star
60

lite-api

Service for handling backend calls in LITE.
Python
3
star
61

infrastructure

Vagrant, Docker and Ansible configuration repository for DIT Digital
Shell
3
star
62

clamav-s3-mirror

Python
3
star
63

legal-basis-api

Legal Basis for Consent Service API Server
Python
3
star
64

dit-helpdesk

HTML
3
star
65

django-zenslackchat

A bot which synchronises support requests from Slack to Zendesk and back.
Python
3
star
66

docker-overlay-network-benchmark

Shell
3
star
67

data-hub-helpcentre

Central point for articles, advice and how-tos for the Data Hub
CSS
3
star
68

market-access-api

The API for Market Access
Python
3
star
69

selling-online-overseas

Department of International Trade marketplace navigator.
Python
3
star
70

flower-runner

Celery Flower runner for GOV.UK PaaS
Python
3
star
71

django-staff-sso-client

A client for staff-sso
Python
3
star
72

vulnerability-priority-list

A command line report on a GitHub organisation's repositories, ordered by priority, and including time-to-SLA for each severity level
Python
3
star
73

global-uk-tariff

GOV.UK service displaying tariffs to trade with the UK from 1 January 2021
CSS
3
star
74

ci-pipeline

DIT CD pipeline
Ruby
2
star
75

investment-opps

JavaScript
2
star
76

aioftps3

FTP in front of AWS S3, powered by asyncio and aiohttp
Python
2
star
77

django-ga-measurement-protocol

Python
2
star
78

copilot-poc

Co Pilot PoC
Python
2
star
79

public-data-api

The source for the Department for International Trade's Public Data API
HTML
2
star
80

directory-form-data

Python
2
star
81

company-matching-service

Python
2
star
82

dit-contact-forms

DIT Contact Forms
HTML
2
star
83

great-styles

Styles for great.gov.uk
SCSS
2
star
84

markdown-gds

Python
2
star
85

directory-ui-supplier

deprecated. Incorporated into https://github.com/uktrade/great-international-ui
Python
2
star
86

data-workspace-visualisation

Template repository for Data Workspace visualisations
R
2
star
87

terraform-module-openshift

HCL
2
star
88

mirror-git-to-s3

Python functions and CLI to mirror git repositories to S3
Python
2
star
89

directory-sso

Python
2
star
90

prototypes-on-paas

HTML
2
star
91

invest-pir-api

Personalised Investment Report
Python
2
star
92

terraform-module-aws_account

HCL
2
star
93

directory-companies-house-search

Python
2
star
94

iigb-beta-content

Content for IIGB website
2
star
95

great-pattern-library

Great Pattern library
CSS
2
star
96

gds-technical-documentation-template

mkdocs template using GOV.UK Design System styles
JavaScript
2
star
97

statement-of-works

Automatic generation for SOW forms
Python
2
star
98

dit-ip

Python
2
star
99

lite-internal-frontend

Application for handling internal information in LITE.
Python
2
star
100

dit-classification-matcher

A microservice developed in Python + Flask to map between CPV <-> HS <-> TARIC codes
Python
2
star