• Stars
    star
    121
  • Rank 293,924 (Top 6 %)
  • Language
    Python
  • License
    Other
  • Created over 7 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Dynamic tiling of raster data for OpenAerialMap + others

Marblecutter

This is a standalone (Python / Flask) and Lambda-based dynamic tiler for S3-hosted GeoTIFFs. It is also a set of utilities for transcoding and otherwise preparing raster data for rendering by the tiler.

More information:

Development

Development is best done using Docker, as there are a number of dependencies, some of which remain unpackaged for common OSes (e.g. GDAL-2.2+).

If you're going to be experimenting with mosaicking in any form, docker-compose makes things even easier, as it packages marblecutter alongside a PostGIS database containing footprints and other metadata about imagery.

# build
docker-compose build

# start (requires that Postgres has been populated with appropriate footprint data)
docker-compose up

The transcoding and metadata tools can be used from the images built by docker-compose:

docker-compose run web bash
process.sh ...

AWS Lambda / API Gateway

project.json.hbs defines an apex project that can be deployed to AWS Lambda.

Create IAM role: tiler_lambda_function with Trust Relationship policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create IAM policy: tiler_lambda_logs with policy document:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Attach policy to the tiler_lambda_function role.

Update project.json with tiler_lambda_function's Role ARN, e.g. arn:aws:iam::670261699094:role/tiler_lambda_function.

Run apex deploy. (Add -l debug to see what's running.) This will build the Docker image defined in deps/ to produce a task.zip containing binary dependencies needed when deploying to the Lambda runtime.

Concepts

PixelCollection

A collection of pixels (as data) associated with Bounds.

Bounds

A bounding box (bounds) with an associated CRS (crs).

More Repositories

1

tessera

A tilelive-based tile server.
JavaScript
322
star
2

metricsd

A metrics aggregator for Graphite
Scala
148
star
3

oauth-proxy

(in-use) An HTTP proxy for signing OAuth requests
Python
143
star
4

switchboard

(formerly-in-use) A toolkit for assembling XMPP clients and interacting with XMPP servers.
Ruby
135
star
5

heroku-buildpack-jemalloc

Heroku buildpack with jemalloc
Shell
116
star
6

marblecutter-virtual

Virtual catalogs for marblecutter
Python
46
star
7

osm2orc

Transcodes OSM PBFs to ORC
Java
46
star
8

tl

An alternate command line interface to tilelive
JavaScript
38
star
9

heroku-buildpack-cairo

Heroku buildpack with Cairo (and Pixman and FreeType)
Makefile
34
star
10

fire-hydrant

(demo) Tools for consuming Fire Eagle's XMPP PubSub feed.
Ruby
32
star
11

osm-pds-pipelines

OSM PDS pipeline
JavaScript
30
star
12

sample-oauth-provider

A sample Rails OAuth provider that doesn't use oauth_plugin (or acts_as_authenticated)
Ruby
26
star
13

active_queue

A toolkit for queueing tasks and creating worker processes
Ruby
20
star
14

oauth-reverse-proxy

(proof-of-concept) A reverse HTTP proxy that verifies signed OAuth requests
Python
18
star
15

linq.js

Git mirror of http://linqjs.codeplex.com/
JavaScript
16
star
16

dovetail

(proof-of-concept) A toolkit for assembling XMPP components.
Ruby
14
star
17

mars

(proof-of-concept) REST over XMPP
Ruby
13
star
18

tilelive-http

An HTTP source for tilelive
JavaScript
12
star
19

marblecutter-tilezen

Python
12
star
20

tilelive-cache

A caching wrapper for tilelive.js
JavaScript
11
star
21

searchable

(formerly-in-use) Annotation-Driven Indexing and Searching with Lucene and Solr
Java
11
star
22

node-oauth-proxy

An HTTP proxy that signs OAuth 1.0a requests
JavaScript
10
star
23

heroku-buildpack-gdal

Heroku buildpack with GDAL
Shell
9
star
24

oauth-signpost

Git mirror of http://code.google.com/p/oauth-signpost/
Java
8
star
25

marblecutter-tools

Tools to process raster data for use with marblecutter
Shell
8
star
26

fireeagle-tutorial

Sample code for using Fire Eagle from Ruby and Python
Python
8
star
27

imap2atom

Create an Atom representation of an IMAP mailbox.
Ruby
7
star
28

marblecutter-openaerialmap

Python
7
star
29

tilelive-merge

A tilelive source that merges sources
JavaScript
7
star
30

placemaker-cli

Command-line interface for Y!'s Placemaker API
7
star
31

tilelive-tmstyle

JavaScript
7
star
32

tilelive-tmsource

A tilelive provider for TM2 sources.
JavaScript
6
star
33

mirror

Mirror SVN repositories to GitHub
Ruby
6
star
34

posm-imagery-api

POSM imagery API
Python
6
star
35

whirlycache

(formerly-in-use) [mirror] A fast, flexible in-process object cache for Java.
6
star
36

sawfish

(incomplete) Barebones Fire Eagle updater for Android
Java
5
star
37

log-nexus

A rebroadcasting syslog server
JavaScript
5
star
38

tilelive-rasterpbf

A tilelive source for outputting PBF-encoded rasters from PostGIS.
JavaScript
5
star
39

tilelive-carto

A Carto style source for tilelive
JavaScript
5
star
40

heroku-buildpack-mapnik

Heroku buildpack with Mapnik and its dependencies
Shell
5
star
41

jupiter

(proof-of-concept) A test bed for ActiveResource over XMPP
5
star
42

bamboo-shooter

(demo) PubSub for Pandas
Ruby
4
star
43

fireeagle-location-provider

Fire Eagle Location Provider for Google Gears
4
star
44

node-jersey

Bridges UDP packets onto the information superhighway.
JavaScript
4
star
45

anemone

Web Hook servicer / batch request system
Ruby
4
star
46

tilelive-xray

tilelive data tile visualization
JavaScript
4
star
47

tp

'tp: putting the tee in HTTP
JavaScript
4
star
48

node-metricsd

metricsd client for Node.js
JavaScript
4
star
49

crud_controller

Abstract resource controller + generator for Rails
4
star
50

tilelive-modules

A listing of known tilelive modules
JavaScript
4
star
51

node-fuse-leveldb

LevelDB-backed FUSE filesystem
JavaScript
4
star
52

geode-held

Geode-HELD + Fire Eagle support
JavaScript
4
star
53

tilelive-raster

tilelive source for simple rasters, both local and remote
JavaScript
4
star
54

planworld.rb

(preliminary) ruby port of Planworld
Ruby
3
star
55

overpass-diff-publisher

Convert Overpass augmented diffs to JSON
JavaScript
3
star
56

osm-replication-streams

OSM replication-related Node.js streams
JavaScript
3
star
57

changeset-replay-tool

JavaScript
3
star
58

lambda-layer-rasterio

Dockerfile
3
star
59

mapshaper-proxy

JavaScript
3
star
60

tilelive-mapbox

A tilelive.js source for mapbox:// URIs
JavaScript
3
star
61

heroku-buildpack-pgsql

Heroku buildpack with PostgreSQL binaries and libraries
Shell
3
star
62

tiled-vector-remix

Mix and match tiled vector layers
JavaScript
3
star
63

gatsby-remark-copy-images

Gatsby Plugin: Copies images referenced from markdown to your `public` folder.
JavaScript
3
star
64

sprout

(formerly-in-use) Annotation-Powered Simplicity for Struts 1.x
Java
3
star
65

tilelive-streaming

Streaming functionality for tilelive modules
JavaScript
3
star
66

mojodna.net

CSS
2
star
67

accelerometer

D3 accelerometer vis + Node websocket server
HTML
2
star
68

vector-tile-wizard

MVT rendering with Dropwizard and Processing
Java
2
star
69

mapnik-styles

(reference, experimental) Mapnik styles
2
star
70

httpclient-oauth

HttpComponent-compatible OAuth client implementation (WIP)
Java
2
star
71

flickr-shape-parser

(utility) Parses and converts the Flickr Shapefiles Public Dataset, Version 1.0
Python
2
star
72

AaronCam

Camera + Flickr app skeleton
Objective-C
2
star
73

mapnik-query-analyzer

Find layer bottlenecks in Mapnik stylesheets
JavaScript
2
star
74

marblecutter-land-cover

Marblecutter-powered land cover data rendering
Python
2
star
75

dji_fc300c

Calibration images for DJI FC300C (Phantom 3 Standard)
2
star
76

tilelive-leveldb

JavaScript
2
star
77

uv_check_sample.node

Node.js addon that demonstrates inconsistent uv_check_* behavior
C++
1
star
78

strava-cli

Strava command line client
JavaScript
1
star
79

firesnake

(demo) Python-powered Google Earth visualizer for Fire Eagle data
Python
1
star
80

node-locking-cache

A locking LRU cache
JavaScript
1
star
81

tilelive-cartodb

A tilelive source for CartoDB
JavaScript
1
star
82

docker-gdal

Curated versions of GDAL as Docker images
Dockerfile
1
star
83

node-strav3

Strava API (v3) client (Node.js)
JavaScript
1
star
84

tilelive-blend

A tilelive provider that blends.
JavaScript
1
star
85

tilelive-csvin

A streaming tilelive source for CSV inputs
JavaScript
1
star
86

node-heroku-runtime-metrics

Consumes Heroku log streams and emits metrics to metricsd
JavaScript
1
star
87

tilelive-null

A noop sink for tilelive
JavaScript
1
star
88

levels

Levels
1
star
89

node-fuse-mbtiles

FUSE filesystem for MBTiles
JavaScript
1
star
90

node-heroku-syslog-metricsd-bridge

A Heroku syslog drain that bridges to metricsd
JavaScript
1
star
91

marblecutter-lambda

Dynamic raster processing on AWS
Python
1
star
92

watercors

River gauge proxy
JavaScript
1
star
93

debian-mapnik

Mirror of git://git.debian.org/pkg-grass/mapnik.git
C++
1
star