• Stars
    star
    231
  • Rank 173,434 (Top 4 %)
  • Language
    JavaScript
  • License
    BSD 2-Clause "Sim...
  • Created almost 9 years ago
  • Updated over 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Peer-to-peer database for OpenStreetMap data

osm-p2p-db

Build Status js-standard-style npm

p2p database for open street map data

Table of Contents

Install

npm install osm-p2p-db

Usage

var hyperlog = require('hyperlog')

var level = require('level')
var db = {
  log: level('/tmp/osm-p2p/log'),
  index: level('/tmp/osm-p2p/index')
}
var fdstore = require('fd-chunk-store')
var storefile = '/tmp/osm-p2p/kdb'

var osmdb = require('osm-p2p-db')
var osm = osmdb({
  log: hyperlog(db.log, { valueEncoding: 'json' }),
  db: db.index,
  store: fdstore(4096, storefile)
})

if (process.argv[2] === 'create') {
  var value = JSON.parse(process.argv[3])
  osm.create(value, function (err, key, node) {
    if (err) console.error(err)
    else console.log(key)
  })
} else if (process.argv[2] === 'query') {
  var q = process.argv.slice(3).map(csplit)
  osm.query(q, function (err, pts) {
    if (err) console.error(err)
    else pts.forEach(function (pt) {
      console.log(pt)
    })
  })
}

function csplit (x) { return x.split(',').map(Number) }

Now we can create a few nodes and search with a bounding box query:

$ mkdir /tmp/osm-p2p
$ node db.js create '{"type":"node","lat":64.6,"lon":-147.8}'
11892499690884077339
$ node db.js create '{"type":"node","lat":64.3,"lon":-148.2}'
1982521011513780909
$ node db.js create '{"type":"node","lat":64.5,"lon":-147.3}'
14062704270722785878
$ node db.js query 64.1,64.6 -148,-147
{ type: 'node',
  lat: 64.5,
  lon: -147.3,
  id: '14062704270722785878',
  version: 'e635d07b9fc0a9d048cdd5d9e97a44a19ba3a0b2a51830d1e3e0fadcb80935fc' }

We can make a way document that refers to a list of node documents:

$ node db.js create '{"type":"way","refs":
["11892499690884077339","1982521011513780909","14062704270722785878"]}'
14666931246975765366

When we query, any ways that have one or more nodes within the bounding box will turn up in the results:

$ node db.js query 64.1,64.6 -148,-147
{ type: 'node',
  lat: 64.5,
  lon: -147.3,
  id: '14062704270722785878',
  version: 'e635d07b9fc0a9d048cdd5d9e97a44a19ba3a0b2a51830d1e3e0fadcb80935fc' }
{ type: 'way',
  refs: [ '11892499690884077339', '1982521011513780909', '14062704270722785878' ],
  id: '14666931246975765366',
  version: 'f4fc0045e298ca4f9373fab78dee4f0561b4056dcd7975eb92f21d0a05e0eede' }

Terminology

  • document: a map element, such a a node or way.
  • document id, osm id: an identifier of a document at its latest known version.
  • version, version id: an identifier of a document at a specific point in time.

API

var osmdb = require('osm-p2p-db')

var osm = osmdb(opts)

Create a new osm instance with:

You may optionally pass in a hyperkv instance as opts.kv, but otherwise one will be created from the opts.log and opts.db.

You may safely delete the index database whenever you like. The index data is automatically regenerated. This is very useful if there are breaking changes to the index code or if the data becomes corrupted. The hyperlog contains the source of truth.

osm.create(doc, opts={}, cb)

Store a new document from doc. cb(err, id, node) fires with the generated OSM id and the node from the underlying hyperlog.

Elements are node, way, and relation. Each element should have a type property that contains the element type as a string.

  • Nodes should have doc.lat and doc.lon coordinates.
  • Ways should have an array of OSM keys as doc.refs.
  • Relations should have an array member objects as doc.members. Each member object has a member.type of the document pointed at by member.ref and optionally a member.role.

Another type of document is a changeset. Each element should have a changeset property that refers to the id of a changeset document.

It is recommended to use tags.comment to store free-form text describing the changeset.

osm.put(id, doc, opts={}, cb)

Replace a document at id with doc.

If the document didn't exist previously, it will be created.

The options opts are passed to the underlying hyperkv instance.

By default, hyperkv will merge the most recent known forks into a single fork. To add modifications to a fork without merging the changes into other forks, set opts.links to an array of only the single key you want to update.

osm.del(id, opts={}, cb)

Delete a document at id.

The options opts are passed to the underlying hyperkv instance.

cb(err, node) fires with the underlying node in the hyperlog.

osm.batch(rows, opts={}, cb)

Atomically insert an array of documents rows.

Each row in rows should have:

  • row.type - 'put' or 'del'
  • row.key or row.id - the id of the document (generated if not specified)
  • row.links - array of links to ancestor keys
  • row.value - for puts, the value to store

osm.get(id, opts={}, cb)

Get a document as cb(err, docs) by its OSM id.

docs is an object mapping hyperlog hashes to current document values. If a document has been deleted, it will only have the properties { id: <osm-id>, version: <osm-version>, deleted: true}.

osm.getByVersion(version, cb)

Fetch a specific OSM element by its version string. Returns null if not found, otherwise the single element.

osm.query(q, opts, cb)

Query for all nodes, ways, and relations in the query given by the array q. Queries are arrays of [[minLat,maxLat],[minLon,maxLon]] specifying a bounding box.

cb(err, res) fires with an array of results res. Each result is the document augmented with an id property and a version property that is the hash key from the underlying hyperlog. If a document has been deleted, it will only have the properties { id: <osm-id>, version: <osm-version>, deleted: true}.

Optionally:

  • opts.order - set to 'type' to order by type: node, way, relation

osm.getReferrers(id, cb)

Fetch a list of all OSM ways and relations that refer to the element with ID id. For a node, this can be ways or relations. For a way or relation, this can only be relations.

Objects of the following form are returned:

{
  id: '...',
  version: '...'
}

osm.ready(cb)

Runs the callback cb once all of osm's internal indexes are caught up to the latest data. cb is called exactly once.

osm.close(cb)

Closes the Level and chunk-store backends associated with the database. cb is called upon completion.

var rstream = osm.queryStream(q, opts)

Return a readable object stream rstream of query results contained in the query q. Queries are arrays of [[minLat,maxLat],[minLon,maxLon]] specifying a bounding box.

Each object in the stream is a document augmented with an id property and a version property that is the hash key from the underlying hyperlog.

Optionally:

  • opts.order - set to 'type' to order by type: node, way, relation

var rstream = osm.getChanges(changeset, [cb])

Get the list of document version ids in a changeset by a changeset id changeset.

If a callback is provided, the version ids are returned as cb(err, versions). Without callback, the versions are provided by the returned readable object stream rstream.

osm.on('error', function (err) {})

Handle errors from the underlying indexes with the 'error' event.

osm.kv

You can get at the hyperkv instance directly to perform more operations using osm.kv.

For example, you can use osm.kv.createReadStream() to list all the id/value pairs in the database.

osm.log

The hyperlog instance is available as the opts.log property if you need to get at it directly later.

Browser

To use this module in the browser, use level-browserify to provide the opts.db instance and idb-chunk-store as the opts.store. Each of these is backed by IndexedDB, a native browser storage interface.

Replication

If you have two hyperlogs log0 and log1, pipe them together and back again to replicate:

var r0 = log0.replicate()
var r1 = log1.replicate()
r0.pipe(r1).pipe(r0)

Insert additional streams as necessary for network transports if the logs live in different processes or machines.

If both logs have made edits to the same IDs, multiple records will appear for the same ID in the results. To merge these "conflicts" back into a single value, use osm.put(id, doc, cb) to store the desired document value.

Architecture

Read about the internal architecture.

Contribute

If you would like to support our work, or if you have ideas about how to use and adapt osm-p2p for your own project, then please dive in. Open an issue with a bug report or feature request, or send us a pull request with a bug-fix or new feature.

We need help right now adding tests and fixing edge-cases with osm-p2p-server and increasing compatibility with other OSM tools such as JOSM.

License

BSD (c) 2016, Digital Democracy.

More Repositories

1

react-dimensions

[Looking for maintainers]
JavaScript
454
star
2

leaflet-side-by-side

A Leaflet control to add a split screen to compare two map overlays
JavaScript
346
star
3

mapeo-desktop

Local-first mapping and monitoring in remote environments
JavaScript
261
star
4

filehub-config

Hacking the RAVPower Filehub to automate SD card transfers and harddrive backups
Shell
126
star
5

mapeo-mobile

Monitor and document the world around you
TypeScript
95
star
6

osm-p2p-server

Peer-to-peer OpenStreetMap API v0.6 Server for osm-p2p-db
JavaScript
84
star
7

mapbox-map-image-export

Export a Mapbox GL map to a hi-res image for printing
JavaScript
83
star
8

leaflet-bing-layer

Bing Maps Layer for Leaflet v1.0.0
JavaScript
83
star
9

osm-p2p

High-level p2p OpenStreetMap database for node and the browser
JavaScript
55
star
10

maplibre-storymap

Storytelling template with MapLibre GL JS, deployed using Node or static html.
HTML
38
star
11

bkd-tree

multi-dimensional spatial database based on the bkd paper
JavaScript
36
star
12

kappa-osm

Peer-to-peer OpenStreetMap database using append-only logs
JavaScript
30
star
13

react-mapfilter

Visualizing, exploring, filtering and printing geographic data and geotagged photos and video
JavaScript
29
star
14

hyperdb-osm

Peer-to-peer OpenStreetMap database over hyperdb.
JavaScript
24
star
15

indexed-tarball

Tar archive with constant-time reads & appends. Supports spanning over multiple archives.
JavaScript
22
star
16

mapeo-core

Library for creating custom geo data and syncronizing via a peer to peer network
JavaScript
22
star
17

geojson-dissolve

Dissolve contiguous GeoJSON (Multi)LineStrings and (Multi)Polygons into single units.
JavaScript
21
star
18

GAS-github-json

Uploads a Google Sheet to a github repository as geojson whenever the spreadsheet is updated
Gosu
21
star
19

unordered-materialized-kv

materialized view key/id store based on unordered log messages
JavaScript
20
star
20

osm2obj

Take a readable stream of OSM XML and output a stream of objects
JavaScript
19
star
21

airtable-github-export

Export airtable tables to json/geojson on Github
JavaScript
18
star
22

simple-odk

A minimal ODK server for receiving form submissions from a mobile device and saving them as JSON on github
JavaScript
18
star
23

cache-blob-store

cache images in local storage for offline web apps
JavaScript
17
star
24

mosaic-image-stream

Streaming mosaic of multiple images into a single image
JavaScript
16
star
25

hyperlog-sneakernet-replicator

peer to peer replication for hyperlog using files you can send around on a USB stick
JavaScript
14
star
26

feature-filter-geojson

Creates filtering function for geojson features
JavaScript
12
star
27

digital-democracy.org

Digital Democracy website
SCSS
12
star
28

map-printer

Export massive hi-res PNGs from Mapbox map styles without memory limits.
JavaScript
11
star
29

github-webhook-middleware

Express middleware for validating Github webhooks
JavaScript
11
star
30

mapeo-server

Mapping web server for managing observations, media, tiles, and various static files.
JavaScript
11
star
31

unordered-materialized-backrefs

materialized view to calculate back-references for unordered log messages
JavaScript
10
star
32

edt-offline

Low-energy device that creates an access-point with the Earth Defender's Toolkit platform running, even without the Internet. It provides applications, documentation, use cases, curated websites, and application bridges.
Shell
10
star
33

multicast-service-discovery

Announce and look up services on a local network using mdns
JavaScript
9
star
34

obj2osm

Generate OSM XML from a stream of objects matching OSM JSON
JavaScript
9
star
35

osm-p2p-syncfile

Replicate OSM data and media using a sneakernet-style sync file.
JavaScript
9
star
36

hyperdb-sneakernet

Peer to peer replication for a hyperdb using files you can send around on a USB stick.
JavaScript
9
star
37

osm-p2p-api

Functions for implementing the OSM API using osm-p2p-db.
JavaScript
9
star
38

xform-to-json

Converts XForm submissions from ODK Collect to json or geojson.
JavaScript
8
star
39

offline-map-editor

offline map editor prototype data model
JavaScript
8
star
40

github-card

Github Card embeds see https://developmentseed.org/blog/2015/05/14/github-cards/
HTML
8
star
41

clearwater-map

An interactive map and stories for ClearWater well installations in the Ecuadorian Amazon. See www.giveclearwater.org for more information.
JavaScript
8
star
42

unordered-materialized-bkd

materialized view spatial tree based on unordered log messages
JavaScript
7
star
43

github-image-resizer

Resizes images pushed to Github and saves them to S3, replacing the original on Github with a lo-res version
JavaScript
7
star
44

tile-downloader

Download tiles for a map region without memory limits.
CSS
6
star
45

osm-p2p-observations

p2p database for monitoring observations
JavaScript
6
star
46

mapeo-docs

Docs site for mapeo
JavaScript
6
star
47

id-presets-builder

Build presets for iD Editor
JavaScript
6
star
48

mapeo-web

A small sync and storage service for Mapeo maps.
JavaScript
6
star
49

satellite-image-processing-environment

Vagrant environment for processing satellite images on AWS or locally
Shell
5
star
50

mapeo-openmaptiles

Lightweight map tiles for mapeo-mobile for global offline map
PLpgSQL
5
star
51

html5-image-cache

Caches images in HTML5 localstorage for offline webapps
JavaScript
5
star
52

vips-resizer

Fast streaming image resizer API using libvips
JavaScript
5
star
53

comapeo-mobile

The next version of Mapeo mobile
TypeScript
5
star
54

mapeo-bridge

A headless Mapeo server that syncronizes with Mapeo devices in order to centralize and manipulate data, exporting to Terrastories csv
JavaScript
5
star
55

mapeo-map-server

Offline map style and tile server
TypeScript
5
star
56

osm-p2p-vector-tile-server

serve vector tiles for an osm-p2p database
JavaScript
4
star
57

hyperdb-osm-server

Peer-to-peer OpenStreetMap server
JavaScript
4
star
58

osm-p2p-vector-tile-index

Vector tile index for osm-p2p-db
JavaScript
4
star
59

HaitiReporter

Database System for reporting gender-based violence in Haiti
4
star
60

react-native-geolocation

GeoLocation module for react-native
JavaScript
4
star
61

ddem-observation-server

server for p2p observations and media
JavaScript
4
star
62

GAS-image-upload

A Google Apps Script for Google Sheets that will scrape any image urls (jpgs only for now) pasted into the spreadsheet and send the image to http://www.blitline.com for resizing and copying to AWS S3
Gosu
4
star
63

mapeo-settings-builder

Build presets and sprites for the Mapeo apps
JavaScript
3
star
64

mapeo-config-deconstructor

JavaScript
3
star
65

openrosa-form-submission-middleware

Express middleware for multiparty to process OpenRosa form submissions from ODK Collect
JavaScript
3
star
66

config-cn

Mapeo config for assisting planning and monitoring community networks
CSS
3
star
67

img-observer

Wrapper for MutationObserver for monitoring added and removed nodes
JavaScript
3
star
68

mapbox-map-image-stream

Create an image stream from a mapbox map any scale
JavaScript
3
star
69

smart-data-table

React component for a table of responses from monitoring data
JavaScript
3
star
70

mapeo-webmaps

Web maps for Mapeo
JavaScript
3
star
71

mapeo-core-next

The upcoming version of Mapeo Core
JavaScript
3
star
72

mapeo-sqlite-indexer

Index Mapeo documents to Sqlite
JavaScript
3
star
73

osm-p2p-mem

Create an osm-p2p-db instance backed entirely by in-memory storage.
JavaScript
3
star
74

mapfilter-desktop

electron app that wraps mapfilter
JavaScript
3
star
75

mapeo-icons

Translate terms, search for icons and download them formated and colored
JavaScript
3
star
76

rpc-reflector

JavaScript
3
star
77

rupununi-mining

Satellite images from mining in the Rupununi, Guyana
JavaScript
3
star
78

digidem-server

Digidem server provisioning and config
Shell
3
star
79

mapeo-crypto

JavaScript
2
star
80

email-templates

Digital Democracy Email Templates for MailChimp
HTML
2
star
81

my-simple-odk

Simple roll-your-own ODK server for receiving ODK Collect form submissions
JavaScript
2
star
82

enketo-noserver

experiments with enketo and ODK forms in the browser
HTML
2
star
83

start-stop-state-machine

A simple state machine for managing a service that asynchronously starts and stops
JavaScript
2
star
84

maptiles

Simple library for writing and reading the maptiles format.
JavaScript
2
star
85

edt-offline-portal

Portal application that links to documentation and applications of Earth Defender's Toolkit Offline
CSS
2
star
86

earthdefenderstoolkit

WordPress Theme for the Earth Defenders Toolkit website, and EDT Toolfinder template
PHP
2
star
87

mapeo-webmaps-public

Publicly shared webmaps from Mapeo
JavaScript
2
star
88

mapeo-config-editor

Work in Progress Mapeo Config Editor
JavaScript
2
star
89

alianza-ceibo-maps

Website maps for Alianza CEIBO
JavaScript
2
star
90

mapa-waorani-block22

Interactive map of Waorani territory showing impact of oil block 22
JavaScript
2
star
91

xhr-offline

Drop-in replacement for XMLHttpRequest adding offline-first caching and retries over slow connections.
JavaScript
2
star
92

mapeo-schema

Data schemas for mapeo data types
JavaScript
2
star
93

mapeo-styles

Default styles & tiles for mapeo backgrounds
JavaScript
2
star
94

osm-p2p-geojson

Export GeoJSON from osm-p2p-db
JavaScript
2
star
95

openrosa-request-middleware

Basic middleware for serving OpenRosa compliant responses for express middleware
JavaScript
2
star
96

guyana-coordinates

convert between coordinate systems commonly used in guyana
HTML
2
star
97

secoya-land-invasion

Interactive map showing a land invasion in indigenous Secoya territory in NE Ecuador
CSS
2
star
98

openrosa-manifest

Create a valid OpenRosa manifest document see https://bitbucket.org/javarosa/javarosa/wiki/FormListAPI
JavaScript
2
star
99

HereSync

Offline Database Desktop Sync
2
star
100

sinangoe-webmap

Sinangoe webmap
JavaScript
2
star