• Stars
    star
    129
  • Rank 279,262 (Top 6 %)
  • Language
    Go
  • License
    MIT License
  • Created almost 10 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

An OpenStreetMap pbf parser which exports json, allows you to cherry-pick tags and handles denormalizing ways and relations. Available as a standalone binary and comes with a convenient npm wrapper.

pbf2json creates a JSON stream of openstreetmap data from any PBF extract, you can pick-and-choose only the bits of the file you want, the library will take care of de-normalizing the relational data (nodes/ways/relations) so you can put it straight in to your favourite document-store, inverted index or graph database.

animated-gif

Run from pre-built binary

Greenkeeper badge

You don't need to have Go installed on your system to use one of the precompiled binaries in ./build:

# AMD 64-bit linux/macOS/windows
$ ./build/pbf2json.linux-x64
$ ./build/pbf2json.darwin-x64
$ ./build/pbf2json.win32-x64

# ARM 64-bit linux/macOS
$ ./build/pbf2json.linux-arm64
$ ./build/pbf2json.darwin-arm64

Usage

To control which tags are output you must pass the -tags= flag to pbf2json and the PBF filepath:

$ ./build/pbf2json.linux-x64 -tags="amenity" /tmp/wellington_new-zealand.osm.pbf
{"id":170603342,"type":"node","lat":-41.289843000000005,"lon":174.7944402,"tags":{"amenity":"fountain","created_by":"Potlatch 0.5d","name":"Oriental Bay Fountain","source":"knowledge"},"timestamp":"0001-01-01T00:00:00Z"}
{"id":170605346,"type":"node","lat":-41.2861039,"lon":174.7711539,"tags":{"amenity":"fountain","created_by":"Potlatch 0.10c","source":"knowledge"},"timestamp":"0001-01-01T00:00:00Z"}

Advanced Usage

Multiple tags can be specified with commas, records will be returned if they match one OR the other:

# all buildings and shops
-tags="building,shop"

Tags can also be grouped with the + symbol, records will only be returned if they match one AND the other:

# only records with BOTH housenumber and street specified
-tags="addr:housenumber+addr:street"

You can also combine the above 2 delimiters to get even more control over what get's returned:

# only highways and waterways which have a name
-tags="highway+name,waterway+name"

If you need to target only specific values for a tag you can specify exactly which values you wish to extract using the ~ symbol:

# only extract cuisine tags which have the value of vegetarian or vegan
-tags="cuisine~vegetarian,cuisine~vegan"

Denormalization

When processing the ways, the node refs are looked up for you and the lat/lon values are added to each way.

Since version 3.0 centroids are also computed for each way, since version 5.0 bounds are now also computed.

Output of the nodes array (as seen below) is optional, this was disabled by default in version 5.0 but can be enabled with the flag --waynodes=true.

{
  "id": 301435061,
  "type": "way",
  "tags": {
    "addr:housenumber": "33",
    "addr:postcode": "N5 1TH",
    "addr:street": "Highbury Park",
    "building": "residential"
  },
  "centroid": {
    "lat": "51.554679",
    "lon": "-0.098485"
  },
  "bounds": {
    "e": "-0.0983673",
    "n": "51.5547179",
    "s": "51.5546574",
    "w": "-0.0985915"
  },
  "nodes": [
    {
      "lat": "51.554663",
      "lon": "-0.098369"
    },
    {
      "lat": "51.554657",
      "lon": "-0.098529"
    },
    {
      "lat": "51.554656",
      "lon": "-0.098592"
    },
    {
      "lat": "51.554676",
      "lon": "-0.098590"
    },
    {
      "lat": "51.554680",
      "lon": "-0.098529"
    },
    {
      "lat": "51.554715",
      "lon": "-0.098529"
    },
    {
      "lat": "51.554720",
      "lon": "-0.098369"
    },
    {
      "lat": "51.554663",
      "lon": "-0.098369"
    }
  ]
}

Relations

Since version 6.0 centroids and bounding boxes are also computed for relations, the calulations are based off the largest member way by area.

Note: if a relation does not contain at least one way then it will not be output.

Leveldb

This library uses leveldb to store the lat/lon info about nodes so that it can denormalize the ways for you.

By default the leveldb path is set to /tmp, you can change where it stores the data with a flag:

$ ./build/pbf2json.linux-x64 -leveldb="/tmp/somewhere"

Batched writes

Since version 3.0 writing of node info to leveldb is done in batches to improve performance.

By default the batch size is 50000, you can change this with the following flag:

$ ./build/pbf2json.linux-x64 -batch="1000"

NPM module

var pbf2json = require('pbf2json'),
    through = require('through2');

var config = {
  file: '/tmp/wellington_new-zealand.osm.pbf',
  tags: [
    'addr:housenumber+addr:street'
  ],
  leveldb: '/tmp'
};

pbf2json.createReadStream( config )
 .pipe( through.obj( function( item, e, next ){
    console.log( item );
    next();
 }));

Run the go code from source

Make sure Go is installed and configured on your system, see: https://gist.github.com/missinglink/4212a81a7d9c125b68d9

Note: You should install the latest version of Golang, at least 1.5+, last tested on 1.6.2

sudo apt-get install mercurial;
go get;
go run pbf2json.go;

Compile source for all supported architecture

If you are doing a release and would like to compile for all supported architectures:

note if this is your first time doing this please read the notes in './compile.sh' to set it all up on your machine.

bash compile.sh;

Compile source for a new architecture

If you would like to compile a version of this lib for an architecture which isn't currently supported you can:

go get;
go build;
chmod +x pbf2json;
mv pbf2json build/pbf2json.{platform}-{arch};

Note you will need to change the variables {platform} and {arch} to match those returned by nodejs for your system:

$ node
> var os=require('os')
> os.platform()
'linux'
> os.arch()
'x64'

Then submit a pull request, you are awesome ;)

More Repositories

1

pelias

Pelias is a modular open-source geocoder using Elasticsearch.
Twig
3,052
star
2

placeholder

stand-alone coarse geocoder
JavaScript
303
star
3

docker

Run the Pelias geocoder in docker containers, including example projects.
Shell
287
star
4

documentation

All things documentation for Pelias
217
star
5

api

HTTP API for Pelias Geocoder
JavaScript
209
star
6

leaflet-plugin

Add Pelias geocoding to your Leaflet map.
JavaScript
189
star
7

openstreetmap

Import pipeline for OSM in to Pelias
JavaScript
107
star
8

polygon-lookup

Fast point-in-polygon intersection for large numbers of polygons.
JavaScript
71
star
9

interpolation

global street address interpolation service (beta)
JavaScript
54
star
10

openaddresses

Pelias import pipeline for OpenAddresses.
JavaScript
46
star
11

geonames

Import pipeline for geonames in to Pelias
JavaScript
43
star
12

parser

natural language classification engine for geocoding
JavaScript
41
star
13

schema

elasticsearch schema files and tooling
JavaScript
38
star
14

libpostal-service

Dockerfile for libpostal-service based on the Who's on First implementation
Dockerfile
33
star
15

spatial

ALPHA: geographic data service backed by spatialite
JavaScript
29
star
16

whosonfirst

Importer for Who's on First gazetteer
JavaScript
26
star
17

pelias-android-sdk

Android sdk for pelias
Java
20
star
18

csv-importer

Import arbitrary data in CSV format to Pelias
JavaScript
17
star
19

polylines

Pelias import pipeline for polyline (road network) data.
JavaScript
17
star
20

pip-service

Pelias point-in-polygon-service
JavaScript
15
star
21

query

geospatial queries used by the pelias api
JavaScript
12
star
22

terraform-elasticsearch

Terraform scripts for running an Elasticsearch cluster
HCL
10
star
23

pelias-ios-sdk

Interact with Mapzen's search & geocoding service
Swift
9
star
24

wof-admin-lookup

Who's on First Admin Lookup for the Pelias Geocoder
JavaScript
9
star
25

config

Configuration file for Pelias
JavaScript
8
star
26

dashboard

Pelias dashboard built with the Dashing framework
JavaScript
7
star
27

scripts-batch-search

JavaScript
6
star
28

model

Pelias data models
JavaScript
6
star
29

transit

Load transit landmarks into the Pelias geocoder
JavaScript
6
star
30

acceptance-tests

Pelias API acceptance tests
4
star
31

presentation

Pelias related talks and presentations.
JavaScript
4
star
32

postal-cities

Scripts to generate mappings of postal codes to 'last line' postal localities (postal cities)
JavaScript
4
star
33

labels

Pelias Label generation
JavaScript
4
star
34

fuzzy-tester

A fuzzy testing library for geocoding
JavaScript
4
star
35

microservice-wrapper

JavaScript
4
star
36

docker-baseimage

Pelias Docker Baseimage
Dockerfile
3
star
37

wof

WhosOnFirst tools
JavaScript
3
star
38

dbclient

Database client for Pelias import pipelines
JavaScript
3
star
39

design

Branding & graphic design guidelines and assets
2
star
40

sorting

JavaScript
2
star
41

loadtest

Scripts for loadtesting pelias
JavaScript
2
star
42

woflint

WhosOnFirst document/collection linter
JavaScript
1
star
43

mars-importer

Importer for Martian data
JavaScript
1
star
44

docker-valhalla-baseimage

Pelias Docker Baseimage with Valhalla additionally installed
Shell
1
star
45

blacklist-stream

Pelias document blacklist stream
JavaScript
1
star
46

analysis

text analysis libraries (work in progress)
JavaScript
1
star
47

ci-tools

Tools for manging CI builds used in other repositories
Shell
1
star