• Stars
    star
    160
  • Rank 234,664 (Top 5 %)
  • Language
    JavaScript
  • License
    MIT License
  • Created almost 8 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Make RPC calls over a Kademlia based DHT.

dht-rpc

Make RPC calls over a Kademlia based DHT.

npm install dht-rpc

Key Features

  • Remote IP / firewall detection
  • Easily add any command to your DHT
  • Streaming queries and updates

Note that internally V5 of dht-rpc differs significantly from V4, due to a series of improvements to NAT detection, secure routing IDs and more.

Usage

Here is an example implementing a simple key value store

First spin up a bootstrap node. You can make multiple if you want for redundancy. There is nothing special about a bootstrap node, except it needs to know it's own host and port, since it knows no other nodes to infer it from.

import DHT from 'dht-rpc'

const bootstrap = DHT.bootstrapper(10001, '127.0.0.1')

Now lets make some dht nodes that can store values in our key value store.

import DHT from 'dht-rpc'
import crypto from 'crypto'

// Let's create 100 dht nodes for our example.
for (var i = 0; i < 100; i++) createNode()

function createNode () {
  const node = new DHT({
    bootstrap: [
      'localhost:10001'
    ]
  })

  const values = new Map()
  const VALUES = 0 // define a command enum

  node.on('request', function (req) {
    if (req.command === VALUES) {
      if (req.token) { // if we are the closest node store the value (ie the node sent a valid roundtrip token)
        const key = hash(req.value).toString('hex')
        values.set(key, req.value)
        console.log('Storing', key, '-->', req.value.toString())
        return req.reply(null)
      }

      const value = values.get(req.target.toString('hex'))
      req.reply(value)
    }
  })
}

function hash (value) {
  return crypto.createHash('sha256').update(value).digest()
}

To insert a value into this dht make another script that does this following

const node = new DHT()

const q = node.query({
  target: hash(val),
  command: VALUES,
  value
}, {
  // commit true will make the query re-request the 20 closest
  // nodes with a valid round trip token to update the values
  commit: true
})

await q.finished()

Then after inserting run this script to query for a value

const target = Buffer.from(hexFromAbove, 'hex')
for await (const data of node.query({ target, command: VALUES })) {
  if (data.value && hash(data.value).toString('hex') === hexFromAbove) {
    // We found the value! Destroy the query stream as there is no need to continue.
    console.log(val, '-->', data.value.toString())
    break
  }
}
console.log('(query finished)')

API

const node = new DHT([options])

Create a new DHT node.

Options include:

{
  // A list of bootstrap nodes
  bootstrap: [ 'bootstrap-node.com:24242', ... ],
  // Optionally pass in array of { host, port } to add to the routing table if you know any peers
  nodes: [{ host, port }, ...],
  // Optionally pass a port you prefer to bind to instead of a random one
  port: 0,
  // Optionally pass a host you prefer to bind to instead of all networks
  host: '0.0.0.0',
  // Optionally pass a UDX instance on which sockets will be created.
  udx,
  // dht-rpc will automatically detect if you are firewalled. If you know that you are not set this to false
  firewalled: true
}

Nodes per default use something called adaptive mode to decide whether or not they want to join other nodes' routing table. This includes things like node uptime, if the node is firewalled etc. Adaptive mode is conservative, so it might take ~20-30 mins for the node to turn persistent. If you are making a test case with your own bootstrap network you'd usually want to turn this off to make sure your test finishes in a timely maner. You can do this by passing ephemeral: false in the constructor. For the vast majority of use-cases you should always use adaptive mode to ensure good DHT health, ie the defaults.

Your DHT routing id is hash(publicIp + publicPort) and will be autoconfigured internally.

const node = DHT.bootstrapper(port, host, [options])

Make a bootstrap node for your DHT. The port and host needs to be it's globally accessable port and host. Note: port and host parameters are used to create the node id. Use options.host if you want to bind to i.e. 127.0.0.1. DHT nodes can use any other DHT node to bootstrap, but a bootstrap node can bootstrap itself, by itself.

await node.ready()

Wait for the node to be fully bootstrapped etc. You don't have to wait for this method, but can be useful during testing.

node.id

Get your own routing ID. Only available when the node is not ephemeral.

node.ephemeral

A boolean indicating if you are currently epheremal or not

node.on('bootstrap')

Emitted when the routing table is fully bootstrapped. Emitted as a conveinience.

node.on('listening')

Emitted when the underlying UDX socket is listening. Emitted as a conveinience.

node.on('ready')

Emitted when the node is fully bootstrapped etc.

node.on('persistent')

Emitted when the node is no longer in ephemeral mode. All nodes start in ephemeral mode, as they figure out their NAT settings. If you set ephemeral: false then this is emitted during the bootstrap phase, assuming you are on an open NAT.

node.on('wake-up')

Emitted when the node has detected that the computer has gone to sleep. If this happens, it will switch from persistent mode to ephemeral again.

node.on('network-change', interfaces)

Emitted when the network interfaces of the computer change.

node.on('nat-update', (host, port) => {})

Emitted when node.host or node.port were changed.

node.on('close')

Will be emitted after node.destroy() is completed.

node.refresh()

Refresh the routing table by looking up a random node in the background. This is called internally periodically, but exposed in-case you want to force a refresh.

node.host

Get your node's public ip, inferred from other nodes in the DHT. If the ip cannot be determined, this is set to null.

node.port

Get your node's public port, inferred from other nodes in the DHT. If your node does not have a consistent port, this is set to 0.

node.firewalled

Boolean indicated if your node is behind a firewall.

This is auto detected by having other nodes trying to do a PING to you without you contacting them first.

const addr = node.address()

Get the local address of the UDP socket bound.

Note that if you are in ephemeral mode, this will return a different port than the one you provided in the constructor (under port), as ephemeral mode always uses a random port.

node.on('request', req)

Emitted when an incoming DHT request is received. This is where you can add your own RPC methods.

  • req.target - the dht target the peer is looking (routing is handled behind the scene)
  • req.command - the RPC command enum
  • req.value - the RPC value buffer
  • req.token - If the remote peer echoed back a valid roundtrip token, proving their "from address" this is set
  • req.from - who sent this request (host, port)

To reply to a request use the req.reply(value) method and to reply with an error code use req.error(errorCode).

In general error codes are up to the user to define, with the general suggestion to start application specific errors from error code 16 and up, to avoid future clashes with dht-rpc internals.

Currently dht-rpc defines the following errors

DHT.OK = 0 // ie no error
DHT.ERROR_UNKNOWN_COMMAND = 1 // the command requested does not exist
DHT.ERROR_INVALID_TOKEN = 2 // the round trip token sent is invalid

reply = await node.request({ token, target, command, value }, to, [options])

Send a request to a specific node specified by the to address ({ host, port }). See the query API for more info on the arguments.

Options include:

{
  retry: true, // whether the request should retry on timeout
  socket: udxSocket // request on this specific socket
}

Normally you'd set the token when commiting to the dht in the query's commit hook.

reply = await node.ping(to, [options])

Sugar for dht.request({ command: 'ping' }, to, options)

Additional options include:

{
  size: 0, // size of the value buffer, filled with zeroes
}

stream = node.query({ target, command, value }, [options])

Query the DHT. Will move as close as possible to the target provided, which should be a 32-byte uniformly distributed buffer (ie a hash).

  • target - find nodes close to this (should be a 32 byte buffer like a hash)
  • command - an enum (uint) indicating the method you want to invoke
  • value - optional binary payload to send with it

If you want to modify state stored in the dht, you can use the commit flag to signal the closest nodes.

{
  // "commit" the query to the 20 closest nodes so they can modify/update their state
  commit: true
}

Commiting a query will just re-request your command to the closest nodes once those are verified. If you want to do some more specific logic with the closest nodes you can specify a function instead, that is called for each close reply.

{
  async commit (reply, dht, query) {
    // normally you'd send back the roundtrip token here, to prove to the remote that you own
    // your ip/port
    await dht.request({ token: reply.token, target, command, value }, reply.from)
  }
}

Other options include:

{
  nodes: [
    // start the query by querying these nodes
    // useful if you are re-doing a query from a set of closest nodes.
  ],
  replies: [
    // similar to nodes, but useful if you have an array of closest replies instead
    // from a previous query.
  ],
  map (reply) {
    // map the reply into what you want returned on the stram
    return { onlyValue: reply.value }
  }
}

The query method returns a stream encapsulating the query, that is also an async iterator. Each data event contain a DHT reply. If you just want to wait for the query to finish, you can use the await stream.finished() helper. After completion the closest nodes are stored in stream.closestNodes array.

If you want to access the closest replies to your provided target you can see those at stream.closestReplies.

stream = node.findNode(target, [options])

Find the node closest to the node with id target. Returns a stream encapsulating the query (see node.query()). options are the same as node.query().

node.destroy()

Shutdown the DHT node.

node.destroyed

Boolean indicating if this has been destroyed.

node.toArray()

Get the routing table peers out as an array of { host, port }

node.addNode({ host, port })

Manually add a node to the routing table.

License

MIT

More Repositories

1

peerflix

Streaming torrent client for node.js
JavaScript
6,094
star
2

playback

Video player built using electron and node.js
JavaScript
2,009
star
3

torrent-stream

The low level streaming torrent engine that peerflix uses
JavaScript
1,941
star
4

why-is-node-running

Node is running but you don't know why? why-is-node-running is here to help you.
JavaScript
1,781
star
5

chromecasts

Query your local network for Chromecasts and have them play media
JavaScript
1,447
star
6

csv-parser

Streaming csv parser inspired by binary-csv that aims to be faster than everyone else
JavaScript
1,413
star
7

torrent-mount

Mount a torrent (or magnet link) as a filesystem in real time using torrent-stream and fuse. AKA MAD SCIENCE!
JavaScript
1,333
star
8

turbo-http

Blazing fast low level http server
JavaScript
996
star
9

is-my-json-valid

A JSONSchema validator that uses code generation to be extremely fast
JavaScript
961
star
10

pump

pipe streams together and close all of them if one of them closes
JavaScript
895
star
11

airpaste

A 1-1 network pipe that auto discovers other peers using mdns
JavaScript
819
star
12

hyperdb

Distributed scalable database
JavaScript
752
star
13

protocol-buffers

Protocol Buffers for Node.js
JavaScript
751
star
14

signalhub

Simple signalling server that can be used to coordinate handshaking with webrtc or other fun stuff.
JavaScript
667
star
15

turbo-json-parse

Turbocharged JSON.parse for type stable JSON data
JavaScript
613
star
16

turbo-net

Low level TCP library for Node.js
JavaScript
598
star
17

peercast

torrent-stream + chromecast
JavaScript
509
star
18

hyperbeam

A 1-1 end-to-end encrypted internet pipe powered by Hyperswarm
JavaScript
482
star
19

multicast-dns

Low level multicast-dns implementation in pure javascript
JavaScript
470
star
20

hyperlog

Merkle DAG that replicates based on scuttlebutt logs and causal linking
JavaScript
466
star
21

hypervision

P2P Television
JavaScript
445
star
22

webcat

Mad science p2p pipe across the web using webrtc that uses your Github private/public key for authentication and a signalhub for discovery
JavaScript
437
star
23

tar-stream

tar-stream is a streaming tar parser and generator.
JavaScript
381
star
24

webrtc-swarm

Create a swarm of p2p connections using webrtc and a signalhub
JavaScript
375
star
25

discovery-swarm

A network swarm that uses discovery-channel to find peers
JavaScript
375
star
26

tar-fs

fs bindings for tar-stream
JavaScript
339
star
27

torrent-docker

MAD SCIENCE realtime boot of remote docker images using bittorrent
JavaScript
314
star
28

fuse-bindings

Notice: We published the successor module to this here https://github.com/fuse-friends/fuse-native
C++
312
star
29

peerwiki

all of wikipedia on bittorrent
JavaScript
308
star
30

awesome-p2p

List of great p2p resources
301
star
31

hyperfs

A content-addressable union file system build on top of fuse, hyperlog, leveldb and node
JavaScript
270
star
32

respawn

Spawn a process and restart it if it crashes
JavaScript
254
star
33

pumpify

Combine an array of streams into a single duplex stream using pump and duplexify
JavaScript
252
star
34

polo

Polo is a zero configuration service discovery module written completely in Javascript.
JavaScript
246
star
35

benny-hill

Play the Benny Hill theme while running another command
JavaScript
242
star
36

streamx

An iteration of the Node.js core streams with a series of improvements.
JavaScript
224
star
37

mp4-stream

Streaming mp4 encoder and decoder
JavaScript
219
star
38

hyperphone

A telephone over Hyperbeam
JavaScript
198
star
39

flat-file-db

Fast in-process flat file database that caches all data in memory
JavaScript
196
star
40

diffy

A tiny framework for building diff based interactive command line tools.
JavaScript
191
star
41

dns-discovery

Discovery peers in a distributed system using regular dns and multicast dns.
JavaScript
190
star
42

duplexify

Turn a writable and readable stream into a streams2 duplex stream with support for async initialization and streams1/streams2 input
JavaScript
185
star
43

browser-server

A HTTP "server" in the browser that uses a service worker to allow you to easily send back your own stream of data.
JavaScript
185
star
44

browserify-fs

fs for the browser using level-filesystem and browserify
JavaScript
184
star
45

ims

Install My Stuff - an opinionated npm module installer
JavaScript
184
star
46

dns-packet

An abstract-encoding compliant module for encoding / decoding DNS packets
JavaScript
181
star
47

jitson

Just-In-Time JSON.parse compiler
JavaScript
178
star
48

dnsjack

A simple DNS proxy that lets you intercept domains and route them to whatever IP you decide.
JavaScript
172
star
49

nanobench

Simple benchmarking tool with TAP-like output that is easy to parse
JavaScript
169
star
50

localcast

A shared event emitter that works across multiple processes on the same machine, including the browser!
JavaScript
165
star
51

level-filesystem

Full implementation of the fs module on top of leveldb
JavaScript
164
star
52

tetris

Play tetris in your terminal - in color
JavaScript
157
star
53

hyperssh

Run SSH over hyperswarm!
JavaScript
146
star
54

end-of-stream

Call a callback when a readable/writable/duplex stream has completed or failed.
JavaScript
145
star
55

flat-tree

A series of functions to map a binary tree to a list
JavaScript
141
star
56

lil-pids

Dead simple process manager with few features
JavaScript
140
star
57

airswarm

Network swarm that automagically discovers other peers on the network using multicast dns
JavaScript
127
star
58

wat2js

Compile WebAssembly .wat files to a common js module
JavaScript
127
star
59

node-modules

Search for node modules
JavaScript
127
star
60

ssh-exec

Execute a script over ssh using Node.JS
JavaScript
126
star
61

add-to-systemd

Small command line tool to simply add a service to systemd
JavaScript
125
star
62

deejay

Music player that broadcasts to everyone on the same network
JavaScript
124
star
63

protocol-buffers-schema

No nonsense protocol buffers schema parser written in Javascript
JavaScript
120
star
64

tree-to-string

Convert a tree structure into a human friendly string
JavaScript
120
star
65

unordered-array-remove

Efficiently remove an element from an unordered array without doing a splice
JavaScript
117
star
66

hyperpipe

Distributed input/output pipe.
JavaScript
116
star
67

abstract-chunk-store

A test suite and interface you can use to implement a chunk based storage backend
JavaScript
113
star
68

shared-structs

Share a struct backed by the same underlying buffer between C and JavaScript
JavaScript
113
star
69

mininet

Spin up and interact with virtual networks using Mininet and Node.js
JavaScript
113
star
70

p2p-workshop

a workshop to learn about p2p
HTML
111
star
71

jsonkv

Single file write-once database that is valid JSON with efficient random access on bigger datasets
JavaScript
109
star
72

ansi-diff-stream

A transform stream that diffs input buffers and outputs the diff as ANSI. If you pipe this to a terminal it will update the output with minimal changes
JavaScript
109
star
73

browser-sync-stream

Rsync between a server and the browser.
JavaScript
108
star
74

docker-registry-server

docker registry server in node.js
JavaScript
108
star
75

dns-socket

Make custom low-level DNS requests from node with retry support.
JavaScript
102
star
76

utp-native

Native bindings for libutp
JavaScript
100
star
77

taco-nginx

Bash script that runs a service and forwards a subdomain to it using nginx when it listens to $PORT
Shell
100
star
78

gunzip-maybe

Transform stream that gunzips its input if it is gzipped and just echoes it if not
JavaScript
98
star
79

merkle-tree-stream

A stream that generates a merkle tree based on the incoming data.
JavaScript
98
star
80

media-recorder-stream

The Media Recorder API in the browser as a readable stream
JavaScript
97
star
81

thunky

Delay the evaluation of a paramless async function and cache the result
JavaScript
97
star
82

peervision

a live p2p streaming protocol
JavaScript
97
star
83

noise-network

Authenticated P2P network backed by Hyperswarm and Noise
JavaScript
96
star
84

soundcloud-to-dat

Download all music from a Soundcloud url and put it into a Dat
JavaScript
96
star
85

blecat

1-1 pipe over bluetooth low energy
JavaScript
95
star
86

debugment

A debug comment -> debugment
JavaScript
93
star
87

hyperdht

A DHT that supports peer discovery and distributed hole punching
JavaScript
93
star
88

docker-browser-console

Forward input/output from docker containers to your browser
JavaScript
90
star
89

srt-to-vtt

Transform stream that converts srt files to vtt files (html5 video subtitles)
JavaScript
89
star
90

speedometer

speed measurement in javascript
JavaScript
88
star
91

mutexify

Bike shed mutex lock implementation
JavaScript
88
star
92

p2p-file-sharing-workshop

A workshop where you learn about distributed file sharing
HTML
88
star
93

mirror-folder

Small module to mirror a folder to another folder. Supports live mode as well.
JavaScript
87
star
94

utp

utp (micro transport protocol) implementation in node
JavaScript
86
star
95

echo-servers.c

A collection of various echo servers in c
C
83
star
96

recursive-watch

Minimal recursive file watcher
JavaScript
82
star
97

docker-browser-server

Spawn and expose docker containers over http and websockets
JavaScript
80
star
98

are-feross-and-mafintosh-stuck-in-an-elevator

Are @feross and @mafintosh stuck in an elevator?
JavaScript
79
star
99

parallel-transform

Transform stream for Node.js that allows you to run your transforms in parallel without changing the order
JavaScript
79
star
100

peer-wire-swarm

swarm implementation for bittorrent
JavaScript
79
star