• This repository has been archived on 07/Jan/2022
  • Stars
    star
    505
  • Rank 87,373 (Top 2 %)
  • Language
    JavaScript
  • License
    MIT License
  • Created over 8 years ago
  • Updated almost 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Node module for creating dat compatible tools on file systems [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]

deprecated

More info on active projects and modules at dat-ecosystem.org


dat-node

dat-node is a high-level module for building Dat applications on the file system.

npm Travis Test coverage Greenkeeper badge

For a lower-level API for building your own applications, use the Dat SDK which works in Node and the Web

Compatibility

Note: Version 4 of dat-node is not compatible with earlier versions (3.5.15 and below).

Dat Project Documentation & Resources

Features

  • High-level glue for common dat:// and hyperdrive modules.
  • Sane defaults and consistent management of storage & secret keys across applications, using dat-storage.
  • Easily connect to the dat:// network with holepunching, using hyperswarm
  • Import files from the file system, using mirror-folder
  • Serve dats over http with hyperdrive-http
  • Access APIs to lower level modules with a single require!

Browser Support

Many of our dependencies work in the browser, but dat-node is tailored for file system applications. See dat-sdk if you want to build browser-friendly applications.

Example

To send files via dat:

  1. Tell dat-node where the files are.
  2. Import the files.
  3. Share the files on the dat network! (And share the link)
var Dat = require('dat-node')

// 1. My files are in /joe/cat-pic-analysis
Dat('/joe/cat-pic-analysis', function (err, dat) {
  if (err) throw err

  // 2. Import the files
  dat.importFiles()

  // 3. Share the files on the network!
  dat.joinNetwork()
  // (And share the link)
  console.log('My Dat link is: dat://' + dat.key.toString('hex'))
})

These files are now available to share over the dat network via the key printed in the console.

To download the files, you can make another dat-node instance in a different folder. This time we also have three steps:

  1. Tell dat where I want to download the files.
  2. Tell dat what the link is.
  3. Join the network and download!
var Dat = require('dat-node')

// 1. Tell Dat where to download the files
Dat('/download/cat-analysis', {
  // 2. Tell Dat what link I want
  key: '<dat-key>' // (a 64 character hash from above)
}, function (err, dat) {
  if (err) throw err

  // 3. Join the network & download (files are automatically downloaded)
  dat.joinNetwork()
})

That's it! By default, all files are automatically downloaded when you connect to the other users.

Dig into more use cases below and please let us know if you have questions! You can open a new issue or talk to nice humans in our chat room.

Example Applications

  • CLI: We use dat-node in the dat CLI.
  • Desktop: The Dat Desktop application manages multiple dat-node instances via dat-worker.
  • See the examples folder for a minimal share + download usage.
  • And more! Let us know if you have a neat dat-node application to add here.

Usage

All dat-node applications have a similar structure around three main elements:

  1. Storage - where the files and metadata are stored.
  2. Network - connecting to other users to upload or download data.
  3. Adding Files - adding files from the file system to the hyperdrive archive.

We'll go through what these are for and a few of the common usages of each element.

Storage

Every dat archive has storage, this is the required first argument for dat-node. By default, we use dat-storage which stores the secret key in ~/.dat/ and the rest of the data in dir/.dat. Other common options are:

  • Persistent storage: Stored files in /my-dir and metadata in my-dir/.dat by passing /my-dir as the first argument.
  • Temporary Storage: Use the temp: true option to keep metadata stored in memory.
// Permanent Storage
Dat('/my-dir', function (err, dat) {
  // Do Dat Stuff
})

// Temporary Storage
Dat('/my-dir', {temp: true}, function (err, dat) {
  // Do Dat Stuff
})

Both of these will import files from /my-dir when doing dat.importFiles() but only the first will make a .dat folder and keep the metadata on disk.

The storage argument can also be passed through to hyperdrive for more advanced storage use cases.

Network

Dat is all about the network! You'll almost always want to join the network right after you create your Dat:

Dat('/my-dir', function (err, dat) {
  dat.joinNetwork()
  dat.network.on('connection', function () {
    console.log('I connected to someone!')
  })
})

Downloading Files

Remember, if you are downloading - metadata and file downloads will happen automatically once you join the network!

dat runs on a peer to peer network, sometimes there may not be anyone online for a particular key. You can make your application more user friendly by using the callback in joinNetwork:

// Downloading <key> with joinNetwork callback
Dat('/my-dir', {key: '<key>'}, function (err, dat) {
  dat.joinNetwork(function (err) {
    if (err) throw err

    // After the first round of network checks, the callback is called
    // If no one is online, you can exit and let the user know.
    if (!dat.network.connected || !dat.network.connecting) {
      console.error('No users currently online for that key.')
      process.exit(1)
    }
  })
})
Download on Demand

If you want to control what files and metadata are downloaded, you can use the sparse option:

// Downloading <key> with sparse option
Dat('/my-dir', {key: '<key>', sparse: true}, function (err, dat) {
  dat.joinNetwork()

  // Manually download files via the hyperdrive API:
  dat.archive.readFile('/cat-locations.txt', function (err, content) {
    console.log(content) // prints cat-locations.txt file!
  })
})

Dat will only download metadata and content for the parts you request with sparse mode!

Importing Files

There are many ways to get files imported into an archive! Dat node provides a few basic methods. If you need more advanced imports, you can use the archive.createWriteStream() methods directly.

By default, just call dat.importFiles() to import from the directory you initialized with. You can watch that folder for changes by setting the watch option:

Dat('/my-data', function (err, dat) {
  if (err) throw err

  var progress = dat.importFiles({watch: true}) // with watch: true, there is no callback
  progress.on('put', function (src, dest) {
    console.log('Importing ', src.name, ' into archive')
  })
})

You can also import from another directory:

Dat('/my-data', function (err, dat) {
  if (err) throw err

  dat.importFiles('/another-dir', function (err) {
    console.log('done importing another-dir')
  })
})

That covers some of the common use cases, let us know if there are more to add! Keep reading for the full API docs.

API

Dat(dir|storage, [opts], callback(err, dat))

Initialize a Dat Archive in dir. If there is an existing Dat Archive, the archive will be resumed.

Storage

  • dir (Default) - Use dat-storage inside dir. This stores files as files, sleep files inside .dat, and the secret key in the user's home directory.
  • dir with opts.latest: false - Store as SLEEP files, including storing the content as a content.data file. This is useful for storing all history in a single flat file.
  • dir with opts.temp: true - Store everything in memory (including files).
  • storage function - pass a custom storage function along to hyperdrive, see dat-storage for an example.

Most options are passed directly to the module you're using (e.g. dat.importFiles(opts). However, there are also some initial opts can include:

opts = {
  key: '<dat-key>', // existing key to create archive with or resume
  temp: false, // Use random-access-memory as the storage.

  // Hyperdrive options
  sparse: false // download only files you request
}

The callback, cb(err, dat), includes a dat object that has the following properties:

  • dat.key: key of the dat (this will be set later for non-live archives)
  • dat.archive: Hyperdrive archive instance.
  • dat.path: Path of the Dat Archive
  • dat.live: archive.live
  • dat.writable: Is the archive writable?
  • dat.resumed: true if the archive was resumed from an existing database
  • dat.options: All options passed to Dat and the other submodules

Module Interfaces

dat-node provides an easy interface to common Dat modules for the created Dat Archive on the dat object provided in the callback:

var network = dat.joinNetwork([opts], [cb])

Join the network to start transferring data for dat.key, using discovery-swarm. You can also use dat.join([opts], [cb]).

If you specify cb, it will be called when the first round of discovery has completed. This is helpful to check immediately if peers are available and if not fail gracefully, more similar to http requests.

Returns a network object with properties:

  • network.connected - number of peers connected
  • network.on('listening') - emitted with network is listening
  • network.on('connection', connection, info) - Emitted when you connect to another peer. Info is an object that contains info about the connection
Network Options

opts are passed to discovery-swarm, which can include:

opts = {
  upload: true, // announce and upload data to other peers
  download: true, // download data from other peers
  port: 3282, // port for discovery swarm
  utp: true, // use utp in discovery swarm
  tcp: true // use tcp in discovery swarm
}

//Defaults from datland-swarm-defaults can also be overwritten:

opts = {
  dns: {
    server: // DNS server
    domain: // DNS domain
  }
  dht: {
    bootstrap: // distributed hash table bootstrapping nodes
  }
}

Returns a discovery-swarm instance.

dat.leaveNetwork() or dat.leave()

Leaves the network for the archive.

var importer = dat.importFiles([src], [opts], [cb])

Archive must be writable to import.

Import files to your Dat Archive from the directory using mirror-folder.

  • src - By default, files will be imported from the folder where the archive was initiated. Import files from another directory by specifying src.
  • opts - options passed to mirror-folder (see below).
  • cb - called when import is finished.

Returns a importer object with properties:

  • importer.on('error', err)
  • importer.on('put', src, dest) - file put started. src.live is true if file was added by file watch event.
  • importer.on('put-data', chunk) - chunk of file added
  • importer.on('put-end', src, dest) - end of file write stream
  • importer.on('del', dest) - file deleted from dest
  • importer.on('end') - Emits when mirror is done (not emitted in watch mode)
  • If opts.count is true:
    • importer.on('count', {files, bytes}) - Emitted after initial scan of src directory. See import progress section for details.
    • importer.count will be {files, bytes} to import after initial scan.
    • importer.putDone will track {files, bytes} for imported files.
Importer Options

Options include:

var opts = {
  count: true, // do an initial dry run import for rendering progress
  ignoreHidden: true, // ignore hidden files  (if false, .dat will still be ignored)
  ignoreDirs: true, // do not import directories (hyperdrive does not need them and it pollutes metadata)
  useDatIgnore: true, // ignore entries in the `.datignore` file from import dir target.
  ignore: // (see below for default info) anymatch expression to ignore files
  watch: false, // watch files for changes & import on change (archive must be live)
}
Ignoring Files

You can use a .datignore file in the imported directory, src, to ignore any the user specifies. This is done by default.

dat-node uses dat-ignore to provide a default ignore option, ignoring the .dat folder and all hidden files or directories. Use opts.ignoreHidden = false to import hidden files or folders, except the .dat directory.

It's important that the .dat folder is not imported because it contains a private key that allows the owner to write to the archive.

var stats = dat.trackStats()

stats.on('update')

Emitted when archive stats are updated. Get new stats with stats.get().

var st = stats.get()

dat.trackStats() adds a stats object to dat. Get general archive stats for the latest version:

{
  files: 12,
  byteLength: 1234,
  length: 4, // number of blocks for latest files
  version: 6, // archive.version for these stats
  downloaded: 4 // number of downloaded blocks for latest
}
stats.network

Get upload and download speeds: stats.network.uploadSpeed or stats.network.downloadSpeed. Transfer speeds are tracked using hyperdrive-network-speed.

var peers = stats.peers
  • peers.total - total number of connected peers
  • peers.complete - connected peers with all the content data

var server = dat.serveHttp(opts)

Serve files over http via hyperdrive-http. Returns a node http server instance.

opts = {
  port: 8080, // http port
  live: true, // live update directory index listing
  footer: 'Served via Dat.', // Set a footer for the index listing
  exposeHeaders: false // expose dat key in headers
}

dat.pause()

Pause all upload & downloads. Currently, this is the same as dat.leaveNetwork(), which leaves the network and destroys the swarm. Discovery will happen again on resume().

dat.resume()

Resume network activity. Current, this is the same as dat.joinNetwork().

dat.close(cb)

Stops replication and closes all the things opened for dat-node, including:

  • dat.archive.close(cb)
  • dat.network.close(cb)
  • dat.importer.destroy() (file watcher)

License

MIT

More Repositories

1

dat-desktop

Peer to peer data syncronization [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
648
star
2

docs

Documentation resources for dat and the surrounding ecosystem [ DEPRECATED - see https://github.com/hypercore-protocol/new-website/tree/master/guides for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
530
star
3

discovery-channel

Search for a key across multiple discovery networks and find peers who answer. [ DEPRECATED - see https://github.com/hyperswarm/dht for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
299
star
4

datBase

Open data sharing powered by Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
248
star
5

awesome-dat

Community curated resources for Dat Project [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
225
star
6

gasket

Build cross platform data pipelines [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
191
star
7

how-dat-works

Dat documentation [ DEPRECATED - see https://github.com/hypercore-protocol/new-website/tree/master/guides for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
HTML
169
star
8

DEPs

Dat Enhancement Proposals. Contains all specs for the Dat protocol, including drafts. [ DEPRECATED - see https://github.com/hypercore-protocol/hypercore-proposals for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
165
star
9

rabin

node native addon for rabin fingerprinting data streams [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
C++
146
star
10

dat-js

A pure JavaScript browser-friendly api for using dat [ DEPRECATED - see https://github.com/datproject/sdk for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
142
star
11

hyperdiscovery

[ DEPRECATED - see https://github.com/hyperswarm/replicator for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
135
star
12

hypercloud

A hosting server for Dat. [ DEPRECATED - see github.com/beakerbrowser/hashbase for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
97
star
13

hyperdrive-http

HTTP request handler for Hyperdrive and Hypercore [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
68
star
14

book

Documentation on how to implement the Dat Protocol [ DEPRECATED - see https://github.com/hypercore-protocol/new-website/tree/master/guides for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
Shell
68
star
15

datproject-discussions

a repo for discussions and other non-code organizing stuff [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
65
star
16

svalbard

A global metadata vault [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
62
star
17

whitepaper

2017 Dat Whitepaper - Added for Archival Purposes (see DEPs for latest spec) [ DEPRECATED - see https://github.com/dat-ecosystem-archive/DEPs for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
TeX
61
star
18

datasilo

Build a home server that adds high speed storage and upload capacity to a global volunteer science network [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
59
star
19

dat-dns

Issue DNS lookups for Dat archives using HTTPS requests to the target host. [ DEPRECATED - see https://github.com/martinheidegger/hyper-dns for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
57
star
20

node-dat-archive

A nodejs API for Dat which is compatible with Beaker's DatArchive API [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
50
star
21

dat.foundation

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
47
star
22

design

Design Resources and Assets [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
SCSS
43
star
23

datscript

A compile-to-JSON data pipeline scripting language [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
43
star
24

dat-storage

Dat specific storage provider for Hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
35
star
25

organization

Dat Project Organization Information (Grants, Accounting, Etc.) [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
28
star
26

dat.json

The WIP specification for the dat.json meta format [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
27
star
27

dat-swarm-defaults

Dat Defaults for Discovery Swarm [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
27
star
28

project-updates

updates from the various projects working on or with dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
27
star
29

dat-http

a http transport/storage provider for dats [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
26
star
30

planning

Dat planning documents [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
25
star
31

old-website

Dat Project Website [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
23
star
32

web-specs

Specifications for the Dat protocol in Web browsers [ DEPRECATED - see https://github.com/dat-ecosystem-archive/DEPs for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
22
star
33

multidat

Manage multiple dat instances [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
22
star
34

dat-ls

Small program that lists all the changes in a dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
22
star
35

hyperdrive-import-files

Import some files and folders into a hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
21
star
36

dat-secret-storage

hyperdrive storage module for dat secret keys [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
21
star
37

dat-link-resolve

Dat link resolver [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
19
star
38

dat-encoding

Dat's way of encoding and decoding dat links [ DEPRECATED - see https://github.com/mafintosh/abstract-encoding and https://github.com/compact-encoding for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
19
star
39

hyperdrive-network-speed

track upload and download speed on a hyperdrive archive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
18
star
40

dat.land

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
17
star
41

multidrive

Manage multiple hyperdrive instances [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
13
star
42

dat-doctor

πŸ₯ extension for dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
13
star
43

projects

Dat Project Projects [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
13
star
44

dat-deep-link

[DEPRECATED] An experiment to define an URI scheme that implements 'hypercore-strong-link'
JavaScript
12
star
45

dat-log

Log History of Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
12
star
46

dat-foundation-website

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
Vue
11
star
47

datprotocol.org

Dat Protocol Website [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
HTML
10
star
48

dat-registry-api

Creates an api and database for tracking users and dats. [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
10
star
49

dat-json

manage dat.json file via toiletdb [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
10
star
50

dat-ignore

Check if you should ignore a file before adding to Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
9
star
51

anacapa-container

A containerized way to run the Anacapa eDNA processing toolkit on your own machine or server [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
Shell
9
star
52

getdat

Bulk data and/or metadata downloader for large lists of HTTP and FTP urls [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
9
star
53

dat-shaka-player-plugin

[DEPRECATED] A plugin for 'shaka-player' to load 'dat://' URI
JavaScript
8
star
54

dat-data.com

OLD website for the dat project [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
8
star
55

hyperdrive-staging-area

Staging area for local, uncommited writes that can sync to a hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
8
star
56

dat-registry-client

Node.js Client for datBase [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
6
star
57

dat-elements

Dat UI elements [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
5
star
58

hyperdrive-to-zip-stream

Create a zipfile from a hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
5
star
59

analytics-server

fair analytics server w/ docker deployment [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
4
star
60

dat.land-roadmap

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
3
star
61

integration-tests

A mininet test suite for Dat implementations.[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
Sass
3
star
62

Code-of-Conduct

Code of Conduct Discussions [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
3
star
63

discovery-deployment

WIP Deploy your own peer discovery infrastructure [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
2
star
64

datproject.org

JavaScript
1
star
65

desktop-website

πŸ–₯️ Dat Desktop project site [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star
66

dat-glob

Glob implementation for dat archives [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star
67

dat-walk

Recursive directory walker for dat archives [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star
68

dat-rm

Delete files from dat archives using glob patterns [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star