• Stars
    star
    224
  • Rank 177,792 (Top 4 %)
  • Language
    TypeScript
  • License
    Other
  • Created over 4 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Multiformats interface (multihash, multicodec, multibase and CID)

multiformats

multiformats.io codecov CI

Interface for multihash, multicodec, multibase and CID

Table of contents

Install

$ npm i multiformats

Browser <script> tag

Loading this module through a script tag will make it's exports available as Multiformats in the global namespace.

<script src="https://unpkg.com/multiformats/dist/index.min.js"></script>

Interfaces

This library defines common interfaces and low level building blocks for various interrelated multiformat technologies (multicodec, multihash, multibase, and CID). They can be used to implement custom base encoders / decoders / codecs, codec encoders /decoders and multihash hashers that comply to the interface that layers above assume.

This library provides implementations for most basics and many others can be found in linked repositories.

import { CID } from 'multiformats/cid'
import * as json from 'multiformats/codecs/json'
import { sha256 } from 'multiformats/hashes/sha2'

const bytes = json.encode({ hello: 'world' })

const hash = await sha256.digest(bytes)
const cid = CID.create(1, json.code, hash)
//> CID(bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea)

Creating Blocks

import * as Block from 'multiformats/block'
import * as codec from '@ipld/dag-cbor'
import { sha256 as hasher } from 'multiformats/hashes/sha2'

const value = { hello: 'world' }

// encode a block
let block = await Block.encode({ value, codec, hasher })

block.value // { hello: 'world' }
block.bytes // Uint8Array
block.cid   // CID() w/ sha2-256 hash address and dag-cbor codec

// you can also decode blocks from their binary state
block = await Block.decode({ bytes: block.bytes, codec, hasher })

// if you have the cid you can also verify the hash on decode
block = await Block.create({ bytes: block.bytes, cid: block.cid, codec, hasher })

Multibase Encoders / Decoders / Codecs

CIDs can be serialized to string representation using multibase encoders that implement MultibaseEncoder interface. This library provides quite a few implementations that can be imported:

import { base64 } from "multiformats/bases/base64"
cid.toString(base64.encoder)
//> 'mAYAEEiCTojlxqRTl6svwqNJRVM2jCcPBxy+7mRTUfGDzy2gViA'

Parsing CID string serialized CIDs requires multibase decoder that implements MultibaseDecoder interface. This library provides a decoder for every encoder it provides:

CID.parse('mAYAEEiCTojlxqRTl6svwqNJRVM2jCcPBxy+7mRTUfGDzy2gViA', base64.decoder)
//> CID(bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea)

Dual of multibase encoder & decoder is defined as multibase codec and it exposes them as encoder and decoder properties. For added convenience codecs also implement MultibaseEncoder and MultibaseDecoder interfaces so they could be used as either or both:

cid.toString(base64)
CID.parse(cid.toString(base64), base64)

Note: CID implementation comes bundled with base32 and base58btc multibase codecs so that CIDs can be base serialized to (version specific) default base encoding and parsed without having to supply base encoders/decoders:

const v1 = CID.parse('bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea')
v1.toString()
//> 'bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea'

const v0 = CID.parse('QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n')
v0.toString()
//> 'QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n'
v0.toV1().toString()
//> 'bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku'

Multicodec Encoders / Decoders / Codecs

This library defines BlockEncoder, BlockDecoder and BlockCodec interfaces. Codec implementations should conform to the BlockCodec interface which implements both BlockEncoder and BlockDecoder. Here is an example implementation of JSON BlockCodec.

/**
 * @template T
 * @type {BlockCodec<0x0200, T>}
 */
export const { name, code, encode, decode } = {
  name: 'json',
  code: 0x0200,
  encode: json => new TextEncoder().encode(JSON.stringify(json)),
  decode: bytes => JSON.parse(new TextDecoder().decode(bytes))
}

Multihash Hashers

This library defines MultihashHasher and MultihashDigest interfaces and convinient function for implementing them:

import * as hasher from 'multiformats/hashes/hasher'

const sha256 = hasher.from({
  // As per multiformats table
  // https://github.com/multiformats/multicodec/blob/master/table.csv#L9
  name: 'sha2-256',
  code: 0x12,

  encode: (input) => new Uint8Array(crypto.createHash('sha256').update(input).digest())
})

const hash = await sha256.digest(json.encode({ hello: 'world' }))
CID.create(1, json.code, hash)

//> CID(bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea)

Traversal

This library contains higher-order functions for traversing graphs of data easily.

walk() walks through the links in each block of a DAG calling a user-supplied loader function for each one, in depth-first order with no duplicate block visits. The loader should return a Block object and can be used to inspect and collect block ordering for a full DAG walk. The loader should throw on error, and return null if a block should be skipped by walk().

import { walk } from 'multiformats/traversal'
import * as Block from 'multiformats/block'
import * as codec from 'multiformats/codecs/json'
import { sha256 as hasher } from 'multiformats/hashes/sha2'

// build a DAG (a single block for this simple example)
const value = { hello: 'world' }
const block = await Block.encode({ value, codec, hasher })
const { cid } = block
console.log(cid)
//> CID(bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea)

// create a loader function that also collects CIDs of blocks in
// their traversal order
const load = (cid, blocks) => async (cid) => {
  // fetch a block using its cid
  // e.g.: const block = await fetchBlockByCID(cid)
  blocks.push(cid)
  return block
}

// collect blocks in this DAG starting from the root `cid`
const blocks = []
await walk({ cid, load: load(cid, blocks) })

console.log(blocks)
//> [CID(bagaaierasords4njcts6vs7qvdjfcvgnume4hqohf65zsfguprqphs3icwea)]

Legacy interface

blockcodec-to-ipld-format converts a multiformats BlockCodec into an interface-ipld-format for use with the ipld package. This can help bridge IPLD codecs implemented using the structure and interfaces defined here to existing code that assumes, or requires interface-ipld-format. This bridge also includes the relevant TypeScript definitions.

Implementations

By default, no base encodings (other than base32 & base58btc), hash functions, or codec implementations are exposed by multiformats, you need to import the ones you need yourself.

Multibase codecs

bases import repo
base16 multiformats/bases/base16 multiformats/js-multiformats
base32, base32pad, base32hex, base32hexpad, base32z multiformats/bases/base32 multiformats/js-multiformats
base64, base64pad, base64url, base64urlpad multiformats/bases/base64 multiformats/js-multiformats
base58btc, base58flick4 multiformats/bases/base58 multiformats/js-multiformats

Other (less useful) bases implemented in multiformats/js-multiformats include: base2, base8, base10, base36 and base256emoji.

Multihash hashers

hashes import repo
sha2-256, sha2-512 multiformats/hashes/sha2 multiformats/js-multiformats
sha3-224, sha3-256, sha3-384,sha3-512, shake-128, shake-256, keccak-224, keccak-256, keccak-384, keccak-512 @multiformats/sha3 multiformats/js-sha3
identity multiformats/hashes/identity multiformats/js-multiformats
murmur3-128, murmur3-32 @multiformats/murmur3 multiformats/js-murmur3
blake2b-*, blake2s-* @multiformats/blake2 multiformats/js-blake2

IPLD codecs (multicodec)

codec import repo
raw multiformats/codecs/raw multiformats/js-multiformats
json multiformats/codecs/json multiformats/js-multiformats
dag-cbor @ipld/dag-cbor ipld/js-dag-cbor
dag-json @ipld/dag-json ipld/js-dag-json
dag-pb @ipld/dag-pb ipld/js-dag-pb
dag-jose dag-jose ceramicnetwork/js-dag-jose

API Docs

License

Licensed under either of

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

More Repositories

1

multihash

Self describing hashes - for future proofing
Shell
884
star
2

multiformats

The main repository for discussing multiformats.
543
star
3

multiaddr

Composable and future-proof network addresses
Go
421
star
4

cid

Self-describing content-addressed identifiers for distributed systems
415
star
5

multicodec

Compact self-describing codecs. Save space by using predefined multicodec tables.
Python
335
star
6

multibase

Self identifying base encodings
277
star
7

go-multiaddr

Composable and future-proof network addresses
Go
263
star
8

go-multihash

Multihash implementation in Go
Go
234
star
9

rust-multihash

multihash implementation in Rust
Rust
150
star
10

js-multihash

multihash implementation in JavaScript
JavaScript
119
star
11

js-multiaddr

JavaScript implementation of multiaddr
TypeScript
109
star
12

js-cid

CID implementation in JavaScript
JavaScript
97
star
13

rust-cid

CID in rust
Rust
86
star
14

rust-multiaddr

multiaddr implementation in rust
Rust
86
star
15

unsigned-varint

unsigned varint in use in multiformat specs
77
star
16

multistream

Make data and streams self-described by prefixing them with human readable codecs.
62
star
17

multistream-select

Friendly protocol negotiation. It enables a multicodec to be negotiated between two entities.
62
star
18

cs-multihash

Multihash implementation in C#
C#
48
star
19

rust-multibase

Multibase in rust
Rust
46
star
20

java-multihash

A Java implementation of Multihash
Java
42
star
21

clj-multihash

Clojure implementation of the Multihash spec
Clojure
40
star
22

go-multistream

an implementation of the multistream protocol in go
Go
39
star
23

java-multibase

A Java implementation of multibase
Java
35
star
24

go-multiaddr-net

DEPRECATED: Please use the "net" subpackage in https://github.com/multiformats/go-multiaddr.
Go
34
star
25

go-multicodec

Go constants for the multicodec table
Go
34
star
26

py-multiaddr

multiaddr implementation in Python
Python
33
star
27

cid-utils-website

A website for decoding CIDs
HTML
33
star
28

go-multibase

Implementation of multibase parser in go
Go
32
star
29

js-multihashing-async

The fast version of js-multihashing
JavaScript
29
star
30

multigram

Protocol negotiation and multiplexing over datagrams
29
star
31

js-multistream-select

JavaScript implementation of multistream-select
JavaScript
29
star
32

go-multiaddr-dns

Go library and CLI tool for /dns4, /dns6, /dnsaddr multiaddr resolution
Go
28
star
33

haskell-multihash

Multihash Haskell implementation
Haskell
27
star
34

specs

Specification work regarding multihash, multiaddr, and others
26
star
35

ex_multihash

Multihash implementation in Elixir
Elixir
24
star
36

js-multibase

JavaScript implementation of the multibase specification
JavaScript
23
star
37

py-multibase

Multibase implementation in Python
Python
22
star
38

ruby-multihash

A simple multihash (https://github.com/multiformats/multihash) implementation for ruby.
Ruby
22
star
39

js-multicodec

JavaScript implementation of the multicodec specification
JavaScript
21
star
40

website

The multiformats website
HTML
20
star
41

py-multicodec

Multicodec implementation in Python
Python
17
star
42

cs-multibase

Multibase implementation in C#
C#
16
star
43

java-multiaddr

Java implementation of multiaddr
Java
15
star
44

js-mafmt

javascript multiaddr validation
TypeScript
15
star
45

py-multihash

Multihash implementation in Python
Python
14
star
46

SwiftMultihash

Swift implementation of multihash
Swift
14
star
47

c-multihash

C implementation of Multihash parsing and encoding (but not hashing)
C
12
star
48

js-multihashing

Use all the functions in multihash.
JavaScript
11
star
49

php-multihash

PHP implementation of multihash
PHP
10
star
50

scala-multihash

Scala multihash implementation
Scala
9
star
51

js-cid-tool

A module and command line tool for converting, formatting and discovering properties of CIDs
JavaScript
8
star
52

SwiftMultiaddr

A Multiaddr implementation in Swift.
Swift
8
star
53

cs-multiaddress

Multiaddress implementation in C#
C#
8
star
54

js-multiaddr-to-uri

Convert a Multiaddr to a URI /dnsaddr/ipfs.io/http -> http://ipfs.io
TypeScript
7
star
55

go-base36

Go
7
star
56

MultiHash.Net

.Net implementation of multihash
PowerShell
7
star
57

go-multigram

Go implementation of multigram
6
star
58

go-varint

Go
6
star
59

clj-multistream

Clojure implementation of multistream codecs
Clojure
6
star
60

haskell-multibase

haskell implementation of the multibase multiformat (project by protocol labs)
Haskell
6
star
61

haskell-multicodec

An implementation of the multicodec specification in haskell.
Haskell
5
star
62

clj-varint

Simple wrapper around Bazel VarInt code.
Java
4
star
63

go-multiaddr-fmt

A declarative validator for multiaddrs.
Go
4
star
64

js-sha3

Multiformats hash functions for SHA3
JavaScript
4
star
65

go-multicodec-packed

DEPRECATED -- see go-multicodec
Go
4
star
66

ma-pipe

multiaddr powered pipes
Go
4
star
67

js-uri-to-multiaddr

Convert a URI to a Multiaddr: https://protocol.ai -> /dns4/protocol.ai/tcp/443/https
TypeScript
3
star
68

js-murmur3

Multiformats hash functions for MurmurHash3
JavaScript
3
star
69

js-blake2

BLAKE2 multihash hashers for JavaScript multiformats
JavaScript
3
star
70

cs-multicodec

Multicodec implementation i C#
C#
3
star
71

c-multihashing

Use all the functions in multihash, in C
3
star
72

docs

Multiformats documentation website
2
star
73

cs-multistream

Multistream
C#
2
star
74

github-mgmt

TypeScript
1
star
75

js-multicodec-table

@multiformats/multicodec-table a JavaScript form of the current multicodec table
1
star
76

js-dns

Resolve DNS queries with browser fallback
TypeScript
1
star