• Stars
    star
    1,105
  • Rank 41,974 (Top 0.9 %)
  • Language
    Go
  • License
    GNU General Publi...
  • Created about 8 years ago
  • Updated about 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

you're invited to a data party!

qri

Qri GoDoc License Codecov CI

Qri CLI

logo
a dataset version control system built on the distributed web

Welcome

Question Answer
"I want to learn about Qri" Read the official documentation
"I want to download Qri" Download Qri or brew install qri-io/qri/qri
"I have a question" Create an issue and use the label 'question'
"I found a bug" Create an issue and use the label 'bug'
"I want to help build the Qri backend" Read the Contributing guides
"I want to build Qri from source" Build Qri from source

qri is a global dataset version control system built on the distributed web

Breaking that down:

  • global so that if anyone, anywhere has published work with the same or similar datasets, you can discover it.
  • Specific to datasets because data deserves purpose-built tools
  • version control to keep data in sync, attributing all changes to authors
  • On the distributed web to make all of the data published on qri simultaneously available, letting peers work on data together.

If you’re unfamiliar with version control, particularly the distributed kind, well you're probably viewing this document on github β€” which is a version control system intended for code. Its underlying technology – git – popularized some magic sauce that has inspired a generation of programmers and popularized concepts at the heart of the distributed web. Qri is applying that family of concepts to four common data problems:

  1. Discovery Can I find data I’m looking for?
  2. Trust Can I trust what I’ve found?
  3. Friction Can I make this work with my other stuff?
  4. Sync How do I handle changes in data?

Because qri is global and content-addressed, adding data to qri also checks the entire network to see if someone has added it before. Since qri is focused solely on datasets, it can provide meaningful search results. Every change on qri is associated with a peer, creating an audit-able trail you can use to quickly see what has changed and who has changed it. All datasets on qri are automatically described at the time of ingest using a flexible schema that makes data naturally inter-operate. Qri comes with tools to turn all datasets on the network into a JSON API with a single command. Finally, all changes in qri are tracked & synced.

Building From Source

To build qri you'll need the go programming language on your machine.

$ git clone https://github.com/qri-io/qri
$ cd qri
$ make install

If this is your first time building, this command will have a lot of output. That's good! Its means it's working :) It'll take a minute or two to build.

After this is done, there will be a new binary qri in your ~/go/bin directory if using go modules, and $GOPATH/bin directory otherwise. You should be able to run:

$ qri help

and see help output.

Building on Windows

To start, make sure that you have enabled Developer Mode. A library that we depend on needs it enabled in order to properly handle symlinks. If not done, you'll likely get the error message "A required privilege is not held by the client".

You should not need to Run As Administrator to build or run qri. We do not recommend using administrator to run qri.

Shell

For your shell, we recommend using msys2. Other shells, such as cmd, Powershell, or cygwin may also be usable, but msys2 makes it easy to install our required dependencies. IPFS also recommends msys2, and qri is built on top of IPFS.

Dependencies

Building depends upon having git and make installed. If using msys2, you can easily install these by using the package manager "pacman". In a shell, type:

pacman -S git make

Assuming you've also installed go using the official Windows installer linked above, you will also need to add go to your PATH by modifying your environment variable. See the next section on "Environment variables" for more information.

Due to how msys2 treats the PATH variable, you also need to add a new environment variable MSYS2_PATH_TYPE, with the value inherit, using the same procedure.

Once these steps are complete, proceed to building.

Building on Rasberry PI

On a Raspberry PI, you'll need to increase your swap file size in order to build. Normal desktop and server linux OSes should be fine to proceed to building.

One symptom of having not enough swap space is the go install command producing an error message ending with:

link: signal: killed

To increase your swapfile size, first turn off the swapfile:

sudo dphys-swapfile swapoff

Then edit /etc/dphys-swapfile as root and set CONF_SWAPSIZE to 1024.

Finally turn on the swapfile again:

sudo dphys-swapfile swapon

Otherwise linux machines with reduced memory will have other ways to increase their swap file sizes. Check documentation for your particular machine.

Packages

Qri is comprised of many specialized packages. Below you will find a summary of each package.

Package Go Docs Go Report Card Description
api Go Docs report user accessible layer, primarily made for communication with our frontend webapp
cmd Go Docs report our command line interface
config Go Docs report user configuration details, includes peer's profile
lib Go Docs report takes arguments from the cmd and api layer and forms proper requests to call to the action layer
p2p Go Docs report the peer to peer communication layer of qri
repo Go Docs report the repository: saving, removing, and storing datasets, profiles, and the config
dataset Go Docs report the blueprint for a dataset, the atoms that make up qri
registry Go Docs report the blueprint for a registry: the service that allows profiles to be unique and datasets to be searchable
starlib Go Docs report the starlark standard library available for qri transform scripts
qfs Go Docs report "qri file sytem" is Qri's file system abstraction for getting & storing data from different sources
ioes Go Docs report package to handle in, out, and error streams: gives us better control of where we send output and errors
jsonschema Go Docs report used to describe the structure of a dataset, so we can validate datasets and determine dataset interop

Outside Libraries

The following packages are not under Qri, but are important dependencies, so we display their latest versions for convenience.

Package Version
ipfs ipfs version
This documentation has been adapted from the Cycle.js documentation.

More Repositories

1

jsonschema

golang implementation of https://json-schema.org drafts 7 & 2019-09
Go
454
star
2

starlib

qri's standard library for starlark
Go
110
star
3

desktop

Qri Desktop
TypeScript
67
star
4

starpg

A web-based starlark playground
Go
31
star
5

data-stories-scripts

A collection of scripts used for data logging in support of _Data Stories_ on qri.io.
Jupyter Notebook
28
star
6

2017-frontend

qri electron & web frontend
JavaScript
23
star
7

dag

tools for working with directed acyclic graphs (DAGs)
Go
20
star
8

website

qri.io static site
JavaScript
20
star
9

deepdiff

structured data differ with near-linear time complexity
Go
16
star
10

jsonpointer

golang implementation of IETF RFC6901: https://tools.ietf.org/html/rfc6901
Go
16
star
11

dataset

qri dataset definition
Go
15
star
12

rfcs

Request For Comments (RFCs) documenting changes to Qri
12
star
13

wnfs-go

go implementation of fissions web-native file system
Go
11
star
14

p2p-testbed

libp2p & request tracing to make sense of distributed stuff
Go
9
star
15

go-ds-s3

IPFS datastore interface implementation for AWS S3
Go
8
star
16

cafs

content addressed file store interface
Go
7
star
17

matlab

golang matlab level 5 reader / writer library
Go
7
star
18

dataset_sql

Qri SQL support
Go
6
star
19

go-ipld-manifest

DAG manifests for IPLD graphs
Go
6
star
20

walk

Webcrawler/sitemapper
Go
6
star
21

community

πŸ‘‹ Welcome to the Data Bazaar!
5
star
22

zarr-go

Go
5
star
23

startf

Starlark transformation syntax for qri datasets
Go
5
star
24

qri-python

qri python client
Python
5
star
25

registry

MOVED. Registry code now lives at: https://github.com/qri-io/qri/tree/master/registry
Go
4
star
26

varName

Provides Utility for converting length titles into condensed but still recognizable variable names
Go
3
star
27

doggos

doggo nick == [a color]_[doggo breed]
Go
3
star
28

qfs

qri filesystem abstraction
Go
3
star
29

ioes

normalized in, out, error streams for go
Go
3
star
30

frontend

frontend application for our qri.cloud site
TypeScript
3
star
31

papers

HTML
2
star
32

jsontable

Frictionless data jsontable format in go
Go
2
star
33

analytics

distributed analytics
Go
2
star
34

ipfs_camp

ipfs camp workshop tutorial!
1
star
35

homebrew-qri

Qri formula for the Homebrew package manager
Ruby
1
star
36

notes

Notes and ideas about the future of qri
1
star
37

docrun

Run examples found in documentation
Go
1
star
38

errors

golang package for rich errors
Go
1
star
39

apiutil

DEPRECATED - apiutil is a small package for writing user-facing APIs
Go
1
star
40

qri-js

JavaScript
1
star
41

mkpkg

package binaries in platform-specific installers
Go
1
star
42

running-a-remote

examples and guides for running a qri remote
1
star
43

qri_install

easy qri building & installation
Go
1
star
44

dsdiff

Utility for Diffing Datasets, currently a very basic placeholder
Go
1
star