• Stars
    star
    385
  • Rank 111,464 (Top 3 %)
  • Language
    Go
  • License
    MIT License
  • Created almost 5 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Multitiered file storage API built on Filecoin and IPFS

Powergate

Made by Textile Chat on Slack GitHub license Go Report Card GitHub action

Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular and extensible.

Join us on our public Slack channel for news, discussions, and status updates. Check out our blog for the latest posts and announcements.

Warning This project is still pre-release and is not ready for production usage.

Table of Contents

Prerequisites

To build from source, you need to have a Go version that is between 1.14.x through 1.17.x. Go 1.18.x and above include a libp2p version that is not currently supported.

Design

Powergate is composed of different modules which can be used independently, and compose together to provide other higher-level modules.

Here's a high-level overview of the main components, and how Powergate interacts with IPFS and a Filecoin client: Powergate Design

Note in the diagram that the Lotus and Filecoin client node doesn't need to be in the same host where Powergate is running. They can, but isn't necessary.

📢 Deals module

The Deals module provides a lower layer of abstraction to a Filecoin client node. It provides simple APIs to store, watch, and retrieve data in the Filecoin network. Currently, it interacts with the Lotus client but we have plans to support other Filecoin clients.

👷 Indices and Reputation scoring

Powergate builds three indexes related to on-chain and off-chain data.

The Miners index provides processed data regarding registered miners (on-chain and off-chain), such as: total miner power, relative power, online status, geolocation, and more!

The Ask index provides a fast-retrieval up to date snapshot of miner's asking prices for data storage.

The Faults index provides history data about miners faults while proving their storage on-chain.

Built on top of the previous indexes, a Reputation module constructs a weighted-scoring system that allows to sort miners considering multiple on-chain and off-chain data, such as: compared price to the median of the market, low storage-fault history, power on network, and external sources (soon!).

FFS

This module provides a multitiered file storage API built on Filecoin and IPFS. Storing data on IPFS and Filecoin is as easy as expressing your desired configuration for storing a Cid.

Want to know more about this Powergate module? Check out the FFS design document and our presentation and demo at the IPFS Pinning Summit:

Video

💫 API + CLI

Powergate exposes an API built from the various modules through gRPC endpoints. You can explore our .proto files to generate your clients, or take advange of a ready-to-use Powergate Go and JS client. 🙌

We have a CLI that supports most of Powergate features.

To build and install the CLI, run:

$ make install-pow

The binary will be placed automatically in $GOPATH/bin. You may have to set the Path variables using the below commands

$ export PATH=$PATH:$(go env GOPATH)/bin
$ export GOPATH=$(go env GOPATH)

You can then run pow in your terminal.

You can read the generated CLI docs in this repo, or run pow with the --help flag to see the available commands:

$ pow --help
A client for storage and retreival of powergate data

Usage:
  pow [flags]
  pow [command]

Available Commands:
  admin        Provides admin commands
  config       Provides commands to interact with cid storage configs
  data         Provides commands to interact with general data APIs
  deals        Provides commands to view Filecoin deal information
  help         Help about any command
  id           Returns the user id
  storage-jobs Provides commands to query for storage jobs in various states
  version      Display version information for pow and the connected server
  wallet       Provides commands about filecoin wallets

Flags:
  -h, --help                   help for pow
      --serverAddress string   address of the powergate service api (default "127.0.0.1:5002")
  -t, --token string           user auth token
  -v, --version                display version information for pow and the connected server

Use "pow [command] --help" for more information about a command.

Installation

Powergate installation involves running external dependencies, and wiring them correctly with Powergate.

External dependencies

Powergate needs external dependencies in order to provide full functionality, in particular a synced Filecoin client and a IPFS node.

Filecoin client

Currently, we support the Lotus Filecoin client but we plan to support other clients.

All described modules of Powergate need to comunicate with Lotus to build indices data, and provide storing and retrieving features in FFS. To install Lotus refer to its official documentation, taking special attention to its dependencies.

Fully syncing a Lotus node can take time, so be sure to check you're fully synced doing ./lotus sync status.

We also automatically generate a public Docker image targeting the master branch of Lotus. This image is a pristine version of Lotus, with a sidecar reverse proxy to provide external access to the containerized API. For more information, refer to textileio/lotus-build and its Dockerhub repository.

In short, a fully-synced Lotus node should be available with its API (127.0.0.1:1234, by default) port accessible to Powergate.

IPFS node

A running IPFS node is needed if you plan to use the FFS module.

If that's the case, you can refer here for installation instructions, or its Dockerhub repository if you want to run a contanerized version. Currently we're supporting v0.5.1. The API endpoint should be accessible to Powergate (port 5001, by default).

Since FFS HotStorage is pinning Cids in the IPFS node, Powergate should be the only party controlling the pinset of the node. Other systems can share the same IPFS node if can guarantee not unpinning Cids pinned by Powergate FFS instances.

Geolite database

Powergate needs an offline geo-location database to resolve miners country using their IP address. The same folder in which powd is executing, should have the Geolite2 database file GeoLite2-City.mmdb or you can pass the --maxminddbfolder flag to powd to specify the path of the folder containing GeoLite2-City.mmdb. You can copy this file from the GitHub repo at iplocation/maxmind/GeoLite2-City.mmdb. If you run Powergate using Docker, this database is bundeled in the image so isn't necessary to have extra considerations.

Server

To build and install the Powergate server, run:

make install-powd

You can run the -h flag to see the configurable flags:

$ powd -h 
Usage of powd:
      --askindexmaxparallel string       Max parallel query ask to execute while updating index (default "3")
      --askindexqueryasktimeout string   Timeout in seconds for a query ask (default "15")
      --askindexrefreshinterval string   Refresh interval measured in minutes (default "60")
      --askindexrefreshonstart           If true it will refresh the index on start
      --autocreatemasteraddr             Automatically creates & funds a master address if none is provided.
      --dealwatchpollduration string     Poll interval in seconds used by Deals Module watch to detect state changes (default "900")
      --debug                            Enable debug log level in all loggers.
      --devnet                           Indicate that will be running on an ephemeral devnet. --repopath will be autocleaned on exit.
      --disableindices                   Disable all indices updates, useful to help Lotus syncing process
      --disablenoncompliantapis          Disable APIs that may not easily comply with US law
      --ffsadmintoken string             FFS admin token for authorized APIs. If empty, the APIs will be open to the public.
      --ffsdealfinalitytimeout string    Deadline in minutes in which a deal must prove liveness changing status before considered abandoned (default "4320")
      --ffsminerselector string          Miner selector to be used by FFS: 'sr2', 'reputation' (default "sr2")
      --ffsminerselectorparams string    Miner selector configuration parameter, depends on --ffsminerselector (default "https://raw.githubusercontent.com/filecoin-project/slingshot/master/miners.json")
      --ffsminimumpiecesize string       Minimum piece size in bytes allowed to be stored in Filecoin (default "67108864")
      --ffsschedmaxparallel string       Maximum amount of Jobs executed in parallel (default "1000")
      --ffsusemasteraddr                 Use the master address as the initial address for all new FFS instances instead of creating a new unique addess for each new FFS instance.
      --gatewaybasepath string           Gateway base path. (default "/")
      --gatewayhostaddr string           Gateway host listening address. (default "0.0.0.0:7000")
      --grpchostaddr string              gRPC host listening address. (default "/ip4/0.0.0.0/tcp/5002")
      --grpcwebproxyaddr string          gRPC webproxy listening address. (default "0.0.0.0:6002")
      --ipfsapiaddr string               IPFS API endpoint multiaddress. (Optional, only needed if FFS is used) (default "/ip4/127.0.0.1/tcp/5001")
      --lotushost string                 Lotus client API endpoint multiaddress. (default "/ip4/127.0.0.1/tcp/1234")
      --lotusmasteraddr string           Existing wallet address in Lotus to be used as source of funding for new FFS instances. (Optional)
      --lotustoken string                Lotus API authorization token. This flag or --lotustoken file are mandatory.
      --lotustokenfile string            Path of a file that contains the Lotus API authorization token.
      --maxminddbfolder string           Path of the folder containing GeoLite2-City.mmdb (default ".")
      --mongodb string                   Mongo database name. (if --mongouri is used, is mandatory
      --mongouri string                  Mongo URI to connect to MongoDB database. (Optional: if empty, will use Badger)
      --repopath string                  Path of the repository where Powergate state will be saved. (default "~/.powergate")
      --walletinitialfund int            FFS initial funding transaction amount in attoFIL received by --lotusmasteraddr. (if set) (default 250000000000000000)

Localnet mode

Having a fully synced Lotus node can take a considerable amount of time and effort to mantain. We have built lotus-devnet which runs a local network with a sectorbuilder mock. This provides a fast way to spinup a local network where the sealing process if mocked, but the rest of the node logic is the same as production The localnet supports both 2Kib and 512Kib sectors, and the speed of block production is configurable. Refer to lotus-devnet readme for more information.

If you're interested in running Powergate and experiment with the CLI, the fastest way is to replace the Lotus client dependency with a running localnet, which runs a local Lotus client connected to a network with local miners.

A simple docker-compose setup is available that will run Powergate connected to a Lotus local network with 512Mib sectors and allows to use the gRPC API or CLI without any extra config flags! 🎊 Note: you will first need to install Docker compose in order to get started.

cd docker
make localnet

This will build Powergate powd, a Lotus local network with BIGSECTORS=true by default, an IPFS node and wire them correctly to be ready to use.

Note: Running BIGSECTORS=false make localnet will create the Lotus devent using 2Kib sectors. This may be more appropriate for certain development or testing scenarios.

Here is a full example of using the local network: Terminal 1:

cd docker
make localnet

Wait for seeing logs about the height of the chain increase in a regular cadence.

Terminal 2:

make build
❯ head -c 700 </dev/urandom > myfile
❯ pow admin user create
{
  "user":  {
    "id":  "c06382e0-2021-4234-be53-6e07a8d40065",
    "token":  "883f57b1-4e66-47f8-b291-7cf8b10f6370"
  }
}
❯ pow data stage -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 myfile
{
  "cid":  "QmQJxVtp61Y7UrdjUKuWvse3TxGHaPDyA7RobrBhFwqcBM"
}
❯ pow config apply -w -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm
{
  "jobId":  "b4110048-5367-4ae5-8508-709bf7969748"
}
                 JOB ID                |       STATUS       | MINER  |  PRICE   |    DEAL STATUS     
---------------------------------------+--------------------+--------+----------+--------------------
  b4110048-5367-4ae5-8508-709bf7969748 | JOB_STATUS_SUCCESS |        |          |                    
                                       |                    | f01000 | 62500000 | StorageDealActive
❯ pow data get -t 883f57b1-4e66-47f8-b291-7cf8b10f6370 QmYaAK8SSsKJsJdtahCbUe7MZzQdkPBybFCcQJJ3dKZpfm myfile2
> Success! Data written to myfile2

In this example we created a random 700 bytes file for the test, but since the localnet supports 512Mib sectors you can store store bigger files.

Production setup

A production setup is also provided in the docker folder. It launches powd connected to lotus and ipfs, plus a set of monitoring components:

  • Prometheus, which is the backend for metrics processing.
  • Grafana, for metrics dashboard.
  • cAdvisor, for container metrics.
  • Lotus, node running on the current mainnet.
  • IPFS, node running to back Powergate FFS.
  • Powergate, wired with all of above components.

Depending on which network you want to connect to, you have to run different commands:

  • make up, to connect to mainnet.

Remember that you should wait for Lotus to be fully-synced which might take a long time; you can check your current node sync status running lotus sync status inside the Lotus container. We also provide automatically generated Dockerhub images of Powergate server, see textile/powergate.

If you're interested in a more detailed explanation about Powergate installation, please refer to the installation docs.

Tests

We have a big set of tests for covering most important Powergate features.

For integration tests, we leverage our textileio/lotus-devnet configured with 2Kib sectors to provide fast iteration and CI runs.

If you want to run tests locally:

make test

It will auto-download any necessary dependencies and run all tests.

Benchmark

There's a dedicated binary to run benchmarks against a Powergate server. For more information see the specific README.

Soon we'll add benchmark results against real miners in mainnet, so stay tuned.

Contributing

This project is a work in progress. As such, there's a few things you can do right now to help out:

  • Ask questions! We'll try to help. Be sure to drop a note (on the above issue) if there is anything you'd like to work on and we'll update the issue to let others know. Also get in touch on Slack.
  • Open issues, file issues, submit pull requests!
  • Perform code reviews. More eyes will help a) speed the project along b) ensure quality and c) reduce possible future bugs.
  • Take a look at the code. Contributions here that would be most helpful are top-level comments about how it should look based on your understanding. Again, the more eyes the better.
  • Add tests. There can never be enough tests.

Before you get started, be sure to read our contributors guide and our contributor covenant code of conduct.

Changelog

Changelog is published to Releases.

License

MIT

More Repositories

1

go-threads

Server-less p2p database built on libp2p
Go
453
star
2

go-textile

[DEPRECATED] Textile is a set of tools and infrastructure for building composable apps and services on the IPFS network
Go
357
star
3

photos

[DEPRECATED] Encrypted, secure, decentralized personal data wallet -- technology behind textile.photos
TypeScript
242
star
4

textile

Textile hub services and buckets lib
Go
227
star
5

community

Textile community repo. Includes a sub-project for documentation 📚 and a discussion board for ideas & questions.
101
star
6

js-textile

Textile's JavaScript Libs. Home of ThreadDB, Buckets, and more. Available on npm as `@textile/hub`.
TypeScript
85
star
7

textile-facebook

[DEPRECATED] simple parsing tool to get your data out of a facebook export
HTML
81
star
8

android-ipfs-lite

Java
78
star
9

ios-ipfs-lite

Objective-C
63
star
10

js-examples

Examples and demos using Textile's Javascript/Typescript libraries and clients.
TypeScript
60
star
11

js-threads

This project has been moved to https://github.com/textileio/js-textile
TypeScript
60
star
12

js-powergate-client

Typescript/Javascript client for Textile's Powergate
TypeScript
45
star
13

react-native-sdk

[DEPRECATED] React Native bindings for https://github.com/textileio/go-textile
Java
40
star
14

lotus-devnet

Contanerized Lotus devnet using a mocked sectorbuilder
Go
39
star
15

grpc-ipfs-lite

A gRPC wrapper around ipfs-lite
Go
35
star
16

notes

[DEPRECATED] Textile Notes App
TypeScript
34
star
17

bidbot

A Filecoin Network sidecar for storage providers to bid in storage deal auctions.
Go
30
star
18

dapp-template

a basic template to build simple ipfs-based browser dapps
CSS
30
star
19

js-http-client

[DEPRECATED] Official Textile JS HTTP Wrapper Client
TypeScript
29
star
20

storage-js

Javascript/Typescript SDK for Textile's blockchain ↔ Filecoin bridge system
TypeScript
26
star
21

near-api-go

NEAR client written in Go
Go
22
star
22

advanced-react-native-boilerplate

[DEPRECATED] React Native boilerplate including react-navigation, redux, and sagas with example Textile management.
TypeScript
21
star
23

react-native-boilerplate

[DEPRECATED] A boilerplate app that shows creating, starting, and managing an IPFS peer using Textile's React Native SDK
TypeScript
20
star
24

photos-desktop

[DEPRECATED] Textile Photos... for desktop!
TypeScript
20
star
25

basin

Basin network interfaces & tooling for scalable subnets & onchain data storage
Rust
19
star
26

go-foldersync

A sample-app of file syncing using Threads V2 and IPFS Lite
Go
17
star
27

desktop

[DEPRECATED] Official Textile Desktop Tray App
TypeScript
17
star
28

cloudflare-update-dnslink

Update Cloudflare DNSLink with IPFS Hash
Shell
16
star
29

android-textile

[DEPRECATED] Android bindings for https://github.com/textileio/go-textile
Java
16
star
30

js-threads-client

This project/library has been moved to https://github.com/textileio/js-threads
TypeScript
15
star
31

go-buckets

File and dynamic directory storage built on Threads, IPFS, and LibP2P
Go
14
star
32

dart-threads-client

Threads client for Dart
Dart
14
star
33

js-threaddb

This project has been moved to https://github.com/textileio/js-textile
TypeScript
14
star
34

go-libp2p-pubsub-rpc

RPC over libp2p pubsub with error handling
Go
13
star
35

lotus-build

Automatic Lotus docker image building
Dockerfile
12
star
36

papers

Textile Whitepapers
TeX
12
star
37

explore

Demonstrator/exploratory projects by the Textile team & community
JavaScript
12
star
38

encryptoid

Browser ĐApp for encrypting and sending ephemeral secret messages over IPFS
JavaScript
12
star
39

github-action-buckets

Push repo to a Textile Bucket
TypeScript
11
star
40

storage-js-basic-demo

Minimal React app demo using @textile/near-storage.
TypeScript
11
star
41

dart-textile

Dart library to build apps on Textile
Dart
10
star
42

ios-textile

[DEPRECATED] iOS bindings for https://github.com/textileio/go-textile
Objective-C
10
star
43

gatsby-ipfs-blog

Template for publishing Gatsby blog over IPFS
JavaScript
10
star
44

go-textile-core

[DEPRECATED] Interfaces, types, and abstractions that make up go-textile
Go
8
star
45

go-eventstore

A lightweight event store in Go
Go
8
star
46

js-todo-demo

A simple todo app build with React, Typescript, Semantic UI, and Textile's Threads
TypeScript
8
star
47

textile.photos

Textile Photos Website
CSS
7
star
48

github-action-bucket-replicate

JavaScript
6
star
49

js-foldersync

Demo: Shared folder synchronization based on Textile Threads
TypeScript
6
star
50

eth-storage-bridge

Reference ETH ↔ Filecoin Bridge Smart Contract (Solidity)
TypeScript
6
star
51

build

Everything you need to know to start building for the decentralized web
6
star
52

go-textile-bots

[DEPRECATED]
Go
5
star
53

opts

[DEPRECATED] A handful of bash scripts for administering textile cafes
Shell
5
star
54

notes-desktop

[DEPRECATED] Example app built with reactjs and @textile/js-http-client
TypeScript
5
star
55

go-did-resolver

Universal did-resolver for Go environments
Go
5
star
56

go-auctions-client

A Go client and CLI for Filecoin Storage Auctions.
Go
4
star
57

go-libp2p-primer-article

Sample app for Libp2p-Primer article
Go
4
star
58

dcrypto

A stream-based encryption library (AES-CTR + AES-512 HMAC)
Go
4
star
59

near-storage-bridge

POC Near Smart Contract for locking funds to enable offline storage
TypeScript
4
star
60

broker-core

Broker for the Filecoin network
Go
4
star
61

ipfs-camp-2019

Material and content for Textile's various IPFS Camp contributions
TypeScript
4
star
62

js-textile-go-daemon

[DEPRECATED] Spawn and control the Textile daemon from Node/Javascript
TypeScript
4
star
63

js-textile-wallet

[DEPRECATED] Official Textile data wallet Javascript implementation
TypeScript
4
star
64

near-storage-dapp-demo

React dapp demo using @textile/near-storage to enable storage from a NEAR app.
TypeScript
4
star
65

broker-utils

A set of util documentation and scripts to interact with a Storage Broker.
Shell
3
star
66

react-native-permissions

Fork of original react-native-permissions but trimmed to only Camera permissions for use with QRCode Scanner library
Objective-C
3
star
67

hub-dashboard

Textile's Hub Dashboard
TypeScript
3
star
68

go-ds-mongo

Mongo implementation of go-datastore
Go
3
star
69

react-native-screen-control

Objective-C
3
star
70

xkcd-dapp-demo

Simple ĐApp for viewing and archiving the web's most precious resource
JavaScript
3
star
71

miner-index-web

Public portal for the Miners Index
HTML
3
star
72

node-starter

Node project with the basic building blocks for a web app
SCSS
3
star
73

node-chat

[DEPRECATED] A simple cli chat app using Textile
TypeScript
3
star
74

react-native-camera-roll

Simple camera roll update polling
Objective-C
3
star
75

js-datastore-ttl

An implementation of the Datastore interface that supports a time-to-live for key-value pairs.
TypeScript
2
star
76

npm-go-textile

Textile binary installation module for Node
JavaScript
2
star
77

react-native-textile-image

Java
2
star
78

go-datastore-extensions

go-datastore extensions
Go
2
star
79

filplus-notary-stats

Jupyter Notebook
2
star
80

near-storage-js

Development has moved to https://github.com/textileio/storage-js
TypeScript
2
star
81

basin-s3

Rust
2
star
82

js-http-playground

[DEPRECATED]
TypeScript
2
star
83

hugo-ipfs-blog

1
star
84

js-threads-shell

This project has been deprecated. No longer maintained.
TypeScript
1
star
85

near-storage-cli

Command-line utilities for Textile's Broker-based data storage system on the Near blockchain.
TypeScript
1
star
86

multiclock

Composable and self-describing clocks
1
star
87

www.boom.fyi

HTML
1
star
88

textile-cookies

Official Textile Cookiecutter Templates
JavaScript
1
star
89

go-ds-badger3

Go
1
star
90

github-action-bucket-remove

Remove a Textile Bucket in a Github Action
Dockerfile
1
star
91

ios-textile-image

[DEPRECATED] UIImage view subclass that loads data from Textile
Objective-C
1
star
92

jekyll-ipfs-blog

Template for publishing Jekyll website over IPFS using Textile Buckets
Ruby
1
star
93

go-bot-interfaces

[DEPRECATED]
1
star
94

filecoin-data-transfer-service

Command-line application to create and manage data migration to Filecoin
Go
1
star
95

workshops

Data and materials for public workshops
Jupyter Notebook
1
star
96

react-native-wait-for

TypeScript
1
star
97

npm-go-textile-dep

Official Textile binary distribution for Node projects
TypeScript
1
star
98

minimal-client-demo

[DEPRECATED] Simple project with minimal http client functionality to add files to an existing local thread
JavaScript
1
star
99

base

Base files we use to configure our repositories
1
star
100

prototype-designs

JavaScript
1
star