• Stars
    star
    236
  • Rank 170,480 (Top 4 %)
  • Language
    TypeScript
  • License
    MIT License
  • Created over 4 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Graph Protocol indexer components and infrastructure

Graph Protocol Indexer Components

CI Docker Image: Indexer Service Docker Image: Indexer Agent

NOTE: THIS PROJECT IS BETA SOFTWARE.

The Graph Network vs. Testnet

For configuration details for The Graph Network and the testnet, see the Mainnet and Testnet Configuration docs.

An overview of Scalar, a microtransaction framework for query fees, can be found here.

Running from NPM packages

The indexer service, agent and CLI can be installed as NPM packages, using

npm install -g @graphprotocol/indexer-service
npm install -g @graphprotocol/indexer-agent

# Indexer CLI is a plugin for Graph CLI, so both need to be installed:
npm install -g @graphprotocol/graph-cli
npm install -g @graphprotocol/indexer-cli

After that, they can be run with the following commands:

# Indexer service
graph-indexer-service start ...

# Indexer agent
graph-indexer-agent start ...

# Indexer CLI
graph indexer ...

Usage

Indexer service

$ graph-indexer-service start --help

Start the service

Ethereum
  --ethereum                   Ethereum node or provider URL [string] [required]
  --ethereum-network           Ethereum network    [string] [default: "mainnet"]
  --ethereum-polling-interval  Polling interval for the Ethereum provider (ms)
                                                        [number] [default: 4000]
  --mnemonic                   Mnemonic for the operator wallet
                                                             [string] [required]
  --indexer-address            Ethereum address of the indexer
                                                             [string] [required]

Indexer Infrastructure
  --port                        Port to serve queries at[number] [default: 7600]
  --metrics-port                Port to serve Prometheus metrics at
                                                        [number] [default: 7300]
  --graph-node-query-endpoint   Graph Node endpoint to forward queries to
                                                             [string] [required]
  --graph-node-status-endpoint  Graph Node endpoint for indexing statuses etc.
                                                             [string] [required]
  --log-level                   Log level            [string] [default: "debug"]

Postgres
  --postgres-host      Postgres host                         [string] [required]
  --postgres-port      Postgres port                    [number] [default: 5432]
  --postgres-username  Postgres username          [string] [default: "postgres"]
  --postgres-password  Postgres password                  [string] [default: ""]
  --postgres-database  Postgres database name                [string] [required]

Network Subgraph
  --network-subgraph-endpoint    Endpoint to query the network subgraph from
                                                             [string] [required]
  --network-subgraph-auth-token  Bearer token to require for /network queries
                                                                        [string]
  --serve-network-subgraph       Whether to serve the network subgraph at
                                 /network             [boolean] [default: false]
  --allocation-syncing-interval  Interval (in ms) for syncing indexer
                                 allocations from the network
                                                      [number] [default: 120000]

Query Fees
  --vector-node                 URL of a vector node                    [string]
  --vector-router               Public identifier of the vector router  [string]
  --vector-transfer-definition  Address of the Graph transfer definition
                                contract              [string] [default: "auto"]

Options:
  --version                Show version number                         [boolean]
  --help                   Show help                                   [boolean]
  --gcloud-profiling       Whether to enable Google Cloud profiling
                                                      [boolean] [default: false]
  --free-query-auth-token  Auth token that clients can use to query for free
                                                                         [array]
  --client-signer-address  Address that signs query fee receipts from a known
                           client                                       [string]

Indexer agent

$ graph-indexer-agent start --help

Start the agent

Ethereum
  --ethereum                   Ethereum node or provider URL [string] [required]
  --ethereum-network           Ethereum network    [string] [default: "mainnet"]
  --ethereum-polling-interval  Polling interval for the Ethereum provider (ms)
                                                        [number] [default: 4000]
  --gas-increase-timeout       Time (in seconds) after which transactions will
                               be resubmitted with a higher gas price
                                                         [number] [default: 240]
  --gas-increase-factor        Factor by which gas prices are increased when
                               resubmitting transactions [number] [default: 1.2]
  --gas-price-max              The maximum gas price (gwei) to use for
                               transactions [deprecated] [number] [default: 100]
  --base-fee-per-gas-max       The maximum base fee per gas (gwei) to use for
                               transactions, for legacy transactions this will
                               be treated as the max gas price          [number]
  --transaction-attempts       The maximum number of transaction attempts (Use 0
                               for unlimited)              [number] [default: 0]
  --mnemonic                   Mnemonic for the operator wallet
                                                             [string] [required]
  --indexer-address            Ethereum address of the indexer
                                                             [string] [required]

Indexer Infrastructure
  --graph-node-query-endpoint           Graph Node endpoint for querying
                                        subgraphs            [string] [required]
  --graph-node-status-endpoint          Graph Node endpoint for indexing
                                        statuses etc.        [string] [required]
  --graph-node-admin-endpoint           Graph Node endpoint for applying and
                                        updating subgraph deployments
                                                             [string] [required]
  --public-indexer-url                  Indexer endpoint for receiving requests
                                        from the network     [string] [required]
  --indexer-geo-coordinates             Coordinates describing the Indexer's
                                        location using latitude and longitude
                                   [array] [default: ["31.780715","-41.179504"]]
  --index-node-ids                      Node IDs of Graph nodes to use for
                                        indexing (separated by commas)
                                                              [array] [required]
  --indexer-management-port             Port to serve the indexer management API
                                        at              [number] [default: 8000]
  --metrics-port                        Port to serve Prometheus metrics at
                                                                        [number]
  --syncing-port                        Port to serve the network subgraph and
                                        other syncing data for indexer service
                                        at              [number] [default: 8002]
  --restake-rewards                     Restake claimed indexer rewards, if set
                                        to 'false' rewards will be returned to
                                        the wallet     [boolean] [default: true]
  --rebate-claim-threshold              Minimum value of rebate for a single
                                        allocation (in GRT) in order for it to
                                        be included in a batch rebate claim
                                        on-chain       [string] [default: "200"]
  --rebate-claim-batch-threshold        Minimum total value of all rebates in an
                                        batch (in GRT) before the batch is
                                        claimed on-chain
                                                      [string] [default: "2000"]
  --rebate-claim-max-batch-size         Maximum number of rebates inside a
                                        batch. Upper bound is constrained by
                                        available system memory, and by the
                                        block gas limit  [number] [default: 100]
  --voucher-redemption-threshold        Minimum value of rebate for a single
                                        allocation (in GRT) in order for it to
                                        be included in a batch rebate claim
                                        on-chain       [string] [default: "200"]
  --voucher-redemption-batch-threshold  Minimum total value of all rebates in an
                                        batch (in GRT) before the batch is
                                        claimed on-chain
                                                      [string] [default: "2000"]
  --voucher-redemption-max-batch-size   Maximum number of rebates inside a
                                        batch. Upper bound is constrained by
                                        available system memory, and by the
                                        block gas limit  [number] [default: 100]
  --log-level                           Log level    [string] [default: "debug"]
  --allocation-management               Indexer agent allocation management
                                        automation mode (auto|manual|oversight)
                                                      [string] [default: "auto"]
  --auto-allocation-min-batch-size                 Minimum number of allocation 
                                        transactions inside a batch for AUTO 
                                        management mode    [number] [default: 1]

Network Subgraph
  --network-subgraph-deployment   Network subgraph deployment           [string]
  --network-subgraph-endpoint     Endpoint to query the network subgraph from
                                                                        [string]
  --allocate-on-network-subgraph  Whether to allocate to the network subgraph
                                                      [boolean] [default: false]

Protocol
  --default-allocation-amount  Default amount of GRT to allocate to a subgraph
                               deployment             [string] [default: "0.01"]
  --register                   Whether to register the indexer on chain
                                                       [boolean] [default: true]
  --epoch-subgraph-endpoint    Endpoint to query epoch start blocks from
                                                             [string] [required]

Cost Models
  --inject-dai  Inject the GRT to DAI/USDC conversion rate into cost model
                variables                              [boolean] [default: true]

Postgres
  --postgres-host      Postgres host                         [string] [required]
  --postgres-port      Postgres port                    [number] [default: 5432]
  --postgres-username  Postgres username          [string] [default: "postgres"]
  --postgres-password  Postgres password                  [string] [default: ""]
  --postgres-database  Postgres database name                [string] [required]

Disputes
  --poi-disputable-epochs   The number of epochs in the past to look for
                            potential POI disputes         [number] [default: 1]
  --poi-dispute-monitoring  Monitor the network for potential POI disputes
                                                      [boolean] [default: false]

Query Fees
  --vector-node                   URL of a vector node                  [string]
  --vector-router                 Public identifier of the vector router[string]
  --vector-transfer-definition    Address of the Graph transfer definition
                                  contract            [string] [default: "auto"]
  --vector-event-server           External URL of the vector event server of the
                                  agent                                 [string]
  --vector-event-server-port      Port to serve the vector event server at
                                                        [number] [default: 8001]
  --collect-receipts-endpoint     Client endpoint for collecting receipts
                                                                        [string]

Options:
  --version             Show version number                            [boolean]
  --help                Show help                                      [boolean]
  --dai-contract        Address of the DAI or USDC contract to use for the
                        --inject-dai conversion rate
                [string] [default: "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"]
  --offchain-subgraphs  Subgraphs to index that are not on chain
                        (comma-separated)                  [array] [default: []]

Indexer CLI

Since indexer CLI is a plugin for @graphprotocol/graph-cli, once installed it is invoked simply by running graph indexer.

$ graph indexer --help
Manage indexer configuration

  indexer status                     Check the status of an indexer                                   
  indexer rules stop (never)         Never index a deployment (and stop indexing it if necessary)     
  indexer rules start (always)       Always index a deployment (and start indexing it if necessary)   
  indexer rules set                  Set one or more indexing rules                                   
  indexer rules prepare (offchain)   Offchain index a deployment (and start indexing it if necessary) 
  indexer rules maybe                Index a deployment based on rules                                
  indexer rules get                  Get one or more indexing rules                                   
  indexer rules delete               Remove one or many indexing rules                                
  indexer rules clear (reset)        Clear one or more indexing rules                                 
  indexer rules                      Configure indexing rules                                         
  indexer disputes get               Cross-check POIs submitted in the network                        
  indexer disputes                   Configure allocation POI monitoring                              
  indexer cost set variables         Update cost model variables                                      
  indexer cost set model             Update a cost model                                              
  indexer cost get                   Get cost models and/or variables for one or all subgraphs        
  indexer cost                       Manage costing for subgraphs                                     
  indexer connect                    Connect to indexer management API                                
  indexer allocations reallocate     Reallocate to subgraph deployment                                
  indexer allocations get            List one or more allocations                                     
  indexer allocations create         Create an allocation                                             
  indexer allocations close          Close an allocation                                              
  indexer allocations                Manage indexer allocations                                       
  indexer actions queue              Queue an action item                                             
  indexer actions get                List one or more actions                                         
  indexer actions execute            Execute approved items in the action queue                       
  indexer actions cancel             Cancel an item in the queue                                      
  indexer actions approve            Approve an action item                                           
  indexer actions                    Manage indexer actions                                           
  indexer                            Manage indexer configuration 

Running from source

Run the following at the root of this repository to install dependencies and build the packages:

yarn

After this, the indexer service and agent can be run with:

# Indexer service
cd packages/indexer-service
./bin/graph-indexer-service start ...

# Indexer agent
cd packages/indexer-agent
./bin/graph-indexer-agent start ...

Docker images

The easiest way to run the indexer service agent is by using Docker. Docker images can either be pulled via

docker pull ghcr.io/graphprotocol/indexer-service:latest
docker pull ghcr.io/graphprotocol/indexer-agent:latest

or built locally with

# Indexer service
docker build \
  -f Dockerfile.indexer-service \
  -t indexer-service:latest \
  .

# Indexer agent
docker build \
  -f Dockerfile.indexer-agent \
  -t indexer-agent:latest \
  .

After this, the indexer agent and service can be run as follows:

  1. Indexer service:

    docker run -p 7600:7600 -it indexer-service:latest ...

    After this, the indexer service should be up and running at http://localhost:7600/.

  2. Indexer Agent

    docker run -p 18000:8000 -it indexer-agent:latest ...

    This starts the indexer agent and serves the so-called indexer management API on the host at port 18000.

Terraform & Kubernetes

The terraform/ and k8s/ directories provide a complete example setup for running an indexer on the Google Cloud Kubernetes Engine (GKE). This setup was also used as the reference setup in the Mission Control testnet and can be a good starting point for those looking to run the indexer in a virtualized environment.

Check out the terraform README for details on how to get started.

Releasing

This repository is managed using Lerna and Yarn workspaces.

chan is used to maintain the following changelogs:

Creating a new release involves the following steps:

  1. Update all changelogs:

    pushd packages/indexer-service
    chan added ...
    chan fixed ...
    chan changed ...
    popd
    
    pushd packages/indexer-agent
    ...
    popd
    
    pushd packages/indexer-cli
    ...
    popd
    
    pushd packages/indexer-common
    ...
    popd
    
  2. Publish the release. This includes committing the changelogs, tagging the new version and publishing packages on npmjs.com.

    yarn release <version>

Copyright

Copyright © 2020-2021 The Graph Foundation

Licensed under the MIT license.

More Repositories

1

graph-node

Graph Node indexes data from blockchains such as Ethereum and serves it over GraphQL
Rust
2,882
star
2

graph-tooling

Monorepo for various tools used by subgraph developers.
TypeScript
387
star
3

contracts

Contracts repository for The Graph protocol
TypeScript
316
star
4

graph-ts

TypeScript/AssemblyScript library for writing mappings for The Graph
214
star
5

example-subgraph

An example to help you get started with The Graph
Solidity
179
star
6

graph-client

The Graph library for building GraphQL-based dapps in a decentralized way.
TypeScript
172
star
7

uniswap-subgraph

This is for uniswap-v1. If you are looking for the uniswap v2 subgraph, please go to https://github.com/uniswap/uniswap-v2-subgraph
TypeScript
105
star
8

everest

Registry of crypto projects
JavaScript
98
star
9

docs

Documentation for The Graph
MDX
78
star
10

graph-network-subgraph

The subgraph, the smart contracts, the tests, and documents for the Graph Explorer Decentralized Application
TypeScript
69
star
11

research

Research, proposals, papers, and specs
TeX
48
star
12

ipfs-sync

Script to sync files from one IPFS node to another
JavaScript
48
star
13

ethdenver-dapp

ETHDenver example dApp built on The Graph
JavaScript
43
star
14

erc20-subgraph

TypeScript
33
star
15

hardhat-graph

TypeScript
33
star
16

decentraland-subgraph

Decentraland data source for The Graph
TypeScript
29
star
17

common-ts

Common TypeScript library for use in Graph Protocol components
TypeScript
24
star
18

ens-rainbow

Rust
22
star
19

ens-subgraph

Official repo: https://github.com/ensdomains/ens-subgraph
TypeScript
22
star
20

aragon-subgraph

DAO - subgraph
TypeScript
21
star
21

mission-control-indexer

Technical indexer documentation and infrastructure templates for the Mission Control testnet
TypeScript
21
star
22

indexer-rs

Rewrite of indexer-service in Rust with TAP payments implementation
Rust
21
star
23

token-distribution

Token distribution contracts
TypeScript
21
star
24

agora

Cost model
Rust
20
star
25

rfcs

Graph Protocol RFCs and Engineering Plans
CSS
16
star
26

mission-control-curator

15
star
27

0x-subgraph

Subgraph that tracks events of the 0x protocol
TypeScript
15
star
28

example-subgraphs

TypeScript
14
star
29

allocation-optimizer

Agents and algorithms for optimizing Indexer decision problems.
Julia
14
star
30

firehose-cosmos

Cosmos Firehose integration
Go
11
star
31

support

Community support for the Hosted Service of The Graph
10
star
32

full-stack-graph-app

Deploy a full stack app leveraging The Graph with one click.
TypeScript
10
star
33

compound-v1-subgraph

Compound is an open-source protocol for algorithmic, efficient Money Markets on the Ethereum blockchain.
TypeScript
9
star
34

crypto-buddies

Gatsby React app interview challenge
JavaScript
8
star
35

augur-subgraph

Subgraph for the Augur protocol
TypeScript
8
star
36

qlog

A tool to summarize and analyze graph-node query logs
Rust
8
star
37

bitsausage

Bitsausage auction project
JavaScript
7
star
38

registry-starter

Starting point for apps that curate data
JavaScript
7
star
39

graph-network-analytics-subgraph

Analytics subgraph for Graph Network
TypeScript
6
star
40

stable-hash

A structured hash that is platform independent and has backward compatibility
Rust
6
star
41

common-subgraphs

Subgraphs that are generally useful and/or good examples
Shell
6
star
42

sportx-subgraph

SportX provides a sports betting platform for Ethereum.
TypeScript
6
star
43

block-oracle

Rust
6
star
44

graph-improvement-proposals

The canonical repository for The Graph's protocol governance. Read more about it: https://thegraph.com/ecosystem/governance/
5
star
45

FOAM-subgraph

The FOAM Proof of Location protocol empowers a permissionless and autonomous network of radio beacons that can offer secure location services.
TypeScript
5
star
46

hackathon-react-apollo-app

React, Apollo & Material UI example app
JavaScript
5
star
47

solidity-bindgen

Rust
5
star
48

hardhat-graph-demo

Demo repository for the hardhat-graph plugin
TypeScript
4
star
49

dharma-subgraph

Dharma protocol subgraph
TypeScript
4
star
50

hackathon-scaffold

A starting point for building dApps on The Graph
JavaScript
4
star
51

mutations

TypeScript
4
star
52

compound-subgraph-wrapper

A wrapper service that extends the Compound subgraph with custom resolvers
TypeScript
4
star
53

cosmoshub-ts

TypeScript
4
star
54

memefactory-subgraph

Memefactory subgraph for The Graph
TypeScript
3
star
55

eip-712-derive

Easy EIP-712
Rust
3
star
56

ipfs-mgm

IPFS cli for content syncing
Go
3
star
57

graph-node-docker

Preconfigured Docker image for running a Graph Node
Dockerfile
3
star
58

adchain-subgraph

Example subgraph project for querying the AdChain TCR
TypeScript
3
star
59

graph-local-state

Shell
2
star
60

thegarii

The Graph Arweave Integration Implementation
Rust
2
star
61

codex

2
star
62

maker-subgraph

Subgraph for maker
TypeScript
2
star
63

governance-subgraph

Subgraph for graph governance
TypeScript
2
star
64

firehose-arweave

The firehose integration with the garii
Go
2
star
65

autotasks

A collection of Defender Autotasks to interact with The Graph protocol contracts
TypeScript
2
star
66

query-examples

Provides examples of how to query Subgraphs published to The Graph Network in a variety of frameworks/languages
2
star
67

DEPRECATED-compound-v2-subgraph

Compound is an open-source protocol for algorithmic, efficient Money Markets on the Ethereum blockchain. DEPRECATED - see the repository under compound-finances github for the current repository https://github.com/graphprotocol/compound-v2-subgraph-1
TypeScript
2
star
68

everest-subgraph

Subgraph for the MetaCartel Everest project registry
1
star
69

hack-for-freedom

1
star
70

ethcc-hackathon

1
star
71

gitcoin-beyond-blockchain-hackathon

A three week virtual hackathon running from June 24th to July 10th powered by Gitcoin
1
star
72

network-support

Graph Network issues and feature requests
1
star
73

subgraph-oracle

The Subgraph Oracle
Rust
1
star
74

graph-pino

Pino log message formatter for The Graph
JavaScript
1
star