QuarkChain
QuarkChain is a sharded blockchain protocol that employs a two-layer architecture - one extensible sharding layer consisting of multiple shard chains processing transactions and one root chain layer securing the network and coordinating cross-shard transactions among shard chains. The capacity of the network scales linearly as the number of shard chains increase while the root chain is always providing strong security guarantee regardless of the number of shards. QuarkChain testnet consistently hit 10,000+ TPS with 256 shards run by 50 clusters consisting of 6450 servers with each loadtest submitting 3,000,000 transactions to the network.
Features
- Cluster implementation allowing multiple processes / physical machines to work together as a single full node
- State sharding dividing global state onto independent processing and storage units allowing the network capacity to scale linearly by adding more shards
- Cross-shard transaction allowing native token transfers among shard chains
- Adding shards dynamically to the network
- Support of different mining algorithms on different shards
- P2P network allowing clusters to join and leave anytime with encrypted transport
- Fully compatible with Ethereum smart contract
For dApp Developers
Please check dApp Development for a step-by-step tutorial.
Design
Check out the Wiki to understand the design of QuarkChain.
Development Setup
QuarkChain should be run using pypy for better performance. The rest of section uses OSX as the reference for environment set-up.
To install pypy3 on OSX, first install Homebrew
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"Then install pypy3 and other dependencies
brew install pypy3 gmp pkg-config opensslIt's is highly recommended to use virtual environment creating an isolated python environment for your project so that the python modules installed later will only affect this environment.
To clone the code to your target directory
git clone https://github.com/QuarkChain/pyquarkchain.git
cd pyquarkchainTo create a virtual environment
mkdir ~/virtualenv
pypy3 -m venv ~/virtualenv/qcAs the virtual env is created with pypy3 once the env is activated all the python and pip commands will point to their pypy3 versions automatically.
To activate the virtual environment
source ~/virtualenv/qc/bin/activate
# the rest of the tutorial assumes virtual environmentInstall rocksdb which is required by the python-rocksdb module in the next step
brew install rocksdbTo install the required modules for the project. Under pyquarkchain dir where setup.py is located
# you may want to set the following if cryptography complains about header files: (https://github.com/pyca/cryptography/issues/3489)
# export CPPFLAGS=-I/usr/local/opt/openssl/include
# export LDFLAGS=-L/usr/local/opt/openssl/lib
pip install -e .Once all the modules are installed, try running all the unit tests under pyquarkchain
python -m pytest
Development Flow
pre-commit is used to manage git hooks.
pip install pre-commit
pre-commit install
black is used to format modified python code, which will be automatically triggered on new commit after running the above commands. Refer to STYLE for coding style suggestions.
Running Clusters
If you are on a private network (e.g. running from a laptop which connects to the Internet through a router), you need to first setup port forwarding for UDP/TCP 38291.
We recommend following the instruction wiki to start clusters using docker.
Running a single cluster for local testing
Start running a local cluster which does not connect to anyone else. The default cluster has 8 shards and 4 slaves.
cd quarkchain/cluster
pypy3 cluster.py --p2p
# add --start_simulated_mining to mine blocks with simulated mining (does not run any hash algorithms)Running multiple clusters for local testing
Run multiple clusters with P2P network on a single machine with simulated mininig:
pypy3 multi_cluster.py --num_clusters=3 --p2p --start_simulated_miningRunning multiple clusters with P2P network on different machines
NOTE this is effectively a private network. If you would like to join our testnet or mainnet, look back a few sections for instructions.
Just follow the same command to run single cluster and provide --bootnodes flag to discover and connect to other clusters. Make sure ports are open and accessible from outside world: this means if you are running on AWS, open the ports (default both UDP and TCP 38291) in security group; if you are running from a LAN (connecting to the internet through a router), you need to setup port forwarding for UDP/TCP 38291. We have a convenience UPNP module as well, but you will need to check if it has successfully set port forwarding.
(Optional) Not needed if you are joining a testnet or mainnet. If you are starting your own network, first start the bootstrap cluster:
# optional, run python quarkchain/tools/newkey.py and note $BOOTSTRAP_PRIV_KEY and $BOOTSTRAP_PUB_KEY
pypy3 cluster.py --p2p --privkey=$BOOTSTRAP_PRIV_KEYThen start other clusters and provide the bootnode.
BOOTSTRAP_ENODE=enode://$BOOTSTRAP_PUB_KEY@$BOOTSTRAP_IP:$BOOTSTRAP_DISCOVERY_PORT
pypy3 cluster.py --p2p --bootnodes=$BOOTSTRAP_ENODEEffectively, newkey.py gives the bootstrap node an identity, and you will need to provide the public key to anyone who wants to connect to the bootnodes for discovery. Read https://github.com/QuarkChain/pyquarkchain/wiki/Networking#commandline-flags-explained for details on the commandline flags.
Monitoring Clusters
Use the stats tool in the repo to monitor the status of a cluster. It queries the given cluster through JSON RPC every 10 seconds and produces an entry.
$ quarkchain/tools/stats --ip=localhost
----------------------------------------------------------------------------------------------------
QuarkChain Cluster Stats
----------------------------------------------------------------------------------------------------
CPU: 8
Memory: 16 GB
IP: localhost
Shards: 8
Servers: 4
Shard Interval: 60
Root Interval: 10
Syncing: False
Mining: False
Peers: 127.0.0.1:38293, 127.0.0.1:38292
----------------------------------------------------------------------------------------------------
Timestamp TPS Pending tx Confirmed tx BPS SBPS ROOT CPU
----------------------------------------------------------------------------------------------------
2018-09-21 16:35:07 0.00 0 0 0.00 0.00 84 12.50
2018-09-21 16:35:17 0.00 0 9000 0.02 0.00 84 7.80
2018-09-21 16:35:27 0.00 0 18000 0.07 0.00 84 6.90
2018-09-21 16:35:37 0.00 0 18000 0.07 0.00 84 4.49
2018-09-21 16:35:47 0.00 0 18000 0.10 0.00 84 6.10JSON RPC
JSON RPCs are defined in jsonrpc.py. Note that there are two JSON RPC ports. By default they are 38491 for private RPCs and 38391 for public RPCs. Since you are running your own clusters you get access to both.
Public RPCs are documented in the Developer Guide. You can use the client library quarkchain-web3.js to query account state, send transactions, deploy and call smart contracts. Here is a simple example to deploy smart contract on QuarkChain using the client library.
You may find a list of accounts with tokens preallocated through the genesis blocks here. Feel free to use any of them to issue transactions.
Loadtest
Follow this wiki page to loadtest your cluster and see how fast it processes large volumn of transacations.
Issue
Please open issues on github to report bugs or make feature requests.
Contribution
All the help from community is appreciated! If you are interested in working on features or fixing bugs, please open an issue first to describe the task you are planning to do. For small fixes (a few lines of change) feel free to open pull requests directly.
Developer Community
Join our developer community on Discord.
License
Unless explicitly mentioned in a folder or a file, all files are licensed under MIT License defined in LICENSE file.