BlockBench

BlockBench is the first benchmarking framework for private blockchain systems. It serves as a fair means of comparison for different platforms and enables deeper understanding of different system design choices.

BlockBench comes with both macro benchmark workloads for evaluating the overall performance and micro benchmark workloads for evaluating performance of individual layers.

Workloads

Macro-benchmark

• YCSB (KVStore).
• SmallBank (OLTP).

Micro-benchmark

• DoNothing (consensus layer).
• IOHeavy (data model layer, read/write oriented).
• Analytics (data model layer, analytical query oriented).
• CPUHeavy (execution layer).

Source file structure

• Smart contract sources are in benchmark/contracts directory.
• Instructions and scripts to run benchmarks for Ethereum, Hyperledger Fabric v0.6, v1.4 and v2.2, Parity and Quorum are in ethereum, hyperledger fabric v0.6 , hyperledger fabric v1.4, hyperledger fabric v2.2, parity , quorum_raft and quorum_vote directories respectively.
• Drivers for benchmark workloads are in src directory.

Dependency

C++ libraries

• restclient-cpp

  Note: we patched this library to include the "Expect: " header in POST requests, which considerably improves the speed for processing RPC request at Parity.

  • The patch file is include in benchmark/parity folder.

  • To patch: go to top-level directory of restclient-cpp, then:

    patch -p4 < $BLOCK_BENCH_HOME/benchmark/parity/patch_restclient

  • The installation can then proceed as normal.

• libcurl

Node.js libraries

Go to micro directory and use npm install to install the dependency libraries

• Web3.js
• zipfian
• bignumber.js
• fabric-v1.4 SDK
• fabric-v2.2 SDK

Blockchain Systems

• geth(ethereum)
• geth(parity)
• geth(quorum)
• hyperledger fabric v0.6
• hyperledger fabric v1.4
• hyperledger fabric v2.2

References

• [1] A. Dinh, J. Wang, G. Chen, R. Liu, B. C. Ooi, K.-L. Tan: BLOCKBENCH: A Framework for Analysing Private Blockchains. ACM SIGMOD 2017.
• [2] The Morning Paper Review 2017
• [3] A. Dinh, R. Liu, M. Zhang, G. Chen, B.C. Ooi, J. Wang: Untangling Blockchain: A Data Processing View of Blockchain Systems. IEEE Transactions on Knowledge and Data Engineering, July 2018.
• [4] FabricSharp -- an faster Fabric blockchain system.
• [5] M. Zhang, Z. Xie, C. Yue, Z. Zhong: Spitz: A Verifiable Database System. Proc. VLDB Endow. 13(12): 3449-3460 (2020).
• [6] P. Ruan, G. Chen, A. Dinh, Q. Lin, D. Loghin, B. C. Ooi, M. Zhang: Blockchains vs. Distributed Databases: Dichotomy and Fusion. To appear in ACM SIGMOD 2021.