Hyperledger Caliper to Provide Benchmarking for Blockchain Systems
The need for a standard
Performance tests are essential for gathering and measuring metrics of any application. When it comes to blockchain, there are quite a few benchmarking tools that already exist, such as BlockBench and Performance Traffic Engine, but most of these provide limited support.
This lack of a standard benchmark solution for blockchain applications is the problem Caliper, a Hyperledger incubation project, is trying to resolve. The tool was initially contributed by developers from Huawei, Hyperchain, Oracle, Bitwise, Soramitsu, IBM, and the Budapest University of Technology and Economics.
According to Nick Lincoln of IBM, who provided an overview of Caliper at the Consensus 2019 conference this May, the framework is fit for comparative testing of blockchain offerings out there.
“Hyperledger Caliper is a performance benchmark framework for blockchains. We’re looking at testing different blockchain solutions with predefined use cases. We’ve designed this specifically for comparative performance studies across different blockchain technologies.”
—Nick Lincoln, IBM
What is Caliper?
One of the primary goals of Caliper is to provide performance results that can be used by other Hyperledger projects, as well as non-ecosystem ones, as reference points when implementing blockchain solutions. The project was initially developed by Huawei and was proposed to Hyperledger early in 2018. It was later accepted and placed into incubation on March 15, 2018.
Based on its documentation, Caliper is ideal for:
- application developers wanting to run performance tests for their smart contracts
- system architects wanting to investigate resource constraints during test loads
A benchmarking layer introduced with Caliper (Image credit)At present, Caliper supports Fabric, Sawtooth, Iroha, Burrow, as well as Composer. The benchmark framework currently provides the following performance indicators:
- success rate
- transaction/read throughput
- transaction/read latency
- resource consumption
Nick emphasized that Caliper not only supports existing Hyperledger solutions, but also non-Hyperledger blockchain technologies.
“We have an open pull request for an Ethereum adaptor. Additionally, Sawtooth itself contains an EVM, which means you can run Ethereum inside Sawtooth. We have have adaptors for all the Hyperledger components. We have an interface that you can follow to provide your own adaptor for any other system. We also give you a CLI package for convenience in running benchmarks. We’re trying to make it as easy as possible to test different blockchain systems and get some performance metrics out of them.”
—Nick Lincoln, IBM
How it works
To better understand how Caliper works, we can reference its benchmark architecture. Starting off, a user will have to define configuration files, which include:
- A benchmark file defining the arguments of a benchmark workload
- A blockchain file specifying the necessary information, which helps to interact with the system being tested
- Smart contracts defining what contracts are going to be deployed
The workflows inside Caliper’s architecture (Image credit)These configuration files are then fed to the Caliper CLI. This creates an admin client and a client factory. The admin client is considered the superuser.
“The admin client feeds through an adaptor. If you’re using Fabric, it goes through the Fabric adaptor. If you’re using Sawtooth, it goes through the Sawtooth adaptor. If you’re testing your own system, it will go through your own adaptor. This defines the system under test.”
—Nick Lincoln, IBM
On the other hand, the client factory creates clients responsible for running test loads. Depending on a benchmark file, the clients could be transacting with the system to add assets or to query. The clients also drive through a rate controller, which dictates the rate of transactions per second (TPS).
“We have a modulized rate controller, which means there are multiple rate controllers you can use. You might want to drive at a steady rate—50 TPS, 150 TPS, or 1000 TPS. You might want to ramp the rate. We have controllers that do that, as well. You might just want to load the system and see what happens. You can drive this with a certain number of pending transactions per client. This tests the limit of the system.” —Nick Lincoln, IBM
During a test, all transactions are saved. The statistics of these transactions are recorded and collated. In addition, a resource monitor also records the consumption of resources. All of this data is pooled together into a single report.
An example of a Hyperledger Caliper report (Image credit)“There’s quite a simple flow throughout where we take the configuration files, pass them into Caliper, and we set up the system. The clients are driving it, collecting all of this information, and publishing a report.” —Nick Lincoln, IBM
The roadmap for Caliper
Caliper is still in the incubation stage of the Hyperledger Project—it is currently in the beta phase of development. Some of the milestones for a release candidate include creating adaptors for Corda, Quorom, and Ethereum, as well as designing enhanced use cases and metrics. The release candidate is expected to be delivered this fall.
“The idea is to have the same use case for each benchmark. It needs to be an apples-to-apples comparison. You should be able to run a use case on one system, run the same use case on another system, and then you have a realistic comparison.” —Nick Lincoln, IBM
Hyperledger Caliper’s roadmap for 2019 (Image credit)Caliper’s development can be tracked in its GitHub repository. Anyone looking to contribute can join the Hyperledger mailing list, as well as the chat channel. There is also a regular community call every Wednesday at 9 a.m. (UTC).
Want details? Watch the video!
In this video, Nick Lincoln of IBM provides an overview of Caliper, describing the project’s architecture and explaining how it works.
You can also read about other Hyperledger projects:
About the expert
