Treasury proposal: KAGOME – the C++ Implementation of Polkadot Host

KAGOME proposal: link to the full proposal

KAGOME is one of the most advanced alternative Polkadot client implementations in the Polkadot ecosystem:

• Written using modern C++
• All code is available on github repo under Apache 2.0 license
• A fully synced Polkadot node can be found in telemetry (navigate to Polkadot Telemetry and type KAGOME)
• Polkadot.JS compatibility (Feel free to open Polkadot JS interface to interact with our node)
• Possibility to run as a validator (without parachain support)
• Implements the Polkadot Host Specification
• Web3 Foundation grant recipient

Our team intends to further support KAGOME by completing its integration in the Polkadot ecosystem and allow node operators to run KAGOME as validating nodes. Moreover, Web3 Foundation and its protocol specification team is willing to support the KAGOME project by helping enhance protocol specification taking into account feedback that will be provided by the KAGOME team on a regular basis.

This proposal describes new features to be implemented as part of a KUSAMA treasury grant.

Proposed feature set:

Parachains integration

Parachains are an essential part of the Polkadot ecosystem. Supporting parachains is what makes Polkadot different from other standalone Substrate-based networks. In order to support parachains, the following parts should be implemented; (following the Parachain Implementation Guide: )

• Collation protocol
• Backing protocol
• Availability and validity protocol
• Dispute resolution

Warp sync

Currently, in order to spin up a node with full chain synchronization, it is necessary to apply every block and reproduce its state or use a fast sync protocol that essentially downloads all the headers from the blockchain and block justifications containing authority changes and its most recent state.
However, there is a third approach: using Warp sync. Warp sync only downloads the headers and justifications of blocks where an authority change is happening, which would allow node runners to trustlessly spin up the node within 10-20 minutes. Also, this is a great foundation for future light client implementations based on KAGOME.

Light client protocol (incoming requests)

In order to send merkle proofs to light clients, KAGOME needs to support the /dot/light/2 substream protocol. As the Polkadot ecosystem has light clients like Smoldot that are capable of running in the browser, support for WebSockets into cpp-libp2p (that is the networking layer of KAGOME) is needed.

Dynamic pruning

Currently KAGOME can only be executed in archive mode; where every state is stored in a Merkle state trie. While this is good for querying the balance of an account at any particular block it consumes a lot of disk space (about 500GB for 11 million blocks in Polkadot).

To make running in archive mode optional, it is necessary to introduce dynamic pruning that would only keep the states of recent N blocks, where N could be set using --pruning <arg> flag.

Lean BEEFY proofs integration

In order to run our validator in the Rococo network, it is necessary to support BEEFY proofs. As BEEFY is currently under development, the Rococo network uses Lean BEEFY.

Having lean BEEFY integrated would allow KAGOME to participate in BEEFY consensus, which is useful for external bridges.

Storage optimization

KAGOME uses RocksDB as the database level for storing blockchain data such as block headers, bodies, justifications, and state. It is needed to logically divide these datasets into their own column families to give room for further optimizations.

Multithreading

Currently, KAGOME utilizes one thread for networking operations and block creation/execution. Additionally, there is a thread pool that creates a new thread for every web socket connection. To optimize it further, it is necessary to introduce separate threads for networking operations and block execution/creation procedures.

General maintenance

Runtime benchmarking

In order to optimize KAGOME’s block creation speed, runtime benchmarks are needed to simplify different WASM engines (Binaryen, WAVM, WASMEdge) testing.

Catching up with Polkadot/Substrate development

It is necessary to implement new RPCs and Host APIs whenever they are introduced and utilized in Polkadot/Substrate, so that KAGOME does not end up in a situation where new blocks cannot be executed due to a lack of functionality or compatibility with PolkadotJS apps.

Libp2p maintenance

KAGOME is the main user of the cpp-libp2p library maintained by Soramitsu. Library needs to be updated whenever any issues are recognized

DevOps support

In order to ensure that the KAGOME node is operational whenever new changes are merged to the master code, we are providing continuous integration on the several AWS environments which require constant maintenance by DevOps team

Submission details

All details are available in the full proposal.

The amount requested for the first submission is $565000.
The amount in KSM is 17953 taking into account KSM EMA (7) rate provided by Subscan service KSM prices (USD) for November, 10th

The amount will be recalculated according to the 7 day average rate, using the high and low of the week on Coingecko, at the day of on-chain action after the discussion process ends.

The 1st milestone includes:

• Parachain support
• Warp sync
• Light client protocol (for incoming requests)
• Dynamic pruning
• Lean BEEFY integration
• Storage optimization
• Multithreading
• General maintenance

Links and references

1. https://soramitsu.co.jp/
2. https://github.com/soramitsu/kagome
3. https://github.com/libp2p/cpp-libp2p
4. Web3 builders: Soramitsu
5. KAGOME: C++ implementation of PRE presentation at DOTCon
6. Web3 Builders: Soramitsu | C++ Implementation of Polkadot Host crowdcast series recording
7. Polkadot JS running on top of KAGOME