Dear fellow Kusama community,

We come to you with a humble but urgent request and need your help.

TL;DR:

The Kintsugi parachain stopped producing blocks at block 1,983,993 [2022-12-01 09:32:18 (+UTC)] due to a Kintsugi-specific issue with the slot duration in Aura
A fix has been found, tested, and reviewed with the help of Parity
No user funds are affected as of this writing
The only way to recover the Kintsugi chain is to update the runtime on the relay-chain via Kusama governance and upgrading all Kintsugi collators

We are asking the Kusama community to vote on recovering the Kintsugi parachain as soon as possible.

This is a technical emergency and we are hence requesting the whitelist track for this proposal, to accelerate recovery of Kintsugi.

Summary

The Kintsugi parachain has stopped producing blocks after a runtime upgrade. The issue is with the slot duration migration and is specific to the Kintsugi chain configuration. No other parachains are affected. See detailed description below.

The only way to recover the chain is via a Kusama governance proposal.

No user funds are affected as of this writing.

**Current status of the network: **All operations are halted. New kBTC cannot be issued and kBTC cannot be redeemed. KINT, kBTC, KSM, LKSM, and any other currency on Kintsugi cannot be moved. There are no KINT rewards for either stakers or vaults. Collateral backing BTC is still locked but collateral thresholds are not enforced at the moment.

Vaults are overall very healthy (https://kintsugi.interlay.io/dashboard/vaults) and no liquidations are expected at this time. There were no pending issues or redeem requests at the time of halting.

Until Kintsugi is back online and producing blocks:

All exchanges have been advised to halt withdrawals
All XCM-connected parachains have been advised to halt XCM channels

Technical Details of the Issue

Problem

The issue was caused by a problem with the slot duration. Specifically, there is a panic when a collator tries to produce a block:

[Parachain] panicked at 'Timestamp slot must match `CurrentSlot`'

Initially, Kintsugi was set at genesis to have a slot duration of 6 seconds but parachains typically have 12 second block times. The other chains built from the interBTC code-base (Interlay, and Interlay and Kintsugi testnets) used 12 seconds. A temporary fix was included in the Kintsugi collators to ensure that block production would succeed even with the 6 second blocktimes. The Kintsugi team did not run a migration as suggested by Parity so the temporary fix remained in place. The missing migration did not cause issues for several runtime upgrades.

However, with the 1.20.0 runtime upgrade we included a recommended change to prevent malicious block producers from potentially building a block. Due to the missing migration, a check in the Aura hook panicked due to an incorrect slot duration calculation which prevented Kintsugi from producing blocks.

Collators are not able to produce new blocks as the relay-chain validation would reject any new blocks with the updated runtime. Unfortunately, this issue was not caught during the release process, code reviews, and testnet upgrades. Also, this issue would not have been caught by the try_runtime testing approach as this issue is specific to collator producing blocks and the runtime migrations as such were not an issue. The issue is specific to the Kintsugi configuration and was hence not caught by the testing process on testnet. We are not aware of any other parachains being affected by this.

A detailed post-mortem will be provided shortly and linked to this proposal.

Solution

There are two parts to solve this issue:

Updating the Kintsugi runtime on Kusama: Collators are unable to produce blocks. Even by using codeSubstitutes Kintsugi cannot recover alone due to the bug described above. Hence, a new wasm needs to be set via Kusama governance on the relay-chain.
Updating all collators on Kintsugi to use the patched wasm: Updating the Kintsugi runtime on Kusama needs to be executed together with providing all collators in the Kintsugi network a patched wasm to be used via the codeSubstitutes approach.

Preimage:

Para: 2092
newCode: https://github.com/interlay/interbtc/releases/download/1.20.0-hotfix/kintsugi_runtime_parachain.compact.compressed.wasm
and

References

Kintsugi, a parachain on the Kusama network, has stopped producing blocks due to a technical issue with the slot duration in Aura. No user funds are affected, but the only way to recover the chain is via a Kusama governance proposal. The Kusama community is being asked to vote on recovering the Kintsugi parachain as soon as possible. All operations are halted, and all exchanges and XCM-connected parachains have been advised to halt withdrawals and channels. The issue was caused by a problem with the slot duration, and a new wasm needs to be set via Kusama governance on the relay-chain, along with providing all collators in the Kintsugi network a patched wasm to be used via the codeSubstitutes approach.