Dear fellow Kusama community,
We come to you with a humble but urgent request and need your help.
We are asking the Kusama community to vote on recovering the Kintsugi parachain as soon as possible.
This is a technical emergency and we are hence requesting the whitelist track for this proposal, to accelerate recovery of Kintsugi.
The Kintsugi parachain has stopped producing blocks after a runtime upgrade. The issue is with the slot duration migration and is specific to the Kintsugi chain configuration. No other parachains are affected. See detailed description below.
The only way to recover the chain is via a Kusama governance proposal.
No user funds are affected as of this writing.
**Current status of the network: **All operations are halted. New kBTC cannot be issued and kBTC cannot be redeemed. KINT, kBTC, KSM, LKSM, and any other currency on Kintsugi cannot be moved. There are no KINT rewards for either stakers or vaults. Collateral backing BTC is still locked but collateral thresholds are not enforced at the moment.
Vaults are overall very healthy (https://kintsugi.interlay.io/dashboard/vaults) and no liquidations are expected at this time. There were no pending issues or redeem requests at the time of halting.
Until Kintsugi is back online and producing blocks:
The issue was caused by a problem with the slot duration. Specifically, there is a panic when a collator tries to produce a block:
[Parachain] panicked at 'Timestamp slot must match `CurrentSlot`'
Initially, Kintsugi was set at genesis to have a slot duration of 6 seconds but parachains typically have 12 second block times. The other chains built from the interBTC code-base (Interlay, and Interlay and Kintsugi testnets) used 12 seconds. A temporary fix was included in the Kintsugi collators to ensure that block production would succeed even with the 6 second blocktimes. The Kintsugi team did not run a migration as suggested by Parity so the temporary fix remained in place. The missing migration did not cause issues for several runtime upgrades.
However, with the 1.20.0 runtime upgrade we included a recommended change to prevent malicious block producers from potentially building a block. Due to the missing migration, a check in the Aura hook panicked due to an incorrect slot duration calculation which prevented Kintsugi from producing blocks.
Collators are not able to produce new blocks as the relay-chain validation would reject any new blocks with the updated runtime. Unfortunately, this issue was not caught during the release process, code reviews, and testnet upgrades. Also, this issue would not have been caught by the try_runtime testing approach as this issue is specific to collator producing blocks and the runtime migrations as such were not an issue. The issue is specific to the Kintsugi configuration and was hence not caught by the testing process on testnet. We are not aware of any other parachains being affected by this.
A detailed post-mortem will be provided shortly and linked to this proposal.
There are two parts to solve this issue:
codeSubstitutes Kintsugi cannot recover alone due to the bug described above. Hence, a new wasm needs to be set via Kusama governance on the relay-chain.codeSubstitutes approach.Threshold