Unlock para 2281 (Kreivo)

Whitelisted Caller

5mos ago

6 Comments

Executed

Content

AI Summary

The Incident

On Saturday, June 29, Kreivo stopped producing blocks after a runtime upgrade to version 0.13.0. This upgrade aimed to introduce pallet contracts, enable bridged DOT reception from Kusama AssetHub, and update our PolkadotSDK dependency from version 1.5 to 1.12. The main goal was to prepare Kreivo for async backing, which would reduce block times from 12 to 6 seconds.

The Issue

Despite thorough testing before the upgrade, the problem only appeared hours after enactment(at the end of the era). The current rapid development of the PolkadotSDK makes it challenging for small teams like ours to constantly update and migrate code. Our main issue stemmed from the significant version jump (1.5 to 1.12), likely missing crucial intermediate migrations.

Impact

While most of our users (DAOs) are aware of Kreivo's beta testing stage and aren't critically affected, two key exceptions are:

Virto team: Our salaries are paid through the DAO's governance, which is currently inaccessible.
KSM Community: This decentralized voice in Kusama can't vote on OpenGov proposals.

Proposed Solution

While this proposal is being voted on, we'll prepare an updated WASM of the runtime that fixes the issue. Unlocking Kreivo means the "manager account" will be able to do a schedule_code_upgrade to manually set the updated parachain's code in the relay chain to match the code manually set by collators with the codeSubstitute feature of the chainspec.

Request

We ask Kusama governance and the Fellowship to approve unlocking the Kreivo parachain so we can restore functionality. After resolution, we'll conduct a public postmortem and implement additional safeguards for the future.

We are glad this kind of issue occurred this early in the lifetime of the chain when impact is manageable and are now more convinced builders need simpler ways to manage this kind of outages, ironically I believe a "governance parachain" like Kreivo where parachains can host their governance can be a great solution to this issue as people can vote to unlock a parachain even if it has halted.

Edited

Status

Decision14d

Confirmation10mins

Attempts

Tally

100%Aye

61.1%Threshold

0%Nay

Aye

≈298.45KKSM

Nay

0KSM

0.0%
0.0%
0.0%

Threshold

Support(5.97%)

≈889.2KKSM

Issuance

≈14.9MKSM

Votes

Nested

Flattened

Calls

How OpenGov works↗

Call

Metadata

Timeline6

Votes Bubble

Statistics

Comments

DG8Q...5nFh

Hello, on behalf of the AIWeb3 DAO (the most vibrant Chinese community within the Polkadot ecosystem, with the mission to amplify the voice of the Chinese-speaking community and support Chinese content creators, ensuring it plays a pivotal role in shaping the future of Polkadot), I would like to invite you to join our weekly Twitter Space and discuss your proposal since we are not familiar with the background. Our Twitter Space is held every Tuesday at 10 AM (UTC+8) on our official Twitter: https://x.com/aiweb3dao. Please join our Telegram group if you have any questions, English TG group: https://t.me/aiweb3dao_eng and Chinese TG group: https://t.me/aiweb3dao
We will make our final decision based on the discussion and input from community members in AIWeb3.

您好，我代表AIWeb3 DAO，这个波卡生态中最具活力的中文社区，旨在放大中文社区的声音，支持中文内容创作者，确保其在塑造波卡未来中发挥重要作用。我们诚挚邀请您参加我们的每周Twitter空间并讨论您的提案。我们的Twitter空间是每周二上午10点（UTC+8），在我们的官方推特：https://x.com/aiweb3dao. 。如有任何问题，请加入我们的Telegram讨论：英文Telegram https://t.me/aiweb3dao_eng ，中文 TG group: https://t.me/aiweb3dao。我们将根据讨论和AIWeb3社区成员的意见做出最终决定。

Twitter space link: https://x.com/aiweb3dao/status/1807637848713257059

EvoL...oVus

For extra context, the team fixed the issue simply by further updating the dependency of PolkadotSDK to version 1.13. Kreivo uses a fork of PolkadotSDK where we introduce governance related features that might make it to the SDK some day(like the editable tracks that enable creating DAOs dynamically), version 1.12 was faulty and the bug fix that was backported wasn't included in the moment we made the fork.

HUf7...WXD4

Hey everyone! We have a post-mortem document explaining what (technically) happened in this incident, in hopes other people can learn from our experience. 😄

DG8Q...5nFh

经过AIWeb3 的Twitter Space讨论和社区反馈意见，AIWeb3社区决定赞成该提案。以下是社区的主要意见：
快速修复严重bug。但是应该思考减少此问题的出现的概率：1完善相关文档 2上线前的测试 2上游流程的优化，或者出问题后的成熟流程等
支持。项目在升级的过程中出现了bug，必须ksm治理授权才能继续更新软件以及通过dao发工资。风险上看，对项目的利益相关者有关，对ksm链无关。
看起来只是影响这条链的运行，支持他们能快速修复bug，同时也希望他们能够分享之后的一些经验，让其他的开发者能够吸取经验教训

我们每周二北京时间早上10点会在AIWeb3 Twitter Space上讨论Kusama的提案，欢迎大家参与讨论。同时，也欢迎大家直接加入我们的Telegram询问任何问题。如果我们对任何提案表达了赞成，AIWeb3社区也会提供中文社区建设的服务和合作，愿意帮项目方一起建设中文社区，希望波卡生态有更多成功的项目。如果被我们拒绝，也不要灰心，我们欢迎申请人代表来我们的推特space分享提案的亮点，同时也请记住，有很多其他的Kusama DV可能会支持你们的提案。

AIWeb3 Community Feedback on Proposal
After discussions on AIWeb3's Twitter Space and feedback from the community, the AIWeb3 community has decided to support the proposal. Below are the main points from the community:

Quickly fix serious bugs. However, we should consider reducing the likelihood of such issues occurring: 1) Improve relevant documentation, 2) Conduct thorough testing before going live, 3) Optimize upstream processes, or establish mature processes for handling issues when they arise.

Support. Bugs have appeared during the project's upgrade process, and authorization through KSM governance is required to continue updating the software and paying salaries through the DAO. From a risk perspective, it affects the project's stakeholders but is irrelevant to the KSM chain.

It seems to only affect the operation of this chain. We support their quick bug fix and also hope they can share their experiences afterward so that other developers can learn from it.

If your proposal is rejected, don't be discouraged. We welcome proposers to join our Twitter Space to share the highlights of their proposals. There are also many other Kusama DV who might support your proposal.