On 25 November, the Robonomics parachain on Kusama stopped producing blocks following a major runtime and node upgrade to v4.0.
This upgrade was intended to transition the parachain consensus from OpenCollator to AURA.
During the upgrade, temporary AURA authorities were expected to be set in the OnRuntimeMigration block.
Although the runtime upgrade executed, collators running OpenConsensus were no longer able to author blocks, while the new AURA authority set was not yet fully established. As a result, the chain halted mid-migration, before the new authorities could take over.
All runtime testing was previously performed using Chopsticks, where this issue did not appear due to differences in the execution environment.
The block production halt affects several Robonomics services:
While this proposal is being voted on, the Robonomics team will prepare an updated runtime WASM that resolves the underlying issue (see: https://github.com/paritytech/polkadot-sdk/issues/10428).
Unlocking the parachain will allow the manager account to perform a schedule_code_upgrade, enabling:
This will restore block production and return the parachain to normal operation.
We respectfully ask Kusama Governance and the Polkadot Fellowship to approve unlocking the Robonomics parachain so that full functionality can be restored.
After resolution, we will publish a public postmortem and implement improved migration testing procedures (including Zombienet) to prevent similar incidents in the future.
On 25 November, the Robonomics parachain on Kusama stopped producing blocks following a major runtime and node upgrade to v4.0.
This upgrade was intended to transition the parachain consensus from OpenCollator to AURA.
During the upgrade, temporary AURA authorities were expected to be set in the OnRuntimeMigration block.
Although the runtime upgrade executed, collators running OpenConsensus were no longer able to author blocks, while the new AURA authority set was not yet fully established. As a result, the chain halted mid-migration, before the new authorities could take over.
All runtime testing was previously performed using Chopsticks, where this issue did not appear due to differences in the execution environment.
The block production halt affects several Robonomics services:
While this proposal is being voted on, the Robonomics team will prepare an updated runtime WASM that resolves the underlying issue (see: https://github.com/paritytech/polkadot-sdk/issues/10428).
Unlocking the parachain will allow the manager account to perform a schedule_code_upgrade, enabling:
This will restore block production and return the parachain to normal operation.
We respectfully ask Kusama Governance and the Polkadot Fellowship to approve unlocking the Robonomics parachain so that full functionality can be restored.
After resolution, we will publish a public postmortem and implement improved migration testing procedures (including Zombienet) to prevent similar incidents in the future.
Threshold
Threshold