PVF execution currently has a memory limit of 128MB, which is the same as Cumulus uses for its block import pipeline. While collators execute state transitions by reading parachain data from disk locally, parachain validators must load state proofs into memory. Therefore, they should have a higher memory limit than collators. Otherwise a parachain block that made full use of memory when executed by the collator could fail to be executed by validators. To fix this, we propose increasing the memory limit for PVF execution to 512MB.
set_config_with_executor_params {}: set_executor_params(RawOrigin::Root,
ExecutorParams::from(&[ExecutorParam::MaxMemoryPages(8192)][..]))
hash: 0x1c356149b664ec3f7d89eeb56a3469bd5f468ae7fafa2b13f2423618db6c922e
calldata: 0x332e040100200000
Submit the preimage for the Fellowship referendum:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.polkadot.io#/extrinsics/decode/0x2000882c001c356149b664ec3f7d89eeb56a3469bd5f468ae7fafa2b13f2423618db6c922e
Open a Fellowship referendum to whitelist the call:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.polkadot.io#/extrinsics/decode/0x17002b0f024076336ce5c2429a0c9ebdc17667391af846008847a8dbc6164ebfc076f172d322000000010a000000
Submit the preimage for the public referendum:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.polkadot.io#/extrinsics/decode/0x2000282c03332e040100200000
Open a public referendum to dispatch the call:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.polkadot.io#/extrinsics/decode/0x15002b0d020ebe5ce9abcaf0e08e88086b99882b0f63862ee2c48f15f84c9c6658526a12d20a000000010a000000
SubSquare proposes to increase the memory limit for PVF execution to 512MB to prevent parachain blocks from failing to be executed by validators. They suggest using the set_executor_params function to set the new memory limit.
To make this change, they need to follow a process that involves submitting a preimage for a Fellowship referendum and opening a Fellowship referendum to whitelist the call. They also need to submit a preimage for a public referendum and open a public referendum to dispatch the call.
The proposed change will help ensure that parachain validators have enough memory to load state proofs and execute parachain blocks successfully.
Threshold
PVF is executed by a polkadot validator during backing, approvals and disputes. It's possible requests from those systems come in simultaneously. I don't know how many of those requests may come in concurrently, but it depends on the configuration. If I were to guess, then I would suppose there would be at most 3-4 PVF executing at the same time. Assuming those numbers, in the pathological case those PVF manage to fill the whole available memory at approx. the same time, it would make at most 512 MiB of committed memory with the current 128 MiB limit. With the new parameters the amount of memory consumed only by PVFs jumps up to 2 GiB.
So that makes me wonder if there were any testing done for what would happen with those pathological values on the standard hardware?