try-runtime: dynamic storage query sizes

[liamaharon]

Closes #13246
polkadot companion: paritytech/polkadot#7111

Changes

• Created get_storage_data_dynamic_batch_size, allowing users to efficiently process a large amount of jsonrpc payloads without knowing the node max payload size configuration. The function calls itself recursively, increasing batch_size when the node responds successfully and reducing it when the node returns an error.

  async fn get_storage_data_dynamic_batch_size(
  	client: &Arc<HttpClient>,
  	payloads: Vec<(String, ArrayParams)>,
  	batch_size: usize,
  ) -> Result<Vec<Option<StorageData>>, String>

• Implemented get_storage_data_dynamic_batch_size for fetching storage keys.
• Switched to using a Http provider for these huge requests. Using a WsClient I continuously encountered unhandleable errors from within the jsonrpsee library such as "961d622f: accumulated message length exceeds maximum" and the node would also randomly refuse to continue servicing WsClient, returning incorrectly formatted errors (missing request IDs) that were also causing more internal errors inside the jsonrpsee library. I'm not sure whether this issue is in the jsonrpsee library or the way the Substrate RPC is configured or worth this is worth opening an issue to investigate, let me know if I should.
• Removed parallelisation from rpc_child_get_keys. It runs extremely fast on just one thread (2 seconds on a Macbook Pro), and the logic to run it across multiple threads was quite complex and causing some difficult to debug issues. I've benchmarked this to confirm negligible performance impact, I do not think parallelising this piece of code is worth it at this time.

Notes

• I compared the speed of loading storage from a local node against master. Performance is identical (+/- <2% between runs)
• I tested this works fetching live state from wss://rpc.polkadot.io:443. it took 76min on my home internet, upload speed was be far the biggest bottleneck.

[liamaharon]

Switching this to a draft and implementing a dynamic batch size as described here #13246

[liamaharon]

Looks good on the whole. Could you please also test the difference with the parity nodes? https://gist.github.com/ggwpez/81db110fe4390ed9a7622f5857dfc4ff I dont expect there to be any - besides it being more fault tolerant.

Confirmed these are working

[liamaharon]

Hey @niklasad1, the polkadot staking-miner rpc expects that remote-externalities uses a WsClient: https://github.com/paritytech/polkadot/blob/5fd2bf8c9a0665b361eb26823030a5e0e65459b4/utils/staking-miner/src/rpc.rs#L108

Do you suggest that I refactor the staking-miner and open a companion PR, or try to modify this PR so that remote-externalities supports either a WsClient or HttpClient?

[niklasad1]

Hey @niklasad1, the polkadot staking-miner rpc expects that remote-externalities uses a WsClient: https://github.com/paritytech/polkadot/blob/5fd2bf8c9a0665b361eb26823030a5e0e65459b4/utils/staking-miner/src/rpc.rs#L108

I think it's fine to use the HTTP client just for the "batch requests/remote externalities" just change https://github.com/paritytech/polkadot/blob/master/utils/staking-miner/src/main.rs#L315-#L323 to take an URL instead (in case it would be a ws:// but you already added converting ws to http URL under the hood)

[liamaharon]

bot merge

[paritytech-processbot]

Waiting for commit status.

[liamaharon]

bot merge force