[Features] Reintroduce sub-du #12439
Conversation
|
So this loops through the pallet+storage items and calculates their keys? I also vaguely remember a TS project which could be web-deployed, but cant find it now… |
This only looks at the storage items that are in the metadata, so yeah any custom key is missed, and probably fine. Exposing |
|
bot rebase |
|
Rebased |
Perhaps a dumb question but what is the purpose of having this on the internet, as opposed to have people run a local binary? Just as a community utility thing? |
Yeah, just a remote api that would conveniently show latest storage of a given network. Possibly with some sort of dumb cache to make sure multiple requests within a short period of time are handled without issues. |
|
bot rebase |
|
Rebased |
|
bot rebase |
|
Rebased |
|
Last time I tried it it would not connect because of some RPC error. |
utils/frame/sub-du/src/main.rs
Outdated
| .expect("Runtime Metadata failed to decode"); | ||
| let metadata = prefixed_metadata.1; | ||
|
|
||
| if let RuntimeMetadata::V12(inner) = metadata { |
There was a problem hiding this comment.
support for metadata v13 and v14? :)
There was a problem hiding this comment.
yeah, that's the plan. I have not touched it much since reintroduction, on it now.
There was a problem hiding this comment.
@niklasad1 do we want to support both v13 and v14? I guess v12 Is probably not needed anymore.
There was a problem hiding this comment.
I guess I'll start with v14
There was a problem hiding this comment.
Sounds good to start with v14, gotcha I was just surprised when I spotted v12 :)
No sure might be nice to query on really old blocks to check how it grows over time not sure how use it will be.
There was a problem hiding this comment.
We should at first only support v14. no need to support the old versions initially. If you can do it, sure, but otherwise no need.
Not yet, getting to it now |
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
|
bot rebase |
|
Rebased |
| @@ -0,0 +1 @@ | |||
| # sub-du | |||
There was a problem hiding this comment.
delete this file please.
There was a problem hiding this comment.
Maybe it's best to actually have a proper README?
There was a problem hiding this comment.
If you are keen on writing some readme please write it as module doc (//!) it will be more worthwhile and our CI will create the READM from it.
| } | ||
|
|
||
| #[derive(Debug, Copy, Clone)] | ||
| pub enum StorageItem { |
There was a problem hiding this comment.
Yeah, it hasn't been touched yet, WIP
There was a problem hiding this comment.
It's interesting, the StorageEntryType actually contains two types now Map and Plain.
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
|
@kianenigma I have removed sub-storage. In the current implementation I am using wrappers done in remote-externalities builder to achieve faster results. Once all of this works - we can discuss having those helpers someplace else shared by the two. But actually making a few methods public might not hurt remote-externalities much. This is still a WIP, any suggestions welcome. |
I should first review the code, which I will do asap. |
|
|
||
| println!("Scraping at block {:?} of {}({})", at, runtime.spec_name, runtime.spec_version,); | ||
|
|
||
| let raw_metadata = rpc_client.metadata(at).await.unwrap(); |
There was a problem hiding this comment.
where is this fn metadata coming from? 🤔
There was a problem hiding this comment.
it's from the substrate_rpc_client::StateApi i.e, https://github.com/paritytech/substrate/blob/master/client/rpc-api/src/state/mod.rs#L79
There was a problem hiding this comment.
I see, then using the RPC stuff directly is almost an identical code, you just use this client:
There was a problem hiding this comment.
You can also consider parallelizing this like my #12537
There was a problem hiding this comment.
I see, then using the RPC stuff directly is almost an identical code, you just use this client:
Makes sense, indeed remote_externalities use seems a bit hacky, especially if it does not yield all that many benefits.
| cfg.transport.map_uri().await.unwrap(); | ||
|
|
||
| // connect to a node. | ||
| let ext = remote_externalities::Builder::<Block>::new().mode(Mode::Online(cfg)); |
There was a problem hiding this comment.
yeah, I agree with Kian that we shouldn't use remote_externalities here instead use
let rpc_client = substrate_rpc_client::ws_client(opt.uri).await.unwrap()
Then we don't need to have this sp_testing::Block which isn't really nice
There was a problem hiding this comment.
Yeah, the sp_testing::Block did not feel like an amazing approach. It's also kinda cryptic.
| let mut modules: Vec<Pallet> = vec![]; | ||
|
|
||
| // potentially replace head with the given hash | ||
| let head = ext.rpc_get_head().await.unwrap(); |
There was a problem hiding this comment.
right, you need to add some ugly stuff here:
let head = ChainApi::<(), _, Header, ()>::finalized_head(&rpc_client).await.unwrap();|
@niklasad1 it might not be bad idea if you entirely take this over, as Roman is busy with other stuff (that's way more high priority). |
|
Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions. |
Yeah, not possible for me to proceed with this atm. But I'll get back to this if it's not done by the time I'm finished with the other stuff. |
|
@harrysolovay would the idea behind this be a good tool to build with CAPI? |
|
This is absolutely something that could be implemented as a Capi pattern lib. Let's try it out & follow up here. |
Resolves paritytech/polkadot-sdk#449
At the moment this is just a copy-paste with updated dependencies.
state.polkadot.networkor something like that.