on_idle hook

[xlc]

Depends on the type of the chain, it usually have peak and off-peak hours. It will be good if there is a way to run some maintenance task on off-peak hours, for example, claim contract states, actively and progressively migrate storage format, doing expensive but not urgent calculation like calculate the winners of an election.

When block producer finished packing all the transactions and still have lots free remaining block weights (with #4058, it is possible to have a good estimation of the free amount before execute on_finalize hooks), it can call the on_idle(block_number: BlockNumber, free_weight: Weight) -> Weight to perform additional non-urgent work. This hook can return a number to indicate how much weight it used, and allow other modules to consume the unused part.

Chain operator can always dispatch an operational transaction to force perform the maintenance task if the the chain is too busy and never calls on_idle hook. Or on_initalize can do the maintenance task if it notices on_idle was never called in last N blocks.

[kianenigma]

Just an overall note that at the moment we do not expose the weight to the client (aka. block producer) as you mentioned. The block producer (basic-authorship) just gets informed once the block weight is full. I theory it should be super simple to expose a runtime API for this.

[xlc]

We really want this feature to allow us to leverage the "free" weights in the block. It could help #6594 as well.

[gui1117]

Sounds good idea, though I think the function should return the weight it consumes ahead of time so that we don't go beyond block weight.

Also transaction-payment adjust fee depending on how much full the blocks are, this logic might be affected.

[kianenigma]

@coriolinus I reckon this is quite similar to what we just talked about today, neh? If implemented, the parachain can also use the same mechanism (Or basically the part that reports block weight to the proposer).

[bkchr]

@coriolinus I reckon this is quite similar to what we just talked about today, neh? If implemented, the parachain can also use the same mechanism (Or basically the part that reports block weight to the proposer).

No. The parachain stuff is not connected in any way to this. Parachain candidates etc are passed as inherent to the relay chain. Inherents are before transactions.

[bkchr]

This means a transaction can not use the weight of an inherent.

Besides that, the proposer will never be aware of the weight or similar stuff. Weight is FRAME specific and other runtimes could use anything.

[xlc]

Good point about Weight is FRAME specific. Can we modify the proposer so an arbitrary data is passed around? And FRAME can interpreted it as weight.

[bkchr]

Data passed from where to where? Can you please explain more what you want to do?

[xlc]

Here is my proposal:

• Type
  • type OpaqueBlockContext = Vec<u8>
  • struct BlockContext(Weight)
• Runtime API:
  • sp_api::Core
    • fn initialize_block(header: &<Block as BlockT>::Header) -> BlockContext
      • or add a new fn new_block_context() -> BlockContext to reduce breaking change
  • sp_block_builder::BlockBuilder
    • fn apply_extrinsic(extrinsic: <Block as BlockT>::Extrinsic, context: BlockContext) -> (ApplyExtrinsicResult, BlockContext)
      • take remaining weight, return new remaining weight
    • fn finalize_block(context: BlockContext) -> <Block as BlockT>::Header
      • call pallet on_idle with remaning weight and then call pallet on_finalize
• sc-block-builder::BlockBuilder:
  • New field context: OpaqueBlockContext
  • context will be created via initialize_block or new_block_context
  • in fn push, pass the context and updates it using the return result from apply_extrinsic
  • in fn build, pass the context to finalize_block

[gui1117]

why should this involve client ? isn't it possible to do everything inside the runtime ?

how I would see it is that just before on_finalize hooks are run we look at the weight and execute some on_idle until reaching the full block.

[bkchr]

why should this involve client ? isn't it possible to do everything inside the runtime ?

how I would see it is that just before on_finalize hooks are run we look at the weight and execute some on_idle until reaching the full block.

I see this exactly in the same way.

[xlc]

Oh the block producer discussion got me confused. Yes we do have everything available in the runtime so we can just update the finalize_block on executive and handle the free weights there.

So we can just update the macro to support on_idle and call it in executive. I guess we may want to wait for #6877?

[gui1117]

adding a hook is not difficult in #6877 thus if you want to make a PR I can "backport" to attribute macro

[athei]

This would be super helpful for #6594 as @xlc already mentioned. I could use the excess weight to do some more storage removal without taking away too much weight from actual extrinsics during busy times.

[apopiak]

This could also be useful for the scheduler, no?
It would be very cool to schedule something and then have it run directly in on_idle if the block isn't full, yet, or have it run in a future block in case the current block is too full.

[apopiak]

@JesseAbram working on this here: #8209