Move finalized state database and filesystem operations to blocking threads to avoid hangs

[teor2345]

Previous Work

We tried to fix this in PR #4199.

It worked, but it didn't help performance much, and the extra threading caused some fragile tests to intermittently fail.

Scheduling

We should only to this ticket if we fix the other hang bugs, and Zebra still hangs frequently.

Is your feature request related to a problem? Please describe.

Zebra runs finalized state database and filesystem operations on async threads, which can block:

• async code running concurrently in the same task, and
• async code from other tasks on the same thread.

Describe the solution you'd like

Instead, Zebra should:

• use tokio's spawn_blocking function to run these tasks, and
• make all callers into async functions (or access the executor directly)

https://docs.rs/tokio/1.6.0/tokio/task/fn.spawn_blocking.html

Describe alternatives you've considered

We can use tokio's block_in_place function, and move test code into a tokio::test. This function only blocks concurrent code in the same task. This makes it suitable for initialization or finalization tasks, because we're not doing much concurrent work at that time.

We could also use block_in_place as we transition code.

Additional context

Blocking functions might be the source of hangs or slowdowns in the inbound service, or state service, or unrelated tasks.

[teor2345]

This doesn't seem to cause bugs in practice, but it could cause:

• Finalizing blocks appears to be slow when checkpointing large blocks #3438