Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use blocking threadpool for all non-trivial message processing #1493

Closed
paulhauner opened this issue Aug 9, 2020 · 1 comment
Closed

Use blocking threadpool for all non-trivial message processing #1493

paulhauner opened this issue Aug 9, 2020 · 1 comment
Labels
bug Something isn't working

Comments

@paulhauner
Copy link
Member

Description

When we receive one of the core a block, attestation, exit messages there's a chance that we might need to access a historical state. Accessing these states can take seconds to load, even 10s of seconds if the system/db is under load.

AIUI, we are not using Tokio's blocking threadpool to process these messages, instead they're just blocking the main pool. This will cause network messages, API calls, etc. to be delayed.

I've seen some attestation processing scenarios take 30 seconds! If we were to process two of these messages on a dual-core CPU, then we'd stop responding to messages for 30s. This would potentially cause us to loose peers and perhaps stop following head.

Steps to resolve

Ensure all non-trivial consensus message processing tasks are spawned on the blocking executor.

@paulhauner paulhauner added the bug Something isn't working label Aug 9, 2020
@paulhauner
Copy link
Member Author

Resolved in #1546, #1518, #1523

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant