Investigate block fill when mempool is full

[kantai]

For the past few weeks, there's consistently been high-congestion on the Stacks network -- many transactions are in the mempool, but there's still often Stacks block with ~10 transactions (as well as blocks with 500-2000 transactions: blocks like these are the goal).

Why is this happening?

One hypothesis was that miner's block assembly is not getting the "second attempt" out quickly enough to be included in the bitcoin sortition. With some of the data provided by a miner in discord, that doesn't seem to be the case:

INFO [1642725692.170701] [testnet/stacks-node/src/neon_node.rs:1988] [relayer] Succeeded assembling Stacks block #45874: 46dccd751f1e169da084e634b6e3bb5b395845a32c3772a0240b6a7a88c33f1a, with 4 txs, attempt 2
INFO [1642725774.741724] [testnet/stacks-node/src/neon_node.rs:1988] [relayer] Succeeded assembling Stacks block #45874: 0248537de81939d7eec4efb97c220cf37695f21bf55ab249dd9e8e9d75efb39d, with 4 txs, attempt 2
INFO [1642726286.764345] [testnet/stacks-node/src/neon_node.rs:1988] [relayer] Succeeded assembling Stacks block #45875: 379194b10dc463756cf4b743964174abc3d8fd413f8ed264f19084752b273837, with 3 txs, attempt 2
INFO [1642726897.061920] [testnet/stacks-node/src/neon_node.rs:1988] [relayer] Succeeded assembling Stacks block #45875: 5b5542a6b333bec1b54cf778e3b3d2e6179f61438937e566406e7ba9261a6c08, with 2 txs, attempt 2

That's only assembling blocks with a handful of transactions even on the second assembly.

The next hypothesis is that block assembly just takes miners a long time. This makes some some logical sense: miners need to iterate over the mempool, evaluate transactions' nonces and account balances before execution, so block assembly actually requires more effort than block validation. There could be different ways to address this, but first we should confirm if this is the case. One way to do this is to use a mock-miner with subsequent_attempt_time_ms set to a large number, say two minutes or three minutes. With this, we should be able to see in the logs that the mock miner is assembling very large blocks while the mempool is full. If this is the case, then we know that is the problem. If the mock miner still assembles small blocks while the mempool is full, we know we need to look elsewhere.

[kantai]

Per @jcnelson, with the following settings:

[miner]
first_attempt_time_ms = 5000
subsequent_attempt_time_ms = 180000
microblock_attempt_time_ms = 30000

A miner would start to fill blocks more easily

$ fgrep 'block-commit' logs.out | grep -v "attempt: 1"
DEBG [1642804044.610182] [testnet/stacks-node/src/neon_node.rs:2084] [relayer] Submit block-commit, block_hash: d9ca4c85ffda2392ec6cb68c129d83785aba2f504cb2a443767e51d305281f04, tx_count: 395, target_height: 45970, parent_consensus_hash: 90d6a4fa065687506f6de1e467930741977c68ce, parent_block_hash: 97dd3a890053e7ceee7605cd5ceccfecc97d2f303a32d600c32e82d4b1806053, parent_microblock_hash: 41db61b58326f4c2376f35dd4cab6fbf39dce823872fbb520f7b998afcfde728, parent_microblock_seq: 0, tip_burn_block_hash: 000000000000000000079a51d76ab103e977fe0611a258a6cba6a5321544f3b2, tip_burn_block_height: 719820, tip_burn_block_sortition_id: 1e441ad163f3646a088c098e2845224005e9c8219fabc29ea8541323c4f5d955, attempt: 2
DEBG [1642804235.445799] [testnet/stacks-node/src/neon_node.rs:2084] [relayer] Submit block-commit, block_hash: 6e8f21de749bbc73ea3b27fbce6652685c5d5b0fa58dcf640c87e53bc92bff41, tx_count: 500, target_height: 45970, parent_consensus_hash: 90d6a4fa065687506f6de1e467930741977c68ce, parent_block_hash: 97dd3a890053e7ceee7605cd5ceccfecc97d2f303a32d600c32e82d4b1806053, parent_microblock_hash: 30b6aa425b052f1e81986f61a960c1455c5f62db850563484e2bc03f4ef5f0df, parent_microblock_seq: 1, tip_burn_block_hash: 000000000000000000079a51d76ab103e977fe0611a258a6cba6a5321544f3b2, tip_burn_block_height: 719820, tip_burn_block_sortition_id: 1e441ad163f3646a088c098e2845224005e9c8219fabc29ea8541323c4f5d955, attempt: 3

There's a couple ways we could make use of this information:

1. Change the default settings to 5000 and 180000
2. Introduce a new setting second_attempt_ms, with default settings like:

[miner]
first_attempt_time_ms = 5000
second_attempt_time_ms = 60000
subsequent_attempt_time_ms = 180000
microblock_attempt_time_ms = 30000

This makes it so that the first attempt is fast, in order to allow miners to continue to progress in flash blocks. The second attempt is slower, but still short enough to make it into most bitcoin blocks, and then subsequent attempts are allowed to take longer.

Any thoughts on the best path of those, @jcnelson ?

[gregorycoppola]

Maybe we can optimize the mining code to get more tx's in for a given timeout.

#3014

[jcnelson]

I think this is addressed as of #3185?

[obycode]

I also think this can be closed.

[blockstack-devops]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.