Throttle staging deals on the lotus-miner markets service and reject new deals, until staged deals are added to sectors.

[nonsense]

Checklist

• This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
• I have a specific, actionable, and well motivated improvement to propose.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Improvement Suggestion

Currently Lotus has no way to throttle incoming storage deals. This results in the deal-staging directory in the markets service repo to become increasingly large without any limits, which results in disk space getting filled to 100% and essentially halting the miner/fullnode.

For example on a small miner configuration, where the sealing area and the full node are hosted on a 2TB SSD, there is room only for a few deals and 1-2 sectors being sealed. However the markets subsystem is accepting and storing/staging too many deals successfully, which fills the SSD very quickly.

Ideally we need a mechanism to specify that we want no more than N deals or M GBs to be in the markets node staging area.

[jennijuju]

is it possible to do a dt rate limiting configuration by size (like byte/sec) based on network speed + ongoing traffic?

[nonsense]

@jennijuju we considered doing this in Lotus, but decided that it is probably best handled by external to Lotus tools, such as tc and other connection traffic shaping tools.

[nonsense]

@jennijuju I think we should close this issue, because it was created for the stopgap tasks of rejecting deals when the staging are is full.

Designing a better backpressure mechanism, and propagating information from the sealing pipeline to the markets process in order for it to decide better how and when to reject deals is certainly going to be more involved, so this is not something that is feasible in a few days.

[jennijuju]

Designing a better backpressure mechanism, and propagating information from the sealing pipeline to the markets process in order for it to decide better how and when to reject deals is certainly going to be more involved,

Agree - do we have a ticket for this? If not, can we create one before closing this?

[jennijuju]

won't fix in lotus - this should be considered in Boost.