Occasional Failure of AddPiece to Initiate on Snapped-Up Sectors

[hyunmoon]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

lotus-miner version 1.23.3+mainnet+git.7bb1f98ac
boostd version 1.7.3+git.188d5f0

Repro Steps

1. Extend a CC sector to accommodate an upcoming storage deal.
2. Snap-up the freshly extended CC sector.
3. Right after, import a single offline storage deal.
4. Replicate steps 1-3 repeatedly (normally requiring over 30 iterations).
5. On some occasions, notice the imported deal fails to land in any sectors. The AddPiece would not start, and the deal persistently shows the 'PublishConfirmed' status without progress.

Describe the Bug

Recently, I began onboarding storage deals via snapping. My approach is:

1. Extend a CC sector to ensure an adequate duration (all sectors expire in less than 540 days).
2. Snap up the extended sector.
3. Import an offline storage deal.

Using an automated script, I've set this process to run continuously.

However, over time, I observed a gradual increase in the number of 'Available' sectors. When cross-referencing with the Boost UI, I discovered a corresponding rise in deals stuck in the 'PublishConfirmed' state. These deals are marked with the 'Adding to Sector' status, which indicates Boost's attempt to transfer the deal to the lotus-miner sealing subsystem.

Interestingly, this failure isn't consistent but occurs occasionally, affecting roughly 10% of the deals by my estimates. A temporary solution I've identified is adjusting the MakeNewSectorForDeals parameter from false to true. When changed, all the previously stagnant deals reactivate, corresponding AddPiece operations launch concurrently, and the sectors shift to the sealing mode (PC1) instead of the expected snapping mode (RU). This is better than having the deals stuck forever, but it's not the outcome I originally intended.

Logging Information

This log doesn't exist for the 'Available' sectors that got skipped.

2023-09-04T09:00:30.815+0900	INFO	sectors	pipeline/input.go:831	new deal sector decision	{"sealing": 12, "maxSeal": 9999999999, "maxUpgrade": 9999999999, "preferNew": false, "canCreate": false, "canUpgrade": true, "shouldUpgrade": true}
2023-09-04T09:00:30.883+0900	INFO	sectors	pipeline/input.go:777	Upgrading sector	{"number": "205297", "type": "deal", "proofType": 8, "expiration": "4729196", "pledge": "0.186159396468063426 FIL", "pieces": 1406, "dealBytesAtExp": 34359738368}
2023-09-04T09:01:32.994+0900	ERROR	sectors	pipeline/input.go:603	we are trying to create a new sector with open sectors map[{1697248 205297}:0xc208e4b470]
2023-09-04T09:01:47.539+0900	INFO	sectors	pipeline/input.go:287	deal added to a sector	{"deal": 53371925, "sector": "205297", "piece": "baga6ea4seaqjkzwri3fivf4hesjoblr27zvza5d4kkqb4gmxdwswkqp6y6u6kaq"}
2023-09-04T09:01:47.541+0900	INFO	sectors	pipeline/input.go:128	starting to seal deal sector	{"sector": 205297, "deals": 1, "trigger": "filled"}
2023-09-04T09:01:47.541+0900	INFO	sectors	pipeline/states_sealing.go:64	performing filling up rest of the sector...	{"sector": "205297"}

[hyunmoon]

I found out the skipped sector failed to be extended properly.