Close ChainNotify channel when it blocks

[Stebalien]

Checklist

• This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
• I have a specific, actionable, and well motivated improvement to propose.

Lotus component

Other

Improvement Suggestion

Currently, if a caller fails to read from the ChainNotify channel, everything will block until they do. This means one slow caller can slow everything else down and cause problems for the entire node (see #6883).

Proposal: If writing to the ChainNotify channel would block (caller isn't reading), close the channel. When the caller resumes reading, they'll see the close, know that they skipped some events, re-subscribe, then figure out which events they missed. Effectively, the caller will just "restart" itself.

[Stebalien]

cc @magik6k any good reasons not to at least try this?

[Stebalien]

A (simpler) alternative solution would be to build the "catchup" part into SubHeadChanges itself.

1. If writing to the output channel blocks (in SubHeadChanges), start dropping notifications (but keep the first "unsent" and last seen notification).
2. If the output channel gets unblocked, walk the dropped part of the chain, sending notifications for each entry.

In terms of efficiency, it's probably better to just close the channel. However, that's a significantly more invasive change.

[Stebalien]

I've gone with the first option and tests seem to pass.

[Stebalien]

Fixed in #7000.