Simplify walproposer code

[petuhovskiy]

Closes neondatabase/neon#1041

• Leave only states in which we wait, others can be executed directly or in a function
• Remove loop in AdvancePollState
• Migrate AsyncReadFixed to AsyncReadMessage
• Once connection is established, do UpdateEventSet only when we need to flush (set WL_SOCKET_WRITEABLE) or when flush is done (unset it).
• Optimize UpdateEventSet calls

[arssher]

Approve on first two checkboxes.

[petuhovskiy]

Updating event set only on flushes is tricky, I optimized UpdateEventSet calls in SS_ACTIVE differently. Now all SS_ACTIVE code is executed only with calls to HandleActiveState, which calls UpdateEventSet in the end. This way we have single place for updating event set and it will not flip in the middle of the handler. Also, if event mask is not updated in the next call to UpdateEventSet, no extra calls will be done, because ModifyWaitEvent has early exit check for such case.

Flushes are tricky mostly because we can have a call stack like HandleActiveState -> RecvAppendResponses -> BroadcastMessage -> SendMessageToNode -> HandleActiveState -> SendAppendRequests. In this case we should be careful about overriding event set, which can cause unsetting WL_SOCKET_WRITEABLE while we still have messages to send.

[arssher]

After looking for some time I now think that UpdateEventSet placement here is better than what I initially proposed :)

[petuhovskiy]

Another thing I remembered, that after HackyRemoveWalProposerEvent (happens on every reconnect) we always UpdateEventSet(WL_SOCKET_READABLE | WL_SOCKET_WRITEABLE) for SS_ACTIVE, because that's how it's defined in WalKeeperStateDesiredEvents.

HandleActiveState isn't picky on events and will fix them in any case, but that's still extra work on every reconnect.

[arssher]

Not that it is extremely expensive, but yeah, it would be nicer to check for flushWrite in WalKeeperStateDesiredEvents to fix it.