Improve typed socket connection robustness

[paulgb]

What we know:

• Sometimes, when there are network issues, some (but not all) drones fail to reconnect. In the logs, it shows up as failed to send heartbeat (from heartbeat.rs), with the err value of Disconnected (typed_socket).
• When we “send” websocket messages, we are actually sending messages to a queue that gets picked up by the websocket event loop asynchronously. Disconnected actually means that the message queue is full, which implies we are disconnected, but is not immediate.
• On a TypedSocket, send() uses Sender::send, while TypedSocketSender's send() uses try_send.
• In tokio, send on a full channel blocks until the channel has capacity. try_send returns immediately if the channel is full.

My leading theory is that when the network is interrupted abruptly, upon reconnecting to the controller, the controller sends a bunch of messages, causing a deadlock:

• the new_client loop is stalled waiting for capacity in send_to_client, which won't happen until socket.recv() is called in the main drone event loop
• the main drone event loop is waiting on a call to socket.send() when acking an action

This PR introduces several changes, which should improve the robustness of reconnects:

• Instead of handling messages from the controller directly in the drone event loop, they are sent to separate tasks. This means that nothing can get in the way of the drone event loop's ability to call socket.recv().
• TypedSocket::send now uses Sender::try_send, for consistency with TypedSocketSender::send. Note that as a result of moving handling out of the drone event loop, messages sent from the drone loop now use TypedSocketSender instead of TypedSocket anyway, so those messages would now use try_send regardless.
• As a result of using try_send, TypedSocket::send no longer needs to be async, so that's removed.
• TypedSocketError::Disconnected is renamed to TypedSocketError::Clogged, to better reflect what the issue is.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
plane	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 10, 2024 10:43pm

[rolyatmax]

"Socket clogged"?

[rolyatmax]

LGTM