update buffers to use Tokio 0.3 MPSC channels

[hawkw]

This branch updates linkerd2-buffer, and linkerd2-proxy-discover's
buffer module to use Tokio 0.3's MPSC channel rather than Tokio 0.2's.
The rest of the proxy still uses Tokio 0.2, including the 0.2 runtime.

Most of the Tokio synchronization primitives lost their poll-based
interfaces in 0.3 as part of the move to intrusive lists of wakers for
synchronization primitives (see tokio-rs/tokio#2325,
tokio-rs/tokio#2509, and tokio-rs/tokio#2861). This change takes
advantage of the inherently pinned nature of async fn and async
blocks to avoid needing a separate heap allocation to store the waiter
state for a task waiting on a synchronization primitive. However, it
means that a synchronization primitive can only be waited on when the
future that waits on it is pinned --- otherwise, there is a potential
dangling pointer. The poll-based APIs allowed waiting on
synchronization primitives from unpinned contexts, so they were removed.

To wait on the synchronization primitives from contexts that may not be
pinned, such as poll_ready, it's necessary to add a Pin<Box<...>>
around the future that's waiting on the synchronization primitive. This
ensures that the future will not move while it's part of the wait list.
It's important to note that this isn't an additional allocation per
waiter versus Tokio 0.2; instead, it's the same allocation that would
have always happened internally to the synchronization primitive in
the 0.2 API. Now, it's moved outside of the tokio::sync type so that
it can be avoided when used with async/await syntax, and added by
the user when polling the sync primitives.

Because we need to poll channel senders in tower::Service
implementations' poll_ready functions, it was necessary to introduce
our own bounded MPSC channel type that exposes a polling-based API. When
the buffer's channel is full, we want to exert backpressure in
poll_ready, so that callers such as load balancers could choose to
call another service rather than waiting for buffer capacity. This
branch adds a new linkerd2-channel crate that implements a pollable
bounded channel, wrapping tokio::sync's unbounded MPSC and using a
tokio::sync::Semaphore to implement bounding. It's worth noting that
this is, essentially, how tokio::sync::mpsc's bounded channel is
implemented --- it also uses the semaphore. However, our implementation
exposes a poll_ready method by boxing the future that waits to acquire
a semaphore permit, which the Tokio channel does not expose.

Finally, I've added some tests for the linkerd2-channel crate, based
on Tokio's tests for the MPSC channel, modified where the APIs differ.
This should help ensure we get similar behavior to what we expect from
Tokio's MPSCs.

This was factored out of PR #732.

Signed-off-by: Eliza Weisman eliza@buoyant.io

[olix0r]

Suggested change

	break;
	return;

[hawkw]

way ahead of you, buddy e2046e3 :D

[olix0r]

@hawkw i think that got the other break, but there were two.. so i think this is still relevant?