Support for different runtimes and TLS libraries

[jakoschiko]

With the introduction of sans I/O (see #158) we are now able to support different runtimes (std, tokio, async-std) and TLS libraries (native TLS, rustls) easily. Currently we only support tokio and rustls by providing the type stream::Stream. In the future we might add more similar implementations. But this raises couple of questions:

• Should they be part of this repo?
• Should they be part of the imap-flow crate and gated via feature flags? Should the feature flags be additive?
• Should they be inside separate crates? A single crate of multiple ones?
• Which combination of runtime and TLS library do we want to support?
• How do we deal with features like backpressure (see How to implement backpressure on imap-flow? #98)? Should all implementations support the same features?

See this discussion.

[duesee]

We have ...

• {tokio-,}openssl
• {tokio-,}rustls
• {tokio}-native-tls
• async-tls (rustls)
• ...

TLS backends are mostly implemented using traits from the runtime, e.g., std::io::{Read,Write}, tokio::Async{Read,Write} or async_std::Async{Read,Write}.

This means that we probably want to write sans I/O adapters that use a runtimes read/write traits (to support most tls backends).

More questions:

• Who configures TLS? E.g., ALPN?
  • We could provide helper methods per runtime+backend.
• STARTTLS helper?

[soywod]

I wanted to share a bit what I've been attempted so far with pimalaya/core/start-tls:

• A StartTls::new declares a new STARTTLS flow using the given extension (or spec).
• An extension (or spec) implements the StartTlsExt trait and is compatible with both blocking and async environments.
• The extension takes a mutable reference to a stream and is responsible for performing I/O operations on it, using Read + Write for blocking environment and AsyncRead + AsyncWrite + Unpin for async environments.
• Extensions are based on structs rather than simple functions (&mut S) -> Result<()> because it can better manager customization, like for example the buffer size, or if handshake should be discarded.
• Only futures is supported, because tokio can be easily supported outside of this crate using tokio_util::compat. There is also compatibility layers to use blocking streams from within futures async via futures::io::AllowStdIo.
• Currently IMAP and SMTP extensions are supported, but custom extensions can be added via the StartTlsExt trait.

Basically, the right .prepared() fn is inferred (either blocking or async), which makes the API easy to use:

// .prepare() for blocking environment (std)
let tcp_stream = std::net::TcpStream::connect(…)
let spec = ImapStartTls::new(&mut tcp_stream);
StartTls::new(spec).prepare().unwrap();

// .prepare().await for async runtime (futures)
let tcp_stream = async_std::net::TcpStream::connect(…)
let spec = SmtpStartTls::new(&mut tcp_stream);
StartTls::new(spec).prepare().await.unwrap();

Examples can be found here:

$ RUST_LOG=debug HOST=posteo.de PORT=25 cargo run --example smtp-std
$ RUST_LOG=debug HOST=posteo.de PORT=143 cargo run --example imap-futures

Keep in mind that this is just an experiment, I would appreciate any feedback on it.

[soywod]

I ran into several issues with the previous implementation. After digging a lot, I unfortunately came to the conclusion that separated structs/traits for sync and async is the actual way to go. There is no happy path ATM.

The actual proposition is the following:

• 2 traits are available: PrepareStartTls<S> and blocking::PrepareStartTls<S>. The prepare fn is the same, except for the return type:

  // blocking, anything that is Read + Write
  fn prepare(&mut self, stream: &mut S) -> Result<()>;
  
  // async, anything that is AsyncRead + AsyncWrite + Unpin
  // (no need for the #[async_trait] macro since now Rust supports return impl T)
  fn prepare(&mut self, stream: &mut S) -> impl Future<Output = Result<()>>;

• 2 implementations are implemented: IMAP and SMTP. They both implements both blocking and async trait.

• The usage is the following (for example, with IMAP):

  ImapStartTls::new()
      .with_read_buffer_capacity(1024)
      .with_handshake_discarded(false)
      .prepare(&mut stream)?;

  After those lines, the stream should be ready for TLS negociation.
  See std and async-std examples.