Channel lacks high watermark flow control

[vdudouyt]

I discovered that russh continues reading into an internal buffer even if my application can't process that bytes in time. This could be a big problem for some applications (like mine), causing unacceptable RAM usage or even a termination by OOM killer.

Steps to reproduce:

• Prepare 1gb.bin:

$ dd if=/dev/urandom of=/tmp/1.bin bs=64M count=16

• Start a minimal reproducible example that just creates a reader and calls tokio::time::sleep().await (you can find full main.rs in attachment):

    async fn channel_open_session(
        &mut self,
        channel: Channel<Msg>,
        _session: &mut Session,
    ) -> Result<bool, Self::Error> {
        let mut channel = channel;
        tokio::spawn(async move {
           {
               let _cin = channel.make_writer();
               let _cout = channel.make_reader();
               tokio::time::sleep(Duration::from_secs(3600)).await;
           }
            // for some reasons russh requires me to call channel.close() explicitly,
            // although I have no problems with that
           let _ = channel.close().await;
        });

        info!("channel_open_session");
        Ok(true)
    }

• Send 1gb.bin with pv and ssh. You can use any password, or an empty password:

$ pv /tmp/1.bin|ssh root@127.0.0.1 -p 2222 foo

Expected behavior: russh stops reading bytes in an internal buffer after high watermark is reached (suppose 16 Mb by default), until my app consumes some bytes from the channel making that internal buffer available for accepting new bytes again.

That's how it normally works for OpenSSH server sessions, as well as for TCP streams, as well as for UNIX domain sockets, as well as for simple UNIX pipes you're familiar with in UNIX shell, ... (you can continue).

Observed behavior: russh just reads whole 1gb.bin into RAM

Is there any way to overcome that issue? (Apart from implementing some self-control at application protocol level, which isn't possible if I have to keep compatibility with existing protocols such as rsync).

As I didn't tested if a similar issue exists for writer, may I also ask what happens if my app starts writing into channel faster than the client can receive that bytes? Will it await the client once buffer high watermark is reached, or just continues storing extra bytes into memory until terminated by OOM killer?

main.rs.zip

P. S. I'm using the latest russh release, which happens to be 0.46.0

[vdudouyt]

Any chances to have that fixed in the next release? This happens to be the last showstopper problem for me :(

[Eugeny]

I've published v0.50.0-beta.1 which is just 0.49 but with this fix on top ✌️

[vdudouyt]

Thank you so much!