awaits (in host functions) are getting stuck in Async wasmtime

[venugopv]

In async wasmtime (using tokio runtime)+block_on usage: awaits from host functions are getting stuck. First instances of await are working fine. After few instances, await is getting stuck without returning any thing.

In the following test case, the first 128 awaits are returned properly, and the next await got stuck infinitely.

We are using block_on to complete futures for instantiate_async and call_async. Without block_on, all awaits are getting executed normal.

Test Case

use anyhow::Result;
use futures::executor::block_on;
use std::future::Future;
use tokio::time::{sleep, Duration};
use wasmtime::*;

type AsyncWasmResult<'a, T> = Box<dyn Future<Output = Result<T, Trap>> + 'a>;

#[tokio::main]
async fn main() -> Result<()> {
    let mut conf = Config::new();
    conf.async_support(true);
    conf.wrap1_host_func_async("test", "host_hello", host_hello);

    let engine = wasmtime::Engine::new(&conf)?;
    // All wasm objects operate within the context of a "store"
    let store = Store::new(&engine);
    // Modules can be compiled through either the text or binary format
    let wat = r#"
        (module
            (import "test" "host_hello" (func $host_hello (param i32)))

            (func (export "hello")
                i32.const 3
                call $host_hello)
        )
    "#;
    let module = Module::new(&engine, wat)?;

    // Host functions can be defined which take/return wasm values and
    // execute arbitrary code on the host.
    fn host_hello<'a>(_caller: Caller<'a>, param: i32) -> AsyncWasmResult<'a, ()> {
        Box::new(async move {
            for x in 1..1000 {
                sleep(Duration::from_millis(1)).await;
                println!("100 ms have elapsed -- {}", x);
            }
            println!("Got {} from WebAssembly", param);
            Ok(())
        })
    }

    let linker = Linker::new(&store);

    match block_on(linker.instantiate_async(&module)) {
        Ok(instance) => match block_on(instance.get_func("hello").unwrap().call_async(&vec![])) {
            Ok(_) => {
                println!("success")
            }
            Err(e) => {
                let err_string = e.to_string();
                println!("{}", err_string);
            }
        },
        Err(e) => {
            let err_string = e.to_string();
            println!("{}", err_string);
        }
    }

    Ok(())
}

Steps to Reproduce

run the above code. Host function get stuck after sleep instances, waiting infinity on await.
We have used cargo to run i.e 'cargo run'

[package]
name = "wasmtime-test"
version = "0.1.0"
authors = ["wsamtimetest"]
edition = "2018"

[dependencies]
wasmtime = "0.26.0"
tokio = { version = "1", features = ["full"] }
anyhow = "1.0.39"
futures = { version = "0.3.13" `

Expected Results

Even with block_on usage, awaits should work normally.

Actual Results

Awaits are getting stuck with block_on usage.

Versions and Environment

Wasmtime version or commit: 0.26
Operating system: Mac
Architecture: x86_64

Extra Info

[bjorn3]

You are using a tokio timer, but futures exector (block_on). This is not allowed. Tokio requires you to use tokio's executor. As you are using #[tokio::main] using .await instead of block_on inside main should work.

[peterhuene]

@venugopv I believe @bjorn3 is correct regarding the need to use tokio's executor in your repro (tokio::main is wrapping your main function in a block_on for tokio's executor anyway, so just .await inside it).

As such I'm closing this issue. Please reopen if you think there's something Wasmtime needs to fix or if @bjorn3's suggestion doesn't help you. Thank you!

[pchickey]

see also: #2832, which includes an example of using wasmtime on tokio.

[venugopv]

@bjorn3 @peterhuene Thanks for your quick response. We are trying to refactor our code to remove block_on and model based on tokio example #2832.
However I am curious to know the limit of 128 awaits for block_on i.e when wasmtime async function(i.e call_async) is called with block_on, exactly 128 awaits are working in host async functions. Subsequent await is getting stuck. It would be great to know from where this 128 limit is coming from.

[bjorn3]

I searched for 128 in the tokio source. I found the following thing that could be the origin: https://github.com/tokio-rs/tokio/blob/b42f21ec3e212ace25331d0c13889a45769e6006/tokio/src/coop.rs#L55

[peterhuene]

There's also some additional context in rust-lang/futures-rs#2157 that seems related to hitting this coop limit when using a tokio timer without yielding back to the tokio executor.