RFC: Make it easier to call the C-ABI from Rust

[jhugman]

The issue I'd like to address here is to make calling the generated C ABI from Rust easier.

Context

Currently, all uniffi bindings works by:

• lowering from the high level foreign language to a low level representation
• squeezing the low level representation through a C ABI
• then lifting that low level representation into high level Rust types.

For languages that can't represent structures directly in C, the bindings need to get to C before calling across into Rust.

For example, the uniffi-bindgen for both gecko-js and React Native, need to generate C++ to call into Rust via the C-ABI.

But:
What about language pairs where compiling C is not feasible?

• WASM

Intent

The issue I'd like to address here is to make calling this C ABI from Rust easier.

It should be additive to the existing generated Rust.

This will enable writing uniffi adapters for new languages or contexts: e.g. node, via napi.rs; tauri; browser WASM via wasm-bindgen.

It may also open up the way to easier to maintain bindgens for languages with specialized Rust bindings crates (e.g. Ruby, Python, Dart, etc).

To ground this however, my current focus is on WASM, re-using the Typescript bindings generated by uniffi-bindgen-react-native. Whether or not we use wasm-bindgen is an implementation detail for the purpose of this issue.

I have been thinking of this proposal as "Rust as a foreign language", though this is not quite accurate: it is only the low-level ABI this proposal is targeting.

Task list

Right now, in order of urgency I think the list of issues that are needed is:

• Ensure that the C-ABI is pub, so it can be accessed from Rust as easily as it can be accessed by C.
• Change gen_ffi_function to add a low-level async caller function to the C-ABI
  • This is so bindings can use specialised implementations of Future.
• Lightweight facade to allow customization of the generation of extern "C" code.
  • This would add a trait (in the same way as log!) to add annotations and code.

For WASM specifically:

• More options to extract the Component Interface from a crate. Either:
  • Ability to read a WASM bundle in library mode.
  • A serialization/deserialization format generated to a well-known location at build time.

For now, I think I can get started with just the first.

With the exception of the last one, I don't think any individual issue is a large amount of work.

I don't think any of this is on Mozilla's roadmap, so would be happy to do the engineering.

Next steps?

• Gather feedback, answer questions
• Break out task list into separate issues
• Hack, test, document, land

Call to action

• Does this conflict with– or align with– existing plans?
• If I do this, would it be landed?
• What would stop this plan working?
• What do you think?

[jhugman]

Ben asks:

This part got my attention, do you have more details on how it would work?

Change gen_ffi_function to add a low-level async caller function to the C-ABI

Currently, the implementation of Futures are tightly coupled with the lifting arguments and lowering of return values.

For example:

#[uniffi::export]
pub async fn say_after(ms: u64, who: String) -> String {}

generates:

#[no_mangle]
pub extern "C" fn uniffi_arithmetical_fn_func_say_after(
    ms: u64,
    who: RustBuffer,
) -> Handle {
    let uniffi_lifted_args = …;
    rust_future_new(
        async move {
            match uniffi_lifted_args {
                Ok(uniffi_args) => Ok(say_after(uniffi_args.0, uniffi_args.1)),
                …
            }
        }
    )
}

If we could have an intermediate function that did the lifting and lowering, we could use other Futures implementations.

#[no_mangle]
pub extern "C" fn uniffi_arithmetical_fn_func_say_after(
    ms: u64,
    who: RustBuffer,
) -> Handle {
    rust_future_new(async move {
        uniffi_arithmetical_fn_lowlevel_func_say_after(ms, who);
    })
}

// A new async function that lifts the args, calls the high level function, then lowers the return.
//
#[no_mangle]
pub async extern "C" fn uniffi_arithmetical_fn_lowlevel_func_say_after(
    ms: u64,
    who: RustBuffer,
) -> RustBuffer{
    let ms = ms.lift();
    let who = who.lift();
    let return_ = say_after(ms, who).await;
    return_.lower()
}

For example, now we could generate a low level FFI which in turn uses (say) wasm-bindgen which has its own specialist implementation of Futures optimised for a given runtime.

#[wasm_bindgen]
pub async fn wasm_arithmetical_fn_lowlevel_func_say_after(
    ms: u64,
    who: RustBuffer
) -> RustBuffer {
    uniffi_arithmetical_fn_lowlevel_func_say_after(ms, who)
}

Another example:

#[napi_rs]
pub async fn napirs_arithmetical_fn_lowlevel_func_say_after(
    ms: u64,
    who: RustBuffer
) -> RustBuffer {
    uniffi_arithmetical_fn_lowlevel_func_say_after(ms, who)
}

It doesn't have to be limited to JS: this may be a good way of improving Ruby support, which doesn't seem to support Futures yet.

[bendk]

IIUC, this could be summarized as allowing a second layer of scaffolding specific to a bindings generator. Seems like good idea to me, although I can't totally picture how it's going to work out.

One question I had was about linking: IIUC, you would build the normal library with the UniFFI scaffolding, then build a second library with an extra layer of scaffolding, then link to both? It seems like it would work to me, but it is introducing an extra layer of complexity. For example, should this extra layer also have checksum/version checks?

My other big-picture question is if any set of bindings would want to share some of this extra scaffolding? That's kind of what I was trying to do with the ffi-buffer feature flag, but I don't think that's the right approach anymore. I wonder what functions you're planning to add and if they would also be useful for the gecko-js bindings.

If we could have an intermediate function that did the lifting and lowering, we could use other Futures implementations.

This seems like a no-brainer to me, I can't see any downside to it. Can an async fn also be extern "C" though? Maybe you'd need to drop that part, but I don't think that affects your plans.

It may also open up the way to easier to maintain bindgens for languages with specialized Rust bindings crates (e.g. Ruby, Python, Dart, etc).

Would this mean something like generating Rust code that uses the pyo3 crate to generate the Python bindings? That could be pretty nice.

[jhugman]

question I had was about linking: IIUC, you would build the normal library with the UniFFI scaffolding, then build a second library with an extra layer of scaffolding, then link to both? It seems like it would work to me, but it is introducing an extra layer of complexity.

I was hoping that this could be made easier by the binding generator pass in a trait which changed the behaviour of the scaffolding generation. i.e. the second layer of generated at the same time as the first, but I hadn't worked out the mechanics of how this works.

if any set of bindings would want to share some of this extra scaffolding?

Yes!

I'd hope that the second layer of scaffolding would be relatively straightforward, but paramterizable: though at the moment, I think I'll be able to use type aliasing to most of the work, and switch out a tiny runtime between wasm-bindgen and napi-rs. The runtime provides the type-aliases and a finite number of inlineable converters.

I'm expecting it to get a bit harder once I start adding callback interfaces.

[jhugman]

If we could have an intermediate function that did the lifting and lowering, we could use other Futures implementations.

This seems like a no-brainer to me, I can't see any downside to it. Can an async fn also be extern "C" though? Maybe you'd need to drop that part, but I don't think that affects your plans.

Brilliant!

Ah yes, I think you're right about the extern "C".

This should be a straightforward change but for one thing: *const c_void can't be send across the thread boundary. I would change the low level representation of an Arc to something else—e.g. a u64—but I thought you'd have a better plan than that.

Other than that I think it's a single file change.