dart2wasm fails with typed FFI classes due to tree-shaking

[nmfisher]

dart2wasm throws various errors when compiling dart:ffi types, I'm guessing due to aggressive tree-shaking removing generic type data when those types are not explicitly instantiated.

All of the below are run with
dart --enable-asserts pkg/dart2wasm/bin/dart2wasm.dart <file> /tmp/out.wasm

Files can be found @ https://github.com/nmfisher/dart2wasm_ffi_examples if that's easier.

1. Pointer where T is AbiSpecificInteger

import 'dart:ffi';

void main() {
  print(Pointer<Char>.fromAddress(0));
}

fails with Constant InstanceConstant(const _FfiAbiSpecificMapping{_FfiAbiSpecificMapping.nativeTypes: const <Type?>[Uint8, Uint8, Int8, Int8, Uint8, Uint8, Int8, Uint8, Int8, Int8, Int8, Uint8, Uint8, Int8, Int8, Uint8, Uint8, Int8, Int8, Int8, Int8, Int8]}) references field _FfiAbiSpecificMapping.nativeTypes which is not retained.

Workaround is to explicitly declare/instantiate an equivalent type:

import 'dart:ffi';

@AbiSpecificIntegerMapping({
  Abi.androidArm: Uint8(),
  Abi.androidArm64: Uint8(),
  Abi.androidIA32: Int8(),
  Abi.androidX64: Int8(),
  Abi.androidRiscv64: Uint8(),
  Abi.fuchsiaArm64: Uint8(),
  Abi.fuchsiaX64: Int8(),
  Abi.fuchsiaRiscv64: Uint8(),
  Abi.iosArm: Int8(),
  Abi.iosArm64: Int8(),
  Abi.iosX64: Int8(),
  Abi.linuxArm: Uint8(),
  Abi.linuxArm64: Uint8(),
  Abi.linuxIA32: Int8(),
  Abi.linuxX64: Int8(),
  Abi.linuxRiscv32: Uint8(),
  Abi.linuxRiscv64: Uint8(),
  Abi.macosArm64: Int8(),
  Abi.macosX64: Int8(),
  Abi.windowsArm64: Int8(),
  Abi.windowsIA32: Int8(),
  Abi.windowsX64: Int8(),
})
final class Dummy extends AbiSpecificInteger {
  const Dummy();
}

void main() {
  var dummy = Dummy();
  print(Pointer<Char>.fromAddress(0));
}

This compiles successfully.

2. Structs

import 'dart:ffi';

void main() {
  print(getStruct());
}

@Native<Pointer<MyStruct> Function()>()
external Pointer<MyStruct> getStruct();

final class MyStruct extends Struct {
  @Int32()
  external int a;

  @Float()
  external double b;
}

fails with Constant InstanceConstant(const _FfiStructLayout{_FfiStructLayout.fieldTypes: const <Type>[Int32, Float], _FfiStructLayout.packing: null}) references field _FfiStructLayout.fieldTypes which is not retained.

This specific issue can be solved by explicitly accessing the ref, e.g.

import 'dart:ffi';

void main() {
  print(getStruct().ref);
}

@Native<Pointer<MyStruct> Function()>()
external Pointer<MyStruct> getStruct();

final class MyStruct extends Struct {
  @Int32()
  external int a;

  @Float()
  external double b;
}

However, that then fails with a different exception Exception in ConstructorInvocation at file:///Users/nickfisher/Documents/dart2wasm_ffi_examples/struct_fix1.dart:4:8 Unhandled exception: Bad state: No element

(presumably it's looking for a single-argument constructor on the Dart class MyStruct to pass the address, but that doesn't exist, hence why it fails).

3. late Pointer treated as null (?)

import 'dart:ffi' as ffi;

class NativeLibrary {
  /// Holds the symbol lookup function.
  final ffi.Pointer<T> Function<T extends ffi.NativeType>(String symbolName)
      _lookup;

  /// The symbols are looked up in [dynamicLibrary].
  NativeLibrary(ffi.DynamicLibrary dynamicLibrary)
      : _lookup = dynamicLibrary.lookup;

  void test(
    ffi.Pointer<ffi.Void> viewer,
  ) {
    return _test(
      viewer,
    );
  }

  late final _testPtr =
      _lookup<ffi.NativeFunction<ffi.Void Function(ffi.Pointer<ffi.Void>)>>(
          'test');
  late final _test =
      _testPtr.asFunction<void Function(ffi.Pointer<ffi.Void>)>();
}

late final ptr = ffi.Pointer<ffi.Void>.fromAddress(0);

void main() {
  var lib = NativeLibrary(ffi.DynamicLibrary.executable());
  lib.test(ptr);
}

fails with:

Unhandled exception:
FFI types can't be nullable
#0      Translator.translateStorageType (package:dart2wasm/translator.dart:553)
#1      Translator.translateType (package:dart2wasm/translator.dart:475)
#2      ClassInfoCollector._generateFields (package:dart2wasm/class_info.dart:463)
#3      ClassInfoCollector.collect (package:dart2wasm/class_info.dart:540)
#4      Translator.translate (package:dart2wasm/translator.dart:272)
#5      compileToModule (package:dart2wasm/compile.dart:153)

If we change ptr to final, it compiles successfully:

import 'dart:ffi' as ffi;

class NativeLibrary {
  /// Holds the symbol lookup function.
  final ffi.Pointer<T> Function<T extends ffi.NativeType>(String symbolName)
      _lookup;

  /// The symbols are looked up in [dynamicLibrary].
  NativeLibrary(ffi.DynamicLibrary dynamicLibrary)
      : _lookup = dynamicLibrary.lookup;

  void test(
    ffi.Pointer<ffi.Void> viewer,
  ) {
    final _testPtr =
        _lookup<ffi.NativeFunction<ffi.Void Function(ffi.Pointer<ffi.Void>)>>(
            'test');
    final _test = _testPtr.asFunction<void Function(ffi.Pointer<ffi.Void>)>();

    return _test(
      viewer,
    );
  }
}

final ptr = ffi.Pointer<ffi.Void>.fromAddress(0);

void main() {
  var lib = NativeLibrary(ffi.DynamicLibrary.executable());
  lib.test(ptr);
}

• Dart version and tooling diagnostic info (dart info)

General info

• Dart 3.3.0-80.0.dev (dev) (Mon Oct 30 21:02:16 2023 -0700) on "macos_arm64"
• on macos / Version 13.5.1 (Build 22G90)
• locale is en-SG

Process info

Memory	CPU	Elapsed time	Command line
68 MB	0.0%	34:18	dart devtools --machine --try-ports 10 --allow-embedding
958 MB	0.0%	34:18	dart language-server --protocol=lsp --client-id=VS-Code --client-version=3.74.0
7225 MB	0.0%	34:16	dart language-server --protocol=lsp --client-id=VS-Code --client-version=3.74.0
146 MB	0.0%	34:18	flutter_tools.snapshot daemon

• Whether you are using Windows, macOS, or Linux (if applicable)
  macOS
• Whether you are using Chrome, Safari, Firefox, Edge (if applicable)
  N/A

[mkustermann]

It may be fair to say (@osa1 / @askeksa correct me if wrong) that dart2wasm doesn't officially support FFI yet (only a subset of the API is working - as it's used by flutter web internals). There's also the issue of multi-memory wasm proposal hasn't been finalized & implemented yet.

That being said, some of the errors may not be intentional and could be fixed.

[IchordeDionysos]

EDIT: This most likely boils down to a lack of WASM support in the package or an incomplete migration to WASM (see https://dart.dev/interop/js-interop/package-web#conditional-imports)

When trying to compile an existing Flutter application, I've run into this issue 🤔

Packages in our project that depended on ffi that break when using WASM:

device_info_plus
connectivity_plus > nm > dbus
open_filex
package_info_plus
rive > rive_common
share_plus
wakelock_plus > dbus

So having support for FFI when running WASM would be important it seems 🤔

[IchordeDionysos]

Hmm okay from the package ecosystem perspective there might be work needed:

e.g. for device_info_plus ffi is only used in Windows and for web there is a conditional export that overrides the windows and Linux versions on web:
https://github.com/fluttercommunity/plus_plugins/blob/58596acafa27ee258925e3a8a15493d13ad94995/packages/device_info_plus/device_info_plus/lib/device_info_plus.dart#L28-L31

export 'src/device_info_plus_linux.dart'
    if (dart.library.html) 'src/device_info_plus_web.dart';
export 'src/device_info_plus_windows.dart'
    if (dart.library.html) 'src/device_info_plus_web.dart';

Though the export does not work on WASM as I understand as the dart.library.html does not exist there as well.

So I guess the code should be changed to something like:

export 'src/device_info_plus_linux.dart'
    if (dart.library.js_interop) 'src/device_info_plus_web.dart';
export 'src/device_info_plus_windows.dart'
    if (dart.library.js_interop) 'src/device_info_plus_web.dart';

Ahh yes, the documentation has been updated to include this change 🙌

[mkustermann]

We don't (officially) support dart:ffi on the web yet (neither in dart2js nor dart2wasm) and will soon prevent importing it. I'm going to close this in favor of two other bugs

• Add dart:ffi support for the web
• Low-level dart:wasm support