Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ship binaries for Dart #4689

Open
robertbastian opened this issue Mar 14, 2024 · 4 comments
Open

Ship binaries for Dart #4689

robertbastian opened this issue Mar 14, 2024 · 4 comments
Assignees
Labels
A-ffi Area: FFI, WebAssembly, Transpilation C-ffi-infra Component: Diplomat, horizontal FFI S-medium Size: Less than a week (larger bug fix or enhancement)

Comments

@robertbastian
Copy link
Member

For Dart we need to ship precompiled binaries, as there is not Rust compiler available in the Dart toolchain (and no appetite to add one). However, the pub.dev package manager has a size limit of 100MB per package, which would be enough if we didn't have to ship separate binaries for all 27 supported platforms.

This means we will need some other mechanism for distributing binaries. Options include:

In the library, we add build.dart build script, which will download the binary from the source we're going with. Some open questions here are

  • How do we configure this? There's currently no mechanism in Dart to parameterise dependencies
  • Do we provide a non-network escape hatch, if the developer provides an ICU4X build themselves?
    • How do we handle versioning?

@mosuem

@robertbastian robertbastian self-assigned this Mar 14, 2024
@robertbastian robertbastian added A-ffi Area: FFI, WebAssembly, Transpilation S-medium Size: Less than a week (larger bug fix or enhancement) C-ffi-infra Component: Diplomat, horizontal FFI labels Mar 14, 2024
@mosuem
Copy link

mosuem commented Mar 14, 2024

Just a few notes: https://github.com/mosuem/generate_icu4x_binaries is where I am currently testing the binary generation (based on your proof of concept), which is then consumed by https://github.com/dart-lang/i18n/blob/17757fc7293fdbbd5e2942135e6867defb8496ec/pkgs/intl4x/build.dart.

For the open questions:

How do we configure this? There's currently no mechanism in Dart to parameterise dependencies

I am not sure I understand - the build.dart gets the target OS and architecture, and would only build/download for that target.

Do we provide a non-network escape hatch, if the developer provides an ICU4X build themselves?

Yes. Currently, the "escape hatch" is provided through the ICU4X_BUILD_MODE environment variable: Either

  • local for a locally stored dylib
  • checkout for a local checkout of the ICU4X repo to build a fresh dylib
  • fetch to get the dylib from the CDN/GitHub releases.

There is an open issue on a better way of passing these parameters than env variables, possibly using Darts pubspec.yaml.

@robertbastian
Copy link
Member Author

Ah yes that issue is what I meant.

@robertbastian
Copy link
Member Author

Conclusion:

  • In dart-lang/i18n we host:
    • per dev-platform dylib for dev (with nice errors and stuff)
    • per dev-platform data filtering tool
    • per target-platform static lib (highly optimized, stripped, panic-immediate-abort)
    • complete data file
  • In the build.dart, we store fingerprints for these.
  • The default behaviour of build.dart is to download. Environment variables can be used to use a local dylib and/or staticlib

LGTM: @Manishearth @mosuem @sffc @robertbastian

@robertbastian
Copy link
Member Author

  • @mosuem - checked with pub.dev team whether we could host artifacts there, they are reluctant. big discussion on the ecosystem team whether a package manager should support that. pub.dev team suggested github releases with fingerprinting. not great because packages are not self-contained anymore, but safe enough if we use fingerprinting
  • @sffc - how do we get above 100MB?
  • @robertbastian - we need 20+ platforms

...

cargo +nightly rustc --crate-type staticlib --profile=release-opt-size --no-default-features --features looping_panic_handler,libc_alloc,buffer_provider,icu_calendar,icu_collator,icu_datetime,icu_decimal,icu_list,icu_locid_transform,icu_plurals,icu_timezone  -p icu_capi -Zbuild-std=core,alloc -Zbuild-std-features=panic_immediate_abort --target=x86_64-unknown-linux-gnu --config=profile.release.strip=true

AI(@robertbastian): - put locale ops behind a feature in icu_capi


@ResourceIdentifier
setLocales(@NeedsToBeConst List<Locale> locales, {@NeedsToBeConst bool withFallback = true}) {
  // pragma:vm_never_remove
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-ffi Area: FFI, WebAssembly, Transpilation C-ffi-infra Component: Diplomat, horizontal FFI S-medium Size: Less than a week (larger bug fix or enhancement)
Projects
None yet
Development

No branches or pull requests

3 participants