Native or wasm compiler backend for Gleam

[lpil]

It would be fun and also useful if we could compile Gleam to native code. This comes up frequently in our discord chats, so let's keep some notes here for future reference.

References

Codegen

https://leonardschuetz.ch/blog/nan-boxing/

https://piotrduperas.com/posts/nan-boxing (with benchmark)

https://bernsteinbear.com/blog/compiling-a-lisp-2/

WASM

https://github.com/WebAssembly/binaryen

https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html (used by TinyGo)

Memory layout

Wren

https://wren.io/performance.html

OCaml

See section 2, the value type https://v2.ocaml.org/manual/intfc.html

https://dev.realworldocaml.org/runtime-memory-layout.html

https://www.janestreet.com/tech-talks/unboxed-types-for-ocaml/

https://lor.sh/@armael@treehouse.systems/109473362564740302 (https://gist.github.com/Armael/aef1b9cec5d4d9591ef19dc6679faa5d)

Grain

https://github.com/grain-lang/grain/blob/main/docs/contributor/data_representations.md

Scheme

https://github.com/racket/ChezScheme/blob/8d134111b5a12ff9d0fecfcdb65e1a7ca64ad5e2/IMPLEMENTATION.md

https://www.youtube.com/watch?v=BcC3KScZ-yA

Skybison

https://github.com/tekknolagi/skybison/blob/trunk/doc/object-model.md

https://github.com/tekknolagi/skybison/blob/trunk/runtime/objects.h

[rockwotj]

Any interest in directly compiling to WASM?

[lpil]

edit: I have changed my mind! wasm is good target.

[dch]

Perhaps it’s possible to have an erts single image binary and slap a gzip of beam files on the end of it?

[rockwotj]

Warp has an Erlang example: https://github.com/dgiagio/warp/blob/master/examples/erlang/README.md

[lpil]

Perhaps it’s possible to have an erts single image binary and slap a gzip of beam files on the end of it?

@dch I would like that but in the far future I'd also like a native Gleam that could be used when more single threaded performance or minimal memory usage is required. It could integrate with Erlang based Gleam using NIFs. One for the far future :)

[dch]

@rockwotj that's neat but basically a self-extracting archive with some smart caching. I'd like to have erts itself be able to handle this case. Not really a topic for here, though!

[dch]

@lpil oh yeah I see what you meant now. I'd love fast-executing command line tools, for example. No need for NIFs initially.

[rockwotj]

For what it's worth I've been (slowly) hacking on a C++ target output for Gleam.

I chose C++ over C because templates, (and lamdbas) are really nice to enforce typesafety and not mess with a bunch of void pointers that plain C would need.

So far math expressions, functions, records (with hierarchy and shared field support) and reference counting via std::shared_ptr are supported. There is certainly a lot of work to go, but I hope to slowly chip away at progress. There are a couple of open questions around things like external types and how the higher level features above the package-compiler will work with complexity of invoking C++ compilers (let alone supporting all the platform specific stuff and things like dependency management).

Initially I really wanted to output golang, as it has a lot of nice parallels with erlang (goroutines, GC, error handling is "result based", etc), but Go has a very different module system than Gleam and generics are lacking in some aspects.

Anyways I am happy to discuss further and I'll make sure to post if I ever "catch up" with all the features in the language to discuss upstreaming some of this work if there is interest.

[lpil]

That's very cool! C++ would probably be what I go for if I were to make a native backend today. Looking forward to seeing what you make :)

I feel there likely is a shared IR that both JS and C++ could use. A simple proceedural language which typed Gleam could be lowered to. Possibly not practical right now, but perhaps in future.

[inoas]

A shared IR would mean we could write arbitrary targets against for that IR? Say one on JVM/Java?

Edit: If FFI could also target that IR language, could we then abstract away FFI for common libs so that they run on all targets supporting the IR?

[lpil]

A shared IR would mean we could write arbitrary targets against for that IR? Say one on JVM/Java?

It would reduce the amount of work to add or maintain procedural language backends, aye.

If FFI could also target that IR language, could we then abstract away FFI for common libs so that they run on all targets supporting the IR?

I think you'd still need code per platform as they'd all have different APIs and characteristics

[rockwotj]

Firefly might be interesting for gleam. You could support WASM by leveraging the existing Erlang target and not have to change semantics. If you wanted to be able to then run them without a wasmengine it seems like wasmer can do that.

[lpil]

Yup! We've spoke in the past about Gleam exposing type information to Firefly so we can enable LLVM type based optimisations too.

[gertvv]

I've been toying a bit with WASM-GC and transcribed some Gleam code into it, taking some inspiration from the JavaScript backend and stdlib implementation: https://gist.github.com/gertvv/39475c39f6419b5a31786e4f64231da6

I'd be interested in attempting to compile a similar subset of Gleam to WASM-GC. Any pointers for where to start on such a thing?

[rockwotj]

A long time ago when Gleam was not stable, I was creating a C++ backend. The first thing was to create an IR to lower gleam to something more procedural: https://github.com/rockwotj/native-gleam/blob/main/compiler-core/src/ir.rs

Although Wasm has some control flow that is a little more functional like so maybe that isn't helpful here.

I would strongly consider using binaryen in the backend layer, it will enable optimizations and you'll have an easier time generating code using it: https://github.com/WebAssembly/binaryen/wiki/Compiling-to-WebAssembly-with-Binaryen#c-api

I didn't see any rust bindings so that will be needed to be generated.

[lpil]

There is actually a prototype wasm compiler that uses wasm GC. It was made by Danielle for a university project.

[gertvv]

Is that prototype somewhere public? I haven't come across it.

Anyway I've just found out glance exists so I might toy with a Gleam implementation to start.

[lpil]

Here it is: https://github.com/DanielleMaywood/fyp-gleam

[timjs]

One of my bachelor students is working on this as well. I'll point him to @DanielleMaywood's work.

Do you already have a thesis about this Danielle? Or some docs about what's working and what not? Could be helpful for @Harmful-Alchemist.

[DanielleMaywood]

One of my bachelor students is working on this as well. I'll point him to @DanielleMaywood's work.

Do you already have a thesis about this Danielle? Or some docs about what's working and what not? Could be helpful for @Harmful-Alchemist.

My dissertation is available here. I don't have any documentation for what is missing, but the main two features that are missing are FFI and support for BitArrays.

[SichangHe]

Firefly might be interesting for gleam. You could support WASM by leveraging the existing Erlang target and not have to change semantics. If you wanted to be able to then run them without a wasmengine it seems like wasmer can do that.

Firefly is unfortunately archived as of recently.

[Harmful-Alchemist]

One of my bachelor students is working on this as well. I'll point him to @DanielleMaywood's work.
Do you already have a thesis about this Danielle? Or some docs about what's working and what not? Could be helpful for @Harmful-Alchemist.

My dissertation is available here. I don't have any documentation for what is missing, but the main two features that are missing are FFI and support for BitArrays.

Very interesting, thank you!

[SichangHe]

Wasmtime GC implementation tracking issue: bytecodealliance/wasmtime#5032

Once Wasmtime implements GC, Lunatic is probably going to get it, making it a very interesting Erlang-like target.

[SichangHe]

Maybe the other good approach would be to compile to Dart, which has the Dynamic type and also compiles to native code.

[lpil]

Closing as we are not working on this currently

[asymmetric]

One of my bachelor students is working on this as well. I'll point him to @DanielleMaywood's work.
Do you already have a thesis about this Danielle? Or some docs about what's working and what not? Could be helpful for @Harmful-Alchemist.

My dissertation is available here. I don't have any documentation for what is missing, but the main two features that are missing are FFI and support for BitArrays.

@DanielleMaywood your website seems to be down (and not in the iA), any chance you could post your dissertation elsewhere? Thanks! :)

[DanielleMaywood]

One of my bachelor students is working on this as well. I'll point him to @DanielleMaywood's work.
Do you already have a thesis about this Danielle? Or some docs about what's working and what not? Could be helpful for @Harmful-Alchemist.

My dissertation is available here. I don't have any documentation for what is missing, but the main two features that are missing are FFI and support for BitArrays.

@DanielleMaywood your website seems to be down (and not in the iA), any chance you could post your dissertation elsewhere? Thanks! :)

@asymmetric Sorry about that, my website is back online now