Use HashLink to run Lime tools

[player-03]

HashLink is the future of Haxe VMs, whereas Neko is unsupported and causes issues like #1307.

Unfortunately, this pull request currently doesn't work because the tools can't load lime.hdll. I assume this is because the ndll-loading code is written for Neko:

switch (System.hostPlatform)
{
	case WINDOWS:
		// var is64 = neko.Lib.load("std", "sys_is64", 0)();
		untyped $loader.path = $array(path + "Windows/", $loader.path);
		if (CFFI.enabled)
		{
			try
			{
				neko.Lib.load("lime", "lime_application_create", 0);
			}
			catch (e:Dynamic)
			{
				untyped $loader.path = $array(path + "Windows64/", $loader.path);
			}
		}

	case MAC:
		// if (System.hostArchitecture == X64) {

		untyped $loader.path = $array(path + "Mac64/", $loader.path);

	// } else {

	//	untyped $loader.path = $array (path + "Mac/", $loader.path);

	// }

	case LINUX:
		var arguments = Sys.args();
		var raspberryPi = false;

		for (argument in arguments)
		{
			if (argument == "-rpi") raspberryPi = true;
		}

		if (raspberryPi || System.hostArchitecture == ARMV6 || System.hostArchitecture == ARMV7)
		{
			untyped $loader.path = $array(path + "RPi/", $loader.path);
		}
		else if (System.hostArchitecture == X64)
		{
			untyped $loader.path = $array(path + "Linux64/", $loader.path);
		}
		else
		{
			untyped $loader.path = $array(path + "Linux/", $loader.path);
		}

	default:
}

Does anyone know the HashLink equivalent? There's no such thing as hl.Lib...

[player-03]

Oh, HashLink isn't even installed for CI. That's another problem for later.

[joshtynjala]

Will Haxe ever bundle HashLink the same way that Neko does? I don't really have evidence either way, but my gut feeling is no.

I suspect that the built-in Haxe interpreter is intended as the true spiritual successor to Neko for running things like Lime tools.

[player-03]

Is the interpreter really fast enough to be practical? The Lime tools may delegate most of the work to Haxe/GCC/Gradle/Xcode/whatever, but they still do some heavy lifting themselves.

I guess the interpreter is worth a shot. I'm certainly hoping Haxe will bundle HashLink at some point, but signs point to that being a long way off. Among other things, Haxelib doesn't accept run.hl as an alternative to run.n.

[player-03]

I looked into the "Failed to load library lime.hdll" error, and it looks like it's completely unrelated to neko.Lib and $loader.path. Presumably those will cause their own errors later, but this is happening first and needs to be resolved.

According to strace, dlopen() is able to locate lime.hdll just fine. It opens it up, scans the first 256 bytes, closes it, and then keeps searching other directories. According to ldd, the only missing dependency is libhl.so, even when I copy that file into the current directory. Then when I run ldd fmt.hdll, it says libhl.so exists and is fine.

This error has been reported a couple times and was usually concluded to have to do with project setup, so I'm hoping that's what it is here. Once, it was suggested that the user was using the wrong version of HashLink, so perhaps there's some kind of version incompatibility going on. I built Haxe from source to debug a prior error, so maybe it's now incompatible.

In the meantime, could someone try running trace(hl.Format.decodeJPG) in HashLink and let me know if it crashes? (I get "Failed to load library fmt.hdll" every time.)

[Apprentice-Alchemist]

Is the interpreter really fast enough to be practical? The Lime tools may delegate most of the work to Haxe/GCC/Gradle/Xcode/whatever, but they still do some heavy lifting themselves.

Looking at https://benchs.haxe.org/, Eval is actually slightly faster than neko.
So while it's not as fast as HL, it won't make things slower.

I looked into the "Failed to load library lime.hdll" error, and it looks like it's completely unrelated to neko.Lib and $loader.path. Presumably those will cause their own errors later, but this is happening first and needs to be resolved.

According to strace, dlopen() is able to locate lime.hdll just fine. It opens it up, scans the first 256 bytes, closes it, and then keeps searching other directories. According to ldd, the only missing dependency is libhl.so, even when I copy that file into the current directory. Then when I run ldd fmt.hdll, it says libhl.so exists and is fine.

Copying to the current directory only helps on Windows, on Linux it usually won't work.
Assuming you're using lime's HL build, it only loads from the directory containing the hl executable, or from standard paths such as /usr/lib.
(With the official build copying to the current directory should still work.)

This error has been reported a couple times and was usually concluded to have to do with project setup, so I'm hoping that's what it is here. Once, it was suggested that the user was using the wrong version of HashLink, so perhaps there's some kind of version incompatibility going on. I built Haxe from source to debug a prior error, so maybe it's now incompatible.

Building Haxe from source shouldn't make a difference, at least not for this specific error.

In the meantime, could someone try running trace(hl.Format.decodeJPG) in HashLink and let me know if it crashes? (I get "Failed to load library fmt.hdll" every time.)

Works for me, both with lime and with my global HL install.

[player-03]

Works for me, both with lime and with my global HL install.

Seems the error was some incompatibility between the official HashLink binary and my built-from-source copy of Haxe. Lime apps (using Lime's bundled version of HashLink) could access hl.Format.decodeJPG no problem, and once I reverted to my package manager's version of Haxe, the official HashLink binary could suddenly access it too.

I also discovered that Lime's bundled HL binary can open lime.hdll, but only attempts to load it from the current directory. Maybe that's special behavior when you request "./lime.hdll" rather than "lime.hdll"? (I don't know why it's hard-coding the dot, but it is.)

Looking at https://benchs.haxe.org/, Eval is actually slightly faster than neko.
So while it's not as fast as HL, it won't make things slower.

Excellent. That seems a lot easier than solving the HL bundling problem.

[player-03]

So while it's not as fast as HL, it won't make things slower.

Excellent. That seems a lot easier than solving the HL bundling problem.

Nope, it actually incurs a ~2 second pause on startup. Maybe it's processing macros, which would normally happen when compiling but now must happen every time.

[Apprentice-Alchemist]

So while it's not as fast as HL, it won't make things slower.

Excellent. That seems a lot easier than solving the HL bundling problem.

Nope, it actually incurs a ~2 second pause on startup. Maybe it's processing macros, which would normally happen when compiling but now must happen every time.

Not macros, it's parsing, typing and all the other normal compiler stuff that causes the slowdown.
I guess that on benchs.haxe.org that's not noticeable because the benchmarks take a while to execute, so the startup cost is hidden.

[player-03]

Makes sense. At least it was easy to get working and doesn't rely on a bundled VM.

[Apprentice-Alchemist]

Works for me, both with lime and with my global HL install.

Seems the error was some incompatibility between the official HashLink binary and my built-from-source copy of Haxe. Lime apps (using Lime's bundled version of HashLink) could access hl.Format.decodeJPG no problem, and once I reverted to my package manager's version of Haxe, the official HashLink binary could suddenly access it too.

That's strange, I wouldn't have expected that error to be in any way related to the Haxe version.
The only way to get that error is for dlopen("fmt.hdll") to fail. And I don't see any way for the Haxe version to influence that...

I also discovered that Lime's bundled HL binary can open lime.hdll, but only attempts to load it from the current directory. Maybe that's special behavior when you request "./lime.hdll" rather than "lime.hdll"? (I don't know why it's hard-coding the dot, but it is.)

Loading from the working directory is not what it's supposed to do.
And it definitely shouldn't be adding a dot, I don't even see how that's possible, this is the loading code:
https://github.com/HaxeFoundation/hashlink/blob/76ca98fee1ace39a3e5843724c226ee809d5928e/src/module.c#L362-L374
There's nothing in there that could add a dot.

[player-03]

The only way to get that error is for dlopen("fmt.hdll") to fail. And I don't see any way for the Haxe version to influence that...

I wonder if my HashLink install is just broken. If I run ldd on HashLink's copy of fmt.hdll, it can't find libturbojpeg.so.0, but Lime's version can find it no problem. Neither directory contains libturbojpeg.so.0, so it must be loading it from the default location.

Also weird: ldd can't find libhl.so in either case, even though it's in both directories.

There's nothing in there that could add a dot.

The function takes a string argument that could already contain a dot.