muvm: init from krun as 0.2.0

[RossComputerGuy]

Things done

Somewhat recently, krun was moved from slp to Asahi Linux and renamed. I've gone ahead and rename it. Unfortunately, there's an upstream issue where muvm doesn't run at all. AsahiLinux/muvm#81

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[alyssais]

Unfortunately, there's an upstream issue where muvm doesn't run at all.

Should this be a draft then?

[RossComputerGuy]

Yes, woops

[alyssais]

Diff looks fine to me, once it actually works :)

[RossComputerGuy]

Yeah, it's not like there's anything bad on our side. It's just upstream has issues.

[yu-re-ka]

mainProgram should be muvm as well?

[Liamolucko]

Are you sure the segfault is actually muvm's fault? I ran into the same issue at first but then realised it was just because muvm was using regular nixpkgs mesa, not Asahi mesa, since replaceRuntimeDependencies doesn't apply to packages you nix build without including in your system.

After that I ran into muvm hardcoding the path to systemd-udevd, which I worked around with this patch + replaceVars:

diff --git a/crates/muvm/src/guest/bin/muvm-guest.rs b/crates/muvm/src/guest/bin/muvm-guest.rs
index 311b2ca..b726547 100644
--- a/crates/muvm/src/guest/bin/muvm-guest.rs
+++ b/crates/muvm/src/guest/bin/muvm-guest.rs
@@ -38,7 +38,7 @@ fn main() -> Result<()> {
     if let Err(err) = mount_filesystems() {
         return Err(err).context("Failed to mount filesystems, bailing out");
     }
-    Command::new("/usr/lib/systemd/systemd-udevd").spawn()?;
+    Command::new("@systemd-udevd@").spawn()?;
 
     setup_fex()?;
 

[RossComputerGuy]

Are you sure the segfault is actually muvm's fault?

It feels like it based on the debug log output and that my shell says this:

[1]    302903 segmentation fault (core dumped)  ./result/bin/muvm sh

I ran into the same issue at first but then realised it was just because muvm was using regular nixpkgs mesa, not Asahi mesa, since replaceRuntimeDependencies doesn't apply to packages you nix build without including in your system.

But I'm not using replaceRuntimeDependencies.

After that I ran into muvm hardcoding the path to systemd-udevd, which I worked around with this patch + replaceVars:

Alright, I'll add that patch.

[Liamolucko]

It feels like it based on the debug log output and that my shell says this:

[1]    302903 segmentation fault (core dumped)  ./result/bin/muvm sh

I got the exact same error too: a proper version mismatch error would have made more sense, but I can attest that running muvm via. environment.systemDependencies rather than its original store path or disabling GPU support fixes it.

But I'm not using replaceRuntimeDependencies.

I'm assuming you're using nixos-apple-silicon: to avoid rebuilding the world, by default when you enable GPU support, it uses replaceRuntimeDependencies to replace any references to nixpkgs' regular mesa with Asahi's patched mesa. My bad if you aren't using that / don't have GPU support enabled!

(I guess replaceRuntimeDependencies isn't really that relevant: the same thing would happen if you used an overlay, the point is just that nixos-apple-silicon doesn't use nixpkgs mesa and so anything that depends on mesa directly won't work outside your system configuration.)

[RossComputerGuy]

Was able to get it to run, the problem is indeed the mesa stuff.

[diegobfernandez]

It builds fine for me but I get this error when running it (on Asahi Linux):

❯ muvm bash
thread 'gpu worker' panicked at src/devices/src/virtio/gpu/virtio_gpu.rs:240:14:
Rutabaga initialization failed!: ComponentError(-1)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: No such file or directory (os error 2)

[RossComputerGuy]

I've rebased it to fix conflicts and got it as 0.2.0 now. Once approved, I'll merge this.

[NickCao]

And maybe meta.mainProgram needs an update?

[NickCao]

Please use rustPlatform.bindgenHook

[RossComputerGuy]

Done

[NickCao]

Suggested change

	repo = pname;
	repo = "muvm";

[RossComputerGuy]

Is this change really necessary? pname is already muvm.

[NickCao]

Purely cosmetic now, but can be problematic in combination with finalAttrs in the future: #310373

[RossComputerGuy]

Ok but we're using buildRustPackage which doesn't support finalAttrs, is there plans to make buildRustPackage support finalAttrs or has that already happened?

[NickCao]

It SHOULD happen I guess? Not aware of the progress though.

[RossComputerGuy]

Interesting. What's the error you get when you don't use gdb?

Just a segfault, no other messages.

[Liamolucko]

Just a segfault, no other messages.

I think you should be able to use handle SIG43 noprint to ignore the SIG43s and get a backtrace for the segfault instead. (You have to do something similar debugging FEX, https://wiki.fex-emu.com/index.php/Development:Debugging_Crash)

[RossComputerGuy]

Thread 14 "gpu worker" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffe8b6ef40 (LWP 445362)]
0x0000fffff79f7948 in __strlen_generic () from /nix/store/5a2rdj5i1338isqdvivsn6yk7hyw4r4i-glibc-2.40-36/lib/libc.so.6
(gdb) bt
#0  0x0000fffff79f7948 in __strlen_generic () from /nix/store/5a2rdj5i1338isqdvivsn6yk7hyw4r4i-glibc-2.40-36/lib/libc.so.6
#1  0x0000fffff79f1344 in strdup () from /nix/store/5a2rdj5i1338isqdvivsn6yk7hyw4r4i-glibc-2.40-36/lib/libc.so.6
#2  0x0000ffffe851e138 in dri2_initialize_drm () from /nix/store/9bsk349jghy4wf9xr2i9nmxw4dp69m59-mesa-24.2.0-drivers/lib/libEGL_mesa.so.0
#3  0x0000ffffe8517f00 in dri2_initialize () from /nix/store/9bsk349jghy4wf9xr2i9nmxw4dp69m59-mesa-24.2.0-drivers/lib/libEGL_mesa.so.0
#4  0x0000ffffe8504e78 in eglInitialize () from /nix/store/9bsk349jghy4wf9xr2i9nmxw4dp69m59-mesa-24.2.0-drivers/lib/libEGL_mesa.so.0
#5  0x0000fffff62e5e70 in virgl_egl_init () from /nix/store/aqw5zki2fi33kb6wz8nldl0vqgflbgnc-virglrenderer-1.1.0/lib/libvirglrenderer.so.1
#6  0x0000fffff62e4c8c in vrend_winsys_init () from /nix/store/aqw5zki2fi33kb6wz8nldl0vqgflbgnc-virglrenderer-1.1.0/lib/libvirglrenderer.so.1
#7  0x0000fffff62a63f0 in virgl_renderer_init () from /nix/store/aqw5zki2fi33kb6wz8nldl0vqgflbgnc-virglrenderer-1.1.0/lib/libvirglrenderer.so.1
#8  0x0000fffff7c8ff2c in devices::virtio::gpu::worker::Worker::work () from /nix/store/xi2ad885wwcjy71gn0wzacwaj0nay5rr-libkrun-1.9.8/lib/libkrun.so.1
#9  0x0000fffff7cb3008 in std::sys::backtrace::__rust_begin_short_backtrace () from /nix/store/xi2ad885wwcjy71gn0wzacwaj0nay5rr-libkrun-1.9.8/lib/libkrun.so.1
#10 0x0000fffff7cc4c68 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /nix/store/xi2ad885wwcjy71gn0wzacwaj0nay5rr-libkrun-1.9.8/lib/libkrun.so.1
#11 0x0000fffff7db0580 in std::sys::pal::unix::thread::Thread::new::thread_start () from /nix/store/xi2ad885wwcjy71gn0wzacwaj0nay5rr-libkrun-1.9.8/lib/libkrun.so.1
#12 0x0000fffff79d86bc in start_thread () from /nix/store/5a2rdj5i1338isqdvivsn6yk7hyw4r4i-glibc-2.40-36/lib/libc.so.6
#13 0x0000fffff7a44c8c in thread_start () from /nix/store/5a2rdj5i1338isqdvivsn6yk7hyw4r4i-glibc-2.40-36/lib/libc.so.6

Ended up being the usual mesa issue so I think it's good on my side once I load it in with the right mesa.

[RossComputerGuy]

Rebased + ping to hopefully get a review, this PR should've been merged months ago.

[NickCao]

One question: this doesn't work on x86_64-linux right? Seems to have some aarch64 specific codes.

[RossComputerGuy]

I'm not sure, maybe give it a try?

[NickCao]

I'm not sure, maybe give it a try?

https://github.com/AsahiLinux/muvm/blob/a336fedf7c91b962d07e8e6b59f42f869dd09d4e/crates/muvm/src/x11bridge/bin/muvm-x11bridge.rs#L650

[NickCao]

I can run bash, but X11 pass-through won't work.

[NickCao]

A few other hard coded paths to be patched:

crates/muvm/src/server/bin/muvm-server.rs
39:            PathBuf::from("/bin/sleep"),

crates/muvm/src/guest/mount.rs
87:    for base in ["/usr/share/fex-emu", "/usr/local/share/fex-emu"] {

crates/muvm/src/monitor.rs
53:    let command = PathBuf::from("/sbin/sysctl");

And ideally, bind mount /run/current-system

[RossComputerGuy]

A few other hard coded paths to be patched:

Done

And ideally, bind mount /run/current-system

The patch has conflicts and it seems upstream wants to go a different direction so it might be better to wait for that.

[marcin-serwin]

Suggested change

	mainProgram = "krun";
	mainProgram = "muvm";

[RossComputerGuy]

Done

[jameschubbuck]

I was trying to run muvm on my machine (M1) with Asahi and NixOS using this module, but I kept getting same thread panic error as @diegobfernandez and @Liamolucko. I tried updating muvm, libkrun, and libkrunfw, along with applying muvm PR #38, but to no avail. Has anyone else found a way of running muvm on Asahi on NixOS? @RossComputerGuy, it seems you have spent some time thinking about this and help would be appreciated. Being able to use wine to run windows executables would be quite nice. Currently, the method proposed by @bhenson in NixOS Discourse, while a bit unorthodox, has been the most functional that I have come across.

[Liamolucko]

I was trying to run muvm on my machine (M1) with Asahi and NixOS using this module, but I kept getting same thread panic error as @diegobfernandez and @Liamolucko. I tried updating muvm, libkrun, and libkrunfw, along with applying muvm PR #38, but to no avail. Has anyone else found a way of running muvm on Asahi on NixOS? @RossComputerGuy, it seems you have spent some time thinking about this and help would be appreciated. Being able to use wine to run windows executables would be quite nice. Currently, the method proposed by @bhenson in NixOS Discourse, while a bit unorthodox, has been the most functional that I have come across.

Did you try using Asahi's fork of virglrenderer like I suggested?

virglrenderer = prev.virglrenderer.overrideAttrs (old: {
  src = final.fetchurl {
    url = "https://gitlab.freedesktop.org/asahi/virglrenderer/-/archive/asahi-20241205.2/virglrenderer-asahi-20241205.2.tar.bz2";
    hash = "sha256-mESFaB//RThS5Uts8dCRExfxT5DQ+QQgTDWBoQppU7U=";
  };
  mesonFlags = old.mesonFlags ++ [ (final.lib.mesonOption "drm-renderers" "asahi-experimental") ];
});

I've been able to get GUI programs working, as well as x86 emulation; I haven't tried Wine, though. GPU acceleration for x86 programs doesn't work, since although a rootfs isn't needed for most Nix stuff thanks to everything using absolute paths to the x86 versions of their dependencies anyway, that isn't the case for GPU drivers: programs that need them will link against whatever's in /run/opengl-driver*. So a rootfs which only contains /run/opengl-driver* will be needed.

[jameschubbuck]

@Liamolucko, thanks for your response.

Did you try using Asahi's fork of virglrenderer like I suggested?

I just tried this, and it did have an effect! Unfortunately, it was just causing the same error to occur much faster than without it.

I've been able to get GUI programs working

I haven't been able to so far, and it even seems that, in whatever cursed setup I have created, muvm doesn't even get as far as processing its arguments. Both muvm wine and muvm foo result in the same OS error 2 (file not found). Perhaps this is because I am using pkgs.callPackage to add muvm to my configuration? I tried reconciling the aforementioned PR in a fork of muvm (v0.3.1), which did build fine, but still had the same error. It's also worth nothing that paths.patch.in didn't apply to this version.

From my understanding, it seems that muvm is trying to access paths which don't exist on NixOS. From our end, wouldn't it be easier just to make some symlinks ourselves to accommodate for where muvm expects to find drivers?

[Liamolucko]

I haven't been able to so far, and it even seems that, in whatever cursed setup I have created, muvm doesn't even get as far as processing its arguments. Both muvm wine and muvm foo result in the same OS error 2 (file not found).

Did you miss patching the path of systemd-udevd? Commenting that out gives me the same symptoms as you're describing. It could also be that you didn't set OPENGL_DRIVER or CURRENT_SYSTEM, since AsahiLinux/muvm#38 doesn't do anything without them.

I've pushed the exact version I'm using at https://github.com/Liamolucko/nix-config (which doesn't require setting OPENGL_DRIVER/CURRENT_SYSTEM).

From my understanding, it seems that muvm is trying to access paths which don't exist on NixOS. From our end, wouldn't it be easier just to make some symlinks ourselves to accommodate for where muvm expects to find drivers?

No, not really (well, that is the case with systemd-udevd, but not the GPU drivers). The problematic accesses to the GPU drivers are coming from x86_64 programs running inside muvm, not muvm itself, which are still coming from nixpkgs and so are still designed to run on NixOS.

The problem is that the path in /run/opengl-driver at which x86_64 programs are expecting to find their x86_64 GPU drivers is already taken up by the host's aarch64 GPU drivers. So we need programs running inside FEX to see x86_64 drivers at the same path that regular programs see aarch64 drivers, which is what the purpose of a FEX rootfs is.

[Liamolucko]

Actually, I suppose a different solution would be to restore OPENGL_DRIVER (and OPENGL_DRIVER_32) as environment variables, and set them to x86_64 and x86 versions of mesa instead of native versions; that would result in aarch64 programs running inside muvm being broken instead, but there wouldn't be much reason to run them there instead of natively anyway.

Personally I still think the rootfs solution is probably cleaner, though. Perhaps it could be integrated into the fex package's default Config.json.