Optional support for Virtualization.framework (on macOS 13)

[AkihiroSuda]

macOS 13 version of Virtualization.framework seems to support UEFI, so we may consider supporting Virtualization.framework in addition to QEMU.

---

EDIT: The current discussion has moved to:

• Support for Virtualization.Framework for macOS 13 #1147

[afbjorklund]

Would it make the code harder to support, when having two "drivers" ? Similar to when adding virtfs in addition to sshfs, with many Mac-only issues.

[jandubois]

Would it make the code harder to support, when having two "drivers" ?

I think it obviously would, but I think the performance improvements we should be able to get from it would make it worthwhile.

[jandubois]

I just learned that on macOS Ventura you can register Rosetta2 as a binfmt_misc handler inside a VM running in arm64 mode. That should mean that instead of using qemu-static emulation we should be able to run amd64 containers "compiled" into arm64, which should provide a big speed boost for code that is not available natively for arm64 (like mssql):

https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta

[abiosoft]

This can only be implemented in Swift/Obj-C, right?

[AkihiroSuda]

This can only be implemented in Swift/Obj-C, right?

Probably yes, but there is a Go wrapper for the objc
https://github.com/Code-Hex/vz

[balajiv113]

I have been working on
https://github.com/mac-vz/macvz as a redo of lima using mac virtualization API.

Am happy to extend my support/contribute over here as well :-)

Currently i have been working on setting up a slirp like network for virtualization API (facing some challenges here).

[jandubois]

Am happy to extend my support/contribute over here as well :-)

I think it would be great if you could contribute here instead of working on a fork/redo; that way we can all work together...

Same way I would like to see required changes to https://github.com/Code-Hex/vz be submitted upstream instead of creating a fork like https://github.com/mac-vz/vz. Forking should really only be a last resort if cooperations turns out to be impossible.

Just start a discussion first before working on large PRs, so we all have agreement how things should go forward. One important feature would be that VZ support needs to be optional; it isn't available on Linux (and Windows, if that becomes a thing), and also not on macOS Catalina, which we want to continue to support.

Also for some people it will be important that we support a fully open source virtualization implementation, even on a proprietary OS.

[balajiv113]

Sure. Am more than happy to work together :-)

I went ahead with fork as i wanted to try few variations first. As you said, the goal was to submit to upstream once finalised/stable.

I will start a discussion with a bit detailed plan of what i had in mind. From there we can take it forward once all in agreement.

[afbjorklund]

This could either use something simple, like the 9p vs sshfs. Or it could use something fancy, like the gRPC in docker-machine

As long as all "drivers" are provided with lima, I don't see the need for adding the complexity of having stand-alone binaries ?

[balajiv113]

@afbjorklund
Yes true, we could have the drivers within lima itself. Only issue we will have is during integration, as virtualization.framework is not available for older versions of macOS

I have opened a discussion #923 covering packaging drivers within lima (Like you said). And also added the complication in doing that and added models on how we can overcome those.

[jandubois]

IMO we need one set of binaries for macOS that works for any supported OS version (well, one set for Intel, and one set for ARM).

It is also not clear to me if parts of the vz will need to run as root (like vmnet)? Or is asking for the entitlement going to be sufficient. If we need to be root, then we need a separate binary that only implements the parts that absolutely must be privileged.

Will take a look at #923 tomorrow.

[balajiv113]

vz itself will not require root. Just signing binary with com.apple.security.virtualization entitlement is sufficient.

Example
Entitlement - https://github.com/mac-vz/macvz/blob/main/vz.entitlements
Sample signing in brew - https://github.com/mac-vz/homebrew-tap/blob/main/Formula/macvz.rb#L15

Except for using bridge network we need to run as root or vmnet entitlement should be added

[jandubois]

Except for using bridge network we need to run as root or vmnet entitlement should be added

The com.apple.vm.networking entitlement is essentially unavailable on macOS, so that part has to run as root in a separate process, like we already do for vde_vmnet.

I would be happy to learn that I'm mistaken on this.

[balajiv113]

@jandubois
As far as i understood,
com.apple.vm.networking entitlement is there but it requires special contract with Apple to sign binaries with that entitlement.

The socket_vmnet explains that well,
https://github.com/lima-vm/socket_vmnet#why-does-socket_vmnet-require-root

But as you said, if we intend to use bridge network in vz (this is optional there are other network models as well, https://developer.apple.com/documentation/virtualization/network), then yes we should run as a root (assuming we will not sign binaries with valid certificate from Apple)

[jandubois]

As far as i understood, com.apple.vm.networking entitlement is there but it requires special contract with Apple to sign binaries with that entitlement.

Yes, but only virtualization software providers that distribute their software through the Appstore can get it, which is not an option for us, I think. It exists only to provide the capability to sandboxed applications that cannot run as root. Applications distributed outside the app store are supposed to just require root.

[AkihiroSuda]

The current discussion has moved to:

• Support for Virtualization.framework #923

[AkihiroSuda]

Linking the PR

• Support for Virtualization.Framework for macOS 13 #1147

(Thanks to @balajiv113)