Update Lima to latest so we can try Virtiofs

[ryancurrah]

Problem Description

The bundled version of Lima does not include Virtiosfs Mount mode support.

Proposed Solution

Sync the forked version of Lima and update lima-and-qemu and create a new release of Rancher Desktop. If it doesn't break things.

Additional Information

No response

[ryancurrah]

@jandubois in the meantime is there a way we can get rancher desktop to use a newer version of lima manually? Like copying the latest lima binary to where it is embedded in the rancher desktop install location?

[jandubois]

in the meantime is there a way we can get rancher desktop to use a newer version of lima manually? Like copying the latest lima binary to where it is embedded in the rancher desktop install location?

I will be looking into this in the next few weeks and see if I can provide a replacement tarball that you can use for testing; not sure if it will require a new ISO as well, or not.

If you follow Lima development, you will have noticed that this is still the bleeding edge, and problems continue to be discovered, so it does not seem to be ready for real usage. I will try to find a way how this can be bundled and enabled on an experimental opt-in basis for the next release, but can't promise yet, if this will be possible.

[jandubois]

Just one example of the issues we need to resolve: the VZ code only works if it has been built on a Ventura machine with the latest XCode.

However, that very likely means that the code will not work correctly on older macOS releases. So maybe we need to build multiple instances of Lima, and choose at runtime which version to use.

We still need to support at least Big Sur and Monterey in addition to Ventura (we will drop support for Catalina though, as it is now completely unsupported by Apple).

[ryancurrah]

That is a challenge. It looks like lima is being built on Monterey and switching to a newer xcode version to enable the macOS 13 SDK. https://github.com/lima-vm/lima/blob/master/.github/workflows/release.yml#L19-L28.

And the oldest macOS version that can be used with xcode 14.1 is Monterey 12.5 or later.

Xcode 14.1 requires a Mac running macOS Monterey 12.5 or later.
https://developer.apple.com/documentation/xcode-release-notes/xcode-14_1-release-notes

If you want to build VZ support (Without supporting multiple builds of lima) you could only support Monterey or newer. Big Sur won't be end of life until 12, Nov, 2023.

It looks like a challenge to support older macOS's and all the features lima is coming out with (VZ). That does put a burden on Rancher Desktop. Does Rancher Desktop try to support all the new features in lima by dropping support for older macOS versions or try to support all non-end of life macOS versions even though lima may not.

Our perspective is we are eager to get our hands on the newest improvements in lima.

Looking at some other projects similar to Rancher Desktop...

Colima does not bundle lima it depends on lima via the brew formula. So Colima supports what lima supports.

Finch builds lima on macOS 10.15 for amd64 and macOS 11.11 for arm64.

I am willing to help figure out solution for this and work on some tickets. Not sure if it is wanted or not, just let me know how I can get involved.

[jandubois]

That is a challenge. It looks like lima is being built on Monterey and switching to a newer xcode version to enable the macOS 13 SDK. https://github.com/lima-vm/lima/blob/master/.github/workflows/release.yml#L19-L28.

And the oldest macOS version that can be used with xcode 14.1 is Monterey 12.5 or later.

I noticed. I just did a test build with lima 0.14.2 and the macOS 13 SDK:

https://github.com/rancher-sandbox/lima-and-qemu/actions/runs/3859700464

I will now test this manually, but I only have Catalina and Monterey Intel Macs anymore, so cannot test the VZ bits right now. Will have to do an ARM build manually to test on Ventura.

If you want to build VZ support (Without supporting multiple builds of lima) you could only support Monterey or newer. Big Sur won't be end of life until 12, Nov, 2023.

I'm not opposed to doing multiple builds to support all platforms; it will just take some extra effort. Let's first find out if it is necessary.

It looks like a challenge to support older macOS's and all the features lima is coming out with (VZ). That does put a burden on Rancher Desktop. Does Rancher Desktop try to support all the new features in lima by dropping support for older macOS versions or try to support all non-end of life macOS versions even though lima may not.

Lima does support older macOS versions too; it just builds without VZ support on them:

https://github.com/lima-vm/lima/blob/13fe50e46db259122ea1b919348dfb8a2b0d5764/Makefile#L19-L22

This works fine for Lima because e.g. Homebrew has separate bottles for Big Sur, Monterey, and Ventura.

We only want to have a single build of RD for macOS (well, one for Intel and one for ARM, at least for now), so we may have to bundle multiple versions and decide at runtime which one to use. We'll see. But we definitely want to support all macOS versions that are not yet EOL'ed by Apple.

Our perspective is we are eager to get our hands on the newest improvements in lima.

Looking at some other projects similar to Rancher Desktop...

Colima does not bundle lima it depends on lima via the brew formula. So Colima supports what lima supports.

It does. But if you observed it closely, you can see that colima ran into various issues during the last few weeks and had to back out some changes to make it work again. If you look closely, you'll notice that I fixed one of the issues for them: lima-vm/lima#1243

We try to avoid this instability by waiting until things have settled down a bit, which I hope they have by now (not completely, but maybe good enough for experimental support).

Finch builds lima on macOS 10.15 for amd64 and macOS 11.11 for arm64.

Finch is using the "lima-and-qemu" packaging mechanism from Rancher Desktop.

I am willing to help figure out solution for this and work on some tickets. Not sure if it is wanted or not, just let me know how I can get involved.

Yes, your help is very much appreciated! I'm trying to get at least the lima-and-qemu bits in place, to you can help with the testing. This will not be my main priority for at least another week, but I'll try to work with you to move it along.

I'll update here when I have some testing results; please do the same if you get the chance!

[jandubois]

I'll update here when I have some testing results

For some reason the bundled lima version is not 0.14.2, but a couple commits later; I still have to figure out how that happened. I thought the submodule was pointing the the right commit.

And testing shows that we definitely need 2 versions of Lima, one for Big Sur+, and one for Ventura. I think we can still package a single version of qemu, to save some space (I know, a losing fight, but we have to try).

[ryancurrah]

I can work on the changes required in Lima-and-qemu to support it. While also updating the submodule.

[jandubois]

I can work on the changes required in Lima-and-qemu to support it. While also updating the submodule.

The submodule turned out to be correct after all. I ran limactl --version instead of ./limactl --version when I checked the commit. 😥

I'm working on the changes right now: I think we can continue to build lima-and-qemu on macOS 11, and then just build lima with the macOS 13 SDK on macOS 12, and just collect limactl as an extra asset.

Then RD will have to be updated to fetch the additional resource and put it into .../lima/bin/limactl.ventura, and on startup copy it over to .../lima/bin/limactl. Or something like this, maybe using symlinks and keeping both files around, or whatever.

But if you want to do it, that works for me too! 😄 Start with the PR for lima-and-qemu, to make the changes I described above. Upload the limactl file for Ventura as a lima.ventura.tgz file.

Once it all works and is merged, I'll create another release, and then you can work on the changes to RD itself.

[ryancurrah]

Ok, I see you made the changes in lima-and-qemu to start building on macOS 12 for VZ support. Do you want to add another workflow step that builds lima with macOS 11 and package up that binary?

[jandubois]

Yes, I would like to revert this to build the whole tarball with macOS 11 again, and then add another step to just build lima on macOS 12, and then package up just limactl as limactl.ventura.tgz as an additional asset.

The XCode switch obviously should move to this new step.

[ryancurrah]

Ok, I'll take a stab at it today.

[jandubois]

@ryancurrah I've created https://github.com/rancher-sandbox/lima-and-qemu/releases/tag/v1.29 for testing (Intel assets only right now). Do you also want to work on the changes required in Rancher Desktop itself?

It will essentially just be the additional download of limactl.ventura.tar.gz into the resources folder, and then moving it into .../lima/bin. I would then rename the regular version to e.g. limactl.bigsur.

At runtime during startup RD would then check the OS version and symlink either of the 2 binaries into the limactl spot, and everything should work normally from then on.

If you don't want to work on that part, please let me know, and I'll look into it probably early next week. At that time I'll also generate the ARM64 binaries for the 1.29 release.

[ryancurrah]

I can take a shot at it.

[jandubois]

Thanks! Let me know if you need help (or if you decide to abandon it).

symlink either of the 2 binaries into the limactl spot

I just realized that we must not make any changes inside the app directory, as that can break the signature. So instead we should have a variable in the paths package that includes the full path to the limactl binary for the current platform.

[ryancurrah]

Ok I will look at tonight after I put the little one to bed. Will update here if/when I get stuck.

[ryancurrah]

Just looking at how it gets installed. Should I create a new class here https://github.com/rancher-sandbox/rancher-desktop/blob/main/scripts/dependencies/lima.ts#L16 called Limactl which downloads the specified limactl archive when building on macOS?

[jandubois]

Should I create a new class

No, just include the download in the LimaAndQemu class that is already there: it is really part of the same release, just in a different file. It will always have the same version number etc. Just a separate/additional download.

[ryancurrah]

Ack thank you.

[ryancurrah]

Well, I got an initial WIP going here #3761.

It will install a limactl.ventura binary in lima/bin. I tried building it with npm run build and the limactl.ventura binary never gets installed in that directory. I even removed the if statement to always run the download limactl code and still no luck. It seems like the code never runs, I'm not super familiar with npm so it may be a user error.

[jandubois]

It will get installed by npm install (or npm run postinstall if you already ran npm install before).

Note that the scripts will not download files again that already exists locally, without checking content checksums, or anything. So if you switch branches, or want to build M1 binaries after building Intel binaries, you'll have to run git clean -dffx before running npm i again.

[ryancurrah]

Thank you, with that information I was able to test that the code is working as intended.

[jandubois]

I've tested this with the latest CI build of #3761 (now merged) from https://github.com/rancher-sandbox/rancher-desktop/actions/runs/3958949827

Switching to VZ requires some manual hackery that we still need to automate:

I installed on Ventura on an M1 machine, ran RD once to get settings setup and directories created. Wait until everything is done (I think there is an issue with qemu remaining running if you quit while RD is still starting up).

Quit RD and wait until all processes are gone. Then

rm -rf ~/Library/Application\ Support/rancher-desktop/lima/0/
echo "vmType: vz" > ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml
rdctl start

This deletes the old vm and creates the override.yaml file to switch to VZ (you cannot convert a qemu vm into a vz vm).

You can see now that qemu is no longer running.

I increased the CPU count in preferences beyond 8 (qemu fails when you configure more than 8 CPUs on arm64) and checked that it worked (after the vm was restarted):

$ rdctl shell nproc
18

So this looks promising, but still needs some settings to switch between VM modes without manually deleting VMs. I'm thinking initially just something like rdctl set --virtual-machine.experimental.vmtype=vz will do until we have time to add it to the UI.

[ryancurrah]

Awesome! We have our devs modify the override.yml file with the yaml tool yq to use the 9p mount type. After the next release we will have them use vz.

[ryancurrah]

@jandubois just realized this now but a lot of people are using limactl cli directly in /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin in their scripts.

For example to get the SSH_AUTH_SOCK location,$(limactl shell 0 printenv SSH_AUTH_SOCK).

Changing the name of the binary to limactl-ventura might break their scripts.

[jandubois]

a lot of people are using limactl cli directly in /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin in their scripts.

I was not aware of that, and it is a really bad idea. We need to tell people to stop doing this.

For example to get the SSH_AUTH_SOCK location,$(limactl shell 0 printenv SSH_AUTH_SOCK).

This can easily be done by rdctl shell printenv SSH_AUTH_SOCK.

I would like to get a list of use-cases why people need to call limactl directly, to make sure all of them can be handled by rdctl.

Changing the name of the binary to limactl-ventura might break their scripts.

I don't think they will break; we are still keeping the regular limactl as well, and I think it only matters for limactl start to use the Ventura version. And I hope nobody tries to use limactl start 0 in their scripts, as it simply won't work correctly.

We may have to break this eventually, but hopefully people have migrated to using rdctl exclusively by then.

[ryancurrah]

Thanks for the thorough reply.

We do some Go development in containers, to clone private git repos with go get we use SSH authentication mechanism which requires SSH Agent socket mounted in the container. We got the solution from this issue #3042.

DOCKER_SSH_SOCK=$(limactl shell 0 printenv SSH_AUTH_SOCK)

docker run \
	--rm \
	--interactive \
	--pull="always" \
	--workdir "$(pwd)" \
	--mount "type=bind,source=$(pwd),target=$(pwd)" \
	--mount "type=bind,source=${HOME}/.ssh/known_hosts,target=/root/.ssh/known_hosts" \
	--mount "type=bind,source=${DOCKER_SSH_SOCK},target=${DOCKER_SSH_SOCK},readonly" \
	-e "SSH_AUTH_SOCK=${DOCKER_SSH_SOCK}" \
	"golang:latest" bash

# Use SSH instead of HTTPS to clone Git repositories
git config --global url.ssh://git@git.acme.com.insteadOf https://git.acme.com/scm

# Go get private go module behind auth
go get git.acme.com/myteam/mygomodule@latest

[ryancurrah]

Also I could make another contribution to resolve #3042 if a fix for this makes sense in Rancher Desktop.

[jandubois]

Also I could make another contribution to resolve #3042 if a fix for this makes sense in Rancher Desktop.

Yes, we should make this easier; I'm just not sure how much we can automate. Obviously we should have some setting like rdctl set virtual-machine.ssh.forward-agent=true so you don't need the override.yaml file, but is there anything else.

E.g. I think it may make sense to have a fixed location for the socket inside the VM, that would be a symlink to the actual socket. That way you don't need to query for SSH_AUTH_SOCK but could use the fixed name.

An additional benefit would be that the forwarded socket would continue to work if you shutdown and restart Rancher Desktop: the symlink would be updated automatically, and the container would restart and just continue to use the same symlink.

Any other ideas?

[ryancurrah]

@jandubois that approach worked in my testing, see below. Would that be an improvement to lima? Seems like something all lima users could benefit from.

Lima VM:

lima-rancher-desktop:/Users/rcurrah$ ln -f "${SSH_AUTH_SOCK}" /tmp/lima-ssh-auth.sock
lima-rancher-desktop:/Users/rcurrah$ ls -l /tmp/lima-ssh-auth.sock
srwxr-xr-x    2 rcurrah  rcurrah          0 Jan 27 22:07 /tmp/lima-ssh-auth.sock

On My Laptop:

╰─ docker run --rm -it --pull="always" --workdir "$(pwd)" --mount "type=bind,source=$(pwd),target=$(pwd)" --mount "type=bind,source=${HOME}/.ssh/known_hosts,target=/root/.ssh/known_hosts" --mount "type=bind,source=/tmp/lima-ssh-auth.sock,target=/tmp/lima-ssh-auth.sock,readonly" -e "SSH_AUTH_SOCK=/tmp/lima-ssh-auth.sock" --entrypoint sh "alpine/git:latest"

latest: Pulling from alpine/git
Digest: sha256:66b210a97bc07bfd4019826bcd13a488b371a6cbe2630a4b37d23275658bd3f2
Status: Image is up to date for alpine/git:latest

/Users/rcurrah # ssh-add -l
256 SHA256:trZksHt15d8sdhjwdouMYB3a0aQuLtagsxhPdr46RQb7Wg ryancurrah@acme.com (ED25519)

/Users/rcurrah # git clone ssh://git@git.acme.com/MYORG/MYREPO.git
Cloning into 'MYREPO'...
remote: Enumerating objects: 894, done.
remote: Counting objects: 100% (894/894), done.
remote: Compressing objects: 100% (881/881), done.
remote: Total 894 (delta 590), reused 12 (delta 8), pack-reused 0
Receiving objects: 100% (894/894), 236.60 KiB | 560.00 KiB/s, done.
Resolving deltas: 100% (590/590), done.

/Users/rcurrah # echo $?
0