colima start --vm-type=vz failing when installed from Nix

[jonathanlking]

Description

Running colima start --vm-type=vz fails to start:

$ colima start --vm-type=vz --very-verbose
TRAC[0000] cmd ["limactl" "info"]                       
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]     
INFO[0000] starting colima                              
INFO[0000] runtime: docker                              
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]     
TRAC[0000] cmd ["pgrep" "oahd"]                         
INFO[0000] starting ...                                  context=vm
TRAC[0000] cmd ["limactl" "start" "colima"]             
> "Non-strict YAML is deprecated and will be unsupported in a future version of Lima" comment="main file \"/Users/jonathan/.lima/colima/lima.yaml\"" error="[80:1] unknown field \"rosetta\"\n      77 |       script: readlink /sbin/fstrim || fstrim -a\n      78 |     - mode: system\n      79 |       script: stat /sys/class/block/vda/queue/write_cache && echo 'write through' > /sys/class/block/vda/queue/write_cache\n    > 80 | rosetta:\n           ^\n      81 |     enabled: true\n      82 |     binfmt: true"
> Using the existing instance "colima"
> errors inspecting instance: [field `mountType` must be "reverse-sshfs" or "9p" , got "virtiofs"]
FATA[0000] error starting vm: error at 'starting': exit status 1 

However colima start works as expected.
When installed through Homebrew it works (see additional context).

Version

Colima Version:

colima version 0.5.2
git commit: 6b5b6fe

Lima Version: limactl version 0.13.0
Qemu Version: qemu-img version 7.2.0

Nixpkgs: https://github.com/NixOS/nixpkgs/blob/2d1dee4f6a6dac81b3bc4a3e7bdf76c9e5ec4008/pkgs/applications/virtualization/colima/default.nix

Operating System

• macOS Intel <= 12 (Monterrey)
• macOS Intel >= 13 (Ventura)
• macOS M1 <= 12 (Monterrey)
• macOS M1 >= 13 (Ventura)
• Linux

Output of colima status

FATA[0000] colima is not running

Reproduction Steps

1. nix-shell -p colima -p qemu -p lima -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/2d1dee4f6a6dac81b3bc4a3e7bdf76c9e5ec4008.tar.gz
2. colima delete (only if you've already run colima before with different settings)
3. colima start --vm-type=vz

Expected behaviour

VM should start.

Additional context

When installed with Homebrew:

Colima Version:

colima version 0.5.2
git commit: 6b5b6fe0540e708f0c9d6e8919fab292c671fc72

Lima Version: limactl version 0.14.2
Qemu Version: qemu-img version 7.2.0

$ colima start --vm-type=vz --verbose
INFO[0000] starting colima                              
INFO[0000] runtime: docker                              
INFO[0000] creating and starting ...                     context=vm
> Terminal is not available, proceeding without opening an editor
> "Attempting to download the image from \"https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.5.0-2/alpine-lima-clm-3.16.2-aarch64.iso\"" digest="sha512:06abfa8c9fd954f8bfe4ce226bf282dd06e9dfbcd09f57566bf6c20809beb5a3367415b515e0a65d6a1638ecfd3a3bb3fb6d654dee3d72164bd0279370448507"
> Using cache "/Users/jonathan/Library/Caches/lima/download/by-url-sha256/c37acb6308026b2fe12f6c0ef3371f690b3e33ee6b5d37d5dc68684f8fd5ee52/data"
> [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/jonathan/.lima/colima/serial.log")
> [hostagent] Setting up Rosetta share
> SSH Local Port: 58153
> [hostagent] new connection from  to 
> [hostagent] Waiting for the essential requirement 1 of 3: "ssh"
> [hostagent] [VZ] - vm state change: running
> [hostagent] 2023/01/03 14:31:22 tcpproxy: for incoming conn 127.0.0.1:58155, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: no route
> [hostagent] Waiting for the essential requirement 1 of 3: "ssh"
> [hostagent] The essential requirement 1 of 3 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 3: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 3 is satisfied
> [hostagent] Waiting for the essential requirement 3 of 3: "the guest agent to be running"
> [hostagent] The essential requirement 3 of 3 is satisfied
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/jonathan/.colima/default/docker.sock" (host)
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/jonathan/.colima/docker.sock" (host)
> [hostagent] The final requirement 1 of 1 is satisfied
> READY. Run `limactl shell colima` to open the shell.
> stat: can't stat '/proc/sys/fs/binfmt_misc/rosetta': No such file or directory
>   File: /proc/sys/fs/binfmt_misc/qemu-x86_64
>   Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
> Device: 31h/49d	Inode: 9550        Links: 1
> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2023-01-03 14:31:30.300000001 +0000
> Modify: 2023-01-03 14:31:30.300000001 +0000
> Change: 2023-01-03 14:31:30.300000001 +0000
INFO[0014] provisioning ...                              context=docker
> colima
> Successfully created context "colima"
> colima
INFO[0014] starting ...                                  context=docker
>  * /var/log/docker.log: creating file
>  * /var/log/docker.log: correcting owner
>  * Starting Docker Daemon ... [ ok ]
INFO[0019] done

[jonathanlking]

Ah, I think the problem might be that Nixpkgs has limactl version 0.13.0 when >= 0.14 is required for vz.

Apologies I only spotted this right after creating the issue! 🤦

This release adds an experimental support for Apple's Virtualization.framework aka vz.

https://github.com/lima-vm/lima/releases/tag/v0.14.0

[abiosoft]

@jonathanlking I'm not very skilled with Nix, otherwise would've modified Colima to only use a specific version of Lima.

The issue with Nix is due to the fixed dependencies. I think a new re-release on Nix should resolve it as newer versions of Lima is now on Nix.

[jonathanlking]

Thanks for such a quick reply!

Unfortunately I think master https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/virtualization/lima/default.nix is still on 0.13.0.

I'm creating an overlay now to test if upgrading to 0.14.2 fixes this issue (which I'm hoping/expecting it will!).

I'll add the overlay below (in case anyone else runs into this issue) and will try to open a PR on nixpkgs to bump the version there too.

[jonathanlking]

It looks like this issue is being discussed/worked on already 👀 (and I haven't got any further than them)

• colima: 0.5.0 -> 0.5.2 NixOS/nixpkgs#208094
• lima: 0.13.0 -> 0.14.2 NixOS/nixpkgs#206285

I'm happy for this issue to be closed, as it appears to be a Nix packaging issue, rather than anything wrong with colima.
Equally happy happy for it to be left open as a signpost to others.

[tcastelly]

I still have the same behavior with up-to-date colima and lima.

colima start --vm-type=vz

> "Non-strict YAML is deprecated and will be unsupported in a future version of Lima" comment="main file \"/Users/tcy/.lima/colima-try/lima.yaml\"" error="[71:1] unknown field \"rosetta\"\n      68 |       script: readlink /sbin/fstrim || fstrim -a\n      69 |     - mode: system\n      70 |       script: stat /sys/class/block/vda/queue/write_cache && echo 'write through' > /sys/class/block/vda/queue/write_cache\n    > 71 | rosetta:\n           ^\n      72 |     enabled: true\n      73 |     binfmt: true"
> field `mountType` must be "reverse-sshfs" or "9p" , got "virtiofs"

OSX: 13.1 (M1 max)
colima version 0.5.2
limactl version 0.14.2

---

edit:
The error message is different now

Terminal is not available, proceeding without opening an editor
> vm driver 'vz' needs macOS 13 or later (Hint: try recompiling Lima if you are seeing this error on macOS 13)
FATA[0000] error starting vm: error at 'creating and starting': exit status 1

[eliasdaler]

Same here. What's strange is that the YAML file mentioned in the comment (main file) doesn't exist.

[jbgosselin]

Hi there, I recently worked on these nix packages to make them up to date.
Colima is up to date, but it is a bit more complex with Lima.
Due to the nature on how packages are build in nix, lima is built in a sandbox that doesn't contains all the recent macos libraries, including vz support. Another contributor added a lima-bin package that pulls the github hosted pre-compiled binaries for lima that are compiled with vz support.
In order to use them you need to override the lima package used by colima. in my home-manager config it looks a little bit like this.

{
  nixpkgs.overlays = [
    (self: super: {
      lima = super.lima-bin;
    })
  ];

  home.packages = with pkgs; [
    ...
    colima
    ...
  ];
  ...
}

Once updated macos toolchain will be updated with vz support, this overlay will not be necessary anymore, but I can't tell when this will be done.

[cecelot]

This is fixed in nixpkgs-unstable with NixOS/nixpkgs#212980