Remote Containers fails to start when containerd experimental Docker feature is enabled

[stazz]

• VSCode Version: 1.71.2 (Universal)
• Local OS Version: Darwin arm64 21.6.0
• Remote OS Version: Debian GNU/Linux 11 (bullseye) 5.10.124-linuxkit ( base image mcr.microsoft.com/vscode/devcontainers/javascript-node:16-bullseye )
• Remote Extension/Connection Type: Containers
• Logs:

<after building the image>
[205890 ms] Start: Run: docker events --format {{json .}} --filter event=start
[205891 ms] Start: Starting container
[205891 ms] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount source=/Users/me/work/clients/the_client/git,target=/Users/me/work/clients/the_client/git,type=bind,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l devcontainer.local_folder=/Users/me/work/clients/the_client/git -e AWS_DEFAULT_REGION=eu-west-1 -e AWS_ACCESS_KEY_ID=foobar -e AWS_SECRET_ACCESS_KEY=foobar --network the_client-nw --entrypoint /bin/sh vsc-git-955b89a1bd3c6bb620098480f09b8a21 -c echo Container started
Container started
[207336 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/Users/me/work/clients/the_client/git
[207409 ms] Start: Run: docker inspect --type container 3af536a06f0b
[207462 ms] Start: Inspecting container
[207462 ms] Start: Run: docker inspect --type container 3af536a06f0bcfc7b098862bec7fdef8e767af6c4d9a309af6fdf73aa06ccf2b
[207510 ms] Start: Run in container: /bin/sh
[207511 ms] Start: Run in container: uname -m
[207566 ms] Shell server terminated (code: 126, signal: null)
[207566 ms] GetResourcePath: BaseFS of container 3af536a06f0bcfc7b098862bec7fdef8e767af6c4d9a309af6fdf73aa06ccf2b is unexpectedly nil
[207566 ms] Start: Run in container: cat /etc/passwd
[207566 ms] Stdin closed!
[207567 ms] Error: An error occurred setting up the container.
[207567 ms]     at SF (/Users/me/.vscode/extensions/ms-vscode-remote.remote-containers-0.251.0/dist/spec-node/devContainersSpecCLI.js:257:2186)
[207567 ms]     at pb (/Users/me/.vscode/extensions/ms-vscode-remote.remote-containers-0.251.0/dist/spec-node/devContainersSpecCLI.js:257:2124)
[207567 ms]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[207567 ms]     at async DF (/Users/me/.vscode/extensions/ms-vscode-remote.remote-containers-0.251.0/dist/spec-node/devContainersSpecCLI.js:263:2006)
[207567 ms]     at async to (/Users/me/.vscode/extensions/ms-vscode-remote.remote-containers-0.251.0/dist/spec-node/devContainersSpecCLI.js:263:3110)
[207567 ms]     at async Ak (/Users/me/.vscode/extensions/ms-vscode-remote.remote-containers-0.251.0/dist/spec-node/devContainersSpecCLI.js:383:8108)
[207567 ms]     at async Ok (/Users/me/.vscode/extensions/ms-vscode-remote.remote-containers-0.251.0/dist/spec-node/devContainersSpecCLI.js:383:7864)
[207569 ms] Exit code 1
[207569 ms] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --ms-enable-electron-run-as-node /Users/me/.vscode/extensions/ms-vscode-remote.remote-containers-0.251.0/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/me/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/me/work/clients/the_client/git --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/me/work/clients/the_client/git --log-level debug --log-format json --config /Users/me/work/clients/the_client/git/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[207569 ms] Exit code 1

Steps to Reproduce:

1. Enable experimental containerd feature of Docker Desktop
2. Use Remote-Containers: Open Folder in Container... command in VS Code to boot up container server and run VS Code via it

Does this issue occur when you try this locally?: Yes
Does this issue occur when you try this locally and all extensions are disabled?: Yes

There is issue like this opened in Docker (for Mac) repo, but I think it is more suitable to be here.
I also have "New Virtualization framework" and "VirtioFS directory sharing" experimental features of Docker (for Mac) enabled, but disabling them did not seem to have any effect on this crash.

[stazz]

Tested this also on Windows computer. The similar error occurs when containerd experimental feature is enabled in Docker Desktop:

GetResourcePath: BaseFS of container c4a8f3c6d8a784483a9d2ea6cba99a1f6717107f921221e9fc076dfc669db49d is unexpectedly nil

[cq01]

Same problem appears in windows 11 with wsl2.

[chrmarti]

This seems to be an issue with Docker's experimental feature. I suggest to reopen docker/for-mac#6468 or open a new issue for Docker.

[stazz]

@chrmarti That is doable, too, but is it possible to get any more info of how this situation looks like from within VS Code? Like the error message talks about BaseFS of container to be unexpectedly nil. Is it possible to get information on how this BaseFS is extracted? I tried docker inspect the container, but found no mention on BaseFS.

I am also thinking - perhaps when containerd feature is active, BaseFS maybe is valid to be nil? The feature discusses about "lazy-pulling" of images, so I am not sure if BaseFS being nil is actually a bug - perhaps it is a signal to perform some additional operation on container?

[stazz]

So I did comparison of docker inspect output for normal vs containerd containers. The findings are:

1. Image property is empty string for containerd -enabled container, while it is hash string for "normal" container
2. Driver property is stargz for containerd -enabled container, while it is overlay2 for "normal" container
3. ExecIDs property is null for containerd -enabled container, while it is an array of strings for "normal" container
4. GraphDriver property is {"name": "stargz", "data": null} for containerd -enabled container, while it is {"Name":"overlay2","Data":{"LowerDir":"path","MergedDir":"path","UpperDir":"path","WorkDir":"path"}} for "normal" container

Does this give any additional help on the issue? Is the GraphDriver.data property the one which referred to as BaseFS in error message?

[chrmarti]

Thanks @stazz for looking into this. We are not using BaseFS in the VS Code extension. This must be something Docker is using, so I don't know if nil is valid.

[stazz]

@chrmarti Ahh, okay! Makes sense then that probably isn't related to VSCode extension. :) But at which point exactly this happens? Is VSCode invoking some docker xyz CLI command, or are you using Docker client library, which then in turn fails to this error on some function call?

This info would help probably also with the linked docker/for-win#13002 if that is indeed correct place to track this error.

[chrmarti]

@stazz Replied there. Thanks.

[stazz]

Thank you @chrmarti ! I guess this issue can be closed then?

[chrmarti]

Let's keep it open to track the issue as it is surfacing in Dev Containers (previously named Remote-Containers).

[ragokan]

Hello, I am also having the same issue. I am using MacOS Monterey latest with M1 chip. There is no problem running the container in regular mode, from the terminal.

These are my outputs:

Error Logs

[2022-10-10T08:09:36.992Z] Start: Run in container: /bin/sh
[2022-10-10T08:09:36.994Z] Start: Run in container: uname -m
[2022-10-10T08:09:37.078Z] Stop (86 ms): Run in container: /bin/sh
[2022-10-10T08:09:37.078Z] Shell server terminated (code: 126, signal: null)
[2022-10-10T08:09:37.078Z] GetResourcePath: BaseFS of container b408891909b4c794834d6d1de86614e18ce6f788bf666c95900aa56df878f318 is unexpectedly nil
[2022-10-10T08:09:37.078Z] Start: Run in container: cat /etc/passwd
[2022-10-10T08:09:37.078Z] Stdin closed!
[2022-10-10T08:09:37.079Z] Error: An error occurred setting up the container.
[2022-10-10T08:09:37.079Z]     at cD (/Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:297:1327)
[2022-10-10T08:09:37.079Z]     at sC (/Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:297:1265)
[2022-10-10T08:09:37.079Z]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-10-10T08:09:37.079Z]     at async bD (/Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:303:2020)
[2022-10-10T08:09:37.079Z]     at async ys (/Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:303:3182)
[2022-10-10T08:09:37.079Z]     at async UL (/Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:423:10319)
[2022-10-10T08:09:37.079Z]     at async ML (/Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:423:10075)
[2022-10-10T08:09:37.081Z] Stop (555 ms): Run: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --ms-enable-electron-run-as-node /Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/ragokan/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/ragokan/Projects/knowy-editor --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/ragokan/Projects/knowy-editor --log-level debug --log-format json --config /Users/ragokan/Projects/knowy-editor/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-10-10T08:09:37.081Z] Exit code 1
[2022-10-10T08:09:37.082Z] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --ms-enable-electron-run-as-node /Users/ragokan/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/ragokan/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/ragokan/Projects/knowy-editor --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/ragokan/Projects/knowy-editor --log-level debug --log-format json --config /Users/ragokan/Projects/knowy-editor/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-10-10T08:09:37.082Z] Exit code 1

[NotMyself]

Hello, I have a user that is running into this issue. She is running MacOS Monterey on M1. I am waiting on confirmation of the state of the Experimental Feature.

Update: Confirmed she had the feature enabled. Disabling it resolved the issue.

This is her log:

Error Logs

[2022-10-18T21:33:05.485Z] Dev Containers 0.255.4 in VS Code 1.72.2 (d045a5eda657f4d7b676dedbfa7aab8207f8a075).
[2022-10-18T21:33:05.485Z] Start: Resolving Remote
[2022-10-18T21:33:05.496Z] Setting up container for folder or workspace: /Users/user/src/repository
[2022-10-18T21:33:05.498Z] Start: Check Docker is running
[2022-10-18T21:33:05.499Z] Start: Run: docker version --format {{.Server.APIVersion}}
[2022-10-18T21:33:05.588Z] Stop (89 ms): Run: docker version --format {{.Server.APIVersion}}
[2022-10-18T21:33:05.588Z] Server API version: 1.43
[2022-10-18T21:33:05.588Z] Stop (90 ms): Check Docker is running
[2022-10-18T21:33:05.589Z] Start: Run: docker volume ls -q
[2022-10-18T21:33:05.648Z] Stop (59 ms): Run: docker volume ls -q
[2022-10-18T21:33:05.663Z] Start: Run: docker ps -q -a --filter label=vsch.local.folder=/Users/user/src/repository --filter label=vsch.quality=stable
[2022-10-18T21:33:05.732Z] Stop (69 ms): Run: docker ps -q -a --filter label=vsch.local.folder=/Users/user/src/repository --filter label=vsch.quality=stable
[2022-10-18T21:33:05.732Z] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/Users/user/src/repository
[2022-10-18T21:33:05.819Z] Stop (87 ms): Run: docker ps -q -a --filter label=devcontainer.local_folder=/Users/user/src/repository
[2022-10-18T21:33:05.819Z] Start: Run: docker inspect --type container b104bf297d89
[2022-10-18T21:33:05.896Z] Stop (77 ms): Run: docker inspect --type container b104bf297d89
[2022-10-18T21:33:05.897Z] Start: Run: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --ms-enable-electron-run-as-node /Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/user/src/repository --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/user/src/repository --log-level debug --log-format json --config /Users/user/src/repository/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-10-18T21:33:06.040Z] (node:6519) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[2022-10-18T21:33:06.040Z] (Use `Code Helper --trace-deprecation ...` to show where the warning was created)
[2022-10-18T21:33:06.041Z] @devcontainers/cli 0.20.0. Node.js v16.14.2. darwin 21.6.0 arm64.
[2022-10-18T21:33:06.041Z] Start: Run: docker buildx version
[2022-10-18T21:33:06.139Z] Stop (98 ms): Run: docker buildx version
[2022-10-18T21:33:06.139Z] github.com/docker/buildx v0.9.1 ed00243a0ce2a0aee75311b06e32d33b44729689
[2022-10-18T21:33:06.139Z] 
[2022-10-18T21:33:06.139Z] Start: Resolving Remote
[2022-10-18T21:33:06.141Z] Start: Run: docker-compose version --short
[2022-10-18T21:33:06.414Z] Stop (273 ms): Run: docker-compose version --short
[2022-10-18T21:33:06.414Z] Docker Compose version: 1.29.2
[2022-10-18T21:33:06.414Z] Start: Run: docker ps -q -a --filter label=com.docker.compose.project=labs-node-working-with-tenants_devcontainer --filter label=com.docker.compose.service=lab
[2022-10-18T21:33:06.486Z] Stop (72 ms): Run: docker ps -q -a --filter label=com.docker.compose.project=labs-node-working-with-tenants_devcontainer --filter label=com.docker.compose.service=lab
[2022-10-18T21:33:06.487Z] Start: Run: docker inspect --type container b104bf297d89
[2022-10-18T21:33:06.544Z] Stop (57 ms): Run: docker inspect --type container b104bf297d89
[2022-10-18T21:33:06.545Z] Start: Inspecting container
[2022-10-18T21:33:06.545Z] Start: Run: docker inspect --type container b104bf297d89ceadc8a60abef6b5fd8cfd42fc2d3dac648e21a4cfab8ffad6c2
[2022-10-18T21:33:06.606Z] Stop (61 ms): Run: docker inspect --type container b104bf297d89ceadc8a60abef6b5fd8cfd42fc2d3dac648e21a4cfab8ffad6c2
[2022-10-18T21:33:06.606Z] Stop (61 ms): Inspecting container
[2022-10-18T21:33:06.607Z] Start: Run in container: /bin/sh
[2022-10-18T21:33:06.608Z] Start: Run in container: uname -m
[2022-10-18T21:33:06.688Z] Stop (81 ms): Run in container: /bin/sh
[2022-10-18T21:33:06.688Z] Shell server terminated (code: 126, signal: null)
[2022-10-18T21:33:06.689Z] GetResourcePath: BaseFS of container b104bf297d89ceadc8a60abef6b5fd8cfd42fc2d3dac648e21a4cfab8ffad6c2 is unexpectedly nil
[2022-10-18T21:33:06.688Z] Start: Run in container: cat /etc/passwd
[2022-10-18T21:33:06.688Z] Stdin closed!
[2022-10-18T21:33:06.689Z] Error: An error occurred setting up the container.
[2022-10-18T21:33:06.689Z]     at _se (/Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1812:2820)
[2022-10-18T21:33:06.689Z]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-10-18T21:33:06.689Z]     at async Xse (/Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1874:2203)
[2022-10-18T21:33:06.689Z]     at async Uf (/Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1874:3193)
[2022-10-18T21:33:06.689Z]     at async Cae (/Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1994:10350)
[2022-10-18T21:33:06.689Z]     at async yae (/Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1994:10104)
[2022-10-18T21:33:06.690Z] Stop (793 ms): Run: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --ms-enable-electron-run-as-node /Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/user/src/repository --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/user/src/repository --log-level debug --log-format json --config /Users/user/src/repository/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-10-18T21:33:06.690Z] Exit code 1
[2022-10-18T21:33:06.693Z] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --ms-enable-electron-run-as-node /Users/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/user/src/repository --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/user/src/repository --log-level debug --log-format json --config /Users/user/src/repository/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-10-18T21:33:06.693Z] Exit code 1

[stazz]

Confirmed to be fixed by Docker Desktop 4.13.0: docker/for-win#13002 (comment) . Worked for me too.