Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"ERRO error unmounting" in overlayfs in buildah 1.20, regression #3126

Closed
sshnaidm opened this issue Apr 4, 2021 · 11 comments
Closed

"ERRO error unmounting" in overlayfs in buildah 1.20, regression #3126

sshnaidm opened this issue Apr 4, 2021 · 11 comments
Labels
locked - please file new issue/PR

Comments

@sshnaidm
Copy link
Member

sshnaidm commented Apr 4, 2021

Description

The error appears in Ansible Buildah connection tests after upgrade to 1.20, was passing fine with 1.19/1.18.

Steps to reproduce the issue:

  1. buildah from --name=buildah-container python:2
  2. buildah run buildah-container echo "dddd"

Describe the results you received:

ERRO error unmounting /home/ubuntu/.local/share/containers/storage/overlay/4a4b51751e6725e0d8bce9ada389436b28da79f1c2f4d7a5c4bf3a9a254423c1/merged: invalid argument 
error mounting container "ff861d96509f62046c803bc4a49cb56934b2273a7f9be35f33608f401f351ea3": error mounting build container "ff861d96509f62046c803bc4a49cb56934b2273a7f9be35f33608f401f351ea3": error creating overlay mount to /home/ubuntu/.local/share/containers/storage/overlay/4a4b51751e6725e0d8bce9ada389436b28da79f1c2f4d7a5c4bf3a9a254423c1/merged, mount_data=",lowerdir=/home/ubuntu/.local/share/containers/storage/overlay/l/LYE46KSU2WBBTJYBMCQ6IBXAN7:/home/ubuntu/.local/share/containers/storage/overlay/l/LJZLZRIP5LSTZFUUFHOYBRQYBJ:/home/ubuntu/.local/share/containers/storage/overlay/l/MAJZZ7ZMYTEACJN5NKXCA3SUM2:/home/ubuntu/.local/share/containers/storage/overlay/l/5JKSCKJ33SUJSUTMMPCMHJOIU6:/home/ubuntu/.local/share/containers/storage/overlay/l/SHYSVFHLKYBFOFMWTLLLCLQQGF:/home/ubuntu/.local/share/containers/storage/overlay/l/HX7SKYQPJTCRNS2TKSSVZEXUCB:/home/ubuntu/.local/share/containers/storage/overlay/l/IQJO7ZNOLQALDG3P7XZYRFRIC6:/home/ubuntu/.local/share/containers/storage/overlay/l/FGNVVLIUAUA7IBDZ4TTERZDNIP:/home/ubuntu/.local/share/containers/storage/overlay/l/V46TJBKJQ2SMKKO4VUQIK3EZVY,upperdir=/home/ubuntu/.local/share/containers/storage/overlay/4a4b51751e6725e0d8bce9ada389436b28da79f1c2f4d7a5c4bf3a9a254423c1/diff,workdir=/home/ubuntu/.local/share/containers/storage/overlay/4a4b51751e6725e0d8bce9ada389436b28da79f1c2f4d7a5c4bf3a9a254423c1/work,userxattr": invalid argument
ERRO exit status 125

With buildah version 1.19.4 I have it working:

$ buildah run buildah-container echo "dddd"
dddd

Describe the results you expected:

Output of rpm -q buildah or apt list buildah:

buildah/unknown,now 100:1.20.0-3 amd6

Output of buildah version:

Version:         1.20.0
Go Version:      go1.15.2
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.10.5
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

Output of cat /etc/*release:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Output of uname -a:

Linux ubuntu20-vmtest 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
[storage.options.overlay]
mountopt = "nodev,metacopy=on"
[storage.options.thinpool]
@rhatdan
Copy link
Member

rhatdan commented Apr 5, 2021

Do you have fuse-overlayfs installed?

@nalind
Copy link
Member

nalind commented Apr 5, 2021

Can you post the output of buildah info or buildah info | jq .store? I think we're trying to use the "userxattr" mount option, but that shouldn't be happening if we're using overlay.mount_program=fuse-overlayfs.

@rhatdan
Copy link
Member

rhatdan commented Apr 5, 2021

@nalind I think storage believes that rootless overlay will work, but since the test mount succeeds. And therefore we go a different code path.

@rhatdan
Copy link
Member

rhatdan commented Apr 5, 2021

containers/storage#871

@sshnaidm
Copy link
Member Author

sshnaidm commented Apr 5, 2021

Yes, seems like I have it installed:

$ dpkg -l | grep  fuse-overlayfs
ii  fuse-overlayfs                 1.4.0~1                               amd64        implementation of overlay+shiftfs in FUSE for rootless containers

Also output of buildah info:

$ buildah info
{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "20.04"
        },
        "MemFree": 1402662912,
        "MemTotal": 2083663872,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 4,
        "hostname": "ubuntu20-vmtest",
        "kernel": "5.4.0-70-generic",
        "os": "linux",
        "rootless": true,
        "uptime": "3m 2.92s"
    },
    "store": {
        "ContainerStore": {
            "number": 8
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/home/ubuntu/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 22
        },
        "RunRoot": "/run/user/1000/containers"
    }
}
`

And all info from CI job:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64
Version:         1.20.0
Go Version:      go1.15.2
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.10.5
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 2
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: fv-az189-703
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 121
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.4.0-1043-azure
  linkmode: dynamic
  memFree: 4918923264
  memTotal: 7292149760
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-45fb5-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 2m 2.68s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/runner/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /tmp/podman-run-1001/containers
  volumePath: /home/runner/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.1

{
    "debug": {
        "buildah version": "1.20.0",
        "compiler": "gc",
        "git commit": "",
        "go version": "go1.15.2"
    },
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "ubuntu",
            "version": "20.04"
        },
        "MemFree": 4928339968,
        "MemTotal": 7292149760,
        "OCIRuntime": "crun",
        "SwapFree": 4294963200,
        "SwapTotal": 4294963200,
        "arch": "amd64",
        "cpus": 2,
        "hostname": "fv-az189-703",
        "kernel": "5.4.0-1043-azure",
        "os": "linux",
        "rootless": true,
        "uptime": "2m 3.7s"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/home/runner/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/var/tmp/containers-user-1001/containers"
    }
}
conmon version 2.0.27
commit: 
runc version 1.0.0-rc93
commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.4.3

@rhatdan
Copy link
Member

rhatdan commented Apr 5, 2021

@sshnaidm Could you try to build podman with containers/storage#871 and see if it fixes your issue?

@sshnaidm
Copy link
Member Author

sshnaidm commented Apr 5, 2021

I'm afraid I will need a guide how to build buildah with a specific containers/storage PR.
Before that I was just cloning buildah repo and running make inside.

@nalind
Copy link
Member

nalind commented Apr 6, 2021

@sshnaidm If you're at the top level of the source tree, given the branch the PR's requesting we merge from, go mod edit -replace github.com/containers/storage=github.com/rhatdan/storage@userxattr && go mod vendor should do it.

@tetchel
Copy link
Contributor

tetchel commented Apr 7, 2021

I think this is the same issue as #3120

@sshnaidm
Copy link
Member Author

sshnaidm commented Apr 7, 2021

@rhatdan I can confirm it works now: (only error message appears)

$ git clone https://github.com/containers/buildah
$ cd buildah/
$ go mod edit -replace github.com/containers/storage=github.com/rhatdan/storage@userxattr && go mod vendor
$ make
$ ./bin/buildah -v
buildah version 1.20.1-dev (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)

$ ./bin/buildah from --name=buildah-container-new python:2
ERRO[0000] overlay test mount with multiple lowers failed, but succeeded with a single lower 
buildah-container-new
$ ./bin/buildah run buildah-container-new echo "dddd"
dddd

@sshnaidm
Copy link
Member Author

I think it can be closed, I use the workaround and fix is merged.
Thanks all.

@rhatdan rhatdan closed this as completed Apr 19, 2021
@wanjunlei wanjunlei mentioned this issue Nov 30, 2021
Closed
4 tasks
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
locked - please file new issue/PR
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nalind @rhatdan @sshnaidm @tetchel