buildah isn't working in ubuntu-20.04 (version: 20210330.1)

[urfin78]

Description

The latest 1.20.0 release of buildah isn't working inside the virtual environment from the github-actions. Maybe it's related to this
buildah issue: containers/buildah#3120

Area for Triage:
Containers

Question, Bug, or Feature?:
Bug

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

Image version
20210330.1

Expected behavior
Image build with buildah suceeds.

Actual behavior
Image build with buildah fails with

STEP 1: FROM debian:buster-slim
Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Getting image source signatures
Copying blob sha256:75646c2fb4101d306585c9b106be1dfa7d82720baabe1c75b64d759ea8adf341
Copying config sha256:2d7852f9772998f25c37ba35c7767bb995b7bdaf9f7ca0cc4a8cc3fe03dc91dd
Writing manifest to image destination
Storing signatures
level=error msg="error unmounting /home/runner/.local/share/containers/storage/overlay/fce8007137e91d9ab3a0da19119413f9d4c25be13b74ea5ef278905603f52df5/merged: invalid argument"
error mounting new container: error mounting build container "285487a42a975e50ac0ff80dd7f77de4115495867faf3334e9d23700a11eb5cd": error creating overlay mount to /home/runner/.local/share/containers/storage/overlay/fce8007137e91d9ab3a0da19119413f9d4c25be13b74ea5ef278905603f52df5/merged, mount_data=",lowerdir=/home/runner/.local/share/containers/storage/overlay/l/6C2IBJ23INYYCXOX573ZQH5EIX,upperdir=/home/runner/.local/share/containers/storage/overlay/fce8007137e91d9ab3a0da19119413f9d4c25be13b74ea5ef278905603f52df5/diff,workdir=/home/runner/.local/share/containers/storage/overlay/fce8007137e91d9ab3a0da19119413f9d4c25be13b74ea5ef278905603f52df5/work,userxattr": invalid argument
level=error msg="exit status 125"
Error: Process completed with exit code 125.

Repro steps
Run the following workflow with the simplest Dockerfile of your choice:

  build:
    name: Build Dockerfile
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: build manual
      run: |
        buildah bud -t test .

[miketimofeev]

Hi @urfin78!
Could you please try the following workaround to build the previous buildah version in runtime? (it takes about 1.5-2min)

    - name: Install buildah 1.19
      shell: bash
      run: |
        sudo apt-get remove buildah
        sudo apt-get -y install software-properties-common
        sudo add-apt-repository -y ppa:alexlarsson/flatpak
        sudo apt-get -y -qq update
        sudo apt-get -y install bats libapparmor-dev libdevmapper-dev libglib2.0-dev libgpgme11-dev libseccomp-dev libselinux1-dev skopeo-containers go-md2man
        sudo apt-get -y install golang-1.13
        mkdir ~/buildah
        cd ~/buildah
        export GOPATH=`pwd`
        git clone https://github.com/containers/buildah ./src/github.com/containers/buildah
        cd ./src/github.com/containers/buildah
        git checkout release-1.19
        PATH=/usr/lib/go-1.13/bin:$PATH make runc all SECURITYTAGS="apparmor seccomp"
        sudo make install install.runc
        buildah version

[urfin78]

Building and using the old version on the same runner seems to work:
https://github.com/urfin78/playground/runs/2247288346?check_suite_focus=true

Run buildah bud -t test .
##[debug]/usr/bin/bash -e /home/runner/work/_temp/8528ec06-c6c7-4596-adcc-4ad6d6014c65.sh
STEP 1: FROM debian:buster-slim
Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Getting image source signatures
Copying blob sha256:75646c2fb4101d306585c9b106be1dfa7d82720baabe1c75b64d759ea8adf341
Copying config sha256:2d7852f9772998f25c37ba35c7767bb995b7bdaf9f7ca0cc4a8cc3fe03dc91dd
Writing manifest to image destination
Storing signatures
STEP 2: RUN echo "test"
test
STEP 3: COMMIT test
Getting image source signatures
Copying blob sha256:346fddbbb0ff19bdb026afb053df490c0c73981a8a985882cf8ee3d716735e87
Copying blob sha256:f16431ff28275933c6f0c5d87c6065e9d6344c043856aa48bc542f646d2c350b
Copying config sha256:6b7698e306e67dc1a1d3585b926e1bfea5ab00b89c0a764e540af970d5e20144
Writing manifest to image destination
Storing signatures

[miketimofeev]

@urfin78 great, let's wait for an answer in containers/buildah#3120

[kpfleming]

Same problem for me, the workaround posted above works in my workflows.

[miketimofeev]

@urfin78 @kpfleming this workaround should be much faster
redhat-actions/buildah-build#45 (comment)

[tetchel]

The Ubuntu-20.04 runner could be modified similarly to the workaround mentioned redhat-actions/buildah-build#45 (comment).
If the virtual environment installed and used fuse-overlayfs by default, that would fix this issue for all users.

/etc/containers/storage.conf would have to be modified to use mount_program=fuse-overlayfs.

https://github.com/containers/buildah/blob/master/vendor/github.com/containers/storage/storage.conf

edit: Looks like this is the script that would have to change

edit: storage.conf not containers.conf

[miketimofeev]

@urfin78 I'm going to close the issue since the problem is external. Please feel free to contact us if you have any concerns.
Thank you!

[urfin78]

@miketimofeev If it's expected behaviour from the buildah side, that container configuration should be mount_program=fuse-overlayfs when using rootless mode and virtual environments in github-actions run as user, than this has to be changed here.
At this state nobody will change a thing (the buildah bug report containers/buildah#3120 was also closed) and we will have to use workarounds to get the old functionality back.

[tetchel]

I agree with @urfin78 that this should be fixed in the virtual environment.

If I had this issue on a system under my control I would edit storage.conf as I mentioned above and the problem would go away. #3080 (comment)

Since Actions users cannot make persistent changes on the GitHub environments, they have to edit that file in every workflow that uses buildah. And there's no way for them to know how to do that without finding this issue.

[miketimofeev]

@urfin78 @tetchel sorry, I might have missed the idea. What config file do we need to change from our side to fix it? It would be great if you can submit a PR with the fix.

[maxim-lobanov]

it is unpredictable how other tools and containers will be affected If we patch storage.conf on image side

[urfin78]

@maxim-lobanov storage.conf is used by tools using the shared container storage - at least the redhat container tools podman, buildah and skopeo. I don't know if any other software use this config. Maybe some k8s related tools?
Beside that, as I read in the different issues about this problem, it seems, that the other linux distributions, which provide these packages have the fuse-overlayfs package as dependency installed. And if I understand it right, when fuse-overlayfsis installed, the mount_program option is automatically set to mount_program=fuse-overlayfs.

Setting this option is also recommended for rootless setups here https://github.com/containers/storage/blob/master/docs/containers-storage.conf.5.md

But surely it has to be tested properly.

[miketimofeev]

@urfin78 in that case maybe it's a better solution to install fuse-overlayfs package? Or it's not possible for ubuntu?

[urfin78]

@miketimofeev It was just a guess from reading one of issues regarding the problem: containers/buildah#1745 (comment)
According to that statement, fuse-overlayfs has to be installed before podman (i think also before the other container tools buildah and skopeo). As it genereates the correct config if the package is available. But I haven't tested it myself.

[tetchel]

As far as I can tell, fuse-overlayfs is already installed on the ubuntu runners. Installing fuse-overlayfs first as suggested may work and is absolutely worth a try.

Otherwise, the storage.conf can be edited as per #3080 (comment) to set it as the default.

[miketimofeev]

@urfin78 @tetchel I'll try to install libfuse3-dev before podman and the other tools installation and get back with the results

[miketimofeev]

Nope, it didn't help, the option is still disabled:

cat /etc/containers/storage.conf | grep mount
# Path to an helper program to use for mounting the file system instead of mounting it
#mount_program = "/usr/bin/fuse-overlayfs"

[tetchel]

FWIW, we added a workaround to https://github.com/redhat-actions/buildah-build so if you use our action, buildah will work without the steps above.