[QUESTION] failing with buildah 1.20.0 (ubuntu-latest)

[jpopelka]

Question

Hello, thank you for this action, it's marvelous.
Today our builds started to fail and I'm wondering what's changed and where can be the cause.

Writing manifest to image destination
Storing signatures
level=error msg="error unmounting /home/runner/.local/share/containers/storage/overlay/0781b6291aebaf9c44638d47503ccc8967e46e541203bd35d79bef4a0566b5ac/merged: invalid argument"
error mounting new container: error mounting build container "b11e0c122471549c9c260843257e96e91b92ed343041b40f9f119667b6979a0f": error creating overlay mount to /home/runner/.local/share/containers/storage/overlay/0781b6291aebaf9c44638d47503ccc8967e46e541203bd35d79bef4a0566b5ac/merged, mount_data=",lowerdir=/home/runner/.local/share/containers/storage/overlay/l/JVFKZCGLHI42F7Y324TMXFQZ4P:/home/runner/.local/share/containers/storage/overlay/l/3PUMEEVI6PIVBZDO2HZF7SXLWO,upperdir=/home/runner/.local/share/containers/storage/overlay/0781b6291aebaf9c44638d47503ccc8967e46e541203bd35d79bef4a0566b5ac/diff,workdir=/home/runner/.local/share/containers/storage/overlay/0781b6291aebaf9c44638d47503ccc8967e46e541203bd35d79bef4a0566b5ac/work,userxattr": invalid argument
level=error msg="exit status 125"

The only difference between the failing one and 20hours old passing previous one I see in logs is the buildah version
1.19.8 (previous, OK) vs. 1.20.0 (failed).

I know it's most likely not your fault, that's why I'm creating a question in case someone else also hits this.

[jpopelka]

Hmm, this still says that ubuntu-latest contains "Buildah 1.19.8".

[divyansh42]

Yeah, they should update that.

[tetchel]

same thing here? #47

Good thing we have the version output

[divyansh42]

same thing here? #47

Good thing we have the version output

Yes

[divyansh42]

FYI: containers/buildah#3120

[divyansh42]

Possible workaround as per buildah troubleshooting that we have did to solve this problem temporarily.

- run: |
    sudo apt-get install fuse-overlayfs
    mkdir -vp ~/.config/containers
    printf "[storage.options]\nmount_program=\"/usr/bin/fuse-overlayfs\"" > ~/.config/containers/storage.conf

[tetchel]

For anyone else coming into this issue, you can also work around this by using runs-on: ubuntu-18.04 for the job that runs buildah. So you'll get an older, working version of buildah while we look for a permanent solution.

Or you can use the workaround above

edit: does not sound like GitHub will fix this on their end, either. actions/runner-images#3080

edit2: fixed in buildah-build@v2

[tetchel]

this is fixed in v2 / v2.4 now.

[tetchel]

if anyone is coming here because they see the warning printed on an ubuntu-18.04 system, don't worry, the action will still work.

however it is recommended to install fuse-overlayfs for better fs performance. upgrading to 20.04 will also remove the warning