Error: failed to get filesystem from image: removing whiteout operation not permitted #1073

tblaisot · 2020-02-23T18:36:11Z

Actual behavior

When using kaniko via Jenkins in a k8s cluster to build an image which contain whiteout file, got this error.

Resolved base name nginxinc/nginx-unprivileged:stable-alpine to nginxinc/nginx-unprivileged:stable-alpine 
Using dockerignore file: /home/jenkins/agent/workspace/mptabilite-front_feature_jenkins/.dockerignore 
Resolved base name nginxinc/nginx-unprivileged:stable-alpine to nginxinc/nginx-unprivileged:stable-alpine 
Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
Built cross stage deps: map[]                
Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
Unpacking rootfs as cmd COPY /dist /usr/share/nginx/html requires it. 
error building image: error building stage: failed to get filesystem from image: removing whiteout etc/nginx/.wh..wh..opq: fstatat /etc/nginx/.wh..opq: operation not permitted

Expected behavior
Expect no error
Note:
Reproducing the bug in a local docker install give no error
I have tried with version 0.9.0 which works and 1.16.0 which don't work but no other version beetween.

To Reproduce
Steps to reproduce the behavior:

Launch a kaniko image on a k8s cluster using /busybox/cat as the entrypoint
Get a shell into the image (/busybox/sh)
Create a Dockerfile containing only one line FROM nginxinc/nginx-unprivileged:stable-alpine
Launch the /kaniko/executor process

Additional Information

Dockerfile

FROM nginxinc/nginx-unprivileged:stable-alpine

Build Context
None
Kaniko Image (fully qualified with digest)
debug-1.17.1

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use `--cache` flag
Please check if your dockerfile is a multistage dockerfile

The text was updated successfully, but these errors were encountered:

cvgw · 2020-02-25T17:01:54Z

This is most likely the same issues as #1039, can you try tags a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79

dren-dk · 2020-02-26T14:30:11Z

Same thing fails for me with: gcr.io/kaniko-project/executor:debug-v0.17.1 and it's reproducible on all versions of Kaniko (I gave up at 0.10.0).

Using this Dockerfile:

FROM gcr.io/kaniko-project/executor:debug-v0.17.1
RUN ["/busybox/sh", "-c", "ls -l"]

I get:

/kaniko/context # /kaniko/executor --context /kaniko/context --verbosity debug --no-push
DEBU[0000] Copying file /kaniko/context/Dockerfile to /kaniko/Dockerfile 
DEBU[0000] Skip resolving path /kaniko/Dockerfile       
DEBU[0000] Skip resolving path /kaniko/context          
DEBU[0000] Skip resolving path /cache                   
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug-v0.17.1 to gcr.io/kaniko-project/executor:debug-v0.17.1 
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug-v0.17.1 to gcr.io/kaniko-project/executor:debug-v0.17.1 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug-v0.17.1 
DEBU[0000] No file found for cache key sha256:bcfcad6b54e9368079232094ed9f8332e5bc642aed77a6b0c1f15fee2b8b13df stat /cache/sha256:bcfcad6b54e9368079232094ed9f8332e5bc642aed77a6b0c1f15fee2b8b13df: no such file or directory 
DEBU[0000] Image gcr.io/kaniko-project/executor:debug-v0.17.1 not found in cache 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug-v0.17.1 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/kaniko-project/executor:debug-v0.17.1 
DEBU[0002] No file found for cache key sha256:bcfcad6b54e9368079232094ed9f8332e5bc642aed77a6b0c1f15fee2b8b13df stat /cache/sha256:bcfcad6b54e9368079232094ed9f8332e5bc642aed77a6b0c1f15fee2b8b13df: no such file or directory 
DEBU[0002] Image gcr.io/kaniko-project/executor:debug-v0.17.1 not found in cache 
INFO[0002] Retrieving image manifest gcr.io/kaniko-project/executor:debug-v0.17.1 
INFO[0002] Unpacking rootfs as cmd RUN ["/busybox/sh", "-c", "ls -l"] requires it. 
DEBU[0002] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/rdma false} {/dev/mqueue false} {/workspace false} {/busybox false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/dev/console false} {/proc/asound false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/kcore false} {/proc/timer_list false} {/proc/sched_debug false} {/sys/firmware false}] 
DEBU[0003] Not adding /kaniko because it is whitelisted 
DEBU[0003] Not adding /kaniko/executor because it is whitelisted 
DEBU[0003] Not adding /kaniko/warmer because it is whitelisted 

-snip-

DEBU[0004] Not adding /kaniko because it is whitelisted 
DEBU[0004] Not adding /kaniko/ssl because it is whitelisted 
DEBU[0004] Whiting out /kaniko/ssl/.wh..wh..opq         
error building image: error building stage: failed to get filesystem from image: removing whiteout kaniko/ssl/.wh..wh..opq: fstatat /kaniko/ssl/.wh..opq: operation not permitted
/kaniko/context #

tblaisot · 2020-02-27T11:02:51Z

This is most likely the same issues as #1039, can you try tags a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79

@cvgw I juste tried the tags you indicate me but I got the same error

�[36mINFO�[0m[0000] Resolved base name nginxinc/nginx-unprivileged:stable-alpine to nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0000] Using dockerignore file: /home/jenkins/agent/workspace/mptabilite-front_feature_jenkins/.dockerignore 
�[36mINFO�[0m[0000] Resolved base name nginxinc/nginx-unprivileged:stable-alpine to nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0000] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0001] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0002] Built cross stage deps: map[]                
�[36mINFO�[0m[0002] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0003] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0003] Unpacking rootfs as cmd COPY /dist /usr/share/nginx/html requires it. 
error building image: error building stage: failed to get filesystem from image: removing whiteout etc/nginx/.wh..wh..opq: fstatat /etc/nginx/.wh..opq: operation not permitted

Btw I don't use the cache or the warmup to make sure it's not related to others bugs.
I have tried the same container (All versions) in local (no k8s) without any problem ...

cvgw · 2020-02-27T23:13:58Z

This is most likely the same issues as #1039, can you try tags a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79

@cvgw I juste tried the tags you indicate me but I got the same error

�[36mINFO�[0m[0000] Resolved base name nginxinc/nginx-unprivileged:stable-alpine to nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0000] Using dockerignore file: /home/jenkins/agent/workspace/mptabilite-front_feature_jenkins/.dockerignore 
�[36mINFO�[0m[0000] Resolved base name nginxinc/nginx-unprivileged:stable-alpine to nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0000] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0001] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0002] Built cross stage deps: map[]                
�[36mINFO�[0m[0002] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0003] Retrieving image manifest nginxinc/nginx-unprivileged:stable-alpine 
�[36mINFO�[0m[0003] Unpacking rootfs as cmd COPY /dist /usr/share/nginx/html requires it. 
error building image: error building stage: failed to get filesystem from image: removing whiteout etc/nginx/.wh..wh..opq: fstatat /etc/nginx/.wh..opq: operation not permitted

Btw I don't use the cache or the warmup to make sure it's not related to others bugs.
I have tried the same container (All versions) in local (no k8s) without any problem ...

Can you please share the exact docker command you are using to execute the kaniko build. Thank you

tejal29 · 2020-03-06T00:25:30Z

Verfied on latest master image gcr.io/tejal-test/executor

FROM nginxinc/nginx-unprivileged:stable-alpine

and

FROM gcr.io/tejal-test/executor:debug
RUN ["/busybox/sh", "-c", "ls -l"]

runs successfully

INFO[0000] Resolved base name gcr.io/tejal-test/executor:debug to gcr.io/tejal-test/executor:debug 
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name gcr.io/tejal-test/executor:debug to gcr.io/tejal-test/executor:debug 
INFO[0000] Retrieving image manifest gcr.io/tejal-test/executor:debug 
INFO[0000] Retrieving image manifest gcr.io/tejal-test/executor:debug 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/tejal-test/executor:debug 
INFO[0001] Retrieving image manifest gcr.io/tejal-test/executor:debug 
INFO[0002] Unpacking rootfs as cmd RUN ["/busybox/sh", "-c", "ls -l"] requires it. 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Resolving paths                              
INFO[0004] RUN ["/busybox/sh", "-c", "ls -l"]           
INFO[0004] cmd: /busybox/sh                             
INFO[0004] args: [-c ls -l]                             
total 28
drwxr-xr-x    2 0        0            12288 Mar  6 00:24 busybox
drwxr-xr-x    5 0        0              340 Mar  6 00:24 dev
drwxr-xr-x    2 0        0             4096 Mar  6 00:24 etc
drwxr-xr-x    1 0        0             4096 Mar  6 00:24 kaniko
dr-xr-xr-x  466 0        0                0 Mar  6 00:24 proc
drwxr-xr-x    3 0        0             4096 Mar  6 00:24 root
dr-xr-xr-x   13 0        0                0 Mar  6 00:24 sys
drwxr-xr-x    8 407936   89939         4096 Mar  6 00:21 workspace
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Resolving paths                              
INFO[0004] No files were changed, appending empty layer to config. No layer added to image.

They both work.

RubenAtPA · 2020-03-09T14:13:09Z

We have had the same error since about a week. After a lot of trial-and-error debugging we've figured out that the problem (in our case) could be traced to the use of an image that contains a symlink to the stdout/stderr. After removing this symlink image in our base image, we could build new images from this base image again.

Is that related to this issue?

tejal29 · 2020-03-19T20:30:07Z

@RubenAtPA Thanks for all the investigation. I will try to reproduce this with symlink to stdout/err and track the progress here. #1147

mschickervxob · 2020-04-14T19:38:27Z

@tejal29
We encounter the same problem with using php:7.4.4-fpm-buster as base image and doing anything that triggers unpacking rootfs. Here for example apt-get update

 $ /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --no-push
 INFO[0000] Resolved base name php:7.4.4-fpm-buster to php:7.4.4-fpm-buster 
 INFO[0000] Resolved base name php:7.4.4-fpm-buster to php:7.4.4-fpm-buster 
 INFO[0000] Retrieving image manifest php:7.4.4-fpm-buster 
 INFO[0001] Retrieving image manifest php:7.4.4-fpm-buster 
 INFO[0002] Built cross stage deps: map[]                
 INFO[0002] Retrieving image manifest php:7.4.4-fpm-buster 
 INFO[0003] Retrieving image manifest php:7.4.4-fpm-buster 
 INFO[0004] Unpacking rootfs as cmd RUN apt-get update requires it. 
 error building image: error building stage: failed to get filesystem from image: removing whiteout var/lib/apt/lists/auxfiles/.wh..wh..opq: fstatat /var/lib/apt/lists/auxfiles/.wh..opq: operation not permitted

Dockerfile:

FROM php:7.4.4-fpm-buster
RUN apt-get update

Environment:
Running on k8s runner from gitlab - tested different kaniko versions including latest with and without debug.

Works from local developer machine with
docker run --rm -v $(pwd):/build gcr.io/kaniko-project/executor:v0.19.0 --context /build --dockerfile /build/Dockerfile --no-push

Gwulior · 2020-04-16T11:15:50Z

Same for me.
Using GitLab shared Kubernetes runner and Kaniko: "gcr.io/kaniko-project/executor:debug-v0.19.0"

Dockerfile:

FROM openjdk:11-jre
RUN useradd -u 230000 unpriv
RUN mkdir /workspace
ARG COMPONENT
ARG VERSION
ARG FOLDER
COPY ./$FOLDER/target/${COMPONENT}-${VERSION}.jar /workspace
RUN chown  -R unpriv /workspace
USER unpriv
WORKDIR /workspace
EXPOSE  8080
CMD [ "java", "-Dspring.profiles.active=cloud", "-jar",  "workspace/${COMPONENT}-${VERSION}.jar" ]

Getting:

INFO[0001] Resolved base name openjdk:11-jre to openjdk:11-jre 
 INFO[0001] Resolved base name openjdk:11-jre to openjdk:11-jre 
 INFO[0001] Retrieving image manifest openjdk:11-jre     
 INFO[0003] Retrieving image manifest openjdk:11-jre     
 INFO[0004] Built cross stage deps: map[]                
 INFO[0004] Retrieving image manifest openjdk:11-jre     
 INFO[0005] Retrieving image manifest openjdk:11-jre     
 INFO[0007] Unpacking rootfs as cmd RUN useradd -u 230000 unpriv requires it. 
 error building image: error building stage: failed to get filesystem from image: chown /etc/gshadow: operation not permitted
sh: exec: line 6: /bin/bash: not found
00:00
 ERROR: Job failed: command terminated with exit code 1

mschickervxob · 2020-04-27T12:53:20Z

Any news on this @tejal29 ? Look like there is something broken/magic/different in these public images.

appcoders · 2020-04-28T19:39:47Z

Tested today with debug-edge image - works.

grollinger · 2020-06-02T13:46:47Z

Tested today with debug-edge image - works.

Tested today with debug (and debug-edge).
It is as @mschickervxob said:

works locally using docker run
doesn't work in GitLab CI using a k8s runner

In my case the base image was dtzar/helm-kubectl:3.2.1

scones · 2020-06-10T11:59:26Z

We encounter the same problem with using php:7.4.4-fpm-buster as base image and doing anything that triggers unpacking rootfs. Here for example apt-get update

confirmed for php:7.4.6-fpm-buster too

krishofmans · 2020-09-18T06:46:08Z

This still fails for us:

 ERROR: Job failed: command terminated with exit code 1

When using gcr.io/kaniko-project/executor:debug-v1.0.0 . on a gitlab k8s runner while unpacking rootfs:

[0006] Unpacking rootfs as cmd RUN apt-get update &&    apt-get install -y --no-install-recommends

Is there a workaround for this issue? Perhaps reopen this ticket?

tejal29 · 2020-09-18T16:08:02Z

@krishofmans Can you please provide some more logs. This issue was related to symlinks not being removed.

krishofmans · 2020-09-18T18:32:51Z

Of course @tejal29, the only thing that changed was the parent image and it broke ...

$ /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$IMAGE_TAG --build-arg SHORT_COMMIT_HASH=$SHORT_COMMIT_HASH
 E0914 12:12:29.051986      71 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
 	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
 INFO[0006] Retrieving image manifest docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Retrieving image docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Retrieving image manifest docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Retrieving image docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Built cross stage deps: map[]                
 INFO[0006] Retrieving image manifest docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Retrieving image docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Retrieving image manifest docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Retrieving image docker-registry.internal.custodix.com/containers/java:master 
 INFO[0006] Executing 0 build triggers                   
 WARN[0006] maintainer is deprecated, skipping           
 INFO[0006] Unpacking rootfs as cmd RUN apt-get update &&    apt-get install -y --no-install-recommends cron gnupg procps openssl openssh-client fontconfig curl tar mysql-client ant ant-optional gettext-base moreutils     gzip ca-certificates git jq httpie libxml2-utils wget coreutils findutils python-pip python3-pip python3-setuptools zip unzip xz-utils xmlstarlet cifs-utils rsync      libgtk2.0-0 libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 xvfb libgtk3.0-cil-dev &&     pip install idna==2.6 && python3.6 -m pip install idna==2.6 &&     pip3 install wheel && pip3 install awscli --upgrade --user &&     mv ~/.local/bin/aws /usr/bin/aws &&     pip3 install yq &&     curl -sSL https://get.docker.com/ | sh  &&     mv /usr/bin/xmlstarlet /usr/bin/xml &&     apt-get clean &&     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* requires it. 
 error building image: error building stage: failed to get filesystem from image: removing whiteout etc/.java/.wh..wh..opq: fstatat /etc/.java/.wh..opq: operation not permitted
 ERROR: Job failed: command terminated with exit code 1

sschueller · 2020-10-15T11:30:33Z

For anyone having this issue (removing whiteout ... operation not permitted ). It appears to be an issue with older version of docker running the aufs storage driver (default in older debian installs). Upgrading to docker 18 or higher and using the storage driver overlay2 fixed this issue for me. To check what storage driver you are using do a docker info.

lynchs61 · 2022-02-15T22:52:40Z

I just ran into this also and I can confirm that the solution from @sschueller worked for me. Here's the relevant information from the original docker info.

root@kube110:~# docker info
Containers: 15
 Running: 8
 Paused: 0
 Stopped: 7
Images: 14
Server Version: 17.03.3-ce
Storage Driver: aufs
...

After the upgrade everything works as expected. Here's the upgraded docker information.

root@kube110-stage:~# docker info
Containers: 18
 Running: 10
 Paused: 0
 Stopped: 8
Images: 7
Server Version: 18.06.3-ce
Storage Driver: overlay2

Also note that overlay2 is the default (at least on Ubuntu 18) so no need to specify.

tblaisot changed the title ~~Got Error: failed to get filesystem from image: removing whiteout operation not permitted~~ Error: failed to get filesystem from image: removing whiteout operation not permitted Feb 24, 2020

cvgw added area/filesystems For all bugs related to kaniko container filesystems (mounting issues etc) fixed-needs-verfication kind/bug Something isn't working regression labels Feb 25, 2020

tejal29 mentioned this issue Mar 6, 2020

release notes for v0.18.0 #1108

Merged

tejal29 removed the fixed-needs-verfication label Mar 6, 2020

tejal29 mentioned this issue Mar 19, 2020

small perf optimizing. Only remove whiteout path if it needs to be included in base image #1147

Merged

tejal29 closed this as completed in #1147 Mar 24, 2020

Yannik mentioned this issue Apr 27, 2020

buildah bud error adding layer /var/lib/apt/lists/auxfiles/.wh..wh..opq: invalid argument containers/buildah#2327

Closed

claudioweiler mentioned this issue Apr 9, 2021

Build failed with "removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system" #1420

Closed

Flauschbaellchen mentioned this issue Apr 16, 2021

error building image: error building stage GCC #763

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: failed to get filesystem from image: removing whiteout operation not permitted #1073

Error: failed to get filesystem from image: removing whiteout operation not permitted #1073

tblaisot commented Feb 23, 2020

cvgw commented Feb 25, 2020

dren-dk commented Feb 26, 2020 •

edited

Loading

tblaisot commented Feb 27, 2020

cvgw commented Feb 27, 2020

tejal29 commented Mar 6, 2020

RubenAtPA commented Mar 9, 2020

tejal29 commented Mar 19, 2020

mschickervxob commented Apr 14, 2020 •

edited

Loading

Gwulior commented Apr 16, 2020

mschickervxob commented Apr 27, 2020

appcoders commented Apr 28, 2020

grollinger commented Jun 2, 2020

scones commented Jun 10, 2020

krishofmans commented Sep 18, 2020

tejal29 commented Sep 18, 2020

krishofmans commented Sep 18, 2020

sschueller commented Oct 15, 2020

lynchs61 commented Feb 15, 2022

Error: failed to get filesystem from image: removing whiteout operation not permitted #1073

Error: failed to get filesystem from image: removing whiteout operation not permitted #1073

Comments

tblaisot commented Feb 23, 2020

cvgw commented Feb 25, 2020

dren-dk commented Feb 26, 2020 • edited Loading

tblaisot commented Feb 27, 2020

cvgw commented Feb 27, 2020

tejal29 commented Mar 6, 2020

RubenAtPA commented Mar 9, 2020

tejal29 commented Mar 19, 2020

mschickervxob commented Apr 14, 2020 • edited Loading

Gwulior commented Apr 16, 2020

mschickervxob commented Apr 27, 2020

appcoders commented Apr 28, 2020

grollinger commented Jun 2, 2020

scones commented Jun 10, 2020

krishofmans commented Sep 18, 2020

tejal29 commented Sep 18, 2020

krishofmans commented Sep 18, 2020

sschueller commented Oct 15, 2020

lynchs61 commented Feb 15, 2022

dren-dk commented Feb 26, 2020 •

edited

Loading

mschickervxob commented Apr 14, 2020 •

edited

Loading