Shared ARG or Whitelist /dev/pts/ptmx in Cloud Build

[dinvlad]

Actual behavior
When running a build in Cloud Build, it fails with

failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

If I move ARG DIST=/opt after FROM (see below), then the build succeeds. However, I keep it above because it's used in multi-staged builds (where a single ARG value is shared among multiple stages). But the error appears even in the absence of other stages.

Expected behavior
Build succeeds.

Alternatively, would it be possible to whitelist /dev/pts?

To Reproduce
Steps to reproduce the behavior:

1. Dockerfile:

ARG DIST=/opt

FROM gcr.io/cloud-builders/wget

ARG DIST
RUN echo ${DIST}

2. cloudbuild.yaml:

steps:
- name: gcr.io/kaniko-project/executor
  args:
  - --destination=gcr.io/${PROJECT_ID}/kaniko-ptmx-repro
  - --cache=true

Additional Information

• Dockerfile (see above)
• Build Context: -
• Kaniko Image (fully qualified with digest): gcr.io/kaniko-project/executor@sha256:565d31516f9bb91763dcf8e23ee161144fd4e27624b257674136c71559ce4493 (v0.17.1)
• Debug log:

BUILD
Pulling image: gcr.io/kaniko-project/executor
Using default tag: latest
latest: Pulling from kaniko-project/executor
Digest: sha256:565d31516f9bb91763dcf8e23ee161144fd4e27624b257674136c71559ce4493
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
DEBU[0000] Copying file /workspace/Dockerfile to /kaniko/Dockerfile 
DEBU[0000] Skip resolving path /kaniko/Dockerfile       
DEBU[0000] Skip resolving path /workspace/              
DEBU[0000] Skip resolving path /cache                   
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
DEBU[0000] Skip resolving path                          
INFO[0000] Resolved base name gcr.io/cloud-builders/wget to gcr.io/cloud-builders/wget 
INFO[0000] Resolved base name gcr.io/cloud-builders/wget to gcr.io/cloud-builders/wget 
INFO[0000] Retrieving image manifest gcr.io/cloud-builders/wget 
DEBU[0000] No file found for cache key sha256:fcbc52cce257e002accec94d8170900429ab51d3ddda629142a6f5a887079d6d stat /cache/sha256:fcbc52cce257e002accec94d8170900429ab51d3ddda629142a6f5a887079d6d: no such file or directory 
DEBU[0000] Image gcr.io/cloud-builders/wget not found in cache 
INFO[0000] Retrieving image manifest gcr.io/cloud-builders/wget 
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest gcr.io/cloud-builders/wget 
DEBU[0000] No file found for cache key sha256:fcbc52cce257e002accec94d8170900429ab51d3ddda629142a6f5a887079d6d stat /cache/sha256:fcbc52cce257e002accec94d8170900429ab51d3ddda629142a6f5a887079d6d: no such file or directory 
DEBU[0000] Image gcr.io/cloud-builders/wget not found in cache 
INFO[0000] Retrieving image manifest gcr.io/cloud-builders/wget 
DEBU[0001] optimize: composite key for command ARG DIST {[sha256:fcbc52cce257e002accec94d8170900429ab51d3ddda629142a6f5a887079d6d ARG DIST]} 
DEBU[0001] optimize: cache key for command ARG DIST 0bfa8f9f752bd48e8540d6790d62fd31bea228295a0311c725368ea2c18834c3 
DEBU[0001] optimize: composite key for command RUN echo ${DIST} {[sha256:fcbc52cce257e002accec94d8170900429ab51d3ddda629142a6f5a887079d6d ARG DIST RUN echo ${DIST}]} 
DEBU[0001] optimize: cache key for command RUN echo ${DIST} 99b2e2e1e2d558a9001845fb5bc050d0a10dc50d718df9f456bbae0cbdf07e9e 
INFO[0001] Checking for cached layer gcr.io/<project>/kaniko-ptmx-repro/cache:99b2e2e1e2d558a9001845fb5bc050d0a10dc50d718df9f456bbae0cbdf07e9e... 
INFO[0001] Using caching version of cmd: RUN echo ${DIST} 
INFO[0001] Skipping unpacking as no commands require it. 
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] ARG DIST                                     
INFO[0001] No files changed in this command, skipping snapshotting. 
DEBU[0001] build: composite key for command ARG DIST &{[sha256:fcbc52cce257e002accec94d8170900429ab51d3ddda629142a6f5a887079d6d ARG DIST]} 
DEBU[0001] build: cache key for command ARG DIST 0bfa8f9f752bd48e8540d6790d62fd31bea228295a0311c725368ea2c18834c3 
INFO[0001] RUN echo ${DIST}                             
INFO[0001] Found cached layer, extracting to filesystem 
DEBU[0001] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/systemd false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/cpu,cpuacct false} {/sys/fs/cgroup/rdma false} {/sys/fs/cgroup/net_cls,net_prio false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/memory false} {/dev/mqueue false} {/workspace false} {/builder/outputs false} {/root/tokencache false} {/builder/home false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/shm false} {/var/run/docker.sock false}] 
DEBU[0001] Whiting out /.wh.dev                         
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor" failed: exit status 1

This might be related to #1001

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[dinvlad]

I should add that this issue started happening for us today (2/6/20) ~1:48 PM EST, without any changes to the Dockerfile or cloudbuild.yaml. Prior to that, a successful build happened earlier today ~10:29 AM EST, which I assume indicates that something changed on Cloud Build infra side..

[michaelfindlater]

Been getting this for two days now on Cloud Build.

For now, having to use --cache=false for builds to pass :(

If anyone has a solution or suggestion, it would be appreciated.

Args we are using are when encountering the error:

         '--cache=true',
         '--cache-ttl=8h',

[tejal29]

Dupe of #1039