ONBUILD COPY fails using multi-stage build

[mbfisher]

Hi everyone, thanks for your work on Kaniko!

Actual behavior

When using ONBUILD instructions with a multi-stage build, Kaniko fails to copy files between stages:

+ docker run --rm -it --volume /Volumes/Users/mbfisher/git-projects/kaniko-multi-stage-bug:/build gcr.io/kaniko-project/executor:debug --context=/build --destination=my-kaniko-bug
INFO[0000] Downloading base image mbfisher/kaniko-multi-stage-bug:builder
2019/01/17 10:05:39 No matching credentials were found, falling back on anonymous
INFO[0001] Executing 1 build triggers
INFO[0001] Unpacking rootfs as cmd RUN mkdir /app && echo "Hello, World!" > /app/index requires it.
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0001] Skipping paths under /build, as it is a whitelisted directory
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory
INFO[0001] Skipping paths under /busybox, as it is a whitelisted directory
INFO[0001] RUN mkdir /app && echo "Hello, World!" > /app/index
INFO[0001] cmd: /bin/sh
INFO[0001] args: [-c mkdir /app && echo "Hello, World!" > /app/index]
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0001] Skipping paths under /build, as it is a whitelisted directory
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory
INFO[0001] Skipping paths under /busybox, as it is a whitelisted directory
INFO[0001] Deleting filesystem...
INFO[0002] Downloading base image mbfisher/kaniko-multi-stage-bug:runtime
2019/01/17 10:05:41 No matching credentials were found, falling back on anonymous
INFO[0002] Executing 1 build triggers
INFO[0002] Taking snapshot of full filesystem...
INFO[0002] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0002] Skipping paths under /build, as it is a whitelisted directory
INFO[0002] Skipping paths under /proc, as it is a whitelisted directory
INFO[0002] Skipping paths under /dev, as it is a whitelisted directory
INFO[0002] Skipping paths under /sys, as it is a whitelisted directory
INFO[0002] Skipping paths under /busybox, as it is a whitelisted directory
INFO[0002] COPY --from=builder /app/* /app/
error building image: error building stage: lstat /kaniko/builder: no such file or directory

Expected behavior
The build should succeed, as it does when using the Docker Engine:

+ docker build --no-cache -t my-multi-stage-build .
Sending build context to Docker daemon  50.69kB
Step 1/2 : FROM mbfisher/kaniko-multi-stage-bug:builder AS builder
# Executing 1 build trigger
 ---> Running in c6bbc064f397
Removing intermediate container c6bbc064f397
 ---> c3d2e2f359f7
Step 2/2 : FROM mbfisher/kaniko-multi-stage-bug:runtime
# Executing 1 build trigger
 ---> 2b4f93925947
Successfully built 2b4f93925947
Successfully tagged my-multi-stage-build:latest
+ docker run --rm my-multi-stage-build
Hello, World!

To Reproduce

See https://github.com/mbfisher/kaniko-multi-stage-bug

Additional Information

• Possibly related to Skip unpacking optimization breaks COPY in multi-stage builds #460?

[grampelberg]

I believe I'm running into the same issue. With this dockerfile - https://github.com/linkerd/linkerd2/blob/master/Dockerfile-proxy, I get errors on this line:

COPY --from=fetch /build/target/proxy/LICENSE ./LICENSE

Adding this line fixes it:

RUN cp /build/target/proxy/LICENSE /build/LICENSE

[laoshancun]

Same issue.

INFO[0093] Executing 2 build triggers                   
INFO[0093] Skipping unpacking as no commands require it. 
INFO[0093] Taking snapshot of full filesystem...        
INFO[0094] Using files from context: [/builds/my-service/data] 
INFO[0094] ADD data /data                               
INFO[0094] Taking snapshot of files...                  
INFO[0094] COPY --from=builder /build/dist/* .          
error building image: error building stage: lstat /kaniko/builder: no such file or directory
INFO[0094] Resolving srcs [/build/dist/*]...      

[tejal29]

I have identified the issue. The resolveStages happens before resolveOnBuild for a stage is called.
resolveStages replaces all the stage names to an index 0, 1 instead of their name in the order in dockerfile. Copy command expects the stage names to be resolved to an index.

The OnBuild command is processed after resolveStages which adds COPY -from=builder .. without resolving the stage name to an index.

The fix here is to resolve stage idx in resolveOnBuild