Multistage COPY failure #1210
Labels
area/filesystems
For all bugs related to kaniko container filesystems (mounting issues etc)
cmd/copy
differs-from-docker
gitlab
help wanted
Looking for a volunteer!
issue/build-fails
issue/no-such-file
kind/bug
Something isn't working
priority/p2
High impact feature/bug. Will get a lot of users happy
uncategorized
works-with-docker
Milestone
Comments
|
Altered issue description to use |
|
@paisleyrob I created the same context from outside the container, and ran kaniko where my /ctxt directory contained your Dockerfile and "empty". The build successfully finishes. Please let me know if the issue can be closed. |
I had to edit your command a little as follow to make it work: docker run --rm -it \
--volume "$(pwd)"/docker:/build \
gcr.io/kaniko-project/executor:debug-v0.19.0@sha256:0d0e34396f47ec6d5fd75aebb9772147a78d96ed2bbb16ec892bd178efdc8307 \
--oci-layout-path /container \
--reproducible \
--no-push \
--context=/build
INFO[0000] Resolved base name scratch to scratch
INFO[0000] Resolved base name scratch to scratch
INFO[0000] Resolved base name scratch to scratch
INFO[0000] Resolved base name scratch to scratch
INFO[0000] Built cross stage deps: map[]
INFO[0000] No base image, nothing to extract
INFO[0000] Skipping unpacking as no commands require it.
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] Resolving paths
INFO[0000] Deleting filesystem...
INFO[0000] No base image, nothing to extract
INFO[0000] Unpacking rootfs as cmd COPY empty / requires it.
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] Resolving paths
INFO[0000] COPY empty /
INFO[0000] Resolving paths
INFO[0000] Taking snapshot of files...
INFO[0000] Skipping push to container registry due to --no-push flagWhich worked. I'm not comfortable closing this issue as I'm using this in a GitLab Runner environment, so I need it work from the kaniko debug image's shell, which it doesn't. |
|
Updated to 0.21.0's output with verification this is still an issue. |
|
Issue still persists with v1.2.0: |
|
we saw the same issue with 1.2.0 here is our log the command we are running |
|
Any updates on this? Still encountering this with the latest |
I mistook this for a request to see if it's still encountered on the latest and I came here to confirm, yes it's still a problem. |
|
I had a quick play, and:
So I suspect this is not to do with the use of Trace logs of the failure and success casesSuccess case (mounted volume from local filesystem) Failure case (context created inside container, not under a mountpoint): The only relevant differences in the trace logs are around the mount-point lists, and I think debug logs would have shown them. Working case: Failure case: The difference is the presence of This is the bug: So I suspect a fix would be to automatically add the context directory to the ignore list. A build-from-git must already do this, or it pulls the context into an existing ignored directory (the latter is probably the case). A workaround might be to bind-mount the context to itself before populating it, but I couldn't get that to work, trivially. On the other hand, now that I know that Kaniko treats its own root as the image root, the following pathological feature works: Given C:\Users\paulh\kantmp\docker\Dockerfile: FROM scratch
RUN ["/kaniko/executor", "--help"]Umm... Gosh. Turns out this last thing is already logged as #1553. |
Worked for me |
|
Minor tweaks as The above comment from @TBBle appears to pinpoint underlying issue. |
|
@paisleyrob @TBBle thanks for digging into this. Unfortunately this project is pretty short-staffed, so if you can find where in the code this is going wrong, or a recent commit that made it go wrong, that would be very helpful. I don't think there's anybody coming to save us here but us. I'm not personally very familiar with the code, but I'd be happy to chat over Slack or video chat if you want to dig in together and throw around ideas. |
|
This issue still exists in |
|
I just ran into a very similar-sounding issue: I'm trying to (Meanwhile, building in Docker works fine.) This did not happen before when my People have complained about similar issues here:
EDIT: It seems I have found a workaround: If in previous stages I use a source folder different from |
|
Hi @codethief, I'm running into a similar error. Could you share the relevant parts from your gitlab-ci yaml file and the Dockerfile that now works for you? That would be awesome! My code and error:
Previous copy statements (with --from kaniko) work but the last one fails. docker-entrypoint.sh is in the same folder as my Dockerfile. |
|
I found this issue while trying to debug an issue - similarly with building from a GitLab Runner. The error-message was similar to what I see here:
The real issue was that Hope this helps someone. |
|
@anneadb My apologies for the delay; for some reason I never got notified about your message! I am afraid I don't remember anymore the full context of the issues I encountered or how I ended up solving them, let alone whether I actually saw the bug that the present bug report is about. I have run into so many Gitlab bugs and sometimes also Kaniko bugs in the last months, that it's difficult to recall things sometimes. 😅 In any case, what I can tell you is that one should absolutely avoid putting any files in $CI_PROJECT_DIR inside a Docker image, since – when the image is later used in a CI job – Gitlab will mount an outside volume there (the cloned repository) and the files won't be visible, anyway. Unfortunately, if you use a different
Again, I'm not sure what ended up resolving the specific issues I mentioned in my previous message but today we no longer use $CI_PROJECT_DIR in any of our Dockerfiles for the reason I mentioned above, and things generally seem to work fine. (Of course, correlation != causation.) The exception to this is this Kaniko bug, which still hits us occasionally, and also that one. Sorry I can't be of greater help! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Actual behavior
Fails to create image.
Expected behavior
Create an image
To Reproduce
Additional Information
emptyis an empty file.touch emptykaniko's call:/kaniko/executor --context /docker --no-push --oci-layout-path /container --reproducibleTriage Notes for the Maintainers
I didn't try with
--cache--cacheflagThe text was updated successfully, but these errors were encountered: