-
Notifications
You must be signed in to change notification settings - Fork 638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot commit and push container image from Kubernetes pods #827
Comments
I can't reproduce this problem on containerd with latest and 1.5.10 version.
In addition, I can't start the corresponding containerd version through minikube. PS: containerd v1.4.x reached EOL. |
I closed it because containerd v1.4.x reached EOL. Feel free to open it up if you have any other questions. |
I am also getting the same error for large images (mine was 7GB, small images were easily pushed) FATA[0000] failed to create a tmp single-platform image "registry.example.com/my-app:0086bf16-21cb-477b-952c-13780780f597-tmp-reduced-platform": content digest sha256:5bed26d33875e6da1d9ff9a1054c5fef3bbeb22ee979e14b72acf72528de007b: not found I have the following config- Server: Please help in resolving this. |
could you please reopen the issue, @junnplus thanks. here is my env info: |
In addition: |
I got the same issue but with |
In addition: nerdctl version 1.5.0 harbor server with port: hub.local: |
could not produce this bug. would you guys mind to give me reproduce step with more detail? like what's registry you use |
Recently, my GKE cluster's worker node upgraded to
Reproduce The commit command: The push command like: Output: Work around Add If I run the push command like: The option is added to avoid Lines 98 to 111 in ce2f63d
Other info The issue is not registry related. I tried different registries, the same error is thrown, even with random invalid registry. The container I am committing from is built on an Ubuntu machine with When it used to work with containerd 1.6.18, I can see the log |
I started a container using the nydus format and then packaged the container into an image. However, I encountered the same problem when I tried to push this image. Steps to reproduce:
nerdctl --snapshotter nydus run --cap-add=CAP_SYS_ADMIN --net host -it registry-test.xx.info/library/nydus-image:old
nerdctl commit ${CONTAINER_ID} registry-test.xx.info/library/nydus-image:new
nerdctl push registry-test.xx.info/library/nydus-image:new
FATA[0000] failed to create a tmp single-platform image " registry-test.xx.info/library/nydus-image:new-tmp-reduced-platform": content digest sha256:cfc7393224358fb100dabf40f7d59c2578c0a6a47781b1a6b56406922aeb6e95: not found env info:
Does the container launched by on-demand loading of images support commit and push operations? |
Hi I'm having the same issue. Is there a specific version of containerd that I can use that doesn't have this problem? |
I am encountering the same issue. This has been an ongoing issue since the past and I am wondering if there have been any follow-ups on this matter. |
This comment was marked as outdated.
This comment was marked as outdated.
By default, commit image will not be unpacked to snapshots. ctr --namespace=k8s.io i check
REF TYPE DIGEST STATUS SIZE UNPACKED
docker.io/lingdie/busybox:commit application/vnd.docker.distribution.manifest.v2+json sha256:a7dce96e72a59c80479569d48a1376a36a2c2b0e9d2ba0e3dca707e2021d5e72 complete (4/4) 28.3 MiB/28.3 MiB false Unpacking the image may solve the problem: ctr --namespace=k8s.io snapshots unpack sha256:a7dce96e72a59c80479569d48a1376a36a2c2b0e9d2ba0e3dca707e2021d5e72
unpacking sha256:a7dce96e72a59c80479569d48a1376a36a2c2b0e9d2ba0e3dca707e2021d5e72 (application/vnd.docker.distribution.manifest.v2+json)...done
nerdctl --namespace k8s.io image convert lingdie/busybox:commit lingdie/busybox:tmp
sha256:a7dce96e72a59c80479569d48a1376a36a2c2b0e9d2ba0e3dca707e2021d5e72
nerdctl --namespace k8s.io image push lingdie/busybox:tmp
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:a7dce96e72a59c80479569d48a1376a36a2c2b0e9d2ba0e3dca707e2021d5e72)
manifest-sha256:a7dce96e72a59c80479569d48a1376a36a2c2b0e9d2ba0e3dca707e2021d5e72: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:55fefd844bcb12134766ab042567efccb311ad9f051e68497c6a207dbdbe7afc: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.0 s total: 2.5 Ki (509.0 B/s) |
this pr may fix this: |
@lingdie testing on main from today, the issue is still there (as described by OP). Wondering what we are missing here... |
kind.yaml:
Create the cluster:
Inside the control plane from above:
More info:
@lingdie let me know if I am missing something here, or if you need anything else. Also, when we get #3296 in we will be able to test for Kube on the CI (in the same conditions, using Kind). |
This comment was marked as outdated.
This comment was marked as outdated.
Regarding the issue of saving snapshots into tar files, has it been resolved? Do I use the latest version 2.0.0-rc1, but it still prompts that some levels of sha256 cannot be found. Is it lost due to garbage collection? |
@ouyangningdong if your issue is the same that is being discussed here, it is not resolved - and the ticket is still open. If your issue is different, I suggest you open a new ticket. Also, it does not seem like a garbage collection problem. |
@lingdie I did add the scenario as a test (https://github.com/containerd/nerdctl/blob/main/cmd/nerdctl/container_commit_linux_test.go#L28) - right now, the test expectation is to fail (until we fix the problem). Interestingly, the test failed once on an unrelated PR: ... meaning the problem might be racy? |
The only thing I can think of is that containerd might have garbage collected some essential layers... |
Signed-off-by: apostasie <spam_blackhole@farcloser.world>
Signed-off-by: apostasie <spam_blackhole@farcloser.world>
I believe I have found the source of the problem. Below is my reproduction process:
kubectl run busybox --image busybox -- sleep 12000
ctr --namespace=k8s.io i check | grep busybox
docker.io/library/busybox:latest application/vnd.oci.image.index.v1+json sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 incomplete (1/2) 390.0 B/1.8 MiB true
docker.io/library/busybox@sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 application/vnd.oci.image.index.v1+json sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 incomplete (1/2) 390.0 B/1.8 MiB true
containerID=$(kubectl get pods -o go-template='{{range .items}}{{range .status.containerStatuses}}{{if eq .name "busybox"}}{{.containerID}}{{"\n"}}{{end}}{{end}}{{end}}' | grep -oP '(?<=containerd://)\w+')
nerdctl commit $containerID docker.io/lingdie/busybox:dev
ctr --namespace=k8s.io i check | grep busybox
docker.io/library/busybox:latest application/vnd.oci.image.index.v1+json sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 incomplete (1/2) 390.0 B/1.8 MiB true
docker.io/library/busybox@sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 application/vnd.oci.image.index.v1+json sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 incomplete (1/2) 390.0 B/1.8 MiB true
docker.io/lingdie/busybox:dev application/vnd.docker.distribution.manifest.v2+json sha256:66b5ca932bd733887d3c4c4d808b6816f9bafed9ea5fb1d350d0a1e84f86781e incomplete (2/3) 791.0 B/1.8 MiB true
nerdctl push docker.io/lingdie/busybox:dev
FATA[0000] failed to create a tmp single-platform image "docker.io/lingdie/busybox:dev-tmp-reduced-platform": content digest sha256:75e8ca8f509fb8f7ee74430edbdc5b78fd863a4f08e06c53828e9a996a79f642: not found
ctr --namespace k8s.io image pull docker.io/library/busybox:latest
docker.io/library/busybox:latest: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e7e097403ca9266ab43ca83b3b55c272c248cf46489a40bcc4864bd7dd945f18: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:75e8ca8f509fb8f7ee74430edbdc5b78fd863a4f08e06c53828e9a996a79f642: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:fd633c23ab56c682e69aacf7fea8d0eaa8d18bd72c4b1c6d82eb5eaace658f1a: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 8.0 s total: 1.0 Mi (128.0 KiB/s)
unpacking linux/arm64/v8 sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41...
done: 9.078793ms
ctr --namespace=k8s.io i check | grep busybox
docker.io/library/busybox:latest application/vnd.oci.image.index.v1+json sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 complete (2/2) 1.8 MiB/1.8 MiB true
docker.io/library/busybox@sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 application/vnd.oci.image.index.v1+json sha256:34b191d63fbc93e25e275bfccf1b5365664e5ac28f06d974e8d50090fbb49f41 complete (2/2) 1.8 MiB/1.8 MiB true
docker.io/lingdie/busybox:dev application/vnd.docker.distribution.manifest.v2+json sha256:66b5ca932bd733887d3c4c4d808b6816f9bafed9ea5fb1d350d0a1e84f86781e complete (3/3) 1.8 MiB/1.8 MiB true
nerdctl push docker.io/lingdie/busybox:dev
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:66b5ca932bd733887d3c4c4d808b6816f9bafed9ea5fb1d350d0a1e84f86781e)
manifest-sha256:66b5ca932bd733887d3c4c4d808b6816f9bafed9ea5fb1d350d0a1e84f86781e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:eb86545fdbaa77abb944627619e66fb8c1853b427fc930493c561988674054be: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:f78592bf867943251d217918b2299bbd7191970ae83dc52a8afc5e7ad98bf71d: done |
Hello folks! 👋
I'm experiencing a strange behavior while committing a container (from Kubernetes pod) and pushing it to some container registry after. I always get the same error from the push command:
content digest sha256:898c46f3b1a1f39827ed135f020c32e2038c87ae0690a8fe73d94e5df9e6a2d6: not found
.I'm filing this issue here but there's a chance the problem is on containerd side because I've already seen it working in an earlier version. I'm trying to find out which version it stopped to work.
I'm still investigating but no idea so far... suggestions are welcome.
Steps to reproduce:
Environment information:
0.17.0
v1.4.12 7b11cfaabd73bb80907dd23182b9347b4245eb5d
Buildroot 2021.02.4
(from minikube)4.19.202
v1.20.6
v1.20.6
v1.25.1
The text was updated successfully, but these errors were encountered: