Jib fails with BLOB_UPLOAD_UNKNOWN error on Sonatype Nexus #2372
Comments
|
Hi @SniXosha,
If that is not the case, I am highly suspicious of the server malfunctioning. In the past, we often did see confirmed cases of server issues in some popular registries when there is a very high level of concurrency (e.g., #1986 (comment) and #2013). From our track record, there's a high chance that this is a server issue. (BTW, Docker CLI by default doesn't allow the same level of high concurrency that Jib outputs.)
Then follow these instructions to capture detailed HTTP logs, except passing Please let me know once you get the network logs for the error. |
Oh, depending on how you set up your multi-module project, you may need to adjust the buildscript accordingly. For example, if you are not applying Jib globally on the root project (that is, you have
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Oh, sorry, forget it. I already have timestamps. Let me get back to you real soon. |
|
This proves that your registry is malfunctioning.
You should contact the registry people and present this evidence that proves that the server is not working correctly. |
|
Thanks for clarification! I'll try to contact them. |
|
If you need assistance, let me know. And please update here once you have more information. That said, what's your registry (Sonatype Nexus, Quay, Harbor, Docker Distribution, etc.)? |
|
It's Sonatype Nexus |
|
Other people did encounter this problem (also with Docker CLI), and there's an open issue on Sonatype Nexus. [NEXUS-20640] docker push may fail with blob upload unknown due to race condition As such, I'll close the issue, but feel free to update or re-open as necessary. |
|
Just got the notification that the Sonatype Nexus bug (NEXUS-20640) is marked fixed. |
|
Adding that this seems to affect ghcr.io (Github Container Registry) as well. Adding |
|
Hi guys, does anybody still has this issue? Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) |
…ssage":"blob upload unknown to registry"}]}` See also GoogleContainerTools/jib#2372
|
For us, the |
|
I can also confirm the same issue when uploading packages to GitHub, and in my experience using What I have noticed is that all jib tasks that fail, seems to fail after trying to retry an upload
|
|
Just something worth nothing that I found out: I think that this is a ghcr.io issue, in my project I have multiple submodules that are uploaded to ghcr.io via jib, and using |
|
Update to my previous comment: That didn't fix the issue... it does help a lot, but it didn't fully fix the issue. What I'm currently testing is running the jib tasks in separate steps AND using |
|
I have raised an issue to the github support team regarding this issue we experienced with ghcr.io and got a quite thorough response with investigation from their engineering team, I'll paraphrase:
also from my support ticket: "one theory being that Jib may be retrying an upload before the previous upload has actually failed. This could also explain the inconsistency of failure you are seeing, as different uploads would take different amounts of time, and thus some would fit within the retry window (thus not failing), while a few might exceed the retry window and thus fail." Since this issue is already closed we probably need to open a new one if we want anything further to happen. |
Environment:
Description of the issue:
Jib fails with BLOB_UPLOAD_UNKNOWN error.
I have a gradle project with about 15 submodules and I want to push images in our private registry. When I run jib for all sumbodules in my gradle project, sometimes this error occurs. Jib doesn't fail immediately, it's often able to create and push image for about 12-13 submodules and only then it crashes. If I rerun jib after waiting a few seconds, all the remaining submodules are processed successfully (currently, this is my workaround).
I asked people responsible for our registry and they said that this is probably jib's problem, because they have not encountered this error using docker cli.
They also suggested to disable parallel execution in gradle, which seemingly made jib more stable, but didn't solve the problem completely.
Expected behavior:
Jib runs successfully regardless of number of modules/execution time without BLOB_UPLOAD_UNKNOWN error.
Steps to reproduce:
jib-gradle-pluginConfiguration:jib { from { image registryBaseImage auth { username registryUsername password registryPassword } } to { image registryRepository + project.name + ':' + project.version auth { username registryUsername password registryPassword } } }Log output:
Additional Information:
I asked people responsible for our registry and they said that this is probably jib's problem, because they have not encountered this error using docker cli.
They also suggested to disable parallel execution in gradle, which seemingly made jib more stable, but didn't solve the problem completely.
The text was updated successfully, but these errors were encountered: