Fails to pull image from registry - registry.redhat.io #2106
Comments
|
403 usually means you successfully authenticated with the server using your username and password but your account does not have sufficient permissions to do the operation. 401 is usually for failing to authenticate. And the log shows that the endpoint returns an HTML page, so another possibility is a typo in your image reference making Jib try a wrong endpoint on the server. A registry endpoint should return a proper registry error JSON for all kinds of errors including unauthorized or forbidden. That said, this seems weird. Jib can use Docker credentials ( |
|
Another thing to try is to pass a totally wrong password and see whether it returns 401 or 403. |
If the Password is wrong, then I get an error |
|
I'm also getting an error |
Reference of the image is correct: https://access.redhat.com/containers/?tab=images#/registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift |
|
That makes sense. Since it returns 401 when the password is incorrect, I think it is that the server does not think your account has sufficient permissions. Make sure your account has correct roles and permissions. |
|
Here is another test which is working using same credentials with docker |
|
Question: How jib knows that it must auth the user against this registry in my case -> |
|
Hmm... I actually suspect the server is malfunctioning. It returns 403 instead of 401, so if it were working correctly, it means you have successfully authenticated with the server but the server intentionally denies your request because you are not allowed to do the operation. If it were returning 401, then it would be certain that Jib failed to auth. But the response is not 401, so the auth must have been succeeded. But since you are using the same username and password, this doesn't make sense. Another reason I suspect server malfunction is the error message: Although this isn't hard evidence, the message "an error occurred while processing your request" and the title "Error" don't really match with 403 (or even 401), and it almost seems like the server had an internal error and it just fell back to returning 403 in such a case. And the fact that it returns a human-readable HTML page is another reason I suspect it's the server malfunctioning. So let's do this. We can actually check in more details how the communication between Jib and |
|
Here is the gist containing the log: https://gist.github.com/cmoulliard/0f872947014af2b2c4668bc50252ffcd#file-gistfile1-txt-L182-L250 @chanseokoh |
|
Yeah, actually, this is what I suspected. We've seen a very similar case with Red Hat Quay. It's basically that the server is not conforming to the HTTP standard. What's happening is that basically, your server returns OK with FYI, the auth had been successful well before Jib requested a manifest for pulling: the registry token auth workflow was already completed, and Jib got the auth token from the server. And there was a successful interaction with the server using the returned auth token. There is no problem with auth. It is just that, the registry fails to see that (so to speak) For the similar Quay issue I mentioned earlier, the Red Hat devs told us that they would file a ticket to fix their non-conformant registry server. Likewise, I urge you to file a bug against Red Hat for your registry so that it conforms to the HTTP standard. BTW, what is the actual registry implementation behind |
|
Many thanks for your help and quick reply/return :-) |
|
@chanseokoh. We are in a deadlock See response hereafter from Red Hat "The ticket is opened, but no solution has been found and, to be honest, won't likely be solved for a very long time. The reality is: the only good solution either requires Jib to fix their bad implementation (which they say they will not do) OR a new custom compilation of nginx on our end to add additional modules (which is a hard sell at this time), so I don't see any resolution coming for a while." |
|
Yeah, I agree we are kind of in a deadlock. This doesn't look good. (FTR, the thread is at https://groups.google.com/forum/#!topic/quay-sig/2y7tMP0h0g0) I'll try to see if there is a way to turn off query string normalization in Google HTTP Client or Apache HttpClient. |
|
I figured that it is Google HTTP Client (more specifically // build low-level HTTP request
String urlString = url.build();
LowLevelHttpRequest lowLevelHttpRequest = transport.buildRequest(requestMethod, urlString);(I don't know if So, it is the String stringValue = CharEscapers.escapeUriQuery(value.toString());
if (stringValue.length() != 0) {
buf.append('=').append(stringValue);
}So, for example, if we test the actual query string value causing this issue, we can see that it escapes the value and returns the exact same escaped value we can see in the actual Jib log. // Note this is the *value* itself. The entire query string including the key is "_auth_=exp=1572..."
String originalQueryStringValue = "exp=1572285389~hmac=f0a387f...";
System.out.println(originalQueryStringValue);
System.out.println(CharEscapers.escapeUriQuery(originalQueryStringValue));Output: So, OpenShift says OK to the first value but denies the second one even though they are just equivalent and an HTTP standard-conforming server should work fine on both. Unfortunately, there is no option to turn this off in |
We could ask Google HTTP Client if they can add an option to disable query string escaping, but I think we should first verify Apache doesn't do escaping either. I feel like it is quite possible that they may escape query strings anyways. |
|
So this is very much a duplicate of the 401 errors in #1986 (but the #1986 issue is wider in that it includes random 500 internal server errors). However, I've noticed one difference between this 403 issue and 401 in #1986. For the 401 issue, it is the unescaping of the query string that confuses Quay, whereas here it is the query string escaping. And worse, I realized unescaping is being done at a different place where But even so, it is still possible that Apache may do something similar. |
|
This pull request: googleapis/google-http-java-client#871 solves the issue. I tested it and it works fine. Currently, the PR has been approved and is waiting for a merge! |
|
Can you help @iocanel to approve/merge the PR please @chanseokoh ? |
|
When is it scheduled to release |
|
@cmoulliard it's likely next year. But it won't get too late. |
|
@cmoulliard 2.0.0 is released, and I expect Jib will work fine with |
|
I did a test and that works |
Issue
We cannot pull an image from the
registry.redhat.ioas jib is returning an error403even if we pass the parameters
-Djib.from.auth.usernameand-Djib.from.auth.passwordError
Steps to reproduce
Docker login
The command
docker login -u xxxxx -p yyyyyregistry.redhat.io works without issues as I can pull the imageThe text was updated successfully, but these errors were encountered: