-
Notifications
You must be signed in to change notification settings - Fork 5.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to query provider "https://argocd-host/api/dex": 502 Bad Gateway: #3975
Comments
I've been facing this same issue for months 😞 I found that this happens when the With I would have expected to see traffic routing This makes me suspect that Argo is trying to egress out of the server in order to talk to dex? See above for a screenshot of Kiali when trying to visit For what it's worth, I can spin up a random pod elsewhere in the cluster and I can communicate with the
Here is the accompanying envoy log for the random pod -> dex-server request:
Here is the envoy log for the argocd-server -> ??? request:
Things I've tried so far:
|
@samwhite, out of curiosity, what version of Istio/k8s are you running in your setup? I've got a story in my backlog to get Istio upgraded and just curious if I'll be able to use this issue to throw some more weight behind that upgrade or not. Glad to know we're not the only ones facing this issue though. Are you just using password auth in the meantime? |
I've seen this in Istio 1.4, 1.5, and 1.6. Same results with each version:
In the meantime we're using password auth, yeah, but sharing a password with an ever-growing team is not wise... |
@samwhite I did some additional testing on our cluster. If I changed the dex server endpoint via the the --dex-server flag to an https endpoint, the error I get back from argocd, is that the wrong protocol was being used when trying to reach the openapi endpoint, so it seems to be reaching dex in that case. I tried disabling MTLS between the argo and dex, and that had no effect either. |
Anyone got any hints for this problem ? Apparently i'm also having the similar problem. Argocd reverse proxy to dex server returned 503. |
same problem with github Oauth |
We have the same problem. In older versions of ArgoCD it worked sometimes. We wondered why and came to the conclusion, that if the Dex server and ArgoCD server pods are running on the same node, it works fine. We then went on to figure out, why it stopped working in newer versions and came to the conclusion, that an anti affinity rule was introduced to the Dex pod template. I honestly don't know, why it behaves in such a way but this could maybe help someone to figure it out. We use ArgoCD 1.8.1 and Istio 1.8.1 |
The problem here is that argocd reverse proxy to dex does not rewrite the |
@akhalyavka that worked -- thank you! ⭐ |
Thanks @akhalyavka it worked! can you please tell me how changing the service name affects it? As the protocol of service is still same (TCP). is it sending TCP requests now to the dex server just because we have changed the name? |
@hanzala1234 Please note that this fix affects load-balancing and telemetry. #6183 fixes it on ArgoCD side, so that you don't need to rename the service. |
Reopening due to continued reports of issues on the fix PR: #6183 |
FWIW Here's a list of changes which are needed for ArgoCD to work with Istio In addition to the list in this link, also the dex service port needs to be renamed from |
Sorry for coming back to this so late. I must have screwed up my testing, it seems like there is at least one more place where the header needs to be rewritten. I'm gonna take another stab at it |
Hit this issue on 2.7.6 and renaming the service made the trick |
hello @tchellomello can u please what changes u maid |
@tchellomello can we connect on linkedin i sent the request |
For this DEX issue, basically renaming the service as explained on the comment above. |
I tried reproducing it and everything works perfectly fine after the fix. Envoy correctly routes all requests to dex-server through I believe the issue you've hit could be related to argocd's own TLS between server and dex. In the default argocd installation the port is named The latter option is far superior: no double-encryption (istio mTLS over argo's TLS), and Istio can parse, manage and meter http/grpc requests in argocd namespace normally. Adding this block to
|
@okhaliavka thank you very much for your input regarding Istio. After considering and applying your advice to disable Argo's TLS to avoid double-encryption, it fixes the Google login. However, every application enters "Unknown" status, becoming unable to compare the commits pushed to git with the app status in k8s. On the UI pop-ups, we can see 2 kinds of errors:
(this first is also available at "Application conditions") and
Can you give any more insightful tips to deal with this issue? ArgoCD version: 2.11.2 (helm chart 7.1.1) |
@r-trigo I fixed it by also adding this config:
|
We are on argo version 2.11.4 And the below annotations on the ingress resource resolved the sudden issue for us:
|
I reached-out about this issue in the ArgoCD slack but received no reply.
Checklist:
argocd version
.Describe the bug
I've gotten ArgoCD
v1.6.1
deployed to an EKS cluster runningv1.15.11-eks-af3caf
along with Istio1.4.7
successfully. Now i'm trying to get SAML auth enabled with Dex via Okta as described in the documentation. However, when I click on theLogin via Okta
button at the login page I'm immediately met with an error page stating:Failed to query provider "https://argocd-host/api/dex": 502 Bad Gateway:
Attempting to authenticate via the CLI returns the same error as well.I've dug through the documentation and the issues here on github and haven't been able to find much but have ensured that the
data.url
config parameter is set correctly. I also haven't been able to find anyone through general googling that seems to have the same sort of setup with running EKS, istio ingress and AWS NLB, so I'm not sure if perhaps there's something about this particular combination that's screwing things up. I do have a few other services/apps running in this same cluster (grafana, jenkins, etc.) that are using Okta authentication without issue, so I don't believe there's any sort of ingress/egress rules getting in the way of things.To Reproduce
Have Istio
1.4.7
running in an EKS1.15
cluster deployed with a configuration very similar to what's documented in this issue with the--insecure
flag set onargocd-server
. The insecure flag is set because I am terminating at the AWS NLB. Configure theargocd-cm
ConfigMap as the following:Go to the configured ArgoCD URL of
https://argocd-host
and click on theLogin Via Okta
button that now appears.Expected behavior
Authentication should happen successfully via Okta and I should be taken to the
Applications
page on my ArgoCD instance.Screenshots
I don't think a screenshot is necessary but the error message displayed after trying login via okta is:
Version
Logs
I have enabled debug logging for the running services, though I didn't too many more log entries as a result
argocd-server log:
argocd-dex log:
The text was updated successfully, but these errors were encountered: