When logging in with Google auth, pages hang on 'Loading' - Logs show "Failed to verify token: failed to verify token: Failed to query provider .../api/dex/.well-known/openid-configuration\": dial tcp 10.x.x.x:443: i/o timeout"

[HariSekhon]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After enabling Google auth and logging in with a Google user the argocd-server pod gets timeouts referring to the internal GKE load balancer.

This doesn't happen when I log in with the local admin user.

A restart of the argocd-server pod fixes this.

Seems like buggy state behaviour, similar to #13526.

To Reproduce

Enable Google auth following these exact steps:

https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/google/#openid-connect-using-dex

Expected behavior

Expected the applications and other pages to load normally like with the admin user, which they do only after an argocd-server pod restart.

Screenshots

Version

argocd: v2.9.5+f943664.dirty
  BuildDate: 2024-01-19T22:59:02Z
  GitCommit: f9436641a616d277ab1f98694e5ce4c986d4ea05
  GitTreeState: dirty
  GoVersion: go1.21.6
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.9.6+ba62a0a

Logs

level=warning msg="Failed to verify token: failed to verify token: Failed to query provider
\"https://argocd-production.domain.co.uk/api/dex\": Get \"https://argocd-production.domain.co.uk/api/dex/.well-known/openid-configuration\": dial tcp 10.x.x.x:443: i/o timeout"

[HariSekhon]

Might be related to #11392

[andrii-korotkov-verkada]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and let us know if the issue is still present, please?