-
Notifications
You must be signed in to change notification settings - Fork 809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Almost standard helm install results in 500 internal error due to redirect loop when connecting to user notebook #3298
Comments
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗 |
Ah breakthrough, port-forwarding port 8000 of the proxy pod does allow me to access it using localhost:8000. I managed to get things to work by manually editing the kubernetes resources. But I am not sure if this is the intended deployment or if I am missing the idea behind the current setup. |
Ok, so I thing something like this might be needed to make the public service work correctly. |
Is your ingress controller pods labelled to be allowed network access to the proxy pod? They are sending traffic to the proxy pod - so they need the label if you have network policy enforcement in your k8s cluster. |
Was the user server restarted since the upgrade - or left running since before? |
Make sure you have read the changelog for 3.0.0 as well before continuing if you havnt, i've forgotten what was breaking so i cant rule out something there is of interest. / From a mobile device |
@consideRatio Thanks for having a look. I think the problem is the selector for the public proxy service is not specific to the Proxy pods and picks up the hub pods too. (But could definitely be missing a trick) I did a complete clean install with the config above, which should be acceptable according to the schema of 3.2.1. |
Also I don't believe I have network policy enforcement enabled ( yet, you know how it is :) ) |
The config doesnt make sense to me, you have both ingress and procy.service.type nodeport. That allows flows in two ways, one directly thorugh the node port to the proxy pod and onwards, and one via an ingress controller. Are you using an ingress controller? When using an incress controller, you wont need proxy service type nodePort and can use ClusterIP instead for the proxy service, which is proxied to by the ingress controller. |
@consideRatio Yeah, I used NodePort to try and debug the issue without going through my traefik ingress. I managed to get things working by applying these patches using ansible:
I couldn't find a way to get soemthing like this to work with the current helm config / templating |
What was the labels on the pod template patched, and the labels on the service selector patched before and after? If they dont target the right pod, its quite weird. |
the public proxy service:
the proxy deployment:
|
The hub deployment which I believe also matches the service selector:
|
Hmmmm, these labels like app.kubernetes.io/instance are not what i expect to see. Is this really a default installation? This chart use old label naming, "app: jupyterhub" etc |
@consideRatio Oh no you are entirely correct. I install jupyterhub as a subchart to a custom chart I helpfully called "jupyterhub". Thank you so much for you help! This thing had me tearing my hair out... |
Ahh, it happens! I'm glad you got it resolved and thank you for following up on the resolution ❤️ 🌻 🎉 |
Bug description
I tried upgrading my jupyterhub deployment from version 2.0.0 of the helm chart to version 3.2.1.
But I am totally stumped by this redirect error.
I tried to reduce the helm config I use to the absolute simplest form and I cannot seem to get things to work.
I found many people with similar issues, but always when trying to run locally.
Where the suggested solution is to use a different port to connect to.
So maybe there is an issue with how the helm chart ingress is setup?
How to reproduce
Your personal set up
Version(s):
Configuration
Helm values:Logs
Hub logs:User notebook:
Configurabe Http Proxy:
Is this a bug, or am I simply missing some combination of config values?
Any help would be greatly appreciated.
The text was updated successfully, but these errors were encountered: