mybinder.org sub-domains: ovh.mybinder.org

[jagwar]

Hello,

OVH is interested in providing myBinder instance on the top of our public cloud. We are aware of https://jupyterhub-team-compass.readthedocs.io/en/latest/binder/subdomains.html

As discussed here: #144 can you provide us a sub-domain ovh.mybinder.org ?

Thanks :)

[betatim]

Sorry for the late reply.

We need a domain name/IP where your instance can be found to setup the domain.

@jupyterhub/binder-team are there other things we should discuss or things to agree first?

[mael-le-gal]

Hello Betatim,

Our binder instance can be reach here :
https://binder.mybinder.ovh/

I think there is points we need to discuss, I just post a related comment on the following issue :
jupyterhub/mybinder.org-deploy#759

[choldgraf]

shall we close this issue and take care of this in the mybinder.org-deploy issue linked above?

[betatim]

I'd keep this open until we have an entry for ovh.mybinder.org that points to binder.mybinder.ovh. We can already create that DNS entry now I think. (By we I mean you because I think you are the only one with access to the account)

[choldgraf]

@betatim then shall we close jupyterhub/mybinder.org-deploy#759 ? I just meant that it seems we have duplicated issues there and here (or maybe this issue is more specific to the sub-domain so we should treat it as a subset of jupyterhub/mybinder.org-deploy#759 ?)

[betatim]

or maybe this issue is more specific to the sub-domain so we should treat it as a subset of jupyterhub/mybinder.org-deploy#759 ?

That is how I see this issue. It is the formal request to get the subdomain and will be closed either when we reject the request or create the DNS entry. The latter can be done before jupyterhub/mybinder.org-deploy#759 is done.

[betatim]

Maybe you can share the details for updating the DNS entries so we grow the number of people who can do this. Bus factors, travel and all that.

[choldgraf]

@jagwar and @mael-le-gal congrats on getting jupyterhub/mybinder.org-deploy#970 merged and taking another step towards N>1 mybinder.org binderhubs!

[choldgraf]

Hey folks - I'm trying to get the domain ovh.mybinder.org pointing to binder.mybinder.ovh. To do this, I thought I would just make a CNAME record, but an early attempt at this resulted in some HTTPS errors.

I think we ran into this with other domains as well - CNAME redirects don't place nicely with HTTPS. I think our next best option would be to use an A record to point that domain to the ovh binderhub. Does that sound right to folks? (and if so, what IP address should I use?)

[betatim]

I think the reason the CNAME setup doesn't work is because the server that answers doesn't have a certificate for ovh.mybinder.org. It only has one for binder.mybinder.ovh. Which means your browser is alarmed.

Right now dig ovh.mybinder.org shows that it is setup with a CNAME that points to mybinder.org. If you change the record back to binder.mybinder.ovh we can then go and update the kubelego configuration for the OVH cluster to obtain a certificate for ovh.mybinder.org. Then everything should start working.

https://github.com/jupyterhub/mybinder.org-deploy/blob/f667f9fefc8d9850dc3c5924242787b8f042cace/config/ovh.yaml#L17-L30

is what needs changing. The hub-binder.mybinder.ovh domains could also be changed if we want to, or left as is. What ever people find more tidy.

[choldgraf]

OK, I just realized that I also have the cloudflare account details. And mybinder.org is there as well!

So I wonder if it's just that my bluehost account acts as if it can control the DNS, when it really cannot. I just set ovh.mybinder.org to point to binder.mybinder.ovh with a CNAME on the cloudflare. Let's see if that updates it properly after it propagates.

[betatim]

$ dig @8.8.8.8 ovh.mybinder.org
;; ANSWER SECTION:
ovh.mybinder.org.	299	IN	A	104.27.160.39
ovh.mybinder.org.	299	IN	A	104.27.161.39

so something isn't right :-(

[choldgraf]

Hmmm, well this is the configuration that's on cloudflare:

I'm not sure why there would be a difference there.

[consideRatio]

Background theory

The host header

Web request are made to an IP, but if you provide a host header it matters greatly. It will affect how traffic is routed.

Ingress controllers routing

Say you have a k8s cluster with an ingress controller, that means that you have something in the k8s cluster that will look at the k8s Ingress resources you can find by writing kubectl get ingress. This ingress controller gets its traffic from a single IP typically, and will route it based on rules described in the Ingress resources and manage certificates as well based on that.

# This will route okay as curl will provide a header of `host: binder.mybinder.ovh`
curl binder.mybinder.ovh -L
<!DOCTYPE html>
...

# This request will route to a default backend that simply returns a string with 404 in it,
# the exact behavior of nginx-ingress-controller I know from experience.
$ curl binder.mybinder.ovh --header "host: ovh.mybinder.org"
default backend - 404

Some terminal inspection written down

erik@xps:~$ nslookup ovh.mybinder.org
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	ovh.mybinder.org
Address: 104.27.161.39
Name:	ovh.mybinder.org
Address: 104.27.160.39
Name:	ovh.mybinder.org
Address: 2606:4700:30::681b:a027
Name:	ovh.mybinder.org
Address: 2606:4700:30::681b:a127

erik@xps:~$ nslookup binder.mybinder.ovh
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	binder.mybinder.ovh
Address: 51.75.198.240

erik@xps:~$ curl 104.27.160.39 --header "host: ovh.mybinder.org"
default backend - 404

erik@xps:~$ curl 104.27.160.39 --header "host: binder.mybinder.ovh"
<!DOCTYPE html>
...

erik@xps:~$ curl 51.75.198.240 -L --header "host: binder.mybinder.ovh"
<!DOCTYPE html>
...

Thoughts

1. I think there are currently two backends the OVH binderhub. I'm confused.
2. The binder.mybinder.ovh binderhub is up and running, but the ingress controller is not routing properly for ovh.mybinder.org. Updating the current ingress object would resolve this. Add one more host is possible but changing it is probably easier when using kube-lego to manage the certificates. Restarting kube-lego stuff is probably required.
3. I'd recommend deploying binderhub with nginx-ingress-controller and cert-manager to handle the TLS stuff with a metachart that requires binderhub, nginx-ingress, and cert-manager, like at https://github.com/consideratio/neurips.mybinder.org-deploy for example do. I note that mybinder.org does not use cert-manager but kube-lego to manage this.

[choldgraf]

hmm, perhaps some of the OVH folks could provide their thoughts? (@jagwar and @mael-le-gal)

[betatim]

$ dig pop.mybinder.org
;; QUESTION SECTION:
;pop.mybinder.org.		IN	A

;; ANSWER SECTION:
pop.mybinder.org.	300	IN	CNAME	mail.mybinder.org.
mail.mybinder.org.	300	IN	A	66.147.242.174

so that works. What does the orange cloud symbol mean in the cloudflare overview?

[jagwar]

Hi, I will talk with @mael-le-gal tomorrow, he was in holiday. I think we will try to deploy binderhub with nginx-ingress-controller and cert-manager to handle the TLS stuff with a metachart that requires binderhub.

[betatim]

I don't think there is a problem with how the OVH cluster is configured. The problem is with the configuration of the ovh.mybinder.org DNS entry. I think the first/initial confusion was that setting up ovh.mybinder.org to point to the IP of the oVH cluster lead to some error messages related to SSL certificates. I think that was expected because the cluster wasn't configured to obtain a certificate for that domain.

The way forward is to sort out the problem with the DNS configuration and then configure the OVH cluster to use ovh.mybinder.org as hostname and obtain a certificate for that.

Switching to cert-manager would be a nice thing for all BinderHubs but won't fix the current problem.

[choldgraf]

Oops! Accidental close

I'm not sure exactly what the orange cloud means, this page is the best I could come up with but I'm not sure why that particular entry is an orange cloud while others are not.

[Carreau]

I'm not sure exactly what the orange cloud means, this page is the best I could come up with but I'm not sure why that particular entry is an orange cloud while others are not.

Orange cloud for binder does not change much as the content is dynamic. It's likely better to avoid it; and cloudflare will not try to be smart about the content and cache it (among other).

[mael-le-gal]

Hello everyone,

Sorry for the answering latency, I took some days off last week. But I'm back ! :)

I added a new ingress in the ovh k8s supporting the ovh.mybinder.org host with kubelego and everything seems to be ok now, you can check by yourself on the URL : https://ovh.mybinder.org/

I will do a PR for adding the deployment of this new ingress on the OVH cluster.

My question is : should we keep the 2 separated access points ? binder.mybinder.ovh + ovh.mybinder.org for binder containers or should we only keep the ovh.mybinder.org ?

On the first case we will deploy 2 ingress whereas in the second case we will deploy only 1.

[betatim]

I think there is no need to have two names that normal users use. I would keep the setup that ovh.mybinder.org is a CNAME and not points directly to an IP though as it makes it easier for you to change the IP when you move the cluster etc.

Nice work all around on getting the domain trouble sorted.

ps. Welcome back from your holiday :)

[choldgraf]

I can confirm that ovh.mybinder.org works for me! woo! So what is left, getting the redirector that @betatim put together deployed on prod?

[betatim]

We are now a federation: https://twitter.com/mybinderteam/status/1136512810475765760

The last missing piece is jupyterhub/mybinder.org-deploy#986. We are currently "losing" about 2% of our traffic in our stats as we are sending 2% to the OVH cluster. As long as it is only 2% it makes no difference to our stats. But before we increase the traffic we need to get our analytics back. Having the numbers to back our case is important for funding/sponsorship/etc to show we are having impact with what we are doing.

Thanks everyone!

[mael-le-gal]

For the current issue, I just pushed a PR here : jupyterhub/mybinder.org-deploy#1010

The travis will then always deploy the ovh specific ingress for ovh.mybinder.org

[choldgraf]

wow, this is so exciting! Thanks to everybody for all of their hard work on this!

[willingc]

Great job folks!

[mael-le-gal]

I guess this issue can be closed @jagwar :)

[choldgraf]

Indeed! I think we have a blog post issue as well so we can use it to track that topic.