BinderHub setup on Azure timesout when trying to contact docker registry

[markdunning]

When following zero to binderhub instructions using Azure, the following command:-

helm install jupyterhub/binderhub --version=0.2.0-3b53fce     --name=sheff-hub     --namespace=sheff-hub -f secret.yaml -f config.yaml

Produced this error message:-

Error: release sheff-hub failed: clusterroles.rbac.authorization.k8s.io "sheff-hub-image-cleaner" is forbidden: attempt to grant extra privileges: [{[get] [] [nodes] [] []} {[patch] [] [nodes] [] []}] user=&{system:serviceaccount:kube-system:tiller ...... [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[[clusterroles.rbac.authorization.k8s.io "system:discovery" not found, clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]]

We managed to fix this using this issue. However, when trying to use my own binderhub with an example repo I get an "Internal server error". I have checked by dockerhub username and password and they are correct.

I can look at the log files with the command

kubectl logs binder-7dd4d6dc95-dzrvr -n sheff-hub

and the following error is seen

[I 190318 15:43:34 repoproviders:441] Using cached ref for https://api.github.com/repos/binder-examples/requirements/commits/master: a73ba121c9847fa38b7c4153230b9bfa9eecfaa7
[I 190318 15:43:38 web:2106] 200 GET /about (10.244.1.1) 1.24ms
[I 190318 15:43:43 web:2106] 200 GET /about (10.244.1.1) 1.14ms
[I 190318 15:43:48 web:2106] 200 GET /about (10.244.1.1) 1.22ms
[I 190318 15:43:53 web:2106] 200 GET /about (10.244.1.1) 1.32ms
[E 190318 15:43:54 web:1621] Uncaught exception GET /build/gh/binder-examples/requirements/master (10.240.0.6)
    HTTPServerRequest(protocol='http', host='40.74.53.217', method='GET', uri='/build/gh/binder-examples/requirements/master', version='HTTP/1.1', remote_ip='10.240.0.6')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/tornado/web.py", line 1543, in _execute
        result = yield result
      File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/usr/local/lib/python3.6/site-packages/binderhub/builder.py", line 273, in get
        image_manifest = await self.registry.get_image_manifest(*'/'.join(image_name.split('/')[-2:]).split(':', 1))
      File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
        yielded = self.gen.throw(*exc_info)
      File "/usr/local/lib/python3.6/site-packages/binderhub/registry.py", line 199, in get_image_manifest
        auth_resp = yield client.fetch(auth_req)
      File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
    tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20001 milliseconds with 0 bytes received

[betatim]

The bit where the error happened is trying to talk to the docker registry you configured. In this case it timed out which sounds like a network problem. It isn't even at the level where it tried the credentials and then failed. Maybe a firewall issue or docker hub (are you using docker hub?)being down/unreachable?

Could you also post your values.yaml and other details about your setup?

[betatim]

Could it be this #811?

[alexmorley]

@betatim I don't think we were using the values.yaml in the setup. But we did check the login details for dockerhub to make sure they were correct (and so I don't think it's #811).

[markdunning]

Thanks for looking into this. Yeah, I don't think I have values.yaml. I was also using my dockerhub name alone, and not any organisation IDs.

[betatim]

I don't have any good ideas :-( Maybe worth moving this discussion to http://discourse.jupyter.org/ in the hopes that it gets more eyes/fresh eyes on it? I'll close this under the assumption that we either move the discussion or that it has somehow solved itself/become low priority. We can reopen this when there is new info/ideas.

[ggorman]

I've run into the same problem. Using kubectl logs I get the following backtrace.

[I 190601 09:03:19 repoproviders:447] Using cached ref for https://api.github.com/repos/opesci/devito/commits/binder: dd2a5f878159bcc2b2523c67384fdc01c18a2d68
[I 190601 09:03:23 web:2106] 200 GET /about (10.244.2.1) 1.62ms
[I 190601 09:03:28 web:2106] 200 GET /about (10.244.2.1) 1.22ms
[I 190601 09:03:33 web:2106] 200 GET /about (10.244.2.1) 1.45ms
[I 190601 09:03:38 web:2106] 200 GET /about (10.244.2.1) 1.39ms
[E 190601 09:03:39 web:1621] Uncaught exception GET /build/gh/opesci/devito/binder (10.244.2.1)
HTTPServerRequest(protocol='http', host='devito.rocks', method='GET', uri='/build/gh/opesci/devito/binder', version='HTTP/1.1', remote_ip='10.244.2.1')
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tornado/web.py", line 1543, in _execute
result = yield result
File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
value = future.result()
File "/usr/local/lib/python3.6/site-packages/binderhub/builder.py", line 273, in get
image_manifest = await self.registry.get_image_manifest(*'/'.join(image_name.split('/')[-2:]).split(':', 1))
File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/site-packages/binderhub/registry.py", line 199, in get_image_manifest
auth_resp = yield client.fetch(auth_req)
File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
value = future.result()
tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20002 milliseconds with 0 bytes received

[ggorman]

Possibly related to https://discourse.jupyter.org/t/trouble-starting-a-binder-after-inactivity-while-my-hub-gently-sleeps/1010

[h4818]

check or remove token_url config item:
config.DockerRegistry.token_url