JupyterHub gets wrong class in MainKernelSpecHandler.get()

[Litchilitchy]

nb2kg/nb2kg/handlers.py

Line 341 in 61863c8

ksm = self.kernel_spec_manager

if we use the same configuration (this config works in jupyter notebook) to start jupyterhub

--NotebookApp.session_manager_class=nb2kg.managers.SessionManager \
--NotebookApp.kernel_manager_class=nb2kg.managers.RemoteKernelManager \
--NotebookApp.kernel_spec_manager_class=nb2kg.managers.RemoteKernelSpecManager

we found this code ksm is KernelSpecManager, but ksm.list_kernel_specs() this method belongs to class RemoteKernelSpecManager, thus would raise an error

However this error does not occur when we use it in jupyter notebook

Any idea?

[kevin-bates]

Hello. You don't mention the error that is actually produced, but I assume its the classic ...

[E 06:46:20.341 NotebookApp] Uncaught exception GET /api/kernelspecs (::1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/kernelspecs', version='HTTP/1.1', remote_ip='::1')
    Traceback (most recent call last):
      File "/opt/anaconda3/envs/enterprise-gateway-dev/lib/python3.6/site-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/opt/anaconda3/envs/enterprise-gateway-dev/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
        yielded = next(result)
      File "/opt/anaconda3/envs/enterprise-gateway-dev/lib/python3.6/site-packages/nb2kg/handlers.py", line 324, in get
        kernel_specs = yield gen.maybe_future(ksm.list_kernel_specs())
    AttributeError: 'KernelSpecManager' object has no attribute 'list_kernel_specs'

This occurs when the class mappings have not been established, yet the nb2kg extension is installed and enabled. This implies your single-user configuration is either not getting detected or is incorrect.

Looking at @lresende's blog post your Notebook configuration is likely missing the 3 class mappings. If you're running in a Kubernetes environment, you can address this by using the elyra/nb2kg image - which is built directly on the jupyterhub/k8s-singleuser-sample image. If you're not running in Kubernetes, then you need to add the class mappings to the per-user notebook configuration used by JupyterHub.

FYI - these configuration issues are dramatically decreased in the 6.0 release of Notebook where NB2KG is actually embedded in Notebook and all you'll need is the one argument --gateway-url ${KG_URL} or equivalent if using the configuration file.

[Litchilitchy]

@kevin-bates yes, exactly this error.

And seems notebook 6.0 release does not exist yet? Currently I suppose what we need to do to connect the gateway is

1. set all the environment variables, modify gateway config, start gateway

2. download kernel, modify kernel.json corresponded (e.g. for Yarn Cluster)

3. download nb2kg and start with nb2kg configuration.

So after 6.0 release we just need to keep step 1 and 2, for step 3, we just need to add --gateway-url config

Am I right? Also want to ask when would 6.0 release?

[kevin-bates]

Yes - you're correct regarding the configuration steps to get Notebook/Lab using Enterprise Gateway. Please note that we provide pre-built kernelspecs for our supported configurations. You can find these as artifacts associated with our releases.

Regarding 6.0, my guess is that it's still a ways away as there are a good number of PRs that are probably destined for 6.0 rather than say, 5.8.0. As a result, you'll need to go with the current configuration steps/documentation for the time being.

[Litchilitchy]

seems its configuration problem, the jupyterhub can not directly read the notebook configuration.

I guess it should take c.Spawner.args and feed the notebook configuration here. Not tested yet, will update when we finish.

[kevin-bates]

I believe you're correct. @lresende has some ansible scripts for setting up Hub with EG in Kubernetes. In there he adds an extraConfig stanza which creates a custom spawner subclass in order to transfer the username properties (which you'll want to do), but also sets the notebook config file.

If you're also configuring for Kubernetes, ignore the setting of namespace_name_template (and was intentionally not highlighted).

[Litchilitchy]

@kevin-bates finally we take it as c.Spawner.args and it works!

To sum up, seems the most important steps are kernel.json config to use remote cluster kernel, and in notebook, just set nb2kg class to be the kernel manager

By the way jupyterhub spawns the same notebook with jupyter notebook if we takes the c.Spawner.args as the same config in notebook.

We will keep our project going and try to keep up with the simplest configuration after notebook 6.0 release

[Litchilitchy]

Also RemoteKernelSpecManager is the default value of NotebookApp.kernel_spec_manager_class, this could be a clue that the config has not been set