Prototype new kernel discovery machinery

[takluyver]

Following jupyter/nbformat#81, this PR introduces extensible kernel finder machinery, to pave the way for different kernel pickers.

The kernel finder machinery would work like this:

• A MetaKernelFinder object holds a collection of different kernel finder objects.
• Each kernel finder has a short ID, and two methods:
  • find_kernels() yields pairs of id (str) and kernel attributes (dict). The attributes contain information about the kernel. At present this has no required fields, but we'll probably want to require things similar to the language info in kernel_info_reply.
  • make_manager(kernel_id) returns an instance of KernelManager, or a subclass, ready to start a kernel of the specified kind.
• The implementation here provides two kernel finders: one using kernel specs (the same way we already look for kernels), and one trying to import ipykernel in the current Python.
• Other kernel finders could find kernels in conda environments, virtualenvs, docker containers, VMs, or remote hosts. These would be distributed separately and found using entry points (not yet implemented).
• The MetaKernelManager namespaces kernels IDs according to the finder responsible for them, so an ID such as spec/python3 means the python3 kernel from the spec finder. Other qualified kernel IDs might look like: conda/myenv or docker/ipython/ipykernel-nightly.
• Specifying a kernel at e.g. a command line will use the qualified ID, e.g. jupyter console --kernel spec/python3. I propose that unqualified names (e.g. python3) are treated as if they were prefixed with spec/, ensuring backwards compatibility.
• Thus, specifying pyimport/kernel (or whatever we change that name to) will always get a kernel running on the same Python executable as the process starting it.
• When we start a kernel for a notebook, a KernelPicker object (not yet implemented) will get the IDs and attributes from find_kernels() and use it to select the most appropriate one.

Implementation notes:

• This involves going back on changes we had previously made to KernelManager to make it aware of kernelspecs. In this design, the kernel discovery machinery sits a level above KernelManager, rather than a level below it. In the long run, I think that we'll be glad of reduced complexity in KernelManager, but in the shorter term, it still needs to be aware of kernelspecs for backwards compatibility.
• We will likely want some finders to cache kernel info, especially if finding kernels involves launching processes or communicating with other systems. I haven't yet tried to design any mechanisms to control gathering or invalidating cache information, but I expect that we'll need these.

[rgbkrk]

Cool, I definitely want to be tracking this.

[damianavila]

Cool, I definitely want to be tracking this.

Me too 😉
Pinging people who is probably interested on this as well: @JanSchulz, @bollwyvl and @Cadair

[Cadair]

👍

[willingc]

Would be very helpful for education folks if things are simplified.

[bollwyvl]

Would love to make sure that nbconvert --execute can take part in this!

[bollwyvl]

These would be distributed separately and found using entry points (not yet implemented).

hooray entry_points!

[takluyver]

Would be very helpful for education folks if things are simplified.

I can't really claim that this will simplify things, unfortunately - it's probably more complex, at least in the default case. My aim is to make it easier to combine different ways of finding kernels.

[willingc]

@takluyver Oddly, I think that will simplify things on a higher level ;-)

[jankatins]

AFAIK env is also possible in the json file

[takluyver]

It certainly is, but the aim of this is not necessarily to expose all the information from the kernelspec file; it's to provide a kernel picker with the information it may need to select a kernel. Maybe env is part of that, though. One of the big parts I haven't yet addressed is what information is required in this dictionary, and what standardised but optional fields we'll provide.

[minrk]

Maybe it should be the whole spec? Just so that we don't have to be responsible for what info the decider uses to select which kernel to run?

[jankatins]

Why this change?

Or better, why not:

if self.extra_env:
   ...

[takluyver]

KernelManager is rather complex because of the evolution of how we use it.

1. Originally it had a configurable kernel command line, which you could use to make it start a different kernel, before Jupyter (then IPython) was really aware of other kernels. But this changed it for every KernelManager in a process, so you couldn't mix kernels in one notebook server, or one Qt console.
2. Then with kernelspecs, it was meant to be given the name of a kernelspec, and it would retrieve argv and env from that. But we left the machinery for 1. in place so that people configuring kernel_cmd wouldn't find it broken as soon as they upgraded.
3. In this PR, I've reused the leftover machinery for 1. to allow kernel_cmd as an input, but not as a config option, and added extra_env to fit with this way of doing it. But I don't want to immediately break things for code using 2. This elif branch is for 3., and the if branch above is for 2. They shouldn't be mixed.

[jankatins]

So If I would want to supply an env I would use

def make_manager(self, name):
    [...]
    env = get_env_for_name(name) # runs activate and would need caching to not do it again the next time
    return KernelManager(.., extra_env=env)

Sounds good to me

[jankatins]

For https://github.com/Cadair/jupyter_environment_kernels/ the most important feature (after an entry point :-) ) would be that it is possible to add env variables which are lazily loaded on kernel creation (in this case by running source activate in the env and outputting all env variables).

In our case, we currently accomplish that by supplying a loader function to a subclassed KernelSpec class:
https://github.com/Cadair/jupyter_environment_kernels/blob/master/environment_kernels/envs_common.py#L51

As far as I understand the new API, the find_kernels return a dict (we had a implementation of a lazy loading dict, but that was basically a big hack and broke on py3.6). If that's so, I would really appreciate if the dict could be changed to an object which can lazy load the env.

Or did I misunderstod the API and that could be accomplished in the make_manager method?

[jankatins]

We will likely want some finders to cache kernel info, especially if finding kernels involves launching processes or communicating with other systems. I haven't yet tried to design any mechanisms to control gathering or invalidating cache information, but I expect that we'll need these.

The jupyter environment kernel manager has some kind of caching nowadays: https://github.com/Cadair/jupyter_environment_kernels/blob/master/environment_kernels/core.py#L110-L124 It currently caches information where kernels are, but not the env variables, which are lazily loaded. Cached values are updated every 3 minutes.

[rolweber]

Is it really necessary to load all environment variables into the kernel spec? If you're going to start a kernel in a conda env, why not start it through a wrapper script that calls source activate before starting the kernel? Seems simpler to me than executing source activate, reading all environment variables, putting them into the kernel spec, and then creating a new environment with the exact same variables to start the kernel with.

[takluyver]

As far as I understand the new API, the find_kernels return a dict (we had a implementation of a lazy loading dict, but that was basically a big hack and broke on py3.6). If that's so, I would really appreciate if the dict could be changed to an object which can lazy load the env.

Or did I misunderstod the API and that could be accomplished in the make_manager method?

Yes, I think that this should be done in make_manager, or in a KernelManager subclass which make_manager returns.

The idea is that the dictionary contains the information needed to select which kernel to use; once you have made a choice, it calls make_manager to get the KernelManager object which will be used to launch the new kernel. This can use extra info which isn't necessary to select the kernel.

It currently caches information where kernels are, but not the env variables, which are lazily loaded.

For kernel specs, I think we will need to cache the language info in a persistent manner; getting the language info requires starting a kernel, making a kernel_info request, waiting for the reply, and shutting the kernel down again. We could save this as e.g. language.json in each kernelspec directory.

I think the kernel finders should have some kind of way to control cached information. Questions I have:

• Should the available control be 'updated cached information now', or clear/invalidate cached information so that it will be updated next time it's needed? Or both?
• Should updating the caches happen asynchronously? What do we do if the data we want isn't there yet?
• Should we distinguish between persistent on-disk caches and ephemeral caches?

[jankatins]

@rolweber

If you're going to start a kernel in a conda env, why not start it through a wrapper script that calls source activate before starting the kernel?

Unfortunately, that has the problem that (at least on windows, which has no exec) you get a process tree like this:

Kernel starter (python) -> batch file (cmd) -> Kernel (python)

This has it's own class of problems, one of them is that the current notebook can't interrupt/kill such kernels when they are busy (#104).

Getting the env variables wasn't actually the problem, the code is battle tested in xonsh. Getting the lazy loading was more work...

[jankatins]

For kernel specs, I think we will need to cache the language info in a persistent manner; getting the language info requires starting a kernel, making a kernel_info request, waiting for the reply, and shutting the kernel down again. We could save this as e.g. language.json in each kernelspec directory.

At least the environment manager will put the language tag into the json: https://github.com/Cadair/jupyter_environment_kernels/blob/master/environment_kernels/envs_common.py#L46

Why should that not work anymore?

[takluyver]

At least the environment manager will put the language tag into the json:

That's just the language name. We'll probably want to include some more of the details from the kernel_info_reply, like the language version and the kernel implementation.

[jankatins]

That's just the language name. We'll probably want to include some more of the details from the kernel_info_reply, like the language version and the kernel implementation.

As long as that is computed when the kernel is started and not when the notebook/lab server starts to know about the kernel, as otherwise the lazy loading is again in vain.

Anyway: we currently do a full import of the kernel classes/library, when we probe for a kernel in the env. So these two informations could already be gotten at that stage.

[takluyver]

That's why I was talking about persistent caching - that information probably requires starting kernel processes, but we'll probably want to know it when picking the kernel (e.g. distinguishing Python 3 vs 2).

[takluyver]

I've added a couple of tests, plus support for entry points (using the group name jupyter_client.kernel_finders).

These are, I think, the big remaining questions:

1. What information about kernels should be available when picking a kernel?
2. Can we rely on this information being rapidly available when we discover kernels (e.g. reading a small number of static files), or do we need mechanisms to gather the information asynchronously (e.g. starting kernels and communicating with them) and cache it?
3. If we do need asynchronous indexing, what happens if you try to use the machinery before all the information is ready? And when do we invalidate caches?

Information:

• Programming language - a Python kernel and a Julia kernel differ in an important way. I think it's reasonable to expect that this information is available from 'outside' the kernel, i.e. without starting it.
• Language version - how precisely to specify this is up to the implementation. Python 3 vs 2 is likely to be significant, but 3.5.1 vs 3.5.2 is probably not. This may be tricky to get without starting the kernel.
• Reference to environment/container? I'm thinking not - you can do equality checks with the kernel ID you get (which might look like conda/myenv), and I think that's the only operation that easily generalises across the different technologies.
• Space for kernel finders to stuff arbitrary information - if you want to use your custom kernel finder together with your custom kernel picker, they should be able to pass any information they care about.

What else might be relevant?

Also, using entry points ties this to Python, which would make it hard for implementations in other languages (e.g. nteract) to replicate this framework. Do we:

1. Provide interfaces (e.g. stdout, REST) to the discovery machinery which other languages can access, and expect them to rely on having Python & jupyter_client installed?
2. Discover the kernel finders using non-Python specific methods, e.g. scripts installed on PATH, which other languages can also use? This would mean significantly more complexity and worse performance for our own implementation, unfortunately.
3. Leave it up to kernel finder authors and frontend authors to write each kernel finder in each frontend language? This could be a nasty n*m problem, though there's a chance that m stays small (so far I only know of Python & Javascript frontends).

[takluyver]

I think the best option for other languages is some variant of 1 (interface with Python & jupyter_client to discover kernels). If frontends can still start their native kernel directly (e.g. a JS kernel for nteract), but require Python to find other kernels, perhaps this is not too onerous.

The variants of this:

1a. As the proposal currently stands, kernels discovered through this mechanism would have to be started through Python as well.
1b. We could serialise the information needed to start a kernel (e.g. argv, env, cwd) and pass it back to the frontend for that to start the kernel. I expect this is more attractive for frontend authors, but it offers significantly less flexibility to kernel finders: they lose the ability to return a kernel manager which talks to another machine, or takes a pre-started kernel from a pool, or 101 other clever things we haven't thought of yet. I guess they could do still do these things by quickly creating a script and asking the frontend to run it, but that feels messy and complex.

@rgbkrk I'd welcome your input from the nteract perspective (& that goes for any other nteract devs watching this). How would you like to interface with an extensible kernel discovery mechanism? Can you see some option I'm missing?

[rgbkrk]

/cc @lgeiger and @BenRussert for Hydrogen

I'll get up to date on the state of this and add new comments.

[minrk]

• Programming language

👍 to including language name

• Language version

In practice, I've found this information not helpful. I don't generally want to distinguish Python 2 from 3 in my notebooks. Separating py2 and py3 may have made sense when we made that decision, but I don't think it does anymore. A Python notebook is a Python notebook, and I think language version belongs more at the same level as env/dependency-specification, and not at the same level as language name. I think we should even consider dropping the python2/python3 default kernel names for IPython, and just use python.

• Reference to environment/container? I'm thinking not - you can do equality checks with the kernel ID you get (which might look like conda/myenv), and I think that's the only operation that easily generalises across the different technologies.

👍 to pushing env info one level down into the Finder spec for the given env type

• Space for kernel finders to stuff arbitrary information - if you want to use your custom kernel finder together with your custom kernel picker, they should be able to pass any information they care about.

👍 to a place for Finders to put their own extra info.

As for where these things reside on Finders being part of the spec, my inclination would be option 3: rely on native implementations. Kernel Finders can choose to define their own specs for implementation in any language. This should be quite doable for the common cases: (base kernelspecs, envs), and I suspect of less interest for the cases where it would be more difficult. I think requiring Finders to emit argv/env misses much of the point of custom kernel finders, which can do things like launch kernels via remote APIs. These shouldn't involve subprocesses at any point. We could have a middle ground, where Finders may emit an argv/env which is accessible via CLI (e.g. jupyter kernels list --json), from which non-argv-supporting kernelspecs would be excluded. But so far, if I recall, nteract has mostly chosen to reimplement things even when Jupyter provides these CLI APIsbecause it avoids pulling in cross-language dependencies. I'll leave it to nteract folks to comment on that, though.

Yet another CLI option is something like jupyter kernel start --json, which would actually handle invoking the kernel finder, manager, etc., and reply with connection info. That would probably be easier for alternative frontends to use, and further remove the need for kernel finders to emit argv or other serializable launch mechanisms.

[blink1073]

"kernel definition", as in, here is how to launch it and its associated metadata?

[dsblank]

Thanks for not using MetaKernel 😸 ... the naming is already a sensitive breakdown of the functionality without confusing it with that other project.

[blink1073]

Wait, haven't we already called this the "kernelspec"?

[takluyver]

Good point re Metakernel, I hadn't even thought of that.

"kernelspec" is also used for the kernel.json file (and the directory containing it) which is just an advertising mechanism for a kernel (sense 1) installed elsewhere. And the kernelspec machinery is somewhat tied to launching a process locally, as it relies on argv. The concept of a type of kernel can extend to kernels that start on a remote machine.

I think so far "kernel type" is the term I can most imagine using. "Kernel class" incorrectly implies a Python class. "Kernel launcher" sounds like the parent process machinery to start a kernel. "Kernel definition" is possible, but suggests static metadata rather than a program you can launch.

E.g. "There are 7 kernel types available on your system" or "You need to install the C# kernel type to run this notebook"

[willingc]

Perhaps we should do a draft doc page as part of this PR that summarizes this PR discussion and complements kernels.rst. It might help clarify functionality of the PR and naming as well. We could mark the doc file as subject to change or under active development.

[takluyver]

That's a good idea. I'll work on it this evening or tomorrow.

[willingc]

I wasn't thinking of anything fancy, @takluyver. More of a recap and nomenclature summary. I'll give the code a more thorough review later today or tomorrow. Thanks.

[rgbkrk]

I tend to think of the running kernel as a kernel instance

[takluyver]

Have some docs. It looks much nicer when run through Sphinx.

[willingc]

@takluyver Thanks for the excellent documentation. It goes a long way to clarifying usage and nomenclature. I've made some suggestions. The suggestions could be addressed in a second PR too.

[willingc]

Perhaps mention that the interface is subject to change.

[willingc]

Perhaps add one short sentence of why one might wish to create a kernel provider.

Add a subheading: Create a kernel provider

[willingc]

environment variables? pip environments? conda environments? Or implementations?

[willingc]

Probably good to specify since environment is a pretty overloaded term.

[willingc]

Subheading: Registering a Kernel Provider

[takluyver]

I've opted not to add a subheading here because it feels like part of the previous section to me. I've implemented all your other suggestions.

[willingc]

Sure no problem.

[willingc]

Perhaps move "Allows ... instances." to end of the paragraph so definition is first.

[willingc]

NIce!

[willingc]

"This...providers." Let's be a bit more detailed and concrete here before comparing to similar API. Perhaps recap at start of section what is a kernel type.

[willingc]

"Basic Kernel Providers included in Jupyter Client"

I'm not totally happy with my suggestion above but I think we need a bigger distinction between prior example section and this reference section.

[willingc]

Are we being consistent with the id before = as described in entrypoints doc section.

[takluyver]

Yep:

https://github.com/takluyver/jupyter_client/blob/c09b8aced5912901b4e96e11d9d4eeb3346b4fd7/jupyter_client/discovery.py#L30

https://github.com/takluyver/jupyter_client/blob/c09b8aced5912901b4e96e11d9d4eeb3346b4fd7/jupyter_client/discovery.py#L55

I can't work out when Github does inline code previews and when it doesn't!

[willingc]

Have some docs. It looks much nicer when run through Sphinx.

Rendered doc page on my test build. For the convenience of others.

[takluyver]

Thanks @willingc !

[willingc]

Thanks @takluyver. Looks great.

[blink1073]

Nicely done @takluyver and @willingc!

[takluyver]

Does anyone else want to review this? If not, I'll push the button in another day or two.

[takluyver]

Button being pressed.

[jankatins]

Is the discovery mechanism already used in any of the jupyter projects? As far as I see the notebook still uses jupyter_client.kernelspec.KernelSpecManager and that one does not yet use the new discovery mechnism :-( Is that correct?

[takluyver]

No, you're right, it's not used anywhere yet. We're still hashing out the details of how it works in different situations. #308 is a follow up to this PR.

[jankatins]

Damn, I just had some time and I wanted to port https://github.com/Cadair/jupyter_environment_kernels to it :-/ Ok, waiting for this to land in notebook land...

Is there a realistic chance that a PR which "just" converts jupyter_client.kernelspec.KernelSpecManager to the new system would be merged?

[takluyver]

If you've got time to experiment with making that work on top of PR #308, that could be useful - we're currently trying to figure out if we've got a useful API design by making prototype things on top of it (I've done an ssh provider and a docker provider).

Kernel specs become one default provider for the new discovery system, so no, we wouldn't convert KernelSpecManager to use the new system.