frontend/extension-config: allow default json files in a .d directory

[maartenbreddels]

Original was in ipython/traitlets#452
It was decided it should move out of traitlets.
This allows for instance a jupyter_notebook_config.d/jupyterlab.json to be placed at install time by pip to avoid users having to run jupyter-serverextension enable. Similar for jupyter-widgets.

Note from Steve:
The intent of this PR is to make it easier for notebook and server extensions to be distributed and consumed by users. By using data_files that populate the appropriate config.d directory, the extension is automatically enabled when installed unless overridden by higher level config. This approach works for both wheels and conda packages.

[blink1073]

Thanks for this!

[blink1073]

Maybe call this config_manager.py?

[maartenbreddels]

yeah, wasn't sure where to put it also, but wanted to start with a clean dup of the original PR. Shall I keep it in the root dir?

[blink1073]

I think so

[blink1073]

Mind adding short docstrings for these methods?

[bollwyvl]

Looking awesome! It will be a fine day for packagers and users when it lands.

Some remaining questions:

• how might we inform a user who is interested/having trouble about which paths will be used and in what order?
  • Perhaps consider a log.debug('Configuration paths for %s: %s', section_name, '\n'.join(paths)) or something
• how might a user/packager actually create one of these files?
  • Short of just touching it on disk, I guess a CLI... or at least start setting a precedent with nbextensions if that's all this covers right now:

jupyter nbextension enable --sys-prefix --py ipyvolume --confd ipyvolume.json

[maartenbreddels]

👍 on the logging (done).
I think package manager just create a file and ship that, I don't think this should be generated from a command line (we document it and people copy from existing projects I think).

[blink1073]

The above command would Just Work™️ when using a conda build though.

[blink1073]

As in, that file would be created and picked up as an artifact.

[blink1073]

Although, it wouldn't help with wheels...

[bollwyvl]

Although, it wouldn't help with wheels...

Seems like you could just include in package_data?

[blink1073]

Using data_files.

[bollwyvl]

Ah, right.

[blink1073]

Kicked the unrelated failing JS test

[damianavila]

Since you are porting this from traitlets, it shows some oldies... let's use the updated header.

[maartenbreddels]

I see another PR failing on that part, so I guess that is not related.

[blink1073]

cc @SylvainCorlay

[blink1073]

Unrelated errors fixed, this should be good to go.

[minrk]

This looks great! Going forward, where extension authors want to enable extensions just by creating a new file (the goal of removing post-link scripts in conda packages), what would an entry point for that be? An extra arg to nbextension enable that writes a new file rather than editing the existing one?

[maartenbreddels]

My feeling about this is that since it's for package creators and meant to put in place using setuptools' data_files, that it's fine that it is just documented somewhere how to do this (a TODO). Automating this doesn't add much value I think.

[bollwyvl]

Re: automation: perhaps it could be reasonable to create/update an existing cookiecutter to reflect how this can be used. I think the jupyterlab/*-cookiecutters have been useful in that regard!

[maartenbreddels]

Indeed, good point. (from mobile phone)

…

On 21 Dec 2017 6:05 pm, "Nicholas Bollweg" ***@***.***> wrote: Re: automation: perhaps it could be reasonable to create/update an existing cookiecutter to reflect how this can be used. I think the jupyterlab/*-cookiecutters have been useful in that regard! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3116 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABryPfID9qVX9xXqPMDn1AW_4Q5tBcZkks5tCo_ZgaJpZM4Q7FH_> .

[blink1073]

No, but when JupyterLab launches it checks for installed extensions that are not part of the JupyterLab bundle.

[stonebig]

ah, so if I want that to happen, I need to launch jupyterlab (in headless/useless mode) once ... or to keep these "labextension" install.

is there a simple way to launch jupyterlab just for that ? a sort of "jupyterlab --check_your_extensions_and_stop" ?

[blink1073]

You can run jupyter lab build at any time, no need to launch JupyterLab. It is a user convenience to auto-build if necessary.

[blink1073]

cf jupyterlab/jupyterlab#3546

[SylvainCorlay]

This is really cool, thanks @maartenbreddels @blink1073 .

[rgbkrk]

Thanks for this!

[jasongrout]

Thank you @maartenbreddels! I think this will solve a lot of the confusion around getting widgets working.

[mpacer]

So I really like this idea. I wish I'd realised this was part of the release sooner, because I really don't think this should be in the notebook library.

I actually am concerned that this is in notebook and not inside jupyter_core.paths, where the rest of the configuration finding mechanisms live, see this function as a partial reference:

https://github.com/jupyter/jupyter_core/blob/master/jupyter_core/paths.py#L180

Frequently, users, both experienced and inexperienced, have ended up in weird states with their config files as it is. The worst part about it is that its really hard to debug because there are so many places config files could live; as expected, these kinds of bugs hit more novice users the worst. That's why I released jupyter_conf_search. What made that utility possible was that there is a single canonical mechanism for finding your configuration files and the order in which they are loaded.

I'm worried that this is going to make these kinds of configuration bugs even harder to debug.

@gnestor Is it possible to remove this from the notebook release and put it inside jupyter core instead? I don't see why it makes more sense here given that it seems to be useful for the entire jupyter stack. edit see #3116 (comment)

[jasongrout]

@mpacer - to raise the visibility of your objections, can you:

1. open a new issue under the 5.3 milestone raising your objection, and
2. note in the 5.3 release issue your objection and reference your issue: Release 5.3 #3183

I'm afraid having a comment here on a closed issue will be less visible.

[minrk]

Will follow-up in the new issue when it’s open, but this is only applied to the very notebook-specific configuration of extensions, not the rest of “normal” traitlets-based configuration that’s shared across packages. You are right that adding .d-style config to that belongs in a higher-level for Jupyter-core or traitlets (see my attempt: ipython/traitlets#242). The fact that extensions use a totally different configuration system is an issue and source of confusion, but not a new one. Adding conf.d to traitlets/core also wouldn’t solve the issue for extensions, because it’s a different setup, so we would have to do it twice anyway. The core.paths are still relevant, this is an additional layer inside a given directory on those paths.

[mpacer]

@minrk thanks for the clarification. My concern was because I didn't understand how the class was being used. In particular, seeing the default value of config_dir=Unicode('.') made my twitch, since that would default to not using the jupyter config system. In all the cases it's used though, it's either inside tests or we're using the juptyer config locations, so my primary concerns about this being a release blocking are moot.

The urgency stemmed from a fear of having to explain to people how to deal with a whole new set of locations that config files might exist in. That isn't the case here, so s'all good for now. I'll open an issue over the next few days in either jupyter_core or traitlets where I can non-urgently express this concern.

[minrk]

Ah, good point. The default value is only because it's technically a generic object. I suspect the default is never used, and a note to that effect might be appropriate. This PR doesn't change where files are looked for (jupyter-paths is still respected), only that wherever we loaded [jupyter-path]/nbconfig/tree.json we can now also load [jupyter-path]/nbconfig/tree.d/*.json. A similar proposal is ongoing in traitlets to get notebook_config.d wherever we already find notebook_config.{py|json}.