From b0142d0607d3c6312c3e031b8ec7a625d9975ac6 Mon Sep 17 00:00:00 2001 From: Zsailer Date: Tue, 12 Feb 2019 08:52:13 -0800 Subject: [PATCH 1/6] first draft of jupyter-server proposal --- jupyter-server/jupyter-server.md | 216 +++++++++++++++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 jupyter-server/jupyter-server.md diff --git a/jupyter-server/jupyter-server.md b/jupyter-server/jupyter-server.md new file mode 100644 index 00000000..20ffd18d --- /dev/null +++ b/jupyter-server/jupyter-server.md @@ -0,0 +1,216 @@ +# Standalone Jupyter server enhancement proposal [active] + +## Problem + +There are now multiple frontends that talk to the backend services provided by the notebook server: the legacy Notebook, the dashboards, JupyterLab, standalone widgets and more. The configuration of legacy notebook and the backend server are tightly coupled. As a consequence, the other applications are forced to load the legacy notebook to use the backend server. + +## Proposed Enhancement + +Decouple the backend server (and its configuration) from the classic notebook frontend. This will require the following steps: + +1. Create `jupyter_server` repository + - Fork of `notebook` repository. + - Pure Python package with notebook backend services. + - `notebook` tornado handlers and frontend logic stay in `notebook` repo. + - Deprecated notebook server APIs do not move to `jupyter_server`. +2. New server extensions mechanism. + - server extensions move to `jupyter_server`. + - new base classes to create applications from server extensions. + - nbextensions stay in `notebook`. + - `jupyter_notebook_config.d` folder becomes `jupyter_server_config.d` + - `notebook` becomes a server extension. +3. Services become server extensions + - Services are just serve extensions with dependencies. + - Add dependency injection system. +5. Namespacing static files and REST API urls. + - Each extension serves static files at the `/static/` url. + - +6. Migrate configuration. + - Notebook App configuration stays in `jupyter_notebook_config.py` + - Server and services configurations move to `jupyter_server_config.py` + - Add a `migrate` application to automate this migration. +7. Add necessary documentation to notebook and jupyter_server repos. + +## Detailed Explanation + +### Create a `jupyter_server` repository + +The first thing to do is fork `notebook`. A new `jupyter_server` repo will keep the server specific logic and remove: +1. the notebook frontend code, +2. deprecated notebook server APIs, and +3. tornado handlers to the classic notebook interface. These pieces will stay in the `notebook` repository. + +Things that stay in notebook: + +- `edit` module: the handlers for the classic notebook text editor. +- `templates` directory: all html templates for the classic notebook +- `terminal` module: handlers for the classic notebook terminal application. +- `view` module: handlers for file viewer component. +- `static` directory: all js and style sheets for notebook frontend. +- `tree` module: a classic notebook file browser+handlers. +- `auth` module? *(Should this move to jupyter_server?)* + +Things that move to jupyter_server: +- `services` module: all jupyter server services, managers, and handlers +- `bundler`: handlers for building download bundles +- `files`: handlers for serving files from contents manager. +- `kernelspec`: handler for getting kernelspec +- `base`: base handler for jupyter apps. +- `i18n`: module for internationalizing the jupyter server + + +Preliminary work resides in [jupyter_server](https://github.com/jupyter/jupyter_server). + +### Server Extensions + +The extension mechanism for the *jupyter server* will be the main area where server extensions differ from notebook's server extensions. + +Enabled server extensions will still be loaded using the `load_jupyter_server_extension` approach when the jupyter server is started. + +**Server extensions as applications** + +In the proposed jupyter_server, extension developers may also create an application from their extension, using a new `JupyterServerExtensionApp` class. Extension developers can subclass the new base class to make server extensions into Jupyter applications (configurable and launchable from the commmand line). This new class loads extension config from file and parses configuration set from the command line. + +For example, the legacy notebook could be started: 1) as an enabled extension or 2) by running the traitlets application from the command line. + +Example extension: +```python +from .extension import load_jupyter_server_extension + +class NotebookApp(JupyterServerExtensionApp): + + name = 'notebook' + description = 'NotebookApp as a server extension.' + load_jupyter_server_extension = staticmethod(load_jupyter_server_extension) +``` + +`JupyterServerExtensionApp` subclasses are configurable using Jupyter's configuration system. Users can generate config files for each extension in the Jupyter config path (see `jupyter --paths`). Each extension's configuration is loaded when the server is initialized or the extension application is launched. + +As will all jupyter applications, users can autogenerate a configuration file for their extension using `jupyter my_extension --generate-config`. + +Initial experimental work resides in [`jupyter_server_extension`](https://github.com/Zsailer/jupyter_server_extension). + +**Classic notebook server extension** + +The classic `NotebookApp` will become a server extension. It will inherit `JupyterServerExtensionApp`. Users can enable the notebook using the extension install/enable mechanism or enable the `notebook` in their `jupyter_server_config.py` file: +```python +c.ServerApp.jpserver_extensions = { + 'notebook': True +} +``` +Users can also launch the notebook application using the (usual)`jupyter notebook` command line interface. + +**Extension installing/enabling mechanism** + +The new extension mechanism in the *jupyter server* will differ from notebook's server extensions. + +* The `--sys-prefix` installation would become the default. (Users are confused when enabling an extension requires more permissions than the installation of the package). Installation in system-wide directories would be done with the `--system` option. +* Installing an extension will include the addition of a 'manifest' file into a conf.d directory (under one of the Jupyter configuration directories, `user / sys-prefix / system`). The placement of such an extension manifest provided by a Python package can be done with `jupyter server extension install --py packagename [--user / --system / --sys-prefix]`. Packages (conda or wheels) carrying server extensions could place such manifests in the sys-prefix path by default effectively installing them. Development installations would also require the call to the installation command. + +* Enabling an extension is separate from the installation. Multiple scenarios are possible: + - enabling an extension at the same level (user / sys-prefix / system) as where it was installed, or at a higher level (user for sys-prefix and system, or sys-prefix for system). + - forcibly disabling an extension that was enabled at a lower level of precedence. + - forcibly enabling an extension that was disabled at a lower level of precedence. + This would be done via two `conf.d` configuration directories managing a list of disabled extensions and list of enabled extensions in the form of empty files having the name of the corresponding extension. If an extension is both disabled and enabled at the same level of precedence, disabling has precedence. Packages (conda or wheels) could place such a enabler file in the sys-prefix path by default. The `jupyter server extension enable` command would be required for development installations. + +(Possibly) when an extension is enabled at a given precedence level, it may only look for the version of the extension installed at the same or lower precedence level. For example, if an extension `foobar` is installed and enabled system wide, but a user installs a version with `--user`, this version will only be picked up if the user also enables it with `--user`. + +### Services become server extensions (with dependency injection) + +Right now, there are two ways to extend and customize the jupyter server: services and server extensions. This is a bit confusing for new contributors. The main difference is that services often (but not always) **depend on other services**. + +On example is the `sessions` service, which depends on the `kernels` and `kernelspec` services. Without these two services, the `sessions` service doesn't work (or even make sense). + +We could reduce complexity around `jupyter_server` by making *everything* a server extension. We would need to add a **dependency injection** system to allow extensions to depend on other extensions. Some good options are [pyinject](https://github.com/google/pinject) or [python-dependency-injector](https://github.com/ets-labs/python-dependency-injector). + +To port a service, it will need to be refactored using extension mechanism mentioned above. + +### Add namespacing to `static` endpoints and REST API urls. + +Currently, the notebook tornado application serves all static files underneath the `/static/` prefix. Jupyter server will add namespacing under the static url and extension REST API urls. Each extension will serve their static files under the `/static/` prefix and their API handlers behind a `/extension/api/` prefix. + +For example, the classic notebook server extension will add static handlers that reroute requests to the `/static/notebook/` endpoints. + +The jupyter_server will provide a new `JupyterExtensionHandler` base class that reroute requests to the extension's namespaced static and REST API endpoints. + +Preliminary experimental work resides in the [`jupyter_server_extension`](https://github.com/Zsailer/jupyter_server_extension) repository. + +### Configuration System + +The configuration of the server and the legacy notebook are currently tightly coupled in a single NotebookApp configurable. The proposed jupyter server has server-specific configurations in a separate config system from the classic notebook (and jupyterlab) application configurations. + +The table below show all the classic notebook traits and where they will live after migration. + +**Overview of configuration structure** + +The notebook and server configuration both live in the `jupyter_notebook_config.py` file. Extensions are configured in separate files (named after the extension) in the `jupyter_notebook_config.d` directory: +``` +~/.jupyter/ +├── jupyter_notebook_config.py +└── jupyter_notebook_config.d + └── my_extension.json +``` + +**New proposed configuration structure** + +The jupyter_server configuration lives in the `jupyter_server_config.py` file. Extensios are configured in separate files in the `jupyter_server_config.d` folder. The notebook configuration is stored in a `notebook.py` file, just like other extensions. +``` +~/.jupyter/ +├── jupyter_server_config.py +└── jupyter_server_config.d + ├── notebook.py|json + ├── lab.py|json + └── my_extension.py|json +``` + +**Migration application** + +To make migration easier on users, the jupyter server will include a `migrate` application to split notebook and server configurations into their appropriate locations (listed above). This application will find any `jupyter_notebook_config.py|json` files in `jupyter --paths`, read configured traits, sort server vs. notebook traits, and write them to the appropriate `jupyter_server_config.py|json` and `jupyter_notebook_config.py|json` files. + + +### How this effects other projects + +[**Classic notebook**]() +In short, the classic notebook will become a server extension application. The rest of this proposal describes the details behind what will change in the notebook repo. +[`JupyterServerExtensionApp`](). + +[**Jupyter Lab**]() +Jupyter lab will also become a server extension application. The new classes described above should simplify the way JupyterLab interfaces with the server. + +[**Kernel Gateway**]() +Kernel Gateway writes custom `kernel` and `kernelmanager` services and load them as server extensions. + +[**Kernel Nanny**]() + +## Pros and Cons + +**Pros** associated with this implementation include + +* Allow various frontends to use the backend services of the jupyter server. +* Provides base classes for extension developers writing server extension applications and handlers. +* Reduce complexity when extending the server by making everything a server extension. +* Organizes the configuration for server and extension in a sane and logical manner (`conf.d` approach). + +**Cons** associated with this implementation include: + +* Break the classic notebook in a backwards incompatible way. +* Affects many projects. The transition may be painful? +* Adding a dependency injection system adds new complexity. + +## Relevent Issues, PRs, and discussion + +Moving to a `conf.d` approach. +* PR [3116](https://github.com/jupyter/notebook/pull/3116), `jupyter/notebook`: extension config in `config.d` directory. +* PR [3782](https://github.com/jupyter/notebook/issues/3782), `jupyter/notebook`: server extension use `conf.d` approach +* PR [2063](https://github.com/jupyter/notebook/issues/2063), `jupyter/notebook`: config merging problems. + +Conversation on server/notebook extensions: +* PR [1706](https://github.com/jupyter/notebook/issues/1706), `jupyter/notebook`: proposal to improve server/notebook extensions +* PR [2824](https://github.com/jupyter/notebook/issues/2824), `jupyter/notebook`: enable nbextensions by default + +Static Namespace: +* PR [21](https://github.com/jupyter/enhancement-proposals/pull/21#issuecomment-248647152)`jupyter/enhancement-proposals`: mention namespacing. + +## Interested + +@Zsailer, @SylvainCorlay, @ellisonbg, @blink1073, @kevin-bates \ No newline at end of file From a9d21a253614d2aec2cf375a37c0d49cd3ad023d Mon Sep 17 00:00:00 2001 From: Zsailer Date: Thu, 28 Feb 2019 10:02:49 -0800 Subject: [PATCH 2/6] add submission table --- jupyter-server/jupyter-server.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/jupyter-server/jupyter-server.md b/jupyter-server/jupyter-server.md index 20ffd18d..fc47454e 100644 --- a/jupyter-server/jupyter-server.md +++ b/jupyter-server/jupyter-server.md @@ -1,5 +1,15 @@ # Standalone Jupyter server enhancement proposal [active] +| Item | Value | +|------------|------------------------------------------------------------------------------------------------------------------------------| +| JEP Number | 10 | +| Title | Standalone Jupyter Server | +| Authors | Zach Sailer ([@Zsailer](https://github.com/Zsailer)) and Sylvain Corlay ([@SylvainCorlay](https://github.com/sylvaincorlay)) | +| Status | Draft | +| Type | S - [Standards Track](https://www.python.org/dev/peps/#pep-types-key) JEP | +| Created | 20 September 2016 | +| History | 20 September 2016, 12 February 2019 | + ## Problem There are now multiple frontends that talk to the backend services provided by the notebook server: the legacy Notebook, the dashboards, JupyterLab, standalone widgets and more. The configuration of legacy notebook and the backend server are tightly coupled. As a consequence, the other applications are forced to load the legacy notebook to use the backend server. From 600cb8cc01c4fc277696a6dde3a76f714f5f8038 Mon Sep 17 00:00:00 2001 From: Zsailer Date: Thu, 28 Feb 2019 13:51:47 -0800 Subject: [PATCH 3/6] revision --- jupyter-server/jupyter-server.md | 141 +++++++++++++++---------------- 1 file changed, 69 insertions(+), 72 deletions(-) diff --git a/jupyter-server/jupyter-server.md b/jupyter-server/jupyter-server.md index fc47454e..b123909e 100644 --- a/jupyter-server/jupyter-server.md +++ b/jupyter-server/jupyter-server.md @@ -18,91 +18,91 @@ There are now multiple frontends that talk to the backend services provided by t Decouple the backend server (and its configuration) from the classic notebook frontend. This will require the following steps: -1. Create `jupyter_server` repository - - Fork of `notebook` repository. +1. Separate `jupyter_server` repository + - A fork of the `notebook` repository. - Pure Python package with notebook backend services. - `notebook` tornado handlers and frontend logic stay in `notebook` repo. - Deprecated notebook server APIs do not move to `jupyter_server`. -2. New server extensions mechanism. +1. Clearly define the smallest unit that is a "Jupyter Server" + - Kernel, KernelSpecs, and Session (?) become the only core services + - Everything else becomes a server extension? +1. Extensions as Applications - server extensions move to `jupyter_server`. + - New `ExtensionApp` class + - Notebook, jupyterlab, etc. become ExtensionApps. +1. New server extensions mechanism. - new base classes to create applications from server extensions. - nbextensions stay in `notebook`. - `jupyter_notebook_config.d` folder becomes `jupyter_server_config.d` - - `notebook` becomes a server extension. -3. Services become server extensions - - Services are just serve extensions with dependencies. - - Add dependency injection system. -5. Namespacing static files and REST API urls. +1. Namespacing static files and REST API urls. - Each extension serves static files at the `/static/` url. - - -6. Migrate configuration. + - New `ExtensionHandlerApp` class +1. Migrate configuration. - Notebook App configuration stays in `jupyter_notebook_config.py` - Server and services configurations move to `jupyter_server_config.py` - Add a `migrate` application to automate this migration. -7. Add necessary documentation to notebook and jupyter_server repos. +1. Add necessary documentation to notebook and jupyter_server repos. ## Detailed Explanation -### Create a `jupyter_server` repository +### Separate `jupyter_server` repository The first thing to do is fork `notebook`. A new `jupyter_server` repo will keep the server specific logic and remove: 1. the notebook frontend code, 2. deprecated notebook server APIs, and 3. tornado handlers to the classic notebook interface. These pieces will stay in the `notebook` repository. -Things that stay in notebook: - +Modules that stay in notebook: - `edit` module: the handlers for the classic notebook text editor. - `templates` directory: all html templates for the classic notebook - `terminal` module: handlers for the classic notebook terminal application. - `view` module: handlers for file viewer component. - `static` directory: all js and style sheets for notebook frontend. - `tree` module: a classic notebook file browser+handlers. -- `auth` module? *(Should this move to jupyter_server?)* -Things that move to jupyter_server: +Modules that move to jupyter_server: - `services` module: all jupyter server services, managers, and handlers - `bundler`: handlers for building download bundles - `files`: handlers for serving files from contents manager. - `kernelspec`: handler for getting kernelspec - `base`: base handler for jupyter apps. - `i18n`: module for internationalizing the jupyter server - +- `auth` module Preliminary work resides in [jupyter_server](https://github.com/jupyter/jupyter_server). -### Server Extensions +### Clearly define the smallest unit that is a "Jupyter Server" + +*(This section needs a lot more discussion.)* -The extension mechanism for the *jupyter server* will be the main area where server extensions differ from notebook's server extensions. +The server-notebook split is an opportunity to clearly define Jupyter's core services. The old notebook server was comprised of many services, and it could be extended by separate "server extensions". However, it isn't clear what should be a core service versus a server extension. Some extensions were "grand-fathered" in as core services to make the notebook application more full-featured (e.g. nbconvert, terminals, contents, ...). Now that we are separating the server from the classic notebook, we need to reevaluate what services are core to the Jupyter Server. -Enabled server extensions will still be loaded using the `load_jupyter_server_extension` approach when the jupyter server is started. +Jupyter server aims to be a core building block for other applications (like dashboards, standalone widgets, etc.). To achieve this, we need to define the simplest "unit" that defines a Jupyter Server: `kernels`, `kernelspec`, and `sessions`. Other services become extensions to the core server (using the `load_jupyter_server_extension` mechanism defined below). -**Server extensions as applications** +### Extensions as Applications -In the proposed jupyter_server, extension developers may also create an application from their extension, using a new `JupyterServerExtensionApp` class. Extension developers can subclass the new base class to make server extensions into Jupyter applications (configurable and launchable from the commmand line). This new class loads extension config from file and parses configuration set from the command line. +A new `ExtensionApp` class will be available in `jupyter_server.extensions.extensionapp`. It enables developers to make server extensions behave like standalone Jupyter applications. Jupyterlab is an example of this kind of server extension. It can be configured, initialized, and launched from the command line. When Lab is launched, it first initializes and configures a jupyter (notebook) server, then loads the configured jupyterlab extension and redirects the use to the Lab landing page. -For example, the legacy notebook could be started: 1) as an enabled extension or 2) by running the traitlets application from the command line. +`ExtensionApp` handles the boilerplate code to make Jupyter applications that configure and launch server extensions directly. It inherits `jupyter_core.JupyterApp` and exposes a command line entry point: `jupyter `. When an extension is launched, it first configures and starts a jupyter_server (`ServerApp`); then, it loads the configured extension and redirects users to the extension's landing page. Extension developers simply inherit the `ExtensionApp` and add the extension's `load_jupyter_server_extension` as a `staticmethod`. -Example extension: ```python +from jupyter_server.extensionapp import ExtensionApp from .extension import load_jupyter_server_extension -class NotebookApp(JupyterServerExtensionApp): +class MyExtension(ExtensionApp): - name = 'notebook' - description = 'NotebookApp as a server extension.' + name = 'myextension' + description = 'My server extension as a Jupyter app.' load_jupyter_server_extension = staticmethod(load_jupyter_server_extension) ``` -`JupyterServerExtensionApp` subclasses are configurable using Jupyter's configuration system. Users can generate config files for each extension in the Jupyter config path (see `jupyter --paths`). Each extension's configuration is loaded when the server is initialized or the extension application is launched. - -As will all jupyter applications, users can autogenerate a configuration file for their extension using `jupyter my_extension --generate-config`. +`ExtensionApp`s can be configured by `jupyter__config.py|json` as well. When the server extension is loaded by a server or launched from the command line, it searches the list of `jupyter --paths` for configured traits in these files. Initial experimental work resides in [`jupyter_server_extension`](https://github.com/Zsailer/jupyter_server_extension). **Classic notebook server extension** -The classic `NotebookApp` will become a server extension. It will inherit `JupyterServerExtensionApp`. Users can enable the notebook using the extension install/enable mechanism or enable the `notebook` in their `jupyter_server_config.py` file: +The classic `NotebookApp` will become a server extension. It will inherit `ExtensionApp`. Users can enable the notebook using the extension install/enable mechanism or enable the `notebook` in their `jupyter_server_config.py` file: ```python c.ServerApp.jpserver_extensions = { 'notebook': True @@ -110,7 +110,7 @@ c.ServerApp.jpserver_extensions = { ``` Users can also launch the notebook application using the (usual)`jupyter notebook` command line interface. -**Extension installing/enabling mechanism** +### New server extensions mechanism. The new extension mechanism in the *jupyter server* will differ from notebook's server extensions. @@ -125,58 +125,54 @@ The new extension mechanism in the *jupyter server* will differ from notebook's (Possibly) when an extension is enabled at a given precedence level, it may only look for the version of the extension installed at the same or lower precedence level. For example, if an extension `foobar` is installed and enabled system wide, but a user installs a version with `--user`, this version will only be picked up if the user also enables it with `--user`. -### Services become server extensions (with dependency injection) - -Right now, there are two ways to extend and customize the jupyter server: services and server extensions. This is a bit confusing for new contributors. The main difference is that services often (but not always) **depend on other services**. - -On example is the `sessions` service, which depends on the `kernels` and `kernelspec` services. Without these two services, the `sessions` service doesn't work (or even make sense). - -We could reduce complexity around `jupyter_server` by making *everything* a server extension. We would need to add a **dependency injection** system to allow extensions to depend on other extensions. Some good options are [pyinject](https://github.com/google/pinject) or [python-dependency-injector](https://github.com/ets-labs/python-dependency-injector). - -To port a service, it will need to be refactored using extension mechanism mentioned above. - ### Add namespacing to `static` endpoints and REST API urls. Currently, the notebook tornado application serves all static files underneath the `/static/` prefix. Jupyter server will add namespacing under the static url and extension REST API urls. Each extension will serve their static files under the `/static/` prefix and their API handlers behind a `/extension/api/` prefix. For example, the classic notebook server extension will add static handlers that reroute requests to the `/static/notebook/` endpoints. -The jupyter_server will provide a new `JupyterExtensionHandler` base class that reroute requests to the extension's namespaced static and REST API endpoints. +A new `ExtensionHandler` class will be available in `jupyter_server.extensions.handlers`. This class inherits `JupyterHandler`. It handles the boilerplate code to reroute requests extension's namespaced static and REST API endpoints. Preliminary experimental work resides in the [`jupyter_server_extension`](https://github.com/Zsailer/jupyter_server_extension) repository. ### Configuration System -The configuration of the server and the legacy notebook are currently tightly coupled in a single NotebookApp configurable. The proposed jupyter server has server-specific configurations in a separate config system from the classic notebook (and jupyterlab) application configurations. - -The table below show all the classic notebook traits and where they will live after migration. - -**Overview of configuration structure** - -The notebook and server configuration both live in the `jupyter_notebook_config.py` file. Extensions are configured in separate files (named after the extension) in the `jupyter_notebook_config.d` directory: -``` -~/.jupyter/ -├── jupyter_notebook_config.py -└── jupyter_notebook_config.d - └── my_extension.json -``` - -**New proposed configuration structure** - -The jupyter_server configuration lives in the `jupyter_server_config.py` file. Extensios are configured in separate files in the `jupyter_server_config.d` folder. The notebook configuration is stored in a `notebook.py` file, just like other extensions. -``` -~/.jupyter/ -├── jupyter_server_config.py -└── jupyter_server_config.d - ├── notebook.py|json - ├── lab.py|json - └── my_extension.py|json -``` +Splitting the server-specific pieces from the classic notebook affects Jupyter's configuration system. This is a non-trivial problem. Changing Jupyter's configuration system affects everyone. We need to consider how to make this transition as painless as possible. At the end of this section, we list some steps that make reduce the friction. + +Here is a list of things the changes on the configuration system: +* Move server-specific configuration from `jupyter_notebook_config.py|json` into `jupyter_server_config.py|json`. +* Notebook configuration will stay in `jupyter_notebook_config.py|json`. +* Server extensions configurations move from `jupyter_notebok_config.d` to `jupyter_server_config.d`. +* The tornado server and webapp configurable applications move to `jupyter_server`. They become `ServerApplication` and `ServerWebApp` +* The `NotebookApp` becomes a server extension. It would only load notebook specific configuration/traitlets, from `jupyter_notebook_config.py|json`. +* Server extensions are found using the `jpserver_extensions` trait instead of the `nbserver_extensions` trait in the `ServerApp`. +* Extension configuration files in `jupyter_server_config.d` must be enabled using the `jpserver_extensions` trait. They are enabled by JSON config files in `jupyter_server_config.d`. +* Extensions can create their own configuration files in `{sys-prefix}/etc/jupyter/` or `~/.jupyter`, i.e. `jupyter__config.py|json`. +* General `jupyter_config.py|json` files must update to set server-specific configuration using the `ServerApp` and notebook specific configuration using `NotebookApp`. + +Some traits will stay in `jupyter_notebook_config.py|json`. Here is a list of those traits (everything else moves to server config files): +* extra_nbextensions_path +* enable_mathjax +* max_body_size +* max_buffer_size +* notebook_dir +* mathjax_url +* get_secure_cookie_kwargs +* mathjax_config +* ignore_minified_js + +Here are some steps we can take to reduce the friction for transitioning: +* **Copy (not move)** `jupyter_notebook_config.py|json` to `jupyter_server_config.py|json` +* **Copy (not move)** `jupyter_notebook_config.d/` to `jupyter_server_config.d`. +* `NotebookApp` becomes `ServerApp` in all copied files. +* Leftover server traits in `jupyter_notebook_config.py|json` get ignored when the notebook extension is initialized. Server traits are only read from `jupyter_server_config.py|json`. +* Document like crazy! + +For an thorough overview of the configuration system and a side-by-side diff between the (current) notebook and (future) server config layout, take a look at [Zsailer/jupyter_config_overview](https://github.com/Zsailer/jupyter_config_overview). **Migration application** -To make migration easier on users, the jupyter server will include a `migrate` application to split notebook and server configurations into their appropriate locations (listed above). This application will find any `jupyter_notebook_config.py|json` files in `jupyter --paths`, read configured traits, sort server vs. notebook traits, and write them to the appropriate `jupyter_server_config.py|json` and `jupyter_notebook_config.py|json` files. - +To make migration easier on users, a `migrate` application will be available to split notebook and server configurations into their appropriate locations (listed above). ### How this effects other projects @@ -188,7 +184,7 @@ In short, the classic notebook will become a server extension application. The r Jupyter lab will also become a server extension application. The new classes described above should simplify the way JupyterLab interfaces with the server. [**Kernel Gateway**]() -Kernel Gateway writes custom `kernel` and `kernelmanager` services and load them as server extensions. +Kernel Gateway can use the new Jupyter Server to server kernels as a service. The new Jupyter server will remove the unwanted services that Kernel Gateway currently removes by using a custom server application. KG will also be able to swap out the kernels and kernelspec manager in the Jupyter Server with its custom classes. [**Kernel Nanny**]() @@ -209,10 +205,11 @@ Kernel Gateway writes custom `kernel` and `kernelmanager` services and load them ## Relevent Issues, PRs, and discussion -Moving to a `conf.d` approach. +Jupyter's configuration system. * PR [3116](https://github.com/jupyter/notebook/pull/3116), `jupyter/notebook`: extension config in `config.d` directory. * PR [3782](https://github.com/jupyter/notebook/issues/3782), `jupyter/notebook`: server extension use `conf.d` approach * PR [2063](https://github.com/jupyter/notebook/issues/2063), `jupyter/notebook`: config merging problems. +* [Overview](https://github.com/Zsailer/jupyter_config_overview) of Jupyter's configuration system Conversation on server/notebook extensions: * PR [1706](https://github.com/jupyter/notebook/issues/1706), `jupyter/notebook`: proposal to improve server/notebook extensions From c2905c1b20c8acfd8bdf5c07d969b0a643e7ab6e Mon Sep 17 00:00:00 2001 From: Zsailer Date: Thu, 28 Feb 2019 13:52:58 -0800 Subject: [PATCH 4/6] remove old migration lines --- jupyter-server/jupyter-server.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/jupyter-server/jupyter-server.md b/jupyter-server/jupyter-server.md index b123909e..b37c54f4 100644 --- a/jupyter-server/jupyter-server.md +++ b/jupyter-server/jupyter-server.md @@ -37,10 +37,7 @@ Decouple the backend server (and its configuration) from the classic notebook fr 1. Namespacing static files and REST API urls. - Each extension serves static files at the `/static/` url. - New `ExtensionHandlerApp` class -1. Migrate configuration. - - Notebook App configuration stays in `jupyter_notebook_config.py` - - Server and services configurations move to `jupyter_server_config.py` - - Add a `migrate` application to automate this migration. +1. Migrate application. 1. Add necessary documentation to notebook and jupyter_server repos. ## Detailed Explanation From 01759467dbbd4987c5ddb2c2cc77503a6ce0ddcd Mon Sep 17 00:00:00 2001 From: Zsailer Date: Thu, 28 Feb 2019 14:00:18 -0800 Subject: [PATCH 5/6] links to sections in proposal --- jupyter-server/jupyter-server.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/jupyter-server/jupyter-server.md b/jupyter-server/jupyter-server.md index b37c54f4..a6fe1dbb 100644 --- a/jupyter-server/jupyter-server.md +++ b/jupyter-server/jupyter-server.md @@ -18,27 +18,27 @@ There are now multiple frontends that talk to the backend services provided by t Decouple the backend server (and its configuration) from the classic notebook frontend. This will require the following steps: -1. Separate `jupyter_server` repository +1. [Separate `jupyter_server` repository](#separate-jupyter_server-repository) - A fork of the `notebook` repository. - Pure Python package with notebook backend services. - `notebook` tornado handlers and frontend logic stay in `notebook` repo. - Deprecated notebook server APIs do not move to `jupyter_server`. -1. Clearly define the smallest unit that is a "Jupyter Server" +1. [Clearly define the smallest unit that is a "Jupyter Server"](#clearly-define-the-smallest-unit-that-is-a-jupyter-server) - Kernel, KernelSpecs, and Session (?) become the only core services - Everything else becomes a server extension? -1. Extensions as Applications +1. [Extensions as Applications](#extensions-as-applications) - server extensions move to `jupyter_server`. - New `ExtensionApp` class - Notebook, jupyterlab, etc. become ExtensionApps. -1. New server extensions mechanism. +1. [New server extensions mechanism.](#new-server-extensions-mechanism) - new base classes to create applications from server extensions. - nbextensions stay in `notebook`. - `jupyter_notebook_config.d` folder becomes `jupyter_server_config.d` -1. Namespacing static files and REST API urls. +1. [Namespacing static files and REST API urls.](#add-namespacing-to-static-endpoints-and-rest-api-urls) - Each extension serves static files at the `/static/` url. - New `ExtensionHandlerApp` class -1. Migrate application. -1. Add necessary documentation to notebook and jupyter_server repos. +1. [Configuration System](#configuration-system) +1. [Migrate application.](#migration-application) ## Detailed Explanation @@ -167,7 +167,7 @@ Here are some steps we can take to reduce the friction for transitioning: For an thorough overview of the configuration system and a side-by-side diff between the (current) notebook and (future) server config layout, take a look at [Zsailer/jupyter_config_overview](https://github.com/Zsailer/jupyter_config_overview). -**Migration application** +### Migration application To make migration easier on users, a `migrate` application will be available to split notebook and server configurations into their appropriate locations (listed above). From a4dbb5854ce6bb38bce177d0cd8777768a2ab67a Mon Sep 17 00:00:00 2001 From: Zsailer Date: Thu, 28 Feb 2019 14:23:18 -0800 Subject: [PATCH 6/6] add nteract to the list of frontends using the server --- jupyter-server/jupyter-server.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jupyter-server/jupyter-server.md b/jupyter-server/jupyter-server.md index a6fe1dbb..44faba26 100644 --- a/jupyter-server/jupyter-server.md +++ b/jupyter-server/jupyter-server.md @@ -12,7 +12,7 @@ ## Problem -There are now multiple frontends that talk to the backend services provided by the notebook server: the legacy Notebook, the dashboards, JupyterLab, standalone widgets and more. The configuration of legacy notebook and the backend server are tightly coupled. As a consequence, the other applications are forced to load the legacy notebook to use the backend server. +There are now multiple frontends that talk to the backend services provided by the notebook server: the classic Notebook, the dashboards, nteract, JupyterLab, standalone widgets and more. The configuration of classic notebook and the backend server are tightly coupled. As a consequence, the other applications are forced to load the legacy notebook to use the backend server. ## Proposed Enhancement @@ -74,7 +74,7 @@ Preliminary work resides in [jupyter_server](https://github.com/jupyter/jupyter_ The server-notebook split is an opportunity to clearly define Jupyter's core services. The old notebook server was comprised of many services, and it could be extended by separate "server extensions". However, it isn't clear what should be a core service versus a server extension. Some extensions were "grand-fathered" in as core services to make the notebook application more full-featured (e.g. nbconvert, terminals, contents, ...). Now that we are separating the server from the classic notebook, we need to reevaluate what services are core to the Jupyter Server. -Jupyter server aims to be a core building block for other applications (like dashboards, standalone widgets, etc.). To achieve this, we need to define the simplest "unit" that defines a Jupyter Server: `kernels`, `kernelspec`, and `sessions`. Other services become extensions to the core server (using the `load_jupyter_server_extension` mechanism defined below). +Jupyter server aims to be a core building block for other applications (like nteract, lab, dashboards, standalone widgets, etc.). To achieve this, we need to define the simplest "unit" that defines a Jupyter Server: `kernels`, `kernelspec`, and `sessions`. Other services become extensions to the core server (using the `load_jupyter_server_extension` mechanism defined below). ### Extensions as Applications