Clean up after Notebook syncing

[Zsailer]

@kevin-bates and I chatted today and developed a plan for syncing notebook and jupyter_server.

1. I will open a series of PRs in jupyter_server that contain batches of PRs from jupyter/notebook. Each sequential PR will build from the previous.
2. @kevin-bates, I, and anyone else is invited to review those PRs.
   • If your review includes a fix or change to the code, please reference this "Clean up" issue (Clean up after Notebook syncing #96).
   • Unless the problem breaks tests, we will not fix it in the porting PR. Instead, we will fix it in step 5 below.
   • We will use this issue to track the proposed fixes (thanks to Github's reference features).
3. Each batch PR will start from the previous batch (alleviating to need to rebase and block future merges).
4. We will merge each batch in order (I'm numbering the batches as I go) to avoid merge conflicts in subsequent PRs.
5. We'll use the "rebase and merge" feature to handle each merge.
6. After the last batch is merged, we'll start a new PR that addresses all the bugfixes in the various batches tracked in this issue.

We're hoping to move fast and merge quickly. Then we can go back and fix things after we get up to sync.

[vidartf]

I've been commenting on the PRs post-merge to indicate things I thing will need a look at for cleanup.

[echarles]

Just taking this train today before it gets full speed :)

1. For my understanding after reading Sync Notebook and Jupyter Server. #53, do the current batches apply consolidated rules discussed in Sync Notebook and Jupyter Server. #53 regarding cherry-pick, squash, merge...?
2. I see Batch 1, Batch 2 and Batch 4 referenced 2 days ago, where is Batch 3?
3. Regarding any regression, I read @Zsailer After the last batch is merged, we'll start a new PR that addresses all the bugfixes - Does this mean that only unit tests are run for now and that there is no minimal end-user testing to ensure that the server can at least serve a real notebook in the browser? If no such tests are run, do we have a branch for this (I am thinking to e.g. a jupyterlab-server branch like [WIP] Jupyter server jupyterlab/jupyterlab#4954 which should be updated).

[Zsailer]

Hi @echarles! Thanks for joining all the fun! 😆

1. We are just "rebasing and merging" these batches. But the plan moving forward is to "squash merge" PRs.
2. We are only referencing batches that need fixing at the end. If the batch didn't have any issues, we won't reference them here.
3. Yeah, the goal is to pass unit tests now and fix end-user testing after. There is a notebook branch and voila branch that depends on the server. We can use these for end-user testing. [WIP] Jupyter server jupyterlab/jupyterlab#4954 is out-of-date. We will update that PR soon.

[echarles]

Thx @Zsailer All 3 branches (notebook, voila, jupyterlab) have conflicts with current server master. What would would advice as next step (one of those, a brand new one)? - I favor jupyterlab as this is my main working env. Happy to help in the coming weeks on this.

[Zsailer]

Awesome!

#48 added a new API for appending frontend applications as extensions to the server. The goal is to use this API for notebook, voila, jupyterlab, etc. We've already ported notebook and voila in those PRs I mentioned. We haven't attempted to port JupyterLab yet. It's a fair amount of work, but if you're feeling ambitious, we'd love the contribution!

If you'd rather spend your time testing the jupyter_server (instead of writing a whole new extension from jupyterlab), notebook and voila will be considerably less work.

[Zsailer]

@echarles I just verified that both branches above work with jupyter_server master. If you're still having issues, let me know.

[echarles]

@Zsailer I will first try with the notebook and voila branches, but looking at the github PR page, the status at the bottom of the pages shows This branch has conflicts that must be resolved. You say that they work with jupyter_server master. Any steps to merge the conflicts?

[echarles]

@Zsailer hacking this closed issue to give you feedback on end-users tests based on your branches.

I can successfully run your voila branch (see [1]).

When I run jupyter server or jupyter notebook with your notebook branch (see [2]), I see the following HTML page.

When I run jupyter notebook ./notebook/docs/source/examples/Notebook/nbpackage/mynotebook.ipynb with your notebook branch (see [2]), I receive AttributeError: 'ServerApp' object has no attribute 'notebook_dir'.

[1]

conda remove -y --name voila-server --all || true
conda create -y -n voila-server python=3.7 && \
  conda activate voila-server && \
  git clone https://github.com/jupyter/jupyter_server.git --branch master --single-branch --depth 1 && \
  cd jupyter_server && python setup.py develop && cd .. && \
  git clone https://github.com/zsailer/voila.git --branch extensionapp --single-branch --depth 1 && \
  cd voila && python setup.py develop && cd .. && \
  voila voila/notebooks/basics.ipynb

[2]

conda remove -y --name jupyter-server --all || true
conda create -y -n jupyter-server python=3.7 && \
  conda activate jupyter-server && \
  git clone https://github.com/jupyter/jupyter_server.git --branch master --single-branch --depth 1 && \
  cd jupyter_server && python setup.py develop && cd .. && \
  git clone https://github.com/zsailer/notebook.git --branch notebook-ext --single-branch --depth 1 && \
  cd notebook && python setup.py develop && cd .. && \
  jupyter notebook

[echarles]

@Zsailer For voila, I realize I was falling back to a repvois voila bin. When I build your branch, I don't have errors, but the voila bin is not generated (well the jupyter-voila bin).

[echarles]

Quick update to my latest comment comment for voila - it is now working for me with latest commits from @maartenbreddels in https://github.com/Zsailer/voila/tree/extensionapp