Allow jupyterhub.overrides in qhub-config.yaml (#930)

* allow jupyterhub.overrides in qhub-config.yaml * Try helm 0.11.1 and hub 1.5.0 * Try 1.2.0 * docs - jupyterhub helm overrides / keycloak docs * networkPolicy disabled for z2jh Helm Chart * vale fixes * fix broken links * remove oauthcallbacks from docs
nebari-dev · Nov 24, 2021 · c0d08bb · c0d08bb
1 parent 90e3448
commit c0d08bb
Show file tree

Hide file tree

Showing 27 changed files with 141 additions and 82 deletions.
diff --git a/docs/source/admin_guide/index.md b/docs/source/admin_guide/index.md
@@ -11,6 +11,8 @@ gpu.md
 preemptible-spot-instances.md
 system_maintenance.md
 monitoring.md
+jupyterhub.md
+keycloak.md
 clearml.md
 prefect.md
 custom-helm-charts.md

diff --git a/docs/source/admin_guide/jupyterhub.md b/docs/source/admin_guide/jupyterhub.md
@@ -0,0 +1,20 @@
+# JupyterHub
+
+QHub has the JupyterHub project at its core.
+
+Within the `qhub deploy` step, JupyterHub is installed using the [Zero2JupyterHub Helm package](https://zero-to-jupyterhub.readthedocs.io/).
+
+It is possible to specify Helm overrides (i.e. your own values for selected fields in the JupyterHub deployment's `values.yaml` file) from the `qhub-config.yaml` file. However, be aware that this may conflict with values that are needed to be set in a certain way in order for QHub to operate correctly.
+
+To set a Helm override, for example enabling auth state:
+
+```
+jupyterhub:
+  overrides:
+    hub:
+      config:
+        Authenticator:
+          enable_auth_state: true
+```
+
+Where it is possible to influence a value using 'native' QHub configuration, you should use that as a preference. For example, you would not set `jupyterhub.overrides.hub.image.name` to use a custom JupyterHub Docker image. Instead you would set `default_images.jupyterhub`.
diff --git a/docs/source/admin_guide/keycloak.md b/docs/source/admin_guide/keycloak.md
@@ -0,0 +1,29 @@
+# Keycloak
+
+QHub includes a deployment of [Keycloak](https://www.keycloak.org/documentation.html) to centralise user management.
+
+Within the `qhub deploy` step, Keycloak is installed using the [Helm chart](https://github.com/codecentric/helm-charts/tree/master/charts/keycloak).
+
+It is possible to specify Helm overrides (i.e. your own values for selected fields in the Keycloak deployment's `values.yaml` file) from the `qhub-config.yaml` file. However, be aware that this may conflict with values that are needed to be set in a certain way in order for QHub to operate correctly.
+
+To set a Helm override, for example:
+
+```
+security:
+  keycloak:
+    initial_root_password: password123
+    overrides:
+      extraEnv: |
+        - name: KEYCLOAK_DEFAULT_THEME
+          value: entqhubtheme
+        - name: KEYCLOAK_WELCOME_THEME
+          value: entqhubtheme
+        - name: PROXY_ADDRESS_FORWARDING
+          value: "true"
+      image:
+        repository: dockerusername/my-qhub-keycloak
+```
+
+If you do set `overrides.extraEnv` as above, you must remember to include `PROXY_ADDRESS_FORWARDING=true`. Otherwise, the Keycloak deployment will not work as you will have overridden an important default Helm value that is required by QHub.
+
+To find out more about using Keycloak in QHub, see [Installation - Login](../installation/login.md)
diff --git a/docs/source/admin_guide/prefect.md b/docs/source/admin_guide/prefect.md
@@ -53,7 +53,7 @@ When you enable prefect via `qhub-config.yml` prefect agent is deployed on the
 QHub's kubernetes cluster, which queries the Prefect Cloud for flow runs.
 
 ## Agent configuration overrides
-You can override your agent configuration without having to modify the helm files directly.  The extra variable `overrides` makes this
+You can override your agent configuration without having to modify the helm files directly. The extra variable `overrides` makes this
 possible by changing the default values for the Agent chart according to the settings presented on your qhub-config.yaml file.
 
 The current variables, originally available in the [Agent helm chart](https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml) that can be overridden include:
@@ -91,7 +91,7 @@ prefect:
      envVars:
        MY_VAR: "<value>"
 ```
-### Adding secrets to you Agent configuration
+### Adding secrets to your Agent configuration
 Overrides also allow you to define extra secrets to pass through your agent configuration, for example, when using [default secrets](https://docs.prefect.io/core/concepts/secrets.html#default-secrets) to automatically authenticate your flow with the listed service. In the Google cloud case, for `GCP_CREDENTIALS` context secret, you can do it by adding that specific key value pair into your configuration:
 
 ```yaml

diff --git a/docs/source/admin_guide/system_maintenance.md b/docs/source/admin_guide/system_maintenance.md
@@ -18,7 +18,7 @@ are three images that are currently built
 - conda-store :: Environment management tool for QHub
 
 Each docker image is customized with its respective directory
-(e.g. `image/Dockerfile.jupyterlab` -> `image/jupyterlab/*`. For
+(for example `image/Dockerfile.jupyterlab` -> `image/jupyterlab/*`. For
 jupyterlab the environment is located at
 `image/jupyterlab/environment.yaml`. Thus to add a package to the
 environment simply submit a pull request with the new package.

diff --git a/docs/source/admin_guide/troubleshooting.md b/docs/source/admin_guide/troubleshooting.md
@@ -47,9 +47,9 @@ After completing these steps. `kubectl` should be able to access the cluster.
 
 #### Debug your Kubernetes cluster
 
- [K9s](https://k9scli.io/) is a terminal-based UI to manage Kubernetes clusters that aims to
- simplify navigating, observing, and managing your applications in K8s.
- K9s continuously monitors Kubernetes clusters for changes and provides
+ [`k9s`](https://k9scli.io/) is a terminal-based UI to manage Kubernetes clusters that aims to
+ simplify navigating, observing, and managing your applications in Kubernetes.
+ `k9s` continuously monitors Kubernetes clusters for changes and provides
  shortcut commands to interact with the observed resources becoming a
  fast way to review and resolve day-to-day issues in Kubernetes. It's
  definitely a huge improvement to the general workflow, and a best-to-have
@@ -59,17 +59,17 @@ Installation can be done on macOS, Windows, and Linux. Instructions
 for each operating system can be found [here](https://github.com/derailed/k9s).
 Complete the installation to follow along.
 
-By default, K9s starts with the standard directory that is set as the
+By default, `k9s` starts with the standard directory that is set as the
 context (in this case Minikube). To view all the current process press `0`:
 
-![Image of K9s termina UI](../images/k9s_UI.png)
+![Image of terminal UI](../images/k9s_UI.png)
 
 > **NOTE**: In some circumstances you will be confronted with the
   need to inspect any services launched by your cluster at your ‘localhost’. For instance, if your cluster has problem
 with the network traffic tunnel configuration, it may limit or block the user's
   access to destination resources over the connection.
 
-K9s port-forward option <kbd>shift</kbd> + <kbd>f</kbd> allows you to access and interact
+`k9s` port-forward option <kbd>shift</kbd> + <kbd>f</kbd> allows you to access and interact
 with internal Kubernetes cluster processes from your localhost you can
 then use this method to investigate issues and adjust your services
 locally without the need to expose them beforehand.

diff --git a/docs/source/admin_guide/upgrade.md b/docs/source/admin_guide/upgrade.md
@@ -41,15 +41,15 @@ You may have made special customizations to your qhub config, such as using your
 
 For CI/CD (GitHub/GitLab) workflows, then as well as generating the updated qhub-config.yaml files as above, you will also need to regenerate the workflow files based on the latest qhub version's templates.
 
-With the newly-upgraded qhub-config.yaml file, run:
+With the newly upgraded qhub-config.yaml file, run:
 
 ```shell
 qhub render -c qhub-config.yaml
 ```
 
 (Note that `qhub deploy` would perform this render step too, but will also immediately redeploy your qhub.)
 
-Commit all the files (qhub-config.yaml and GitHub/GitLab workflow files) back to the remote repo. All files need to be commited together in the same commit.
+Commit all the files (qhub-config.yaml and GitHub/GitLab workflow files) back to the remote repo. All files need to be committed together in the same commit.
 
 ## Known versions that require re-deployment
 

diff --git a/docs/source/dev_guide/minikube.md b/docs/source/dev_guide/minikube.md
@@ -17,7 +17,7 @@ Currently, **QHub local deployment is primarily compatible with Linux-based
 Operating Systems**. The main limitation for the installation on
 MacOS relates to [Docker Desktop for
 Mac](https://docs.docker.com/docker-for-mac/networking/#known-limitations-use-cases-and-workarounds)
-being unable to route traffic to containers.  Theoretically, the
+being unable to route traffic to containers. Theoretically, the
 installation of HyperKit Driver could solve the issue, although the
 proposed solution has not yet been tested. There are notes on our current attempts at workarounds for
 running [Minikube on Mac below](#minikube-on-mac).
@@ -203,7 +203,7 @@ The output should be `The 'metallb' addon is enabled`.
 <details>
   <summary>Click to expand note</summary>
 
-The browser can have trouble reaching the load balancer running on WSL2. A workaround is to port forward the proxy-... pod to the host (ip 0.0.0.0). Get the ip address of the WSL2 machine via ```ip a```, it should be a 127.x.x.x address. To change the port forwarding after opening k9s you can type ```:pods <enter>```, hover over the proxy-... pod and type ```<shift-s>```, and enter the ip addresses.
+The browser can have trouble reaching the load balancer running on WSL2. A workaround is to port forward the proxy-... pod to the host (ip 0.0.0.0). Get the ip address of the WSL2 machine via ```ip a```, it should be a 127.x.x.x address. To change the port forwarding after opening `k9s` you can type ```:pods <enter>```, hover over the proxy-... pod and type ```<shift-s>```, and enter the ip addresses.
 </details>
 
 ## Deploy QHub
@@ -487,7 +487,7 @@ qhub deploy --config qhub-config.yaml --disable-prompt
 
 ## Enable Kubernetes access from Mac
 
-This step is optional, but will allow you to use kubectl and K9s directly from your Mac. It is not needed
+This step is optional, but will allow you to use kubectl and `k9s` directly from your Mac. It is not needed
 if you are happy to use kubectl within an SSH session on AWS instead.
 
 On your Mac laptop:

diff --git a/docs/source/dev_guide/testing.md b/docs/source/dev_guide/testing.md
@@ -27,7 +27,7 @@ variable. In that case, Docker tags and workflow `pip install qhub`
 commands will be based on the qhub version specified in the
 qhub/version.py file, but these tags and releases may not yet exist,
 perhaps if the version has been updated to include a beta/dev
-component which has not been released.  So you may need to manually
+component which has not been released. So you may need to manually
 modify the qhub-config.yaml to 'downgrade' the tags to a full release
 version.
 
@@ -80,9 +80,9 @@ Hadolint will report `error`, `warning`, `info` and `style` while linting Docker
 
 ## Debug Kubernetes clusters
 
- To debug Kubernetes clusters, we advise you to use [K9s](https://k9scli.io/), a terminal-based UI that aims to
+ To debug Kubernetes clusters, we advise you to use [`k9s`](https://k9scli.io/), a terminal-based UI that aims to
  simplify navigation, observation, and management of applications in Kubernetes.
- K9s continuously monitors Kubernetes clusters for changes and provides
+ `k9s` continuously monitors Kubernetes clusters for changes and provides
  shortcut commands to interact with the observed resources becoming a
  fast way to review and resolve day-to-day issues in deployed clusters.
 
@@ -92,7 +92,7 @@ check out the [Troubleshooting documentation](https://docs.qhub.dev/en/stable/so
 
 ## Cypress Tests
 
-Cypress automates testing within a web browser environment. It is integrated into the GitHub Actions tests.yaml workflows in this repo, and 
+Cypress automates testing within a web browser environment. It is integrated into the GitHub Actions tests.yaml workflows in this repo, and
 you can also run it locally. To do so:
 
 ```
@@ -107,9 +107,9 @@ npm run cypress:open
 ```
 
 The Base URL can point anywhere that should be accessible - it can be the URL of a QHub cloud deployment.
-The QHub Config Path should point to the associated yaml file for that site. Most importantly, the tests will inspect the yaml file to understand 
-what tests are relevant. To start with, it checks security.authentication.type to determine what should be available on the login page, and 
-how to test it. If the login type is 'password' then it uses the value in CYPRESS_EXAMPLE_USER_PASSWORD as the password (default username is 
+The QHub Config Path should point to the associated yaml file for that site. Most importantly, the tests will inspect the yaml file to understand
+what tests are relevant. To start with, it checks security.authentication.type to determine what should be available on the login page, and
+how to test it. If the login type is 'password' then it uses the value in CYPRESS_EXAMPLE_USER_PASSWORD as the password (default username is
 `example-user` but this can be changed by setting CYPRESS_EXAMPLE_USER_NAME).
 
 The final command above should open the Cypress UI where you can run the tests manually and see the actions in the browser.

diff --git a/docs/source/installation/configuration.md b/docs/source/installation/configuration.md
@@ -142,7 +142,6 @@ security:
     config:
       client_id: <CLIENT_ID>
       client_secret: <CLIENT_SECRET>
-      oauth_callback_url: https://do.qhub.dev/hub/oauth_callback
 ```
 
 ### Omitting sensitive values
@@ -161,7 +160,6 @@ security:
     config:
       client_id: QHUB_SECRET_github_client_id
       client_secret: QHUB_SECRET_github_client_key
-      oauth_callback_url: https://do.qhub.dev/hub/oauth_callback
 ```
 
 ### Authentication
@@ -224,7 +222,7 @@ security:
 
 ### Keycloak
 
-The security.keycloak section allows you to specify an initial password for the `root` user (to login at https://myqhubsite.com/auth/admin/) to manage your Keycloak database, e.g. add users/groups.
+The `security.keycloak` section allows you to specify an initial password for the `root` user (to login at `https://myqhubsite.com/auth/admin/`) to manage your Keycloak database, e.g. add users/groups.
 
 You should change this after deployment. Future deployments will not reset the password to any specified in the YAML file.
 
@@ -553,7 +551,7 @@ closely follows the
 [KubeSpawner](https://jupyterhub-kubespawner.readthedocs.io/en/latest/spawner.html)
 API. The only exception is that two keys are added `users` and
 `groups` which allow restriction of profiles to a given set of groups
-and users.  We recommend using groups to manage profile access.
+and users. We recommend using groups to manage profile access.
 
 Finally, we allow for configuration of the Dask workers. In general,
 similar to the JupyterLab instances you only need to configuration the
@@ -746,7 +744,6 @@ security:
     config:
       client_id: CLIENT_ID
       client_secret: CLIENT_SECRET
-      oauth_callback_url: https://jupyter.do.qhub.dev/hub/oauth_callback
 
   users:
     example-user:

diff --git a/docs/source/installation/login.md b/docs/source/installation/login.md
@@ -5,7 +5,7 @@ Keycloak is the name of the open source user management software that is automat
 If you ran `qhub init` to create your `qhub-config.yaml` configuration file in the [Usage](usage.md) step, you will have been provided with a
 "random password for Keycloak root user".
 
-The password will also be visible in the `qhub-config.yaml` file under the security.keycloak.initial_root_password field.
+The password will also be visible in the `qhub-config.yaml` file under the `security.keycloak.initial_root_password` field.
 
 Note that when your QHub is first deployed, the password given at that location will be set for the root Keycloak user. If added or changed for subsequent deployments, that password value will have no effect.
 
@@ -28,13 +28,11 @@ This will create a user called bob with the initial password provided. Omit the
 
 ### Add user using Keycloak console
 
-To add a QHub user from the web console for Keycloak, visit:
-
-https://myqhubsite.com/auth/admin/
+To add a QHub user from the web console for Keycloak, visit `https://myqhubsite.com/auth/admin/`
 
 (Switch 'myqhubsite.com' for the domain you provided for your QHub deployment.)
 
-![Root Login to Keycloak](/source/images/keycloak_master_login.png)
+![Root Login to Keycloak](../images/keycloak_master_login.png)
 
 Login using the username `root` and the password provided for the initial Keycloak root password.
 
@@ -46,25 +44,25 @@ Click 'Users' along the left-hand side of the page.
 
 Click the 'Add user' button and you will see the new user form:
 
-![Add User to Keycloak](/source/images/keycloak_adduser.png)
+![Add User to Keycloak](../images/keycloak_adduser.png)
 
 Enter the name you would like for the user then click Save.
 
 Once the user has been created, you can set a password (not needed for GitHub/Auth0 login):
 
-![Set Password in Keycloak](/source/images/keycloak_user_password.png)
+![Set Password in Keycloak](../images/keycloak_user_password.png)
 
 It is best to unset the 'Temporary' on/off button so the user won't be forced to change the password on first login.
 
 ## Login to QHub
 
 Your new user can now log into QHub proper (not Keycloak's admin console).
 
-Visit https://myqhubsite.com/ (or whatever domain you have chosen for your QHub).
+Visit `https://myqhubsite.com/` (or whatever domain you have chosen for your QHub).
 
 Click 'Sign in with Keycloak'.
 
-![Login to Keycloak](/source/images/keycloak_qhub_login.png)
+![Login to Keycloak](../images/keycloak_qhub_login.png)
 
 Enter the username and password you chose when you added a user to QHub above.
 
@@ -74,17 +72,17 @@ If you chose GitHub or Auth0 login, click the 'GitHub' button to be taken to a G
 
 You should change your root password for Keycloak now that you've got things running.
 
-Back in https://myqhubsite.com/auth/admin/ you can click on the 'Root' dropdown in the top right of the screen, and select 'Manage account'.
+Back in `https://myqhubsite.com/auth/admin/` you can click on the 'Root' dropdown in the top right of the screen, and select 'Manage account'.
 
 Under 'Account Security' click 'Signing In'.
 
 In the Password box, click the 'Update' button. This will guide you through entering your existing root password, and then entering a new password.
 
-From this point, the security.keycloak.initial_root_password field in `qhub-config.yaml` has no effect. If you redeploy QHub it will not reset the password back to the old one (or anything else that might be in the field in your YAML file). You can delete that line from your YAML file if you wish.
+From this point, the `security.keycloak.initial_root_password` field in `qhub-config.yaml` has no effect. If you redeploy QHub it will not reset the password back to the old one (or anything else that might be in the field in your YAML file). You can delete that line from your YAML file if you wish.
 
 # Groups
 
-Add Groups in the same Keycloak backend as you can add users - that is, login as `root` to https://myqhubsite.com/auth/admin/. Click Groups on the left-hand side.
+Add Groups in the same Keycloak backend as you can add users - that is, login as `root` to `https://myqhubsite.com/auth/admin/`. Click Groups on the left-hand side.
 
 Groups named `users` and `admin` will have been created automatically by QHub. All users will be added to the `users` group automatically when you create them. You should never remove them from the `users` group as that group must contain all users.