[RELEASE] - merge develop into main

.github/ISSUE_TEMPLATE/release-checklist.md

@@ -55,12 +55,16 @@ Release captain responsible - <@gh_username>
 
 ## Cut the official release
 
+_If there were changes to the following packages, handle their releases before cutting a new release for Nebari_
+- [ ] [Cut PyPI release for `nebari-workflow-controller`](https://github.com/nebari-dev/nebari-workflow-controller)
+- [ ] [Cut PyPI release for `argo-jupyter-scheduler`](https://github.com/nebari-dev/argo-jupyter-scheduler)
+
 _These steps must be actioned in the order they appear in this checklist._
 
 - [ ] [Tag, build and push docker images](https://github.com/nebari-dev/nebari-docker-images/releases/new)
 - [ ] [Update and cut release for `nebari-dask` meta package on Conda-Forge.](https://github.com/conda-forge/nebari-dask-feedstock)
 - [ ] [Cut PyPI release via GHA release workflow.](https://github.com/nebari-dev/nebari/releases/new)
   - Avoid appending `v` to tag.
     - Copy release notes from `RELEASE.md`.
-- [ ] Merge release branch into `main`
 - [ ] [Merge automated release PR for `nebari` on Conda-Forge.](https://github.com/conda-forge/nebari-feedstock)
+- [ ] Merge release branch into `main`

.github/ISSUE_TEMPLATE/testing-checklist.md

@@ -18,14 +18,14 @@ However, the following core services still need to be manually validated (until
 
 At minimum, the following services will need to be tested:
 
-- [ ] [Log into keycloak as root user](https://nebari-docs.netlify.app/how-tos/configuring-keycloak#change-keycloak-root-password)
-  - [ ] [Add a user](https://nebari-docs.netlify.app/how-tos/configuring-keycloak#adding-a-nebari-user)
-- [ ] [Log into conda-store and create](https://nebari-docs.netlify.app/tutorials/creating-new-environments)
+- [ ] [Log into keycloak as root user](https://www.nebari.dev/docs/how-tos/configuring-keycloak/#change-keycloak-root-password)
+  - [ ] [Add a user](https://www.nebari.dev/docs/how-tos/configuring-keycloak/#adding-a-nebari-user)
+- [ ] [Log into conda-store and create](https://www.nebari.dev/docs/tutorials/creating-new-environments)
   - [ ] a conda environment in a shared namespace and,
   - [ ] a conda environment in your personal namespace
-- [ ] [Launch dask-gateway cluster, test auto-scaler and](https://nebari-docs.netlify.app/tutorials/using_dask)
-  - [ ] [Validate that the dask-labextention is working](https://nebari-docs.netlify.app/tutorials/using_dask#step-5---viewing-the-dashboard-inside-of-jupyterlab)
-- [ ] [Create a basic CDS Dashboard](https://nebari-docs.netlify.app/tutorials/creating-cds-dashboard)
-- [ ] [Open VS-Code extension](https://nebari-docs.netlify.app/tutorials/using-vscode)
-  - [ ] [Add the Python extension](https://nebari-docs.netlify.app/tutorials/using-vscode#adding-extensions)
-  - [ ] [Create a `.py` file and run it](https://nebari-docs.netlify.app/tutorials/using-vscode#running-python-code)
+- [ ] [Launch dask-gateway cluster, test auto-scaler and](https://www.nebari.dev/docs/tutorials/using_dask)
+  - [ ] [Validate that the dask-labextention is working](https://www.nebari.dev/docs/tutorials/using_dask/#step-4---understand-dasks-diagnostic-tools)
+- [ ] [Confirm that a notebook can be submitted via Jupyter-Scheduler](https://nebari.dev/docs/tutorials/jupyter-scheduler)
+- [ ] [Open VS-Code extension](https://www.nebari.dev/docs/how-tos/using-vscode)
+  - [ ] [Add the Python extension](https://www.nebari.dev/docs/how-tos/using-vscode#adding-extensions)
+  - [ ] [Create a `.py` file and run it](https://www.nebari.dev/docs/how-tos/using-vscode#running-python-code)

.github/actions/publish-from-template/action.yml

@@ -16,8 +16,7 @@ runs:
       shell: bash
       env: ${{ env }}
       run:
-        python .github/actions/publish-from-template/render_template.py ${{
-        inputs.filename }}
+        python ${{ github.action_path }}/render_template.py ${{inputs.filename }}
 
     - uses: JasonEtco/create-an-issue@v2
       # Only render template and create an issue in case the workflow is a scheduled one

.github/workflows/generate_cli_doc.yml

@@ -0,0 +1,76 @@
+name: Update API docs
+
+on:
+  pull_request:
+    paths:
+    - "src/_nebari/subcommands/**"
+    - "src/_nebari/cli.py"
+  push:
+    branches:
+      - main
+    paths:
+    - "src/_nebari/subcommands/**"
+    - "src/_nebari/cli.py"
+  workflow_dispatch:
+
+jobs:
+  update_api:
+    permissions:
+      contents: write
+      pull-requests: write
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -l {0}
+        working-directory: ./docs-sphinx
+    steps:
+      - name: Check out repository 🛎️
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: "3.10"
+
+      - name: Install nebari and docs dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e "../[docs]"
+
+      - name: Generate new API docs
+        run: |
+          make html
+
+      - name: Copy cli doc
+        run: |
+          cp _build/html/cli.html cli.html
+
+      - name: Look for changes to generated docs
+        uses: tj-actions/verify-changed-files@v12
+        id: verify-changed-files
+        with:
+          files: |
+            docs-sphinx/cli.html
+
+      - name: Create Pull Request in code repo
+        id: create_pull_request
+        uses: peter-evans/create-pull-request@v4
+        if: steps.verify-changed-files.outputs.files_changed == 'true' && github.event_name != 'pull_request'
+        with:
+          token: ${{ secrets.NEBARI_SENSEI_API_DOCS_PR_OPENER }}
+          commit-message: Update api docs
+          committer: GitHub <noreply@github.com>
+          author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
+          signoff: false
+          branch: auto_cli_doc_update
+          delete-branch: true
+          title: '[AUTO] Update CLI doc'
+          body: |
+            Update CLI doc
+            - Auto-generated by [create-pull-request][1]
+
+            [1]: https://github.com/peter-evans/create-pull-request
+          labels: |
+            "area: documentation 📖"
+          draft: false
+          base: ${{ github.head_ref }}

.github/workflows/kubernetes_test.yaml

@@ -5,12 +5,11 @@ on:
     paths:
       - ".github/workflows/kubernetes_test.yaml"
       - "tests/**"
-      - "tests_deployment/**"
-      - "tests_e2e/**"
       - "scripts/**"
       - "src/**"
       - "pyproject.toml"
       - "pytest.ini"
+      - ".cirun.yml"
   push:
     branches:
       - main
@@ -19,12 +18,11 @@ on:
     paths:
       - ".github/workflows/kubernetes_test.yaml"
       - "tests/**"
-      - "tests_deployment/**"
-      - "tests_e2e/**"
       - "scripts/**"
       - "src/**"
       - "pyproject.toml"
       - "pytest.ini"
+      - ".cirun.yml"
   workflow_call:
     inputs:
       pr_number:
@@ -148,14 +146,14 @@ jobs:
         env:
           CYPRESS_BASE_URL: https://github-actions.nebari.dev/
         with:
-          working-directory: tests_e2e
+          working-directory: tests/tests_e2e
 
       - name: Playwright Tests
         env:
           KEYCLOAK_USERNAME: ${{ env.CYPRESS_EXAMPLE_USER_NAME }}
           KEYCLOAK_PASSWORD: ${{ env.CYPRESS_EXAMPLE_USER_PASSWORD }}
           NEBARI_FULL_URL: https://github-actions.nebari.dev/
-        working-directory: tests_e2e/playwright
+        working-directory: tests/tests_e2e/playwright
         run: |
           # create environment file
           envsubst < .env.tpl > .env
@@ -168,17 +166,18 @@ jobs:
         with:
           name: e2e-cypress
           path: |
-            ./tests_e2e/cypress/screenshots/
-            ./tests_e2e/cypress/videos/
-            ./tests_e2e/playwright/videos/
+            ./tests/tests_e2e/cypress/screenshots/
+            ./tests/tests_e2e/cypress/videos/
+            ./tests/tests_e2e/playwright/videos/
 
       - name: Deployment Pytests
         run: |
           export KEYCLOAK_USERNAME=${CYPRESS_EXAMPLE_USER_NAME}
           export KEYCLOAK_PASSWORD=${CYPRESS_EXAMPLE_USER_PASSWORD}
-          pytest tests_deployment/ -v -s
+          pytest tests/tests_deployment/ -v -s
 
       - name: JupyterHub Notebook Tests
+        timeout-minutes: 2
         # run jhub-client after pytest since jhubctl can cleanup
         # the running server
         run: |
@@ -190,7 +189,7 @@ jobs:
                                 --validate --no-verify-ssl \
                                 --kernel python3 \
                                 --stop-server \
-                                --notebook tests_deployment/assets/notebook/simple.ipynb \
+                                --notebook tests/tests_deployment/assets/notebook/simple.ipynb \
 
       ### CLEANUP AFTER TESTS
       - name: Cleanup nebari deployment

.github/workflows/test-provider.yaml

@@ -5,12 +5,11 @@ on:
     paths:
       - ".github/workflows/kubernetes_test.yaml"
       - "tests/**"
-      - "tests_deployment/**"
-      - "tests_e2e/**"
       - "scripts/**"
       - "src/**"
       - "pyproject.toml"
       - "pytest.ini"
+      - ".cirun.yml"
   push:
     branches:
       - main
@@ -19,12 +18,11 @@ on:
     paths:
       - ".github/workflows/kubernetes_test.yaml"
       - "tests/**"
-      - "tests_deployment/**"
-      - "tests_e2e/**"
       - "scripts/**"
       - "src/**"
       - "pyproject.toml"
       - "pytest.ini"
+      - ".cirun.yml"
   workflow_call:
     inputs:
       pr_number:
@@ -148,14 +146,14 @@ jobs:
         env:
           CYPRESS_BASE_URL: https://github-actions.nebari.dev/
         with:
-          working-directory: tests_e2e
+          working-directory: tests/tests_e2e
 
       - name: Playwright Tests
         env:
           KEYCLOAK_USERNAME: ${{ env.CYPRESS_EXAMPLE_USER_NAME }}
           KEYCLOAK_PASSWORD: ${{ env.CYPRESS_EXAMPLE_USER_PASSWORD }}
           NEBARI_FULL_URL: https://github-actions.nebari.dev/
-        working-directory: tests_e2e/playwright
+        working-directory: tests/tests_e2e/playwright
         run: |
           # create environment file
           envsubst < .env.tpl > .env
@@ -168,17 +166,18 @@ jobs:
         with:
           name: e2e-cypress
           path: |
-            ./tests_e2e/cypress/screenshots/
-            ./tests_e2e/cypress/videos/
-            ./tests_e2e/playwright/videos/
+            ./tests/tests_e2e/cypress/screenshots/
+            ./tests/tests_e2e/cypress/videos/
+            ./tests/tests_e2e/playwright/videos/
 
       - name: Deployment Pytests
         run: |
           export KEYCLOAK_USERNAME=${CYPRESS_EXAMPLE_USER_NAME}
           export KEYCLOAK_PASSWORD=${CYPRESS_EXAMPLE_USER_PASSWORD}
-          pytest tests_deployment/ -v -s
+          pytest tests/tests_deployment/ -v -s
 
       - name: JupyterHub Notebook Tests
+        timeout-minutes: 2
         # run jhub-client after pytest since jhubctl can cleanup
         # the running server
         run: |
@@ -190,7 +189,7 @@ jobs:
                                 --validate --no-verify-ssl \
                                 --kernel python3 \
                                 --stop-server \
-                                --notebook tests_deployment/assets/notebook/simple.ipynb \
+                                --notebook tests/tests_deployment/assets/notebook/simple.ipynb \
 
       ### CLEANUP AFTER TESTS
       - name: Cleanup nebari deployment

.github/workflows/test.yaml

@@ -30,6 +30,9 @@ jobs:
   test-general:
     name: "Pytest"
     runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -el {0}
     strategy:
       matrix:
         python-version:
@@ -38,22 +41,33 @@ jobs:
           - "3.10"
           - "3.11"
       fail-fast: false
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}-${{ matrix.python-version }}
+      cancel-in-progress: true
     steps:
       - name: "Checkout Infrastructure"
         uses: actions/checkout@v3
         with:
           fetch-depth: 0
+
       - name: Setup miniconda
         uses: conda-incubator/setup-miniconda@v2
         with:
+          auto-update-conda: true
           python-version: ${{ matrix.python-version }}
-          channel-priority: strict
-          channels: conda-forge
+          channels: conda-forge,defaults
+          activate-environment: nebari-dev
+
       - name: Install Nebari
         run: |
-          pip install .[dev]
-          conda install --quiet --yes conda-build
+          python --version
+          pip install -e .[dev]
+
       - name: Test Nebari
         run: |
           pytest --version
-          pytest tests/tests_unit -vvv
+          pytest --cov=src --cov-report=xml --cov-config=pyproject.toml tests/tests_unit
+
+      - name: Report Coverage
+        run: |
+          coverage report -m