Skip to content

Commit

Permalink
Merge develop
Browse files Browse the repository at this point in the history
  • Loading branch information
iameskild committed Oct 24, 2023
2 parents 6f7a93d + 376ffba commit e0fb856
Show file tree
Hide file tree
Showing 421 changed files with 12,397 additions and 5,425 deletions.
6 changes: 5 additions & 1 deletion .github/ISSUE_TEMPLATE/release-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,16 @@ Release captain responsible - <@gh_username>

## Cut the official release

_If there were changes to the following packages, handle their releases before cutting a new release for Nebari_
- [ ] [Cut PyPI release for `nebari-workflow-controller`](https://github.com/nebari-dev/nebari-workflow-controller)
- [ ] [Cut PyPI release for `argo-jupyter-scheduler`](https://github.com/nebari-dev/argo-jupyter-scheduler)

_These steps must be actioned in the order they appear in this checklist._

- [ ] [Tag, build and push docker images](https://github.com/nebari-dev/nebari-docker-images/releases/new)
- [ ] [Update and cut release for `nebari-dask` meta package on Conda-Forge.](https://github.com/conda-forge/nebari-dask-feedstock)
- [ ] [Cut PyPI release via GHA release workflow.](https://github.com/nebari-dev/nebari/releases/new)
- Avoid appending `v` to tag.
- Copy release notes from `RELEASE.md`.
- [ ] Merge release branch into `main`
- [ ] [Merge automated release PR for `nebari` on Conda-Forge.](https://github.com/conda-forge/nebari-feedstock)
- [ ] Merge release branch into `main`
18 changes: 9 additions & 9 deletions .github/ISSUE_TEMPLATE/testing-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ However, the following core services still need to be manually validated (until

At minimum, the following services will need to be tested:

- [ ] [Log into keycloak as root user](https://nebari-docs.netlify.app/how-tos/configuring-keycloak#change-keycloak-root-password)
- [ ] [Add a user](https://nebari-docs.netlify.app/how-tos/configuring-keycloak#adding-a-nebari-user)
- [ ] [Log into conda-store and create](https://nebari-docs.netlify.app/tutorials/creating-new-environments)
- [ ] [Log into keycloak as root user](https://www.nebari.dev/docs/how-tos/configuring-keycloak/#change-keycloak-root-password)
- [ ] [Add a user](https://www.nebari.dev/docs/how-tos/configuring-keycloak/#adding-a-nebari-user)
- [ ] [Log into conda-store and create](https://www.nebari.dev/docs/tutorials/creating-new-environments)
- [ ] a conda environment in a shared namespace and,
- [ ] a conda environment in your personal namespace
- [ ] [Launch dask-gateway cluster, test auto-scaler and](https://nebari-docs.netlify.app/tutorials/using_dask)
- [ ] [Validate that the dask-labextention is working](https://nebari-docs.netlify.app/tutorials/using_dask#step-5---viewing-the-dashboard-inside-of-jupyterlab)
- [ ] [Create a basic CDS Dashboard](https://nebari-docs.netlify.app/tutorials/creating-cds-dashboard)
- [ ] [Open VS-Code extension](https://nebari-docs.netlify.app/tutorials/using-vscode)
- [ ] [Add the Python extension](https://nebari-docs.netlify.app/tutorials/using-vscode#adding-extensions)
- [ ] [Create a `.py` file and run it](https://nebari-docs.netlify.app/tutorials/using-vscode#running-python-code)
- [ ] [Launch dask-gateway cluster, test auto-scaler and](https://www.nebari.dev/docs/tutorials/using_dask)
- [ ] [Validate that the dask-labextention is working](https://www.nebari.dev/docs/tutorials/using_dask/#step-4---understand-dasks-diagnostic-tools)
- [ ] [Confirm that a notebook can be submitted via Jupyter-Scheduler](https://nebari.dev/docs/tutorials/jupyter-scheduler)
- [ ] [Open VS-Code extension](https://www.nebari.dev/docs/how-tos/using-vscode)
- [ ] [Add the Python extension](https://www.nebari.dev/docs/how-tos/using-vscode#adding-extensions)
- [ ] [Create a `.py` file and run it](https://www.nebari.dev/docs/how-tos/using-vscode#running-python-code)
3 changes: 1 addition & 2 deletions .github/actions/publish-from-template/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@ runs:
shell: bash
env: ${{ env }}
run:
python .github/actions/publish-from-template/render_template.py ${{
inputs.filename }}
python ${{ github.action_path }}/render_template.py ${{inputs.filename }}

- uses: JasonEtco/create-an-issue@v2
# Only render template and create an issue in case the workflow is a scheduled one
Expand Down
76 changes: 76 additions & 0 deletions .github/workflows/generate_cli_doc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Update API docs

on:
pull_request:
paths:
- "src/_nebari/subcommands/**"
- "src/_nebari/cli.py"
push:
branches:
- main
paths:
- "src/_nebari/subcommands/**"
- "src/_nebari/cli.py"
workflow_dispatch:

jobs:
update_api:
permissions:
contents: write
pull-requests: write
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l {0}
working-directory: ./docs-sphinx
steps:
- name: Check out repository 🛎️
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: "3.10"

- name: Install nebari and docs dependencies
run: |
python -m pip install --upgrade pip
pip install -e "../[docs]"
- name: Generate new API docs
run: |
make html
- name: Copy cli doc
run: |
cp _build/html/cli.html cli.html
- name: Look for changes to generated docs
uses: tj-actions/verify-changed-files@v12
id: verify-changed-files
with:
files: |
docs-sphinx/cli.html
- name: Create Pull Request in code repo
id: create_pull_request
uses: peter-evans/create-pull-request@v4
if: steps.verify-changed-files.outputs.files_changed == 'true' && github.event_name != 'pull_request'
with:
token: ${{ secrets.NEBARI_SENSEI_API_DOCS_PR_OPENER }}
commit-message: Update api docs
committer: GitHub <noreply@github.com>
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
signoff: false
branch: auto_cli_doc_update
delete-branch: true
title: '[AUTO] Update CLI doc'
body: |
Update CLI doc
- Auto-generated by [create-pull-request][1]
[1]: https://github.com/peter-evans/create-pull-request
labels: |
"area: documentation 📖"
draft: false
base: ${{ github.head_ref }}
1 change: 1 addition & 0 deletions .github/workflows/kubernetes_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,7 @@ jobs:
pytest tests_deployment/ -v -s
- name: JupyterHub Notebook Tests
timeout-minutes: 2
# run jhub-client after pytest since jhubctl can cleanup
# the running server
run: |
Expand Down
21 changes: 10 additions & 11 deletions .github/workflows/test-provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,11 @@ on:
paths:
- ".github/workflows/kubernetes_test.yaml"
- "tests/**"
- "tests_deployment/**"
- "tests_e2e/**"
- "scripts/**"
- "src/**"
- "pyproject.toml"
- "pytest.ini"
- ".cirun.yml"
push:
branches:
- main
Expand All @@ -19,12 +18,11 @@ on:
paths:
- ".github/workflows/kubernetes_test.yaml"
- "tests/**"
- "tests_deployment/**"
- "tests_e2e/**"
- "scripts/**"
- "src/**"
- "pyproject.toml"
- "pytest.ini"
- ".cirun.yml"
workflow_call:
inputs:
pr_number:
Expand Down Expand Up @@ -148,14 +146,14 @@ jobs:
env:
CYPRESS_BASE_URL: https://github-actions.nebari.dev/
with:
working-directory: tests_e2e
working-directory: tests/tests_e2e

- name: Playwright Tests
env:
KEYCLOAK_USERNAME: ${{ env.CYPRESS_EXAMPLE_USER_NAME }}
KEYCLOAK_PASSWORD: ${{ env.CYPRESS_EXAMPLE_USER_PASSWORD }}
NEBARI_FULL_URL: https://github-actions.nebari.dev/
working-directory: tests_e2e/playwright
working-directory: tests/tests_e2e/playwright
run: |
# create environment file
envsubst < .env.tpl > .env
Expand All @@ -168,17 +166,18 @@ jobs:
with:
name: e2e-cypress
path: |
./tests_e2e/cypress/screenshots/
./tests_e2e/cypress/videos/
./tests_e2e/playwright/videos/
./tests/tests_e2e/cypress/screenshots/
./tests/tests_e2e/cypress/videos/
./tests/tests_e2e/playwright/videos/
- name: Deployment Pytests
run: |
export KEYCLOAK_USERNAME=${CYPRESS_EXAMPLE_USER_NAME}
export KEYCLOAK_PASSWORD=${CYPRESS_EXAMPLE_USER_PASSWORD}
pytest tests_deployment/ -v -s
pytest tests/tests_deployment/ -v -s
- name: JupyterHub Notebook Tests
timeout-minutes: 2
# run jhub-client after pytest since jhubctl can cleanup
# the running server
run: |
Expand All @@ -190,7 +189,7 @@ jobs:
--validate --no-verify-ssl \
--kernel python3 \
--stop-server \
--notebook tests_deployment/assets/notebook/simple.ipynb \
--notebook tests/tests_deployment/assets/notebook/simple.ipynb \
### CLEANUP AFTER TESTS
- name: Cleanup nebari deployment
Expand Down
24 changes: 19 additions & 5 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ jobs:
test-general:
name: "Pytest"
runs-on: ubuntu-latest
defaults:
run:
shell: bash -el {0}
strategy:
matrix:
python-version:
Expand All @@ -38,22 +41,33 @@ jobs:
- "3.10"
- "3.11"
fail-fast: false
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}-${{ matrix.python-version }}
cancel-in-progress: true
steps:
- name: "Checkout Infrastructure"
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Setup miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
python-version: ${{ matrix.python-version }}
channel-priority: strict
channels: conda-forge
channels: conda-forge,defaults
activate-environment: nebari-dev

- name: Install Nebari
run: |
pip install .[dev]
conda install --quiet --yes conda-build
python --version
pip install -e .[dev]
- name: Test Nebari
run: |
pytest --version
pytest tests/tests_unit -vvv
pytest --cov=src --cov-report=xml --cov-config=pyproject.toml tests/tests_unit
- name: Report Coverage
run: |
coverage report -m
86 changes: 86 additions & 0 deletions .github/workflows/test_aws_integration.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
name: test-aws-integration

on:
schedule:
- cron: "0 0 * * MON"
workflow_dispatch:
inputs:
branch:
description: 'Nebari branch to deploy, test, destroy'
required: true
default: develop
type: string
image-tag:
description: 'Nebari image tag created by the nebari-docker-images repo'
required: true
default: main
type: string
tf-log-level:
description: 'Change Terraform log levels'
required: false
default: info
type: choice
options:
- info
- warn
- debug
- trace
- error


env:
AWS_DEFAULT_REGION: "us-west-2"
NEBARI_GH_BRANCH: ${{ github.event.inputs.branch || 'develop' }}
NEBARI_IMAGE_TAG: ${{ github.event.inputs.image-tag || 'main' }}
TF_LOG: ${{ github.event.inputs.tf-log-level || 'info' }}


jobs:
test-aws-integration:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Checkout
uses: actions/checkout@v3
with:
ref: ${{ env.NEBARI_GH_BRANCH }}
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.11

- name: Install Nebari
run: |
pip install .[dev]
playwright install
- name: Retrieve secret from Vault
uses: hashicorp/vault-action@v2.5.0
with:
method: jwt
url: "https://quansight-vault-public-vault-b2379fa7.d415e30e.z1.hashicorp.cloud:8200"
namespace: "admin/quansight"
role: "repository-nebari-dev-nebari-role"
secrets: |
kv/data/repository/nebari-dev/nebari/amazon_web_services/nebari-dev-ci role_name | AWS_ROLE_ARN;
kv/data/repository/nebari-dev/nebari/cloudflare/internal-devops@quansight.com/nebari-dev-ci token | CLOUDFLARE_TOKEN;
- name: Authenticate to AWS
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: ${{ env.AWS_ROLE_ARN }}
role-session-name: github-action
aws-region: ${{ env.AWS_DEFAULT_REGION }}

- name: Integration Tests
run: |
pytest --version
pytest tests/tests_integration/ -vvv -s --cloud aws
env:
NEBARI_SECRET__default_images__jupyterhub: "quay.io/nebari/nebari-jupyterhub:${{ env.NEBARI_IMAGE_TAG }}"
NEBARI_SECRET__default_images__jupyterlab: "quay.io/nebari/nebari-jupyterlab:${{ env.NEBARI_IMAGE_TAG }}"
NEBARI_SECRET__default_images__dask_worker: "quay.io/nebari/nebari-dask-worker:${{ env.NEBARI_IMAGE_TAG }}"
Loading

0 comments on commit e0fb856

Please sign in to comment.