Initial playwright setup (#1665)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Philip Meier <github.pmeier@posteo.de>
Co-authored-by: Amit Kumar <dtu.amit@gmail.com>

.github/workflows/kubernetes_test.yaml

@@ -69,6 +69,7 @@ jobs:
         run: |
           conda install --quiet --yes -c anaconda pip
           pip install .[dev]
+          playwright install
       - name: Download and Install Kubectl
         run: |
           mkdir -p bin
@@ -149,6 +150,18 @@ jobs:
         with:
           working-directory: tests_e2e
 
+      - name: Playwright Tests
+        env:
+          KEYCLOAK_USERNAME: ${{ env.CYPRESS_EXAMPLE_USER_NAME }}
+          KEYCLOAK_PASSWORD: ${{ env.CYPRESS_EXAMPLE_USER_PASSWORD }}
+          NEBARI_FULL_URL: https://github-actions.nebari.dev/
+        working-directory: tests_e2e/playwright
+        run: |
+          # create environment file
+          envsubst < .env.tpl > .env
+          # run playwright pytest tests in headed mode with the chromium browser
+          xvfb-run pytest --browser chromium
+
       - name: Save Cypress screenshots and videos
         if: always()
         uses: actions/upload-artifact@v3
@@ -157,6 +170,7 @@ jobs:
           path: |
             ./tests_e2e/cypress/screenshots/
             ./tests_e2e/cypress/videos/
+            ./tests_e2e/playwright/videos/
 
       - name: Deployment Pytests
         run: |

.github/workflows/test.yaml

@@ -6,7 +6,7 @@ on:
       - ".github/workflows/test.yaml"
       - "tests/**"
       - "tests_deployment/**"
-      - "tests_e2e/**"
+      - "tests_e2e/cypress/**"
       - "scripts/**"
       - "src/**"
       - "pyproject.toml"
@@ -20,7 +20,7 @@ on:
       - ".github/workflows/test.yaml"
       - "tests/**"
       - "tests_deployment/**"
-      - "tests_e2e/**"
+      - "tests_e2e/cypress/**"
       - "scripts/**"
       - "src/**"
       - "pyproject.toml"
@@ -56,4 +56,4 @@ jobs:
       - name: Test Nebari
         run: |
           pytest --version
-          pytest --ignore=tests_deployment
+          pytest --ignore=tests_deployment --ignore=tests_e2e/playwright

.gitignore

@@ -1,3 +1,8 @@
+# docs
+.nox
+_build
+.env
+
 # setuptools scm
 src/_nebari/_version.py
 
@@ -7,6 +12,9 @@ terraform.tfstate
 terraform.tfstate.backup
 .terraform.tfstate.lock.info
 
+# tests
+videos/
+
 # python
 __pycache__
 build/
@@ -41,5 +49,6 @@ nebari-config.yaml
 
 .vscode/
 .pytest_cache
-
+.ipynb_checkpoints
+.DS_Store
 /.ruff_cache

pyproject.toml

@@ -76,6 +76,7 @@ dev = [
     "black==22.3.0",
     "dask-gateway",
     "diagrams",
+    "python-dotenv",
     "escapism",
     "flake8==3.8.4",
     "importlib-metadata<5.0",
@@ -84,6 +85,7 @@ dev = [
     "pre-commit",
     "pytest",
     "pytest-timeout",
+    "pytest-playwright",
     "grayskull",
     "build",
     "jinja2",

tests_e2e/playwright/.env.tpl

@@ -0,0 +1,3 @@
+KEYCLOAK_USERNAME="USERNAME_OR_GOOGLE_EMAIL"
+KEYCLOAK_PASSWORD="PASSWORD"
+NEBARI_FULL_URL="https://nebari.quansight.dev/"

tests_e2e/playwright/README.md

@@ -0,0 +1,199 @@
+# Nebari integration testing with Playwright
+
+
+## How does it work?
+
+Playwright manages interactions with any website. We are using it to interact
+with a deployed Nebari instance and test the various integrations that are
+included.
+
+For our test suite, we utilize Playwright's synchronous API. The first task
+is to launch the web browser you'd like to test in. Options in our test suite
+are `chromium`, `webkit`, and `firefox`. Playwright uses browser contexts to
+achieve test isolation. The context can either be created by default or
+manually (for the purposes of generating multiple contexts per test in the case
+of admin vs user testing). Next the page on the browser is created. For all
+tests this starts as a blank page, then during the test, we navigate to a given
+url. This is all achieved in the `setup` method of the `Navigator` class.
+
+## Setup
+
+Install Nebari with the development requirements (which include Playwright)
+
+`pip install -e ".[dev]"`
+
+Then install playwright itself (required).
+
+`playwright install`
+
+> If you see the warning `BEWARE: your OS is not officially supported by Playwright; downloading fallback build., it is not critical.` Playwright will likely still work microsoft/playwright#15124
+
+### Create environment file
+
+Create a copy of the `.env` template file
+
+```bash
+cd tests_e2e/playwright
+cp .env.tpl .env
+```
+
+Fill in the newly created `.env` file with the following values:
+
+* KEYCLOAK_USERNAME: Nebari username for username/password login OR Google email address or Google sign in
+* KEYCLOAK_PASSWORD: Password associated with USERNAME
+* NEBARI_FULL_URL: full url path including scheme to Nebari instance, e.g. "https://nebari.quansight.dev/"
+
+This user can be created with the following command (or you can use an existing non-root user):
+
+```
+nebari keycloak adduser --user <username> <password> --config <NEBARI_CONFIG_PATH>
+```
+
+## Running the Playwright tests
+
+The playwright tests are run inside of pytest using
+
+```python
+pytest tests_e2e/playwright/test_playwright.py
+```
+
+Videos of the test playback will be available in `$PWD/videos/`.
+To see what is happening while the test is run, pass the `--headed` option to `pytest`.
+You can also add the `--slowmo=$MILLI_SECONDS` option to add a delay before each action
+by Playwright and thus slowing down the process.
+
+Another option is to run playwright methods outside of pytest. Both
+`navigator.py` and `run_notebook.py` can be run as scripts. For example,
+
+```python
+    import os
+
+    import dotenv
+    # load environment variables from .env file
+    dotenv.load_dotenv()
+    # instantiate the navigator class
+    nav = Navigator(
+        nebari_url="https://nebari.quansight.dev/",
+        username=os.environ["KEYCLOAK_USERNAME"],
+        password=os.environ["KEYCLOAK_PASSWORD"],
+        auth="password",
+        instance_name="small-instance",
+        headless=False,
+        slow_mo=100,
+    )
+    # go through login sequence (defined by `auth` method in Navigator class)
+    nav.login()
+    # start the nebari server (defined by `instance_type` in Navigator class)
+    nav.start_server()
+    # reset the jupyterlab workspace to ensure we're starting with only the
+    # Launcher screen open, and we're in the root directory.
+    nav.reset_workspace()
+    # instantiate our test application
+    test_app = RunNotebook(navigator=nav)
+    # Write the sample notebook on the nebari instance
+    notebook_filepath_in_repo = (
+        "tests_e2e/playwright/test_data/test_notebook_output.ipynb"
+    )
+    notebook_filepath_on_nebari = "test_notebook_output.ipynb"
+    with open(notebook_filepath_in_repo, "r") as notebook:
+        test_app.nav.write_file(
+            filepath=notebook_filepath_on_nebari, content=notebook.read()
+        )
+    # run a sample notebook
+    test_app.run_notebook(
+        path="nebari/tests_e2e/playwright/test_data/test_notebook_output.ipynb",
+        expected_output_text="success: 6",
+        conda_env="conda-env-default-py",
+    )
+    # close out playwright and its associated browser handles
+    nav.teardown()
+```
+
+## Writing Playwright tests
+
+In general most of the testing happens through `locators` which is Playwright's
+way of connecting a python object to the HTML element on the page.
+The Playwright API has several mechanisms for getting a locator for an item on
+the page (`get_by_role`, `get_by_text`, `get_by_label`, `get_by_placeholder`,
+etc).
+
+```python
+button = self.page.get_by_role("button", name="Sign in with Keycloak")
+```
+
+Once you have a handle on a locator, you can interact with it in different ways,
+depending on the type of object. For example, clicking
+a button:
+
+```python
+button.click()
+```
+
+Occasionally you'll need to wait for things to load on the screen. We can
+either wait for the page to finish loading:
+
+```python
+self.page.wait_for_load_state("networkidle")
+```
+
+or we can wait for something specific to happen with the locator itself:
+
+```python
+button.wait_for(timeout=3000, state="attached")
+```
+
+Note that waiting for the page to finish loading may be deceptive inside of
+Jupyterlab since things may need to load _inside_ the page, not necessarily
+causing network traffic - or causing several bursts network traffic, which
+would incorrectly pass the `wait_for_load_state` after the first burst.
+
+Playwright has a built-in auto-wait feature which waits for a timeout period
+for some actionable items. See https://playwright.dev/docs/actionability .
+
+### Workflow for creating new tests
+
+An example of running a new run notebook test might look like this:
+
+```python
+    import os
+
+    import dotenv
+    # load environment variables from .env file
+    dotenv.load_dotenv()
+    # instantiate the navigator class
+    nav = Navigator(
+        nebari_url="https://nebari.quansight.dev/",
+        username=os.environ["KEYCLOAK_USERNAME"],
+        password=os.environ["KEYCLOAK_PASSWORD"],
+        auth="password",
+        instance_name="small-instance",
+        headless=False,
+        slow_mo=100,
+    )
+    # go through login sequence (defined by `auth` method in Navigator class)
+    nav.login()
+    # start the nebari server (defined by `instance_type` in Navigator class)
+    nav.start_server()
+    # reset the jupyterlab workspace to ensure we're starting with only the
+    # Launcher screen open, and we're in the root directory.
+    nav.reset_workspace()
+    # instantiate our test application
+    test_app = RunNotebook(navigator=nav)
+    # Write the sample notebook on the nebari instance
+    notebook_filepath_in_repo = (
+        "tests_e2e/playwright/test_data/test_notebook_output.ipynb"
+    )
+    notebook_filepath_on_nebari = "test_notebook_output.ipynb"
+    with open(notebook_filepath_in_repo, "r") as notebook:
+        test_app.nav.write_file(
+            filepath=notebook_filepath_on_nebari, content=notebook.read()
+        )
+    # run a sample notebook
+    test_app.run_notebook(
+        path="nebari/tests_e2e/playwright/test_data/test_notebook_output.ipynb",
+        expected_output_text="success: 6",
+        conda_env="conda-env-default-py",
+    )
+    # close out playwright and its associated browser handles
+    nav.teardown()
+```

tests_e2e/playwright/conftest.py

@@ -0,0 +1,59 @@
+import logging
+import os
+from pathlib import Path
+
+import dotenv
+import pytest
+from navigator import Navigator
+
+logger = logging.getLogger()
+
+
+@pytest.fixture(scope="session")
+def _navigator_session(browser_name, pytestconfig):
+    """Set up a navigator instance, login with username/password, start
+    a server. Teardown when session is complete.
+    Do not use this for individual tests, use `navigator` fixture
+    for tests."""
+    dotenv.load_dotenv()
+    # try/except added here in attempt to reach teardown after error in
+    # order to close the browser context which will save the video so I debug
+    # the error.
+    try:
+        nav = Navigator(
+            nebari_url=os.environ["NEBARI_FULL_URL"],
+            username=os.environ["KEYCLOAK_USERNAME"],
+            password=os.environ["KEYCLOAK_PASSWORD"],
+            headless=not pytestconfig.getoption("--headed"),
+            slow_mo=pytestconfig.getoption("--slowmo"),
+            browser=browser_name,
+            auth="password",
+            instance_name="small-instance",  # small-instance included by default
+            video_dir="videos/",
+        )
+    except Exception as e:
+        logger.debug(e)
+        raise
+
+    try:
+        nav.login_password()
+        nav.start_server()
+        yield nav
+    except Exception as e:
+        logger.debug(e)
+        raise
+    finally:
+        nav.teardown()
+
+
+@pytest.fixture(scope="function")
+def navigator(_navigator_session):
+    """High level navigator instance with a reset workspace."""
+    _navigator_session.reset_workspace()
+    yield _navigator_session
+
+
+@pytest.fixture(scope="session")
+def test_data_root():
+    here = Path(__file__).parent
+    return here / "test_data"