Add UI/visual regression tests (#105)

* Seed ui-tests from template.

- Populate ui-tests yarn.lock
- Add `ui-tests` to `.eslintignore`

* Implement minimal integration tests, and windowing test

* Clean utils function

* Update changelog

.eslintignore

@@ -2,4 +2,5 @@ node_modules
 dist
 coverage
 **/*.d.ts
-jupyterlab_execute_time
+jupyterlab_execute_time
+ui-tests

.github/workflows/build.yml

@@ -76,3 +76,68 @@ jobs:
         jupyter labextension list
         jupyter labextension list 2>&1 | grep -ie "jupyterlab-execute-time.*OK"
         python -m jupyterlab.browser_check --no-browser-test
+
+  integration-tests:
+    name: Integration tests
+    needs: build
+    runs-on: ubuntu-latest
+
+    env:
+      PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/pw-browsers
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v3
+
+    - name: Install node
+      uses: actions/setup-node@v3
+      with:
+       node-version: '18.x'
+
+    - name: Install Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+        architecture: 'x64'
+
+    - name: Download extension package
+      uses: actions/download-artifact@v3
+      with:
+        name: extension-artifacts
+
+    - name: Install the extension
+      run: |
+        set -eux
+        python -m pip install "jupyterlab>=4.0.0,<5" jupyterlab_execute_time*.whl
+
+    - name: Install dependencies
+      working-directory: ui-tests
+      env:
+        YARN_ENABLE_IMMUTABLE_INSTALLS: 0
+        PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
+      run: jlpm install
+
+    - name: Set up browser cache
+      uses: actions/cache@v3
+      with:
+        path: |
+          ${{ github.workspace }}/pw-browsers
+        key: ${{ runner.os }}-${{ hashFiles('ui-tests/yarn.lock') }}
+
+    - name: Install browser
+      run: jlpm playwright install chromium
+      working-directory: ui-tests
+
+    - name: Execute integration tests
+      working-directory: ui-tests
+      run: |
+        jlpm playwright test
+
+    - name: Upload Playwright Test report
+      if: always()
+      uses: actions/upload-artifact@v3
+      with:
+        name: jupyterlab_execute_time-playwright-tests
+        path: |
+          ui-tests/test-results
+          ui-tests/playwright-report

.github/workflows/update-integration-tests.yml

@@ -0,0 +1,50 @@
+name: Update Playwright Snapshots
+
+on:
+  issue_comment:
+    types: [created, edited]
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  update-snapshots:
+    if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, 'please update snapshots') }}
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: React to the triggering comment
+        run: |
+          gh api repos/${{ github.repository }}/issues/comments/${{ github.event.comment.id }}/reactions --raw-field 'content=+1'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Checkout the branch from the PR that triggered the job
+        run: gh pr checkout ${{ github.event.issue.number }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Base Setup
+        uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1
+
+      - name: Install dependencies
+        run: python -m pip install -U "jupyterlab>=4.0.0,<5"
+
+      - name: Install extension
+        run: |
+          set -eux
+          jlpm
+          python -m pip install .
+
+      - uses: jupyterlab/maintainer-tools/.github/actions/update-snapshots@v1
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          # Playwright knows how to start JupyterLab server
+          start_server_script: 'null'
+          test_folder: ui-tests

CHANGELOG.md

@@ -9,10 +9,11 @@
 ### Fixed
 
 - Font size of the execute time indicator [#103](https://github.com/deshaw/jupyterlab-execute-time/pull/103)
+- Fix node attachment issue with windowed notebook [#102](https://github.com/deshaw/jupyterlab-execute-time/pull/102)
 
-### Fixed
+### Maintenance
 
-- Fix node attachment issue with windowed notebook [#102](https://github.com/deshaw/jupyterlab-execute-time/pull/102)
+- Added UI/visual regression tests [#105](https://github.com/deshaw/jupyterlab-execute-time/pull/105)
 
 ## [3.0.1](https://github.com/deshaw/jupyterlab-execute-time/compare/v3.0.0...v3.0.1) (2023-08-02)
 

ui-tests/README.md

@@ -0,0 +1,167 @@
+# Integration Testing
+
+This folder contains the integration tests of the extension.
+
+They are defined using [Playwright](https://playwright.dev/docs/intro) test runner
+and [Galata](https://github.com/jupyterlab/jupyterlab/tree/main/galata) helper.
+
+The Playwright configuration is defined in [playwright.config.js](./playwright.config.js).
+
+The JupyterLab server configuration to use for the integration test is defined
+in [jupyter_server_test_config.py](./jupyter_server_test_config.py).
+
+The default configuration will produce video for failing tests and an HTML report.
+
+> There is a new experimental UI mode that you may fall in love with; see [that video](https://www.youtube.com/watch?v=jF0yA-JLQW0).
+
+## Run the tests
+
+> All commands are assumed to be executed from the root directory
+
+To run the tests, you need to:
+
+1. Compile the extension:
+
+```sh
+jlpm install
+jlpm build:prod
+```
+
+> Check the extension is installed in JupyterLab.
+
+2. Install test dependencies (needed only once):
+
+```sh
+cd ./ui-tests
+jlpm install
+jlpm playwright install
+cd ..
+```
+
+3. Execute the [Playwright](https://playwright.dev/docs/intro) tests:
+
+```sh
+cd ./ui-tests
+jlpm playwright test
+```
+
+Test results will be shown in the terminal. In case of any test failures, the test report
+will be opened in your browser at the end of the tests execution; see
+[Playwright documentation](https://playwright.dev/docs/test-reporters#html-reporter)
+for configuring that behavior.
+
+## Update the tests snapshots
+
+> All commands are assumed to be executed from the root directory
+
+If you are comparing snapshots to validate your tests, you may need to update
+the reference snapshots stored in the repository. To do that, you need to:
+
+1. Compile the extension:
+
+```sh
+jlpm install
+jlpm build:prod
+```
+
+> Check the extension is installed in JupyterLab.
+
+2. Install test dependencies (needed only once):
+
+```sh
+cd ./ui-tests
+jlpm install
+jlpm playwright install
+cd ..
+```
+
+3. Execute the [Playwright](https://playwright.dev/docs/intro) command:
+
+```sh
+cd ./ui-tests
+jlpm playwright test -u
+```
+
+> Some discrepancy may occurs between the snapshots generated on your computer and
+> the one generated on the CI. To ease updating the snapshots on a PR, you can
+> type `please update playwright snapshots` to trigger the update by a bot on the CI.
+> Once the bot has computed new snapshots, it will commit them to the PR branch.
+
+## Create tests
+
+> All commands are assumed to be executed from the root directory
+
+To create tests, the easiest way is to use the code generator tool of playwright:
+
+1. Compile the extension:
+
+```sh
+jlpm install
+jlpm build:prod
+```
+
+> Check the extension is installed in JupyterLab.
+
+2. Install test dependencies (needed only once):
+
+```sh
+cd ./ui-tests
+jlpm install
+jlpm playwright install
+cd ..
+```
+
+3. Start the server:
+
+```sh
+cd ./ui-tests
+jlpm start
+```
+
+4. Execute the [Playwright code generator](https://playwright.dev/docs/codegen) in **another terminal**:
+
+```sh
+cd ./ui-tests
+jlpm playwright codegen localhost:8888
+```
+
+## Debug tests
+
+> All commands are assumed to be executed from the root directory
+
+To debug tests, a good way is to use the inspector tool of playwright:
+
+1. Compile the extension:
+
+```sh
+jlpm install
+jlpm build:prod
+```
+
+> Check the extension is installed in JupyterLab.
+
+2. Install test dependencies (needed only once):
+
+```sh
+cd ./ui-tests
+jlpm install
+jlpm playwright install
+cd ..
+```
+
+3. Execute the Playwright tests in [debug mode](https://playwright.dev/docs/debug):
+
+```sh
+cd ./ui-tests
+jlpm playwright test --debug
+```
+
+## Upgrade Playwright and the browsers
+
+To update the web browser versions, you must update the package `@playwright/test`:
+
+```sh
+cd ./ui-tests
+jlpm up "@playwright/test"
+jlpm playwright install
+```

ui-tests/jupyter_server_test_config.py

@@ -0,0 +1,12 @@
+"""Server configuration for integration tests.
+
+!! Never use this configuration in production because it
+opens the server to the world and provide access to JupyterLab
+JavaScript objects through the global window variable.
+"""
+from jupyterlab.galata import configure_jupyter_server
+
+configure_jupyter_server(c)
+
+# Uncomment to set server log level to debug level
+# c.ServerApp.log_level = "DEBUG"