Test: E2E tests for new Component Testing features #29826

JReinhold · 2024-12-05T20:19:06Z

Telescopes on to #29808

Closes #29745

What I did

Added E2E tests for Focused Tests, Coverage, and a combination of both
Modified DOM a11y slightly
Modified existing E2E tests slightly
Added tests for preview-head.html functionality. They don't need to be E2E tested, they can be tested by just running Vitest. So I added that to CI too, skipping the tests that fail on purpose.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

Add or update documentation reflecting your changes
If you are deprecating/removing a feature, make sure to update
MIGRATION.MD

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts
Make sure this PR contains one of the labels below:
Available labels
- bug: Internal changes that fixes incorrect behavior.
- maintenance: User-facing maintenance tasks.
- dependencies: Upgrading (sometimes downgrading) dependencies.
- build: Internal-facing build tooling & test updates. Will not show up in release changelog.
- cleanup: Minor cleanup style change. Will not show up in release changelog.
- documentation: Documentation only changes. Will not show up in release changelog.
- feature request: Introducing a new feature.
- BREAKING CHANGE: Changes that break compatibility in some way with current major version.
- other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	77.7 MB	77.7 MB	0 B	0.63	0%
initSize	136 MB	136 MB	-1 B	0.42	0%
diffSize	58.4 MB	58.4 MB	-1 B	0.42	0%
buildSize	7.24 MB	7.24 MB	-1 B	0.23	0%
buildSbAddonsSize	1.88 MB	1.88 MB	0 B	0.23	0%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.86 MB	1.86 MB	-1 B	1.02	0%
buildSbPreviewSize	0 B	0 B	0 B	-	-
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	3.93 MB	3.93 MB	-1 B	0.23	0%
buildPreviewSize	3.3 MB	3.3 MB	0 B	-0.27	0%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	26.8s	7.2s	-19s -653ms	-0.85	-272.3%
generateTime	19.7s	20.7s	970ms	0.4	4.7%
initTime	14.8s	13.8s	-1s -59ms	-0.33	-7.7%
buildTime	8.3s	8.6s	264ms	-0.77	3.1%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	5.1s	4.9s	-234ms	-0.3	-4.7%
devManagerResponsive	3.9s	3.7s	-178ms	-0.07	-4.7%
devManagerHeaderVisible	884ms	723ms	-161ms	1.18	-22.3%
devManagerIndexVisible	973ms	751ms	-222ms	0.98	-29.6%
devStoryVisibleUncached	1.9s	2s	120ms	1.52	🔺5.8%
devStoryVisible	970ms	752ms	-218ms	0.96	-29%
devAutodocsVisible	679ms	717ms	38ms	2.69	🔺5.3%
devMDXVisible	718ms	625ms	-93ms	0.95	-14.9%
buildManagerHeaderVisible	633ms	677ms	44ms	0.87	6.5%
buildManagerIndexVisible	753ms	772ms	19ms	0.78	2.5%
buildStoryVisible	589ms	641ms	52ms	1.07	8.1%
buildAutodocsVisible	485ms	509ms	24ms	0.82	4.7%
buildMDXVisible	499ms	498ms	-1ms	0.4	-0.2%

Greptile Summary

Here's my concise summary of the PR changes:

Added end-to-end tests and infrastructure for Storybook's component testing features, focusing on Vitest integration and coverage reporting.

Added E2E tests in portable-stories-kitchen-sink/react/e2e-tests/component-testing.spec.ts for focused tests and coverage reporting
Modified code/addons/test/src/node/vitest-manager.ts to handle coverage options and test filtering
Added coverage reporter in code/addons/test/src/node/coverage-reporter.ts with watermark support
Added accessibility improvements to test UI components in code/addons/test/src/components/TestProviderRender.tsx
Added new test stories and configurations in portable-stories-kitchen-sink/react/ for validating component testing features

…e-tests-for-testing-improvements

…ybookjs/storybook into 29745-e2e-tests-for-testing-improvements

…ts-for-testing-improvements

…:storybookjs/storybook into 29745-e2e-tests-for-testing-improvements

test-storybooks/portable-stories-kitchen-sink/react/package.json

code/lib/test/package.json

nx-cloud · 2024-12-06T12:33:44Z

☁️ Nx Cloud Report

CI is running/has finished running commands for commit a19b0f5. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

✅ Successfully ran 1 target

nx run-many -t build --parallel=3

Sent with 💌 from NxCloud.

…ts-for-testing-improvements

…ybookjs/storybook into 29745-e2e-tests-for-testing-improvements

…:storybookjs/storybook into 29745-e2e-tests-for-testing-improvements

JReinhold · 2024-12-06T13:55:43Z

test-storybooks/portable-stories-kitchen-sink/react/e2e-tests/component-testing.spec.ts

+    await expect(page.locator('#storybook-explorer-menu').getByRole('status', { name: 'Test status: error' })).toHaveCount(2);
+  });
+
+  test("should run focused tests without coverage, even when enabled", async ({


this test is slightly excessive.

It:

enables coverage

do a focus test

assert that coverage is not collected

do a full test

assert that coverage is collected

I'm not sure if we should keep it or not, you could also argue that it's largely covered by the unit test here:

https://github.com/storybookjs/storybook/pull/29749/files#diff-18fe411ce6b83d11273334c241ee323db41b0c412c66ce04ceef562e5e39a899R172-R201

WDYT @yannbf ?

If it's too taxing or flake-prone, I'd say remove it. Seems to be well tested elsewhere, but this E2E test does involve more elements in a real scenario. However I think these elements are tested in the other tests above

…ybookjs/storybook into 29745-e2e-tests-for-testing-improvements

greptile-apps

_{12 file(s) reviewed, 10 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

code/addons/test/src/components/TestProviderRender.tsx

test-storybooks/portable-stories-kitchen-sink/react/.storybook/main.ts

test-storybooks/portable-stories-kitchen-sink/react/e2e-tests/component-testing.spec.ts

test-storybooks/portable-stories-kitchen-sink/react/package.json

test-storybooks/portable-stories-kitchen-sink/react/stories/AddonTest.stories.tsx

test-storybooks/portable-stories-kitchen-sink/react/stories/OtherComponent.stories.tsx

…e-tests-for-testing-improvements

…storybook into 29745-e2e-tests-for-testing-improvements

…e-tests-for-testing-improvements

storybook-pr-benchmarking · 2024-12-12T14:13:40Z

Package Benchmarks

^{Commit: a19b0f5, ran on 17 December 2024 at 12:45:07 UTC}

No significant changes detected, all good. 👏

JReinhold · 2024-12-12T20:26:58Z

test-storybooks/portable-stories-kitchen-sink/react/package.json

@@ -12,8 +12,9 @@
    "playwright-ct": "playwright test -c playwright-ct.config.ts",
    "playwright-e2e": "playwright test -c playwright-e2e.config.ts",
    "preview": "vite preview",
-    "storybook": "storybook dev -p 6006",
-    "vitest": "echo 'not running'"
+    "storybook": "NODE_OPTIONS=\"--preserve-symlinks --preserve-symlinks-main\" storybook dev -p 6006",


this speeds up the story loading of the kitchen sink, at least on my machine.

code/addons/test/src/components/TestProviderRender.tsx

Co-authored-by: Norbert de Langen <ndelangen@me.com>

yannbf · 2024-12-17T10:31:01Z

.circleci/config.yml

@@ -757,6 +757,10 @@ jobs:
          name: Run E2E tests
          command: yarn playwright-e2e
          working_directory: test-storybooks/portable-stories-kitchen-sink/react
+      - run:
+          name: Run Vitest


Why do we need this? If you want the vitest script to be tested as part of CI, just move the contents of vitest-skip-fail-on-purpose and move it to vitest, which will be automatically run as part of this CI step:
https://github.com/storybookjs/storybook/blob/29745-e2e-tests-for-testing-improvements/.circleci/config.yml#L794

ah yeah that's much better, it's to test these scenarios: https://github.com/storybookjs/storybook/pull/29826/files#diff-0550ef2a360c5242f25677cd964d3afce1c0412497a76eded7d03c0838941611R68-R92

Another alternative would be to move those stories into the UI Storybook instead, no strong opinion.

JReinhold and others added 15 commits December 3, 2024 20:39

ugprade vitest dependencies to 2.1.6

82e0c62

improve type-safety in coverage reporter

1373b36

add coverage E2E test, a11y labels to coverage UI

e111a2d

Merge branch 'next' of github.com:storybookjs/storybook into 29745-e2…

291a70a

…e-tests-for-testing-improvements

Merge branch 'norbert/vitest-support-preview-html' of github.com:stor…

1dd20b7

…ybookjs/storybook into 29745-e2e-tests-for-testing-improvements

commit stuff - not done

c91f604

add an experimentalLoadStorybook api

eb1a526

cleanup

41a0c7a

fix coverage+watch test

6f48ea8

fix tests

269f1a2

Merge branch 'norbert/vitest-support-preview-html' into 29745-e2e-tes…

0c76a0e

…ts-for-testing-improvements

fix story types

bc52357

Merge branch '29745-e2e-tests-for-testing-improvements' of github.com…

b4154f5

…:storybookjs/storybook into 29745-e2e-tests-for-testing-improvements

better comments

87fa9ba

add non-failing-vitest to ci

23f4967

yannbf reviewed Dec 6, 2024

View reviewed changes

test-storybooks/portable-stories-kitchen-sink/react/package.json Outdated Show resolved Hide resolved

yannbf reviewed Dec 6, 2024

View reviewed changes

code/lib/test/package.json Outdated Show resolved Hide resolved

JReinhold added 3 commits December 6, 2024 12:51

fix vitest/ui dep

d1441aa

simplify coverage+focus test

b5b65b0

revert vitest upgrades

1b86bf3

JReinhold and others added 4 commits December 6, 2024 14:36

Merge branch 'norbert/vitest-support-preview-html' into 29745-e2e-tes…

a5034d8

…ts-for-testing-improvements

Merge branch 'norbert/vitest-support-preview-html' of github.com:stor…

ab5be7f

…ybookjs/storybook into 29745-e2e-tests-for-testing-improvements

Merge branch '29745-e2e-tests-for-testing-improvements' of github.com…

5a9ba43

…:storybookjs/storybook into 29745-e2e-tests-for-testing-improvements

revert yarn.lock changes.

b8c9393

JReinhold commented Dec 6, 2024

View reviewed changes

JReinhold added 4 commits December 9, 2024 11:24

add staticDirs test

f47acc5

add test for vitefinal

0a5a607

Merge branch 'norbert/vitest-support-preview-html' of github.com:stor…

b4f4965

…ybookjs/storybook into 29745-e2e-tests-for-testing-improvements

Merge branch 'norbert/vitest-support-preview-html' of github.com:stor…

5813cd4

…ybookjs/storybook into 29745-e2e-tests-for-testing-improvements

JReinhold added build Internal-facing build tooling & test updates addon: test labels Dec 10, 2024

greptile-apps bot reviewed Dec 10, 2024

View reviewed changes

Base automatically changed from norbert/vitest-support-preview-html to next December 10, 2024 12:55

Merge branch 'next' of github.com:storybookjs/storybook into 29745-e2…

0367631

…e-tests-for-testing-improvements

JReinhold linked an issue Dec 10, 2024 that may be closed by this pull request

E2E tests for Testing improvements #29745

Closed

6 tasks

JReinhold requested a review from ndelangen December 10, 2024 13:00

JReinhold added 4 commits December 10, 2024 14:03

cleanup

11a25dc

Merge branch 'jeppe/fix-duplicate-include' of github.com:storybookjs/…

58afdd4

…storybook into 29745-e2e-tests-for-testing-improvements

Merge branch 'next' of github.com:storybookjs/storybook into 29745-e2…

e7a5eb2

…e-tests-for-testing-improvements

fix e2e tests 🤞

65f6411

JReinhold added 2 commits December 12, 2024 15:16

speedup kitchen sink storybook

0225d09

add fail-on-purpose tag to Button story

1765262

JReinhold commented Dec 12, 2024

View reviewed changes

JReinhold and others added 2 commits December 15, 2024 23:13

Merge branch 'next' into 29745-e2e-tests-for-testing-improvements

64d9692

Merge branch 'next' into 29745-e2e-tests-for-testing-improvements

8128b6f

ndelangen reviewed Dec 16, 2024

View reviewed changes

code/addons/test/src/components/TestProviderRender.tsx Outdated Show resolved Hide resolved

ndelangen reviewed Dec 16, 2024

View reviewed changes

code/addons/test/src/components/TestProviderRender.tsx Outdated Show resolved Hide resolved

ndelangen approved these changes Dec 16, 2024

View reviewed changes

JReinhold and others added 3 commits December 17, 2024 08:45

Update code/addons/test/src/components/TestProviderRender.tsx

041d195

Co-authored-by: Norbert de Langen <ndelangen@me.com>

Merge branch 'next' into 29745-e2e-tests-for-testing-improvements

a2dc9e1

fix format

1fca1f6

yannbf reviewed Dec 17, 2024

View reviewed changes

yannbf approved these changes Dec 17, 2024

View reviewed changes

reuse existing vitest script

a19b0f5

JReinhold merged commit 438393a into next Dec 17, 2024
56 of 58 checks passed

JReinhold deleted the 29745-e2e-tests-for-testing-improvements branch December 17, 2024 19:33

This was referenced Dec 17, 2024

Release: Prerelease 8.5.0-beta.2 #30085

Merged

Release: Prerelease 8.5.0-beta.2 #30097

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test: E2E tests for new Component Testing features #29826

Test: E2E tests for new Component Testing features #29826

JReinhold commented Dec 5, 2024 •

edited by storybook-pr-benchmarking bot

Loading

nx-cloud bot commented Dec 6, 2024 •

edited

Loading

JReinhold Dec 6, 2024

yannbf Dec 6, 2024

greptile-apps bot left a comment

storybook-pr-benchmarking bot commented Dec 12, 2024 •

edited

Loading

JReinhold Dec 12, 2024

yannbf Dec 17, 2024 •

edited

Loading

JReinhold Dec 17, 2024

Test: E2E tests for new Component Testing features #29826

Test: E2E tests for new Component Testing features #29826

Conversation

JReinhold commented Dec 5, 2024 • edited by storybook-pr-benchmarking bot Loading

What I did

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

🦋 Canary release

Greptile Summary

nx-cloud bot commented Dec 6, 2024 • edited Loading

☁️ Nx Cloud Report

JReinhold Dec 6, 2024

Choose a reason for hiding this comment

yannbf Dec 6, 2024

Choose a reason for hiding this comment

greptile-apps bot left a comment

Choose a reason for hiding this comment

storybook-pr-benchmarking bot commented Dec 12, 2024 • edited Loading

Package Benchmarks

JReinhold Dec 12, 2024

Choose a reason for hiding this comment

yannbf Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

JReinhold Dec 17, 2024

Choose a reason for hiding this comment

JReinhold commented Dec 5, 2024 •

edited by storybook-pr-benchmarking bot

Loading

nx-cloud bot commented Dec 6, 2024 •

edited

Loading

storybook-pr-benchmarking bot commented Dec 12, 2024 •

edited

Loading

yannbf Dec 17, 2024 •

edited

Loading