Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky widgets e2e tests by waiting for the snackbar correctly #33317

Merged
merged 1 commit into from
Jul 15, 2021

Conversation

kevin940726
Copy link
Member

@kevin940726 kevin940726 commented Jul 9, 2021

Description

Related to #33275, but probably won't fix that entirely.

I think one reason behind the flaky e2e tests is that we are waiting for the wrong elements when the screen is being saved. We are waiting for elements with text of Widgets saved., expecting it to be the snackbar element. But it turns out that there will always be a a11y element that contains that text.

<div id="a11y-speak-polite" class="a11y-speak-region" style="position: absolute;margin: -1px;padding: 0;height: 1px;width: 1px;overflow: hidden;clip: rect(1px, 1px, 1px, 1px);-webkit-clip-path: inset(50%);clip-path: inset(50%);border: 0;word-wrap: normal !important;" aria-live="polite" aria-relevant="additions text" aria-atomic="true">Widgets saved.&nbsp;</div>

Hence, the waiting resolves immediately, the widgets has not yet been saved. This is not always failing because we wait for another 500ms after saving, which is the root of the flakiness.

The solution is to correctly wait for the snackbar elements, and dismiss them once it's shown. We have to also enable reduced motion mode to disable the snackbar transition. (I think we should enable it for all of our e2e tests though, WDYT?)

How has this been tested?

E2E tests should pass.

Types of changes

Bug fix

Checklist:

  • My code is tested.
  • My code follows the WordPress code style.
  • My code follows the accessibility standards.
  • I've tested my changes with keyboard and screen readers.
  • My code has proper inline documentation.
  • I've included developer documentation if appropriate.
  • I've updated all React Native files affected by any refactorings/renamings in this PR (please manually search all *.native.js files for terms that need renaming or removal).

@kevin940726 kevin940726 added [Type] Bug An existing feature does not function as intended [Feature] Widgets Screen The block-based screen that replaced widgets.php. labels Jul 9, 2021
@github-actions
Copy link

github-actions bot commented Jul 9, 2021

Size Change: 0 B

Total Size: 1.07 MB

ℹ️ View Unchanged
Filename Size
build/a11y/index.min.js 1.12 kB
build/admin-manifest/index.min.js 1.41 kB
build/annotations/index.min.js 2.93 kB
build/api-fetch/index.min.js 2.44 kB
build/autop/index.min.js 2.28 kB
build/blob/index.min.js 673 B
build/block-directory/index.min.js 6.61 kB
build/block-directory/style-rtl.css 1.02 kB
build/block-directory/style.css 1.02 kB
build/block-editor/index.min.js 126 kB
build/block-editor/style-rtl.css 14 kB
build/block-editor/style.css 14 kB
build/block-library/blocks/archives/editor-rtl.css 61 B
build/block-library/blocks/archives/editor.css 60 B
build/block-library/blocks/archives/style-rtl.css 65 B
build/block-library/blocks/archives/style.css 65 B
build/block-library/blocks/audio/editor-rtl.css 58 B
build/block-library/blocks/audio/editor.css 58 B
build/block-library/blocks/audio/style-rtl.css 112 B
build/block-library/blocks/audio/style.css 112 B
build/block-library/blocks/audio/theme-rtl.css 125 B
build/block-library/blocks/audio/theme.css 125 B
build/block-library/blocks/block/editor-rtl.css 161 B
build/block-library/blocks/block/editor.css 161 B
build/block-library/blocks/button/editor-rtl.css 475 B
build/block-library/blocks/button/editor.css 474 B
build/block-library/blocks/button/style-rtl.css 603 B
build/block-library/blocks/button/style.css 602 B
build/block-library/blocks/buttons/editor-rtl.css 315 B
build/block-library/blocks/buttons/editor.css 315 B
build/block-library/blocks/buttons/style-rtl.css 375 B
build/block-library/blocks/buttons/style.css 375 B
build/block-library/blocks/calendar/style-rtl.css 208 B
build/block-library/blocks/calendar/style.css 208 B
build/block-library/blocks/categories/editor-rtl.css 84 B
build/block-library/blocks/categories/editor.css 83 B
build/block-library/blocks/categories/style-rtl.css 79 B
build/block-library/blocks/categories/style.css 79 B
build/block-library/blocks/code/style-rtl.css 90 B
build/block-library/blocks/code/style.css 90 B
build/block-library/blocks/code/theme-rtl.css 131 B
build/block-library/blocks/code/theme.css 131 B
build/block-library/blocks/columns/editor-rtl.css 190 B
build/block-library/blocks/columns/editor.css 190 B
build/block-library/blocks/columns/style-rtl.css 475 B
build/block-library/blocks/columns/style.css 476 B
build/block-library/blocks/cover/editor-rtl.css 670 B
build/block-library/blocks/cover/editor.css 670 B
build/block-library/blocks/cover/style-rtl.css 1.22 kB
build/block-library/blocks/cover/style.css 1.23 kB
build/block-library/blocks/embed/editor-rtl.css 486 B
build/block-library/blocks/embed/editor.css 486 B
build/block-library/blocks/embed/style-rtl.css 401 B
build/block-library/blocks/embed/style.css 400 B
build/block-library/blocks/embed/theme-rtl.css 124 B
build/block-library/blocks/embed/theme.css 124 B
build/block-library/blocks/file/editor-rtl.css 301 B
build/block-library/blocks/file/editor.css 300 B
build/block-library/blocks/file/style-rtl.css 255 B
build/block-library/blocks/file/style.css 255 B
build/block-library/blocks/file/view.min.js 780 B
build/block-library/blocks/freeform/editor-rtl.css 2.44 kB
build/block-library/blocks/freeform/editor.css 2.44 kB
build/block-library/blocks/gallery/editor-rtl.css 704 B
build/block-library/blocks/gallery/editor.css 705 B
build/block-library/blocks/gallery/style-rtl.css 1.06 kB
build/block-library/blocks/gallery/style.css 1.06 kB
build/block-library/blocks/gallery/theme-rtl.css 122 B
build/block-library/blocks/gallery/theme.css 122 B
build/block-library/blocks/group/editor-rtl.css 160 B
build/block-library/blocks/group/editor.css 160 B
build/block-library/blocks/group/style-rtl.css 57 B
build/block-library/blocks/group/style.css 57 B
build/block-library/blocks/group/theme-rtl.css 93 B
build/block-library/blocks/group/theme.css 93 B
build/block-library/blocks/heading/editor-rtl.css 152 B
build/block-library/blocks/heading/editor.css 152 B
build/block-library/blocks/heading/style-rtl.css 76 B
build/block-library/blocks/heading/style.css 76 B
build/block-library/blocks/home-link/style-rtl.css 259 B
build/block-library/blocks/home-link/style.css 259 B
build/block-library/blocks/html/editor-rtl.css 281 B
build/block-library/blocks/html/editor.css 281 B
build/block-library/blocks/image/editor-rtl.css 729 B
build/block-library/blocks/image/editor.css 727 B
build/block-library/blocks/image/style-rtl.css 481 B
build/block-library/blocks/image/style.css 485 B
build/block-library/blocks/image/theme-rtl.css 124 B
build/block-library/blocks/image/theme.css 124 B
build/block-library/blocks/latest-comments/style-rtl.css 286 B
build/block-library/blocks/latest-comments/style.css 286 B
build/block-library/blocks/latest-posts/editor-rtl.css 137 B
build/block-library/blocks/latest-posts/editor.css 137 B
build/block-library/blocks/latest-posts/style-rtl.css 526 B
build/block-library/blocks/latest-posts/style.css 524 B
build/block-library/blocks/list/style-rtl.css 63 B
build/block-library/blocks/list/style.css 63 B
build/block-library/blocks/media-text/editor-rtl.css 263 B
build/block-library/blocks/media-text/editor.css 264 B
build/block-library/blocks/media-text/style-rtl.css 492 B
build/block-library/blocks/media-text/style.css 489 B
build/block-library/blocks/more/editor-rtl.css 434 B
build/block-library/blocks/more/editor.css 434 B
build/block-library/blocks/navigation-link/editor-rtl.css 474 B
build/block-library/blocks/navigation-link/editor.css 473 B
build/block-library/blocks/navigation-link/style-rtl.css 94 B
build/block-library/blocks/navigation-link/style.css 94 B
build/block-library/blocks/navigation/editor-rtl.css 1.69 kB
build/block-library/blocks/navigation/editor.css 1.69 kB
build/block-library/blocks/navigation/style-rtl.css 1.65 kB
build/block-library/blocks/navigation/style.css 1.66 kB
build/block-library/blocks/navigation/view.min.js 2.87 kB
build/block-library/blocks/nextpage/editor-rtl.css 395 B
build/block-library/blocks/nextpage/editor.css 395 B
build/block-library/blocks/page-list/editor-rtl.css 310 B
build/block-library/blocks/page-list/editor.css 311 B
build/block-library/blocks/page-list/style-rtl.css 240 B
build/block-library/blocks/page-list/style.css 240 B
build/block-library/blocks/paragraph/editor-rtl.css 157 B
build/block-library/blocks/paragraph/editor.css 157 B
build/block-library/blocks/paragraph/style-rtl.css 247 B
build/block-library/blocks/paragraph/style.css 248 B
build/block-library/blocks/post-author/editor-rtl.css 209 B
build/block-library/blocks/post-author/editor.css 209 B
build/block-library/blocks/post-author/style-rtl.css 183 B
build/block-library/blocks/post-author/style.css 184 B
build/block-library/blocks/post-comments-form/style-rtl.css 140 B
build/block-library/blocks/post-comments-form/style.css 140 B
build/block-library/blocks/post-comments/style-rtl.css 360 B
build/block-library/blocks/post-comments/style.css 359 B
build/block-library/blocks/post-content/editor-rtl.css 139 B
build/block-library/blocks/post-content/editor.css 139 B
build/block-library/blocks/post-excerpt/editor-rtl.css 73 B
build/block-library/blocks/post-excerpt/editor.css 73 B
build/block-library/blocks/post-excerpt/style-rtl.css 69 B
build/block-library/blocks/post-excerpt/style.css 69 B
build/block-library/blocks/post-featured-image/editor-rtl.css 338 B
build/block-library/blocks/post-featured-image/editor.css 338 B
build/block-library/blocks/post-featured-image/style-rtl.css 141 B
build/block-library/blocks/post-featured-image/style.css 141 B
build/block-library/blocks/post-template/editor-rtl.css 100 B
build/block-library/blocks/post-template/editor.css 99 B
build/block-library/blocks/post-template/style-rtl.css 379 B
build/block-library/blocks/post-template/style.css 380 B
build/block-library/blocks/post-terms/style-rtl.css 73 B
build/block-library/blocks/post-terms/style.css 73 B
build/block-library/blocks/post-title/style-rtl.css 60 B
build/block-library/blocks/post-title/style.css 60 B
build/block-library/blocks/preformatted/style-rtl.css 103 B
build/block-library/blocks/preformatted/style.css 103 B
build/block-library/blocks/pullquote/editor-rtl.css 183 B
build/block-library/blocks/pullquote/editor.css 183 B
build/block-library/blocks/pullquote/style-rtl.css 318 B
build/block-library/blocks/pullquote/style.css 318 B
build/block-library/blocks/pullquote/theme-rtl.css 169 B
build/block-library/blocks/pullquote/theme.css 169 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B
build/block-library/blocks/query-pagination/editor-rtl.css 270 B
build/block-library/blocks/query-pagination/editor.css 262 B
build/block-library/blocks/query-pagination/style-rtl.css 168 B
build/block-library/blocks/query-pagination/style.css 168 B
build/block-library/blocks/query-title/editor-rtl.css 86 B
build/block-library/blocks/query-title/editor.css 86 B
build/block-library/blocks/query/editor-rtl.css 131 B
build/block-library/blocks/query/editor.css 132 B
build/block-library/blocks/quote/style-rtl.css 169 B
build/block-library/blocks/quote/style.css 169 B
build/block-library/blocks/quote/theme-rtl.css 221 B
build/block-library/blocks/quote/theme.css 221 B
build/block-library/blocks/rss/editor-rtl.css 201 B
build/block-library/blocks/rss/editor.css 202 B
build/block-library/blocks/rss/style-rtl.css 290 B
build/block-library/blocks/rss/style.css 290 B
build/block-library/blocks/search/editor-rtl.css 211 B
build/block-library/blocks/search/editor.css 211 B
build/block-library/blocks/search/style-rtl.css 359 B
build/block-library/blocks/search/style.css 362 B
build/block-library/blocks/search/theme-rtl.css 64 B
build/block-library/blocks/search/theme.css 64 B
build/block-library/blocks/separator/editor-rtl.css 99 B
build/block-library/blocks/separator/editor.css 99 B
build/block-library/blocks/separator/style-rtl.css 251 B
build/block-library/blocks/separator/style.css 251 B
build/block-library/blocks/separator/theme-rtl.css 172 B
build/block-library/blocks/separator/theme.css 172 B
build/block-library/blocks/shortcode/editor-rtl.css 476 B
build/block-library/blocks/shortcode/editor.css 476 B
build/block-library/blocks/site-logo/editor-rtl.css 465 B
build/block-library/blocks/site-logo/editor.css 465 B
build/block-library/blocks/site-logo/style-rtl.css 154 B
build/block-library/blocks/site-logo/style.css 154 B
build/block-library/blocks/site-tagline/editor-rtl.css 87 B
build/block-library/blocks/site-tagline/editor.css 87 B
build/block-library/blocks/site-title/editor-rtl.css 85 B
build/block-library/blocks/site-title/editor.css 85 B
build/block-library/blocks/social-link/editor-rtl.css 164 B
build/block-library/blocks/social-link/editor.css 165 B
build/block-library/blocks/social-links/editor-rtl.css 800 B
build/block-library/blocks/social-links/editor.css 799 B
build/block-library/blocks/social-links/style-rtl.css 1.34 kB
build/block-library/blocks/social-links/style.css 1.34 kB
build/block-library/blocks/spacer/editor-rtl.css 308 B
build/block-library/blocks/spacer/editor.css 308 B
build/block-library/blocks/spacer/style-rtl.css 48 B
build/block-library/blocks/spacer/style.css 48 B
build/block-library/blocks/table/editor-rtl.css 478 B
build/block-library/blocks/table/editor.css 478 B
build/block-library/blocks/table/style-rtl.css 480 B
build/block-library/blocks/table/style.css 480 B
build/block-library/blocks/table/theme-rtl.css 188 B
build/block-library/blocks/table/theme.css 188 B
build/block-library/blocks/tag-cloud/style-rtl.css 146 B
build/block-library/blocks/tag-cloud/style.css 146 B
build/block-library/blocks/template-part/editor-rtl.css 551 B
build/block-library/blocks/template-part/editor.css 550 B
build/block-library/blocks/template-part/theme-rtl.css 101 B
build/block-library/blocks/template-part/theme.css 101 B
build/block-library/blocks/term-description/editor-rtl.css 90 B
build/block-library/blocks/term-description/editor.css 90 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B
build/block-library/blocks/text-columns/editor.css 95 B
build/block-library/blocks/text-columns/style-rtl.css 166 B
build/block-library/blocks/text-columns/style.css 166 B
build/block-library/blocks/verse/style-rtl.css 87 B
build/block-library/blocks/verse/style.css 87 B
build/block-library/blocks/video/editor-rtl.css 569 B
build/block-library/blocks/video/editor.css 570 B
build/block-library/blocks/video/style-rtl.css 173 B
build/block-library/blocks/video/style.css 173 B
build/block-library/blocks/video/theme-rtl.css 124 B
build/block-library/blocks/video/theme.css 124 B
build/block-library/common-rtl.css 1.29 kB
build/block-library/common.css 1.29 kB
build/block-library/editor-rtl.css 9.81 kB
build/block-library/editor.css 9.81 kB
build/block-library/index.min.js 147 kB
build/block-library/reset-rtl.css 514 B
build/block-library/reset.css 515 B
build/block-library/style-rtl.css 10.3 kB
build/block-library/style.css 10.3 kB
build/block-library/theme-rtl.css 692 B
build/block-library/theme.css 693 B
build/block-serialization-default-parser/index.min.js 1.3 kB
build/block-serialization-spec-parser/index.min.js 3.06 kB
build/blocks/index.min.js 47.2 kB
build/components/index.min.js 196 kB
build/components/style-rtl.css 16.1 kB
build/components/style.css 16.1 kB
build/compose/index.min.js 10.2 kB
build/core-data/index.min.js 12.4 kB
build/customize-widgets/index.min.js 10.3 kB
build/customize-widgets/style-rtl.css 1.48 kB
build/customize-widgets/style.css 1.48 kB
build/data-controls/index.min.js 830 B
build/data/index.min.js 7.22 kB
build/date/index.min.js 31.8 kB
build/deprecated/index.min.js 738 B
build/dom-ready/index.min.js 576 B
build/dom/index.min.js 4.78 kB
build/edit-navigation/index.min.js 13.9 kB
build/edit-navigation/style-rtl.css 3.12 kB
build/edit-navigation/style.css 3.12 kB
build/edit-post/classic-rtl.css 483 B
build/edit-post/classic.css 483 B
build/edit-post/index.min.js 59.5 kB
build/edit-post/style-rtl.css 7.25 kB
build/edit-post/style.css 7.24 kB
build/edit-site/index.min.js 26 kB
build/edit-site/style-rtl.css 5.04 kB
build/edit-site/style.css 5.03 kB
build/edit-widgets/index.min.js 16.2 kB
build/edit-widgets/style-rtl.css 3.88 kB
build/edit-widgets/style.css 3.89 kB
build/editor/index.min.js 38.7 kB
build/editor/style-rtl.css 3.88 kB
build/editor/style.css 3.88 kB
build/element/index.min.js 3.44 kB
build/escape-html/index.min.js 739 B
build/format-library/index.min.js 5.71 kB
build/format-library/style-rtl.css 668 B
build/format-library/style.css 669 B
build/hooks/index.min.js 1.76 kB
build/html-entities/index.min.js 628 B
build/i18n/index.min.js 3.73 kB
build/is-shallow-equal/index.min.js 709 B
build/keyboard-shortcuts/index.min.js 1.74 kB
build/keycodes/index.min.js 1.43 kB
build/list-reusable-blocks/index.min.js 2.07 kB
build/list-reusable-blocks/style-rtl.css 842 B
build/list-reusable-blocks/style.css 842 B
build/media-utils/index.min.js 3.08 kB
build/notices/index.min.js 1.07 kB
build/nux/index.min.js 2.31 kB
build/nux/style-rtl.css 745 B
build/nux/style.css 742 B
build/plugins/index.min.js 1.99 kB
build/primitives/index.min.js 1.06 kB
build/priority-queue/index.min.js 790 B
build/react-i18n/index.min.js 924 B
build/redux-routine/index.min.js 2.82 kB
build/reusable-blocks/index.min.js 2.56 kB
build/reusable-blocks/style-rtl.css 256 B
build/reusable-blocks/style.css 256 B
build/rich-text/index.min.js 10.8 kB
build/server-side-render/index.min.js 1.64 kB
build/shortcode/index.min.js 1.68 kB
build/token-list/index.min.js 847 B
build/url/index.min.js 1.95 kB
build/viewport/index.min.js 1.28 kB
build/warning/index.min.js 1.16 kB
build/widgets/index.min.js 6.48 kB
build/widgets/style-rtl.css 1.04 kB
build/widgets/style.css 1.05 kB
build/wordcount/index.min.js 1.24 kB

compressed-size-action

@kevin940726 kevin940726 force-pushed the fix/editing-widgets-flaky-e2e branch 3 times, most recently from 24e84bd to be0d41c Compare July 10, 2021 05:03
'gutenberg-test-plugin-disables-the-css-animations'
);
// Reduced motion is needed to immediately show and dismiss the snackbars.
await page.emulateMediaFeatures( [
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to enable it globally too.

@kevin940726 kevin940726 marked this pull request as ready for review July 12, 2021 08:49
Copy link
Contributor

@talldan talldan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, though I feel the snapshot thing should be in a separate PR.

I don't mind if you merge it though 😄

Comment on lines 5 to 8
### New Features

- Bail out tests if a prior snapshot failed. Fixed a bug which failing snapshots won't trigger screenshots [#33317](https://github.com/WordPress/gutenberg/pull/33317).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems unrelated, should it be in a separate PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just being lazy 😅, I can do that in another PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it here: #33448

Comment on lines +906 to +909
// Close the snackbar.
const savedSnackbar = await find( savedSnackbarQuery );
await savedSnackbar.click();
await expect( savedSnackbarQuery ).not.toBeFound();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it need to be dismissed for the test to work? If so, I think that's ok, but otherwise it should probably be in a separate test rather than something that happens any time widgets are saved.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They should be dismissed, or else there will multiple snackbars on the page, and we won't know when the widgets are saved. Which was the reason it was flaky before 😅

Copy link
Contributor

@talldan talldan Jul 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. If only the snackbar disappearing speed was proportional to the speed the e2e tests move. 😄

Could consider adding a comment like 'avoids multiple snackbars being present on the screen'.

@kevin940726 kevin940726 force-pushed the fix/editing-widgets-flaky-e2e branch from 6faf1a9 to 5e941fe Compare July 15, 2021 08:18
@kevin940726 kevin940726 merged commit e3e8416 into trunk Jul 15, 2021
@kevin940726 kevin940726 deleted the fix/editing-widgets-flaky-e2e branch July 15, 2021 09:02
@github-actions github-actions bot added this to the Gutenberg 11.2 milestone Jul 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Feature] Widgets Screen The block-based screen that replaced widgets.php. [Type] Bug An existing feature does not function as intended
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants