Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore error when axe core fails to inject into blank iframes #26590

Closed
wants to merge 2 commits into from

Conversation

talldan
Copy link
Contributor

@talldan talldan commented Oct 30, 2020

Description

Previously #26527, #26535

Axe core tests fail intermittently:

Template Part › Template part placeholder › Should insert template part when preview is selected

    expect(jest.fn()).not.toHaveErrored(expected)

    Expected mock function not to be called but it was called with:
    ["Failed to inject axe-core into frame (about:blank)"],["Failed to inject axe-core into frame (about:blank)"]

This occurs when axe attempts to inject scripts into iframes created by the react-resize-aware utility in the useResizeObserver hook, mostly when tests briefly interact with block previews.

This PR adds the error message to our list of exceptions for things that will fail tests.

Axe itself won't fail tests when this injection fails, so it seems we're being stricter than Axe:
https://github.com/dequelabs/axe-core-npm/blob/1f06cb67e2bbbf1922fd9d2e6bcee74f092cfa1c/packages/puppeteer/src/axePuppeteer.ts#L67-L69

How has this been tested?

CI tests should regularly pass.

@talldan talldan added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Oct 30, 2020
@talldan talldan self-assigned this Oct 30, 2020
@talldan talldan changed the title Ignore when axe core fails to inject into blank iframes Ignore error when axe core fails to inject into blank iframes Oct 30, 2020
@github-actions
Copy link

github-actions bot commented Oct 30, 2020

Size Change: +208 B (0%)

Total Size: 1.21 MB

Filename Size Change
build/block-editor/index.js 130 kB +207 B (0%)
build/block-library/index.js 146 kB +20 B (0%)
build/components/index.js 172 kB +30 B (0%)
build/components/style-rtl.css 15.2 kB +2 B (0%)
build/components/style.css 15.2 kB +3 B (0%)
build/compose/index.js 9.81 kB -1 B
build/data/index.js 8.77 kB -2 B (0%)
build/edit-site/index.js 22.1 kB -50 B (0%)
build/editor/index.js 43.1 kB -1 B
build/element/index.js 4.65 kB -1 B
build/list-reusable-blocks/index.js 3.11 kB +1 B
build/media-utils/index.js 5.34 kB +1 B
build/nux/index.js 3.42 kB +1 B
build/rich-text/index.js 13.2 kB -1 B
build/server-side-render/index.js 2.77 kB -1 B
ℹ️ View Unchanged
Filename Size Change
build/a11y/index.js 1.14 kB 0 B
build/annotations/index.js 3.78 kB 0 B
build/api-fetch/index.js 3.45 kB 0 B
build/autop/index.js 2.84 kB 0 B
build/blob/index.js 665 B 0 B
build/block-directory/index.js 8.72 kB 0 B
build/block-directory/style-rtl.css 943 B 0 B
build/block-directory/style.css 942 B 0 B
build/block-editor/style-rtl.css 11.1 kB 0 B
build/block-editor/style.css 11.1 kB 0 B
build/block-library/editor-rtl.css 8.97 kB 0 B
build/block-library/editor.css 8.97 kB 0 B
build/block-library/style-rtl.css 7.83 kB 0 B
build/block-library/style.css 7.84 kB 0 B
build/block-library/theme-rtl.css 837 B 0 B
build/block-library/theme.css 838 B 0 B
build/block-serialization-default-parser/index.js 1.88 kB 0 B
build/block-serialization-spec-parser/index.js 3.1 kB 0 B
build/blocks/index.js 48.1 kB 0 B
build/core-data/index.js 12.3 kB 0 B
build/data-controls/index.js 772 B 0 B
build/date/index.js 31.8 kB 0 B
build/deprecated/index.js 768 B 0 B
build/dom-ready/index.js 571 B 0 B
build/dom/index.js 4.46 kB 0 B
build/edit-navigation/index.js 11.2 kB 0 B
build/edit-navigation/style-rtl.css 881 B 0 B
build/edit-navigation/style.css 885 B 0 B
build/edit-post/index.js 306 kB 0 B
build/edit-post/style-rtl.css 6.4 kB 0 B
build/edit-post/style.css 6.39 kB 0 B
build/edit-site/style-rtl.css 3.88 kB 0 B
build/edit-site/style.css 3.88 kB 0 B
build/edit-widgets/index.js 26.4 kB 0 B
build/edit-widgets/style-rtl.css 3.12 kB 0 B
build/edit-widgets/style.css 3.12 kB 0 B
build/editor/editor-styles-rtl.css 480 B 0 B
build/editor/editor-styles.css 482 B 0 B
build/editor/style-rtl.css 3.85 kB 0 B
build/editor/style.css 3.85 kB 0 B
build/escape-html/index.js 735 B 0 B
build/format-library/index.js 7.7 kB 0 B
build/format-library/style-rtl.css 547 B 0 B
build/format-library/style.css 548 B 0 B
build/hooks/index.js 2.13 kB 0 B
build/html-entities/index.js 623 B 0 B
build/i18n/index.js 3.57 kB 0 B
build/is-shallow-equal/index.js 712 B 0 B
build/keyboard-shortcuts/index.js 2.52 kB 0 B
build/keycodes/index.js 1.94 kB 0 B
build/list-reusable-blocks/style-rtl.css 476 B 0 B
build/list-reusable-blocks/style.css 476 B 0 B
build/notices/index.js 1.79 kB 0 B
build/nux/style-rtl.css 671 B 0 B
build/nux/style.css 668 B 0 B
build/plugins/index.js 2.56 kB 0 B
build/primitives/index.js 1.43 kB 0 B
build/priority-queue/index.js 791 B 0 B
build/redux-routine/index.js 2.85 kB 0 B
build/reusable-blocks/index.js 3.06 kB 0 B
build/shortcode/index.js 1.69 kB 0 B
build/token-list/index.js 1.27 kB 0 B
build/url/index.js 4.06 kB 0 B
build/viewport/index.js 1.84 kB 0 B
build/warning/index.js 1.14 kB 0 B
build/wordcount/index.js 1.22 kB 0 B

compressed-size-action

Copy link
Contributor

@Addison-Stavlo Addison-Stavlo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow! A bit disappointing and surprising that the approach in #26535 didn't work out.

Axe itself won't fail tests when this injection fails, so it seems we're being stricter than Axe

Interesting! It seems to make sense to not fail our tests as a result of this axe error. Especially since all other approaches seem to not work out. Il re-run the tests a few times and see if its consistent.

@Addison-Stavlo
Copy link
Contributor

🤦‍♀️ - It seems to still fail on the error... 🤷‍♀️

Expected mock function not to be called but it was called with:
    ["Failed to inject axe-core into frame (about:blank)"],["Failed to inject axe-core into frame (about:blank)"]

@talldan
Copy link
Contributor Author

talldan commented Nov 2, 2020

@Addison-Stavlo 🤔 I'm guessing text is an array of arrays or something given the output.

@talldan
Copy link
Contributor Author

talldan commented Nov 2, 2020

Nope, that didn't work either.

I think I have to call it quits there. I've spent quite a bit of my time looking into these issues. It's about time that others on the project stepped up and helped.

@talldan talldan closed this Nov 2, 2020
@talldan talldan deleted the ignore/axe-core-injection-errors branch November 2, 2020 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants