Fix flaky playwright tests #4000
Annotations
19 errors, 25 warnings, and 6 notices
[Chrome] › room_options/marked_unread.spec.ts:22:9 › Mark as Unread › should mark a room as unread:
playwright/e2e/room_options/marked_unread.spec.ts#L51
1) [Chrome] › room_options/marked_unread.spec.ts:22:9 › Mark as Unread › should mark a room as unread
Error: expect(locator).toBeVisible()
Locator: getByLabel('The mark unread test room Unread messages.')
Expected: visible
Received: <element(s) not found>
Call log:
- expect.toBeVisible with timeout 5000ms
- waiting for getByLabel('The mark unread test room Unread messages.')
49 | await page.getByRole("menuitem", { name: "Mark as unread" }).click();
50 |
> 51 | expect(page.getByLabel(TEST_ROOM_NAME + " Unread messages.")).toBeVisible();
| ^
52 | });
53 | });
54 |
at /home/runner/work/element-web/element-web/playwright/e2e/room_options/marked_unread.spec.ts:51:71
|
[Chrome] › share-dialog/share-dialog.spec.ts:19:9 › Share dialog › should share a room @screenshot:
playwright/element-web-test.ts#L321
2) [Chrome] › share-dialog/share-dialog.spec.ts:19:9 › Share dialog › should share a room @screenshot
Error: expect.toHaveScreenshot(share-dialog-room.png): Test ended.
Call log:
- expect.toHaveScreenshot(share-dialog-room.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for getByRole('dialog', { name: 'Share room' })
at ../element-web-test.ts:321
319 |
320 | const screenshotName = sanitizeFilePathBeforeExtension(name);
> 321 | await baseExpect(receiver).toHaveScreenshot(screenshotName, options);
| ^
322 |
323 | await style.evaluate((tag) => tag.remove());
324 |
at Object.toMatchScreenshot (/home/runner/work/element-web/element-web/playwright/element-web-test.ts:321:9)
|
[Chrome] › share-dialog/share-dialog.spec.ts:32:9 › Share dialog › should share a room member @screenshot:
playwright/element-web-test.ts#L321
3) [Chrome] › share-dialog/share-dialog.spec.ts:32:9 › Share dialog › should share a room member @screenshot
Error: expect.toHaveScreenshot(share-dialog-user.png): Test ended.
Call log:
- expect.toHaveScreenshot(share-dialog-user.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for getByRole('dialog', { name: 'Share User' })
at ../element-web-test.ts:321
319 |
320 | const screenshotName = sanitizeFilePathBeforeExtension(name);
> 321 | await baseExpect(receiver).toHaveScreenshot(screenshotName, options);
| ^
322 |
323 | await style.evaluate((tag) => tag.remove());
324 |
at Object.toMatchScreenshot (/home/runner/work/element-web/element-web/playwright/element-web-test.ts:321:9)
|
[Chrome] › share-dialog/share-dialog.spec.ts:49:9 › Share dialog › should share an event @screenshot:
playwright/element-web-test.ts#L321
4) [Chrome] › share-dialog/share-dialog.spec.ts:49:9 › Share dialog › should share an event @screenshot
Error: expect.toHaveScreenshot(share-dialog-event.png): Test ended.
Call log:
- expect.toHaveScreenshot(share-dialog-event.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for getByRole('dialog', { name: 'Share Room Message' })
- locator resolved to <div role="dialog" class="mx_ShareDialog" data-focus-lock-disabled="false" aria-describedby="mx_Dialog_content" aria-labelledby="mx_BaseDialog_title">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
at ../element-web-test.ts:321
319 |
320 | const screenshotName = sanitizeFilePathBeforeExtension(name);
> 321 | await baseExpect(receiver).toHaveScreenshot(screenshotName, options);
| ^
322 |
323 | await style.evaluate((tag) => tag.remove());
324 |
at Object.toMatchScreenshot (/home/runner/work/element-web/element-web/playwright/element-web-test.ts:321:9)
|
[Chrome] › sliding-sync/sliding-sync.spec.ts:215:5 › Sliding Sync › should update user settings promptly:
playwright/e2e/sliding-sync/sliding-sync.spec.ts#L86
5) [Chrome] › sliding-sync/sliding-sync.spec.ts:215:5 › Sliding Sync › should update user settings promptly
Test timeout of 30000ms exceeded while running "beforeEach" hook.
84 |
85 | // Load the user fixture for all tests
> 86 | test.beforeEach(({ user }) => {});
| ^
87 |
88 | test("should render the Rooms list in reverse chronological order by default and allowing sorting A-Z", async ({
89 | page,
at /home/runner/work/element-web/element-web/playwright/e2e/sliding-sync/sliding-sync.spec.ts:86:10
|
[Chrome] › sliding-sync/sliding-sync.spec.ts:215:5 › Sliding Sync › should update user settings promptly:
playwright/element-web-test.ts#L191
5) [Chrome] › sliding-sync/sliding-sync.spec.ts:215:5 › Sliding Sync › should update user settings promptly
Error: page.waitForSelector: Test timeout of 30000ms exceeded.
Call log:
- waiting for locator('.mx_MatrixChat') to be visible
at ../element-web-test.ts:191
189 | user: async ({ pageWithCredentials: page, credentials }, use) => {
190 | await page.goto("/");
> 191 | await page.waitForSelector(".mx_MatrixChat", { timeout: 30000 });
| ^
192 | await use(credentials);
193 | },
194 |
at Object.user (/home/runner/work/element-web/element-web/playwright/element-web-test.ts:191:20)
|
[Chrome] › spotlight/spotlight.spec.ts:362:5 › Spotlight › should be able to navigate results via keyboard:
playwright/e2e/spotlight/spotlight.spec.ts#L369
1) [Chrome] › spotlight/spotlight.spec.ts:362:5 › Spotlight › should be able to navigate results via keyboard
Error: Timed out 5000ms waiting for expect(locator).toHaveCount(expected)
Locator: locator('[role=dialog][aria-label="Search Dialog"]').locator('.mx_SpotlightDialog_section.mx_SpotlightDialog_results .mx_SpotlightDialog_option')
Expected: 2
Received: 31
Call log:
- expect.toHaveCount with timeout 5000ms
- waiting for locator('[role=dialog][aria-label="Search Dialog"]').locator('.mx_SpotlightDialog_section.mx_SpotlightDialog_results .mx_SpotlightDialog_option')
2 × locator resolved to 1 element
- unexpected value "1"
7 × locator resolved to 31 elements
- unexpected value "31"
367 |
368 | let resultLocator = spotlight.results;
> 369 | await expect(resultLocator).toHaveCount(2);
| ^
370 | await expect(resultLocator.first()).toHaveAttribute("aria-selected", "true");
371 | await expect(resultLocator.last()).toHaveAttribute("aria-selected", "false");
372 |
at /home/runner/work/element-web/element-web/playwright/e2e/spotlight/spotlight.spec.ts:369:37
|
[Chrome] › audio-player/audio-player.spec.ts:168:9 › Audio player › should be correctly rendered - dark theme @no-firefox @no-webkit @screenshot:
playwright/e2e/audio-player/audio-player.spec.ts#L125
1) [Chrome] › audio-player/audio-player.spec.ts:168:9 › Audio player › should be correctly rendered - dark theme @no-firefox @no-webkit @screenshot
Test timeout of 30000ms exceeded while running "beforeEach" hook.
123 | };
124 |
> 125 | test.beforeEach(async ({ page, app, user }) => {
| ^
126 | await app.client.createRoom({ name: "Test Room" });
127 | await app.viewRoomByName("Test Room");
128 |
at /home/runner/work/element-web/element-web/playwright/e2e/audio-player/audio-player.spec.ts:125:10
|
[Chrome] › audio-player/audio-player.spec.ts:168:9 › Audio player › should be correctly rendered - dark theme @no-firefox @no-webkit @screenshot:
playwright/pages/ElementAppPage.ts#L96
1) [Chrome] › audio-player/audio-player.spec.ts:168:9 › Audio player › should be correctly rendered - dark theme @no-firefox @no-webkit @screenshot
Error: locator.click: Test timeout of 30000ms exceeded.
Call log:
- waiting for getByRole('tree', { name: 'Rooms' }).locator('[title="Test Room"],[aria-label="Test Room"]').first()
at ../pages/ElementAppPage.ts:96
94 | .locator(`[title="${name}"],[aria-label="${name}"]`)
95 | .first()
> 96 | .click();
| ^
97 | }
98 |
99 | public async viewRoomById(roomId: string): Promise<void> {
at ElementAppPage.viewRoomByName (/home/runner/work/element-web/element-web/playwright/pages/ElementAppPage.ts:96:14)
at /home/runner/work/element-web/element-web/playwright/e2e/audio-player/audio-player.spec.ts:127:19
|
[Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere:
playwright/testcontainers/utils.ts#L73
1) [Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere
Error: Request to http://localhost:44657/_matrix/client/_matrix/client/v3/room_keys/version failed with status 404: {"errcode":"M_UNRECOGNIZED","error":"Unrecognized request"}
at ../testcontainers/utils.ts:73
71 |
72 | if (!res.ok()) {
> 73 | throw new Error(
| ^
74 | `Request to ${url} failed with status ${res.status()}: ${JSON.stringify(await res.json())}`,
75 | );
76 | }
at TestClientServerAPI.request (/home/runner/work/element-web/element-web/playwright/testcontainers/utils.ts:73:19)
at /home/runner/work/element-web/element-web/playwright/e2e/crypto/backups-mas.spec.ts:75:28
|
[Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere:
playwright/testcontainers/utils.ts#L73
1) [Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: Request to http://localhost:43789/_matrix/client/_matrix/client/v3/room_keys/version failed with status 404: {"errcode":"M_UNRECOGNIZED","error":"Unrecognized request"}
at ../testcontainers/utils.ts:73
71 |
72 | if (!res.ok()) {
> 73 | throw new Error(
| ^
74 | `Request to ${url} failed with status ${res.status()}: ${JSON.stringify(await res.json())}`,
75 | );
76 | }
at TestClientServerAPI.request (/home/runner/work/element-web/element-web/playwright/testcontainers/utils.ts:73:19)
at /home/runner/work/element-web/element-web/playwright/e2e/crypto/backups-mas.spec.ts:75:28
|
[Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere:
playwright/testcontainers/utils.ts#L73
1) [Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: Request to http://localhost:41645/_matrix/client/_matrix/client/v3/room_keys/version failed with status 404: {"errcode":"M_UNRECOGNIZED","error":"Unrecognized request"}
at ../testcontainers/utils.ts:73
71 |
72 | if (!res.ok()) {
> 73 | throw new Error(
| ^
74 | `Request to ${url} failed with status ${res.status()}: ${JSON.stringify(await res.json())}`,
75 | );
76 | }
at TestClientServerAPI.request (/home/runner/work/element-web/element-web/playwright/testcontainers/utils.ts:73:19)
at /home/runner/work/element-web/element-web/playwright/e2e/crypto/backups-mas.spec.ts:75:28
|
[Chrome] › login/soft_logout_oauth.spec.ts:40:9 › Soft logout with SSO user › shows the soft-logout page when a request fails:
playwright/e2e/login/utils.ts#L38
2) [Chrome] › login/soft_logout_oauth.spec.ts:40:9 › Soft logout with SSO user › shows the soft-logout page when a request fails, and allows a re-login
Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
Locator: getByRole('heading', { name: 'Create your account' })
Expected: visible
Received: <element(s) not found>
Call log:
- expect.toBeVisible with timeout 5000ms
- waiting for getByRole('heading', { name: 'Create your account' })
at login/utils.ts:38
36 |
37 | // Synapse prompts us to pick a user ID
> 38 | await expect(page.getByRole("heading", { name: "Create your account" })).toBeVisible();
| ^
39 | await page.getByRole("textbox", { name: "Username (required)" }).fill(`alice_${testInfo.testId}`);
40 |
41 | // wait for username validation to start, and complete
at doTokenRegistration (/home/runner/work/element-web/element-web/playwright/e2e/login/utils.ts:38:78)
at Object.user (/home/runner/work/element-web/element-web/playwright/e2e/login/soft_logout_oauth.spec.ts:30:26)
|
[Chrome] › right-panel/memberlist.spec.ts:41:9 › Memberlist › Renders correctly @screenshot:
playwright/e2e/right-panel/memberlist.spec.ts#L1
3) [Chrome] › right-panel/memberlist.spec.ts:41:9 › Memberlist › Renders correctly @screenshot ───
Test timeout of 60000ms exceeded.
|
[Chrome] › right-panel/memberlist.spec.ts:41:9 › Memberlist › Renders correctly @screenshot:
playwright/pages/ElementAppPage.ts#L96
3) [Chrome] › right-panel/memberlist.spec.ts:41:9 › Memberlist › Renders correctly @screenshot ───
Error: locator.click: Test timeout of 60000ms exceeded.
Call log:
- waiting for getByRole('tree', { name: 'Rooms' }).locator('[title="Test room"],[aria-label="Test room"]').first()
at ../pages/ElementAppPage.ts:96
94 | .locator(`[title="${name}"],[aria-label="${name}"]`)
95 | .first()
> 96 | .click();
| ^
97 | }
98 |
99 | public async viewRoomById(roomId: string): Promise<void> {
at ElementAppPage.viewRoomByName (/home/runner/work/element-web/element-web/playwright/pages/ElementAppPage.ts:96:14)
at /home/runner/work/element-web/element-web/playwright/e2e/right-panel/memberlist.spec.ts:42:19
|
[Chrome] › widgets/layout.spec.ts:73:9 › Widget Layout › should be set properly @screenshot:
playwright/e2e/widgets/layout.spec.ts#L33
4) [Chrome] › widgets/layout.spec.ts:73:9 › Widget Layout › should be set properly @screenshot ───
Test timeout of 30000ms exceeded while running "beforeEach" hook.
31 | let roomId: string;
32 | let widgetUrl: string;
> 33 | test.beforeEach(async ({ webserver, app, user }) => {
| ^
34 | widgetUrl = webserver.start(WIDGET_HTML);
35 |
36 | roomId = await app.client.createRoom({ name: ROOM_NAME });
at /home/runner/work/element-web/element-web/playwright/e2e/widgets/layout.spec.ts:33:10
|
[Chrome] › widgets/layout.spec.ts:73:9 › Widget Layout › should be set properly @screenshot:
playwright/pages/ElementAppPage.ts#L96
4) [Chrome] › widgets/layout.spec.ts:73:9 › Widget Layout › should be set properly @screenshot ───
Error: locator.click: Test timeout of 30000ms exceeded.
Call log:
- waiting for getByRole('tree', { name: 'Rooms' }).locator('[title="Test Room"],[aria-label="Test Room"]').first()
at ../pages/ElementAppPage.ts:96
94 | .locator(`[title="${name}"],[aria-label="${name}"]`)
95 | .first()
> 96 | .click();
| ^
97 | }
98 |
99 | public async viewRoomById(roomId: string): Promise<void> {
at ElementAppPage.viewRoomByName (/home/runner/work/element-web/element-web/playwright/pages/ElementAppPage.ts:96:14)
at /home/runner/work/element-web/element-web/playwright/e2e/widgets/layout.spec.ts:70:19
|
Run Tests [Chrome] 6/6
Process completed with exit code 1.
|
end-to-end-tests
Process completed with exit code 1.
|
Slow Test:
[Chrome] › pinned-messages/pinned-messages.spec.ts#L1
[Chrome] › pinned-messages/pinned-messages.spec.ts took 59.3s
|
Slow Test:
[Chrome] › right-panel/right-panel.spec.ts#L1
[Chrome] › right-panel/right-panel.spec.ts took 24.9s
|
Slow Test:
[Chrome] › one-to-one-chat/one-to-one-chat.spec.ts#L1
[Chrome] › one-to-one-chat/one-to-one-chat.spec.ts took 24.9s
|
Slow Test:
[Chrome] › right-panel/file-panel.spec.ts#L1
[Chrome] › right-panel/file-panel.spec.ts took 19.8s
|
Slow Test:
[Chrome] › polls/polls.spec.ts#L1
[Chrome] › polls/polls.spec.ts took 19.4s
|
Slow Test:
[Chrome] › messages/messages.spec.ts#L1
[Chrome] › messages/messages.spec.ts took 58.4s
|
Slow Test:
[Chrome] › crypto/invisible-crypto.spec.ts#L1
[Chrome] › crypto/invisible-crypto.spec.ts took 27.0s
|
Slow Test:
[Chrome] › knock/knock-into-room.spec.ts#L1
[Chrome] › knock/knock-into-room.spec.ts took 19.3s
|
Slow Test:
[Chrome] › integration-manager/kick.spec.ts#L1
[Chrome] › integration-manager/kick.spec.ts took 18.9s
|
Slow Test:
[Chrome] › editing/editing.spec.ts#L1
[Chrome] › editing/editing.spec.ts took 16.4s
|
Slow Test:
[Chrome] › room/room-header.spec.ts#L1
[Chrome] › room/room-header.spec.ts took 30.3s
|
Slow Test:
[Chrome] › settings/account-user-settings-tab.spec.ts#L1
[Chrome] › settings/account-user-settings-tab.spec.ts took 17.3s
|
Slow Test:
[Chrome] › timeline/timeline.spec.ts#L1
[Chrome] › timeline/timeline.spec.ts took 1.3m
|
Slow Test:
[Chrome] › spaces/spaces.spec.ts#L1
[Chrome] › spaces/spaces.spec.ts took 53.8s
|
Slow Test:
[Chrome] › spaces/threads-activity-centre/threadsActivityCentre.spec.ts#L1
[Chrome] › spaces/threads-activity-centre/threadsActivityCentre.spec.ts took 48.0s
|
Slow Test:
[Chrome] › threads/threads.spec.ts#L1
[Chrome] › threads/threads.spec.ts took 19.1s
|
Slow Test:
[Chrome] › crypto/event-shields.spec.ts#L1
[Chrome] › crypto/event-shields.spec.ts took 54.4s
|
Slow Test:
[Chrome] › crypto/decryption-failure-messages.spec.ts#L1
[Chrome] › crypto/decryption-failure-messages.spec.ts took 50.5s
|
Slow Test:
[Chrome] › composer/RTE.spec.ts#L1
[Chrome] › composer/RTE.spec.ts took 36.7s
|
Slow Test:
[Chrome] › crypto/crypto.spec.ts#L1
[Chrome] › crypto/crypto.spec.ts took 34.7s
|
Slow Test:
[Chrome] › accessibility/keyboard-navigation.spec.ts#L1
[Chrome] › accessibility/keyboard-navigation.spec.ts took 28.9s
|
Slow Test:
[Chrome] › login/login-consent.spec.ts#L1
[Chrome] › login/login-consent.spec.ts took 48.1s
|
Slow Test:
[Chrome] › app-loading/guest-registration.spec.ts#L1
[Chrome] › app-loading/guest-registration.spec.ts took 25.1s
|
Slow Test:
[Chrome] › user-onboarding/user-onboarding-new.spec.ts#L1
[Chrome] › user-onboarding/user-onboarding-new.spec.ts took 24.3s
|
Slow Test:
[Chrome] › register/register.spec.ts#L1
[Chrome] › register/register.spec.ts took 16.6s
|
🎭 Playwright Run Summary
1 skipped
35 passed (3.0m)
|
🎭 Playwright Run Summary
1 skipped
65 passed (4.6m)
|
🎭 Playwright Run Summary
5 flaky
[Chrome] › room_options/marked_unread.spec.ts:22:9 › Mark as Unread › should mark a room as unread
[Chrome] › share-dialog/share-dialog.spec.ts:19:9 › Share dialog › should share a room @screenshot
[Chrome] › share-dialog/share-dialog.spec.ts:32:9 › Share dialog › should share a room member @screenshot
[Chrome] › share-dialog/share-dialog.spec.ts:49:9 › Share dialog › should share an event @screenshot
[Chrome] › sliding-sync/sliding-sync.spec.ts:215:5 › Sliding Sync › should update user settings promptly
1 skipped
44 passed (4.7m)
|
🎭 Playwright Run Summary
1 flaky
[Chrome] › spotlight/spotlight.spec.ts:362:5 › Spotlight › should be able to navigate results via keyboard
3 skipped
56 passed (5.2m)
|
🎭 Playwright Run Summary
1 flaky
[Chrome] › audio-player/audio-player.spec.ts:168:9 › Audio player › should be correctly rendered - dark theme @no-firefox @no-webkit @screenshot
55 passed (5.9m)
|
🎭 Playwright Run Summary
1 failed
[Chrome] › crypto/backups-mas.spec.ts:48:9 › Key backup reset from elsewhere › Key backup is disabled when reset from elsewhere
3 flaky
[Chrome] › login/soft_logout_oauth.spec.ts:40:9 › Soft logout with SSO user › shows the soft-logout page when a request fails, and allows a re-login
[Chrome] › right-panel/memberlist.spec.ts:41:9 › Memberlist › Renders correctly @screenshot ────
[Chrome] › widgets/layout.spec.ts:73:9 › Widget Layout › should be set properly @screenshot ────
40 passed (7.7m)
|
Artifacts
Produced during runtime
Name | Size | |
---|---|---|
all-blob-reports-Chrome-1
|
4.27 MB |
|
all-blob-reports-Chrome-2
|
577 KB |
|
all-blob-reports-Chrome-3
|
122 KB |
|
all-blob-reports-Chrome-4
|
15.9 MB |
|
all-blob-reports-Chrome-5
|
5.25 MB |
|
all-blob-reports-Chrome-6
|
17.9 MB |
|
html-report
|
41.5 MB |
|
webapp
|
28.7 MB |
|