Add CI workflow that runs only changed E2E tests, and succeeds when no tests have changed

[tpaksu]

Context: p1737719556299039-slack-C01B8KNUYSW

Changes proposed in this Pull Request

Testing instructions

This PR adds a workflow that runs only the modified E2E tests in it, helping us see the errors faster (~5-10 minutes) without waiting for all tests to complete (Which takes around 35-40 minutes to complete now, before optimizations). Also updates Playwright to 1.50, since the --only-changed option was added in v1.46, and we were using v1.43.

---

• Run npm run changelog to add a changelog file, choose patch to leave it empty if the change is not significant. You can add multiple changelog files in one PR by running this command a few times.
• Covered with tests (or have a good reason not to test in description ☝️)
• Tested on mobile (or does not apply)

Post merge

• Link to testing instructions from release testing doc following these instructions : Add link here / 'QA Testing Not Applicable'
• Add or update critical flows and testing instructions for critical flows, if applicable.
• Add what's changed (description, screenshot, demo videos etc.) to the release announcement post, if applicable.

[alopezari]

Thanks for working on this @tpaksu !

I asked some questions related to the changes.

Also, are we going to consider running the actual E2E tests only if the only-changed tests passed (similar to how smoke tests work) or we would consider that a b-plan?

[alopezari]

The only-changed flag points to origin/develop, but the workflow will run when the PR targets develop or trunk. Should we keep 2 test:e2e-pw-ci-only-changed npm scripts, one for trunk and the other one for develop so that we could choose one or another depending on the branch targeted by the PR?

[alopezari]

The E2E tests - All workflow runs tests against different versions of WP and WC (7.7.0, latest and beta, for example). Would we want to consider this in this workflow? There will be times when tests will pass against latest but fail against beta, like with the Coming Soon mode introduced in the latest WC release.

[alopezari]

What would happen if E2E tests are run as usual and then someone adds a new review? Would the tests trigger again?

[tpaksu]

Yes, you're right, pull_request.ready_for_review action seems like the correct event that should trigger this.

Edit: It ran again when I added an answer on the main thread :D Confirmed this.

[tpaksu]

@alopezari thanks for the review, I was focused on another issue, so I couldn't put my eyes on this earlier.

Also, are we going to consider running the actual E2E tests only if the only-changed tests passed (similar to how smoke tests work) or we would consider that a b-plan?

I think this is a good candidate to be in this PR, but there's one thing that makes me have second thoughts of doing that way; when I changed something in the dependent file, it ran 114 tests out of 123, and it made me wonder, if it only picked the method that was changed to define the changed tests, or did it run all tests that its file imported that changed dependency file. If it's the second, then it will nearly be a double run, right? And if the E2E workflow took 40 minutes to complete, it would take 80 minutes for someone to be able to merge a PR 😱 That's why I'd prefer running a single workflow related to the PR's draft status.

I'll answer the other questions in their respective threads.

[alopezari]

I think this is a good candidate to be in this PR, but there's one thing that makes me have second thoughts of doing that way; when I changed something in the dependent file, it ran 114 tests out of 123, and it made me wonder, if it only picked the method that was changed to define the changed tests, or did it run all tests that its file imported that changed dependency file. If it's the second, then it will nearly be a double run, right? And if the E2E workflow took 40 minutes to complete, it would take 80 minutes for someone to be able to merge a PR 😱 That's why I'd prefer running a single workflow related to the PR's draft status.

Thanks for the explanation @tpaksu! I think I don't fully understand it though. With my suggestion, I meant:

1. Run E2E Playwright Tests - Pull Request - Only Changed / WC - latest always, no matter if it's draft or not.
2. If it passed, then run E2E Playwright Tests - Pull Request / WC - latest (pull_request). Otherwise, if it failed, don't run it.

As it is right now, it:

1. Runs E2E Playwright Tests - Pull Request - Only Changed / WC - latest if the PR is in draft, otherwise it runs E2E Playwright Tests - Pull Request / WC - latest (pull_request).
2. After we switch from draft to ready for review, it runs E2E Playwright Tests - Pull Request / WC - latest (pull_request_review).

What's the difference between both approaches that would make us double the E2E Playwright Tests - Pull Request / WC - latest (pull_request) run? Thanks!

[tpaksu]

With the suggested approach:

• Run E2E Playwright Tests - Pull Request - Only Changed / WC - latest always, no matter if it's draft or not.
• If it passed, then run E2E Playwright Tests - Pull Request / WC - latest (pull_request). Otherwise, if it failed, don't run it.

A change of a helper function may cause 100+ tests to run in the "only changed" workflow. Right? And once it passes, it'll start the next 100+ tests workflow. So it will take 40 minutes for the only changed workflow, and after that, another 40+ minutes for the full test workflow.

[alopezari]

A change of a helper function may cause 100+ tests to run in the "only changed" workflow. Right? And once it passes, it'll start the next 100+ tests workflow. So it will take 40 minutes for the only changed workflow, and after that, another 40+ minutes for the full test workflow.

Right, that might be problematic. However, the same thing would happen with the current scenario, right? It would take 40 mins for the only-changed workflow in draft mode + 40 with the full test run. The only difference I can see is that with the current approach we can control when we want to run the only-changed workflow by setting the PR in draft mode. Hmm a potential solution could be running the only-changed tests if there is any change in the tests directory.

However, the only reason for advocating for that approach is to provide a better developer experience when running the only-changed tests, as the process would be smoother than making sure everyone knows they can put the PR on draft to run the tests, etc. But if making this smoother gets tricky, I'm happy with the current approach as it is right now 👍

[tpaksu]

However, the same thing would happen with the current scenario, right? It would take 40 mins for the only-changed workflow in draft mode + 40 with the full test run.

Nope, we will only run the "only changed" ones when the PR is in draft mode, and when it's marked as "not draft", it will only run the full test suite. So it will be max 40 when in draft if all tests are affected, and not in draft, the regular run.

[alopezari]

Nope, we will only run the "only changed" ones when the PR is in draft mode, and when it's marked as "not draft", it will only run the full test suite. So it will be max 40 when in draft if all tests are affected, and not in draft, the regular run.

Yep, that's what I mean: 40m if all tests are affected (in draft mode) + 40m regular run (in review mode).

The alternative approach would be 40m if all tests are affected + 40m regular run (all in review mode and the regular run running only if the initial - only-changed - test run passed.

It would take the same time, isn't it? There are some differences though:

• With the draft mode approach, we can control manually when we want to run the only-changed tests by setting the PR status to draft. The alternative approach would be automated and - unless we tweak the process - it would run always the the combination of workflows.
• The draft mode approach would take around 80m in the worst scenario (all tests needs to be run on draft mode if all tests are affected) but in 2 different stages: 40m once it's in draft + 40m once it's ready for review. On the other hand, the alternative approach would take around 80m all together, on a continuous basis since one workflow is linked to the other.
• The draft mode requires that all devs learn about this change in the process whereas the alternative approach doesn't require any manual intervention as devs wouldn't need to change the PR status to trigger one workflow or another.
• The alternative approach could be optimized to run only the only-affected tests if there is any change in the tests directory, followed by the full test suite if the only-changed ones passed. Otherwise, no changes would be added to tests, the we could skip the only-changed workflow and directly run the full one.

I tried to gather all thoughts on this message to make sure we're not missing any details and we're aligned on how the understanding of both approaches. Please let me know if I missed anything!

Personally, I'm happy with both approaches. Both of them have pros and cons, and I'm not strongly opinionated on anyone in particular, so I'll support whatever you decide 👍

[tpaksu]

@alopezari Nevermind, I guess I'm going to close this PR as we may break the tests in different workflows per scope (merchant, shopper, subscriptions, etc) in different containers, so the time will be shorter, and it'll be more like what it was with Puppeteer.