[Feature] add CLI flag for making a build but not running tests afterwards

[tom2drum]

Currently, as I see during the build process Playwright creates a file metainfo.json stored in the .cache directory. This file contains information about all dependencies for every test suite that will be run after the build stage. I would like to use this info for solving my problem. I try to implement in our company CI workflow a script that will run only those test suites which were affected by the changes in the current branch comparing to main branch. We run Playwright tests for every pull request in our repo, and this enhancement in the workflow will allow us optimize billable time of our CI provider. So, I can figure out what files was change in the pull request, but I still need to match this file list against the dependencies of each test suite in order to get which one should be run.

So, I would love to see in the Playwright an option that will allow to make the actual build and not run all tests afterwards.

[dgozman]

@tom2drum Are you running component tests only? I don't think we use .cache directory for end-to-end tests.

[tom2drum]

@dgozman I should have mentioned it, yes, we are running components visual tests exclusively.

[dgozman]

@tom2drum I don't think we would like anyone to build on top of metainfo.json and then break because we changed the format, since we do not consider it to be stable.

However, we are open to an option/flag similar to Jest's --onlyChanged, which looks exactly like your end goal. Let me know if I understood you correctly. If so, you can take a stab at it and send a pull request, or wait for us to implement it.

[tom2drum]

Yeah, the flag similar to Jest's one would be perfect. Actually I wanted propose this idea too as alternative.

I am not sure if I am able to implement it by myself, so I will probably wait for an implementation from your side.

Thanks! 🙌

[dj-stormtrooper]

@dgozman since the issue is about CI exclusively, what do you think, maybe it's better to implement Jest's --findRelatedTests instead? So the user could pass the list of changed sources and playwright will run only affected tests based on this list

The onlyChanged option is good for local development, it's hard to configure properly on CI because we need to specify the comment or branch and unstage the sources to make it work

For example Jest has whole bunch of options for CI-friendly usage, we could also choose this approach:

• https://jestjs.io/docs/cli#--changedfileswithancestor
• https://jestjs.io/docs/cli#--lastcommit
• https://jestjs.io/docs/cli#--changedsince

Also with VCS-based option it's hard to handle caveats with package.json dependencies. For example if the component depends on the external package we need to figure out which package-lock.json changes affect the component.
I think in that case it's better to leave decision making on the user side

P.S. I'm interested in this solution and perhaps I could try to implement it so let me know which approach do you think is the best

[dj-stormtrooper]

@dgozman could you take a look at this idea? ^

[ilkerbynder]

Just came to this thread by searching. I would like to express that this is a great idea, and a feature that we all need

[dgozman]

@dj-stormtrooper @ilkerbynder We are experimenting in this area, and v1.42 release now has a new experimental command find-related-test-files. If you would like to try it and give us feedback, that would be great.

npx playwright find-related-test-files  file1.ts file2.ts