ci: Find duplicate and slow tests

[dplewis]

Pull Request

• Report security issues confidentially.
• Any contribution is under this license.
• Link this pull request to an issue.

Issue

With over 3000+ tests and counting, over the years the repository has gone from 15+ minutes of CI time down to under 6 minutes with community efforts #7262. I had the idea years ago to find slow tests and duplicate tests with the same name and functionality using a custom Jasmine Spec Reporter.

This can also help find bottlenecks in performance or tests improvements for instance
This takes 3 seconds and I don't know why maybe it's internal or a Jasmine best practice thats missing.

it('polygon loop is not valid', done => {
    const coords = [
      [0, 0],
      [0, 1],
      [1, 0],
      [1, 1],
    ];
    const obj = new TestObject();
    obj.set('polygon', new Parse.Polygon(coords));
    obj.save().then(done.fail, () => done());
});

Optimizing tests as they are created is the goal.

Approach

• Remove duplicate tests
• Remove unnecessary setTimeout
• Cleanup console warnings
• Display tests taking longer that 2 seconds
• Saves ~10 seconds
• Rename test with same name but different functionality
• I left it('supports bytes range if start and end undefined) just to make sure it displays in CI

Sub 5 minutes

Executed 3087 of 3173 specs INCOMPLETE (86 PENDING) in 4 mins 50 secs. 

[parse-github-assistant]

Thanks for opening this pull request!

• ❌ Please link an issue that describes the reason for this pull request, otherwise your pull request will be closed. Make sure to write it as Closes: #123 in the PR description, so I can recognize it.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.14%. Comparing base (872b9eb) to head (756d110).
Report is 51 commits behind head on alpha.

Additional details and impacted files

@@            Coverage Diff             @@
##            alpha    #9188      +/-   ##
==========================================
- Coverage   94.16%   94.14%   -0.02%     
==========================================
  Files         186      186              
  Lines       14751    14751              
==========================================
- Hits        13890    13888       -2     
- Misses        861      863       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mtrezza]

It still logs duplicate tests:

Duplicate spec: Auth should load auth without a config
Duplicate spec: Auth should load auth with a config
Duplicate spec: AuthenticationProviders should throw error when Facebook request appId is wrong data type
Duplicate spec: apple signin auth adapter (using client id as string) should throw error with with invalid jwt issuer
Duplicate spec: facebook limited auth adapter (using client id as string) should throw error with with invalid jwt issuer
Duplicate spec: Auth Adapter features should throw if no triggers found
Duplicate spec: InstallationsRouter query installations with count = 1
Duplicate spec: MongoStorageAdapter should use index for caseInsensitive query
Duplicate spec: Parse.File testing Gridstore Range supports bytes range if start and end undefined
Duplicate spec: Parse.Query hint query find with hint object

[mtrezza]

Some minor log improvements

[dplewis]

@mtrezza I removed all the dups, looks like there around 75 tests that take longer than a 1000ms.

This is ready for review.

[mtrezza]

The clean-up and optimization done with this PR is great. But the changes in the test names can be problematic going forward.

I understand that a test needs a unique ID to reference the test. But this also means the title suddenly is not just a test description but has a function that influences this logic. It requires manual data duplication, i.e. add the DB name and version to the text, even though this is defined in the test methods used.

Maintenance: If there are duplicate tests, the CI passes, if I'm not mistaken. That means neither contributors nor reviewers can consider the result of this check. In the end, we'll accumulate test duplicates until someone looks at the logs and cleans it up.

Review effort: This adds an additional review step to verify that the test title includes the methods that are used for the test, DB type and version.

Error prone: New tests are often created by copy/paste/modify. When changing the method, the text description now also has to adapt accordingly, including any specified DB and version. If a test description includes an incorrect MongoDB version, the failing test's title in the logs is misleading.

I'm thinking about how this could be addressed...

Worst case, we throw out the test duplication detection until we found a solution and keep the - I believe by itself already very valuable - execution time analysis.

We may be able to uniquely identify a test by its name, file and code line number without relying on a unique test description. We are also introducing test IDs for the external Oracle DB adapter. A solution could be to add IDs to all tests, with VSC and regex scripting maybe not much of an effort.

[dplewis]

@mtrezza I reverted the DB specific duplicate tests. They can be handled in a separate PR.

[parseplatformorg]

🎉 This change has been released in version 7.3.0-alpha.2

[parseplatformorg]

🎉 This change has been released in version 7.3.0-beta.1

[parseplatformorg]

🎉 This change has been released in version 7.3.0