[BUG]: Intermittent ace.Checker is not a constructor errors in Jest tests

[caleballen]

Project

a11y checker

Browser

No response

Operating system

Other

Description

Hi there,

We import accessibility-checker into our node environment for use in Jest testing both locally and as part of our CI pipeline. We have begun seeing intermittent failures, usually all tests will pass without issue but occasionally the tests encounter errors:

    TypeError: ace.Checker is not a constructor

      at node_modules/src-ts/lib/ACEngineManager.ts:112:31
      at node_modules/graceful-fs/graceful-fs.js:61:14

which leads to

    TypeError: Cannot read properties of undefined (reading 'check')

      at node_modules/src-ts/lib/ACHelper.ts:400:45
      at step (node_modules/accessibility-checker/lib/ACHelper.js:33:23)
      at Object.next (node_modules/accessibility-checker/lib/ACHelper.js:14:53)
      at node_modules/accessibility-checker/lib/ACHelper.js:8:71
      at Object.<anonymous>.__awaiter (node_modules/accessibility-checker/lib/ACHelper.js:4:12)
      at getComplianceHelperLocal (node_modules/accessibility-checker/lib/ACHelper.js:466:12)
      at node_modules/src-ts/lib/ACHelper.ts:216:22
      at step (node_modules/accessibility-checker/lib/ACHelper.js:33:23)
      at Object.next (node_modules/accessibility-checker/lib/ACHelper.js:14:53)
      at fulfilled (node_modules/accessibility-checker/lib/ACHelper.js:5:58)

in the same test.

When these errors occur often several tests still pass. All issues can be mitigated by using the Jest flag --runInBand forcing all tests to run sequentially although this is not ideal. This issue is highly reproducible on my 8 thread laptop.

Steps to reproduce

I'll put together a reproducible repo if that would be helpful but I believe the steps are as follows

1. Set up accessibility-checker with Jest. Extend the expect functionality with the accessibility-checker compliance assertion.
2. Have several accessibility tests.
3. Run on a multithreaded system to allow for Jest workers.
4. Repeat several times.

[aliunwala]

Moving this issue into Triage pipeline to be prioritized to be looked at.

[tombrunet]

I ran this boilerplate: https://github.com/IBMa/equal-access/tree/jest-puppeteer/accessibility-checker/boilerplates/jest-puppeteer-ts with 5 suites running 50 tests each using puppeteer and am not running into this. Are you sure you aren't sharing pages across tests and trying to load two tests into the same page at the same time?

[aliunwala]

Triage meeting:
We are closing this for now as we cannot reproduce it reliably. If you get a test environment that sees this consistently you can share feel free to open this issue.

[tombrunet]

Going to re-open this. Got some new reports. Will try again to reproduce this.

[tombrunet]

Finally able to reproduce this on my end. Looking...

[tombrunet]

Should be fixed in 3.1.44