Automated WCAG/ARIA testing in CI/CD

[mattxwang]

Long story short: I want to run automated accessibility tests on our website, and make sure we beat a certain threshold/not introduce accessibility regressions.

This may be a large undertaking (depending on tool configuration, CI/CD), and we may catch quite a few issues that we haven't anticipated before. However, it's a great ticket for developers who are looking to learn quite a bit about accessibility, CI/CD, or tooling!

Some thoughts from me:

• it seems like the JS pipeline for testing may be easier than with Ruby; given that we may already be adding other tests in JS, I think this is a fine
• I think axe is a good place to start, and it's one of the only free/open-source options available.
• one simple approach, then, is to run axe (with node) on our compiled HTML files.
• this blog post by Jason Dalton seems interesting, though here we're setting up a docker container and using puppeteer - it may be overkill!

That being said, this is only a cursory look; I would encourage whoever takes on this ticket to dive in if they'd like!

[BryanPan342]

Ooo this seems very interesting 😊

its been a while since i looked at jekyll, but this Github Action lets you run an action after another..

Maybe you can just run it on the deploy preview with the axe cli The PR number is pretty easy to get so finding the website is trivial.

You can also probably just add the <script/> tag whenever the environment isn't prod or smth so that the site doesnt get too thic lol

[BryanPan342]

I might look into this more when i have free time but its an interesting problem aha and i think the above solution is pretty portable for pretty much any repo that uses Netlify.. only drawback is you might eat up your Github Actions minutes.. in that case maybe travis??

[mattxwang]

Ooh, I didn't even think about running on the deploy preview will the pull request number - that's a great idea! And yeah, I don't think we'll get this done before my term w/ Teach LA is up, but it's definitely something I'm interested in.

A slight frustration is that Jekyll is within the Ruby sphere, while almost all of the automated testing tools I've seen for accessibility are in JS. I do think it's pretty portable - I'm planning on making another template repo (similar to CL's template) that has a very opinionated stack and would include something like this. Hopefully it's something I can start to roll out ACM-wide (and possibly across most of UCLA's CS community 😊 ).

[mattxwang]

Oh also re: Actions minutes, we might be able to apply for extra minutes because we're a nonprofit/OSS org, but I'd need to look into that. I've been able to save on a ton of minutes by manually caching deps (Actions doesn't do this by default).

[BryanPan342]

Yeah.. the action minutes shouldn't be a constraint for most UCLA Organizations

There should be a UCLA Open Source organization or smth to put some of this stuff into! It would require a lot of management lmao but I think it would be cool if we had that.

I think once we test how fast axe is will we know if we need something other than a GitHub action..

Another solution is to use AccessLint.. tho that seems like an issue with jekyll..

Or build ur own third party app 😊

[mattxwang]

Oh I totally forgot to respond to this:

• the UCLA OSS org is an interesting idea, I'm a bit concerned about governance/momentum - it's already been a struggle managing perms for just uclaacm
• yup, definitely testing axe is a good first step
• I think for AccessLint we'd have to build it against a live gh-pages branch that auto-updates on every change, which seems tricky to do :(
• our own third-party app .... say less 👁️