Replaceaxe-puppeteer

[domoscargin]

What

Use @axe-core/puppeteer or jest-axe as a dependency rather than axe-puppeteer.

Why

axe-puppeteer has been deprecated in favour of @axe-core/puppeteer, and our automated accessibility testing is therefore not up to date with Axe's latest changes.

We will likely need to ignore issues relating to our iFrames, which might render our tests a bit toothless and require a bit of a rethink. See the discussion here: #2442

We currently use jest-axe on govuk-frontend, and we highlight this fact in our accessibility strategy, so we might want to consider moving to that instead.

This should mostly unblock #2498, however, @axe-core/puppeteer currently only supports Puppeteer up to version 18.

Who needs to work on this

Developer

Who needs to review this

Developer, Accessibility specialist

Done when

• axe-puppeteer has been replaced
• [Optional] Puppeteer is up to date
• All our accessibility tests pass

[domoscargin]

Violation output

One limitation is that we're only testing certain pages (because we use templates), so specific issues might not get caught.

Summary of violations:

• Elements must only use allowed ARIA attributes
  • A result of conditional radios having the aria-expanded attribute. We are aware of this issue: Radios that conditionally reveal content use aria-expanded which is not a valid aria-attribute for those roles govuk-frontend#979
• Heading levels should only increase by one
  • We demonstrate <h1> tags in iFrames, which are often followed by <h3> in the main page content. These headings all get jumbled up together in screenreaders. We're not sure there's much we can do to fix it.
• Document does not have a main landmark (Another iFrame issue)

Radios (component page example)

[
  {
    "id": "aria-allowed-attr",
    "impact": "critical",
    "tags": ["cat.aria", "wcag2a", "wcag412"],
    "description": "Ensures ARIA attributes are allowed for an element's role",
    "help": "Elements must only use allowed ARIA attributes",
    "helpUrl": "https://dequeuniversity.com/rules/axe/4.5/aria-allowed-attr?application=axe-puppeteer",
    "nodes": [
      {
        "any": [
          {
            "id": "aria-allowed-attr",
            "data": ["aria-expanded=\"false\""],
            "relatedNodes": [],
            "impact": "critical",
            "message": "ARIA attribute is not allowed: aria-expanded=\"false\""
          }
        ],
        "all": [],
        "none": [],
        "impact": "critical",
        "html": "<input class=\"govuk-radios__input\" id=\"contact\" name=\"contact\" type=\"radio\" value=\"email\" aria-controls=\"conditional-contact\" aria-expanded=\"false\">",
        "target": ["#iFrameResizer6", "#contact"],
        "failureSummary": "Fix any of the following:\n  ARIA attribute is not allowed: aria-expanded=\"false\""
      },
      {
        "any": [
          {
            "id": "aria-allowed-attr",
            "data": ["aria-expanded=\"false\""],
            "relatedNodes": [],
            "impact": "critical",
            "message": "ARIA attribute is not allowed: aria-expanded=\"false\""
          }
        ],
        "all": [],
        "none": [],
        "impact": "critical",
        "html": "<input class=\"govuk-radios__input\" id=\"contact-2\" name=\"contact\" type=\"radio\" value=\"phone\" aria-controls=\"conditional-contact-2\" aria-expanded=\"false\">",
        "target": ["#iFrameResizer6", "#contact-2"],
        "failureSummary": "Fix any of the following:\n  ARIA attribute is not allowed: aria-expanded=\"false\""
      },
      {
        "any": [
          {
            "id": "aria-allowed-attr",
            "data": ["aria-expanded=\"false\""],
            "relatedNodes": [],
            "impact": "critical",
            "message": "ARIA attribute is not allowed: aria-expanded=\"false\""
          }
        ],
        "all": [],
        "none": [],
        "impact": "critical",
        "html": "<input class=\"govuk-radios__input\" id=\"contact-3\" name=\"contact\" type=\"radio\" value=\"text\" aria-controls=\"conditional-contact-3\" aria-expanded=\"false\">",
        "target": ["#iFrameResizer6", "#contact-3"],
        "failureSummary": "Fix any of the following:\n  ARIA attribute is not allowed: aria-expanded=\"false\""
      },
      {
        "any": [
          {
            "id": "aria-allowed-attr",
            "data": ["aria-expanded=\"true\""],
            "relatedNodes": [],
            "impact": "critical",
            "message": "ARIA attribute is not allowed: aria-expanded=\"true\""
          }
        ],
        "all": [],
        "none": [],
        "impact": "critical",
        "html": "<input class=\"govuk-radios__input\" id=\"contact\" name=\"contact\" type=\"radio\" value=\"email\" checked=\"\" aria-controls=\"conditional-contact\" aria-expanded=\"true\">",
        "target": ["#iFrameResizer9", "#contact"],
        "failureSummary": "Fix any of the following:\n  ARIA attribute is not allowed: aria-expanded=\"true\""
      },
      {
        "any": [
          {
            "id": "aria-allowed-attr",
            "data": ["aria-expanded=\"false\""],
            "relatedNodes": [],
            "impact": "critical",
            "message": "ARIA attribute is not allowed: aria-expanded=\"false\""
          }
        ],
        "all": [],
        "none": [],
        "impact": "critical",
        "html": "<input class=\"govuk-radios__input\" id=\"contact-2\" name=\"contact\" type=\"radio\" value=\"phone\" aria-controls=\"conditional-contact-2\" aria-expanded=\"false\">",
        "target": ["#iFrameResizer9", "#contact-2"],
        "failureSummary": "Fix any of the following:\n  ARIA attribute is not allowed: aria-expanded=\"false\""
      },
      {
        "any": [
          {
            "id": "aria-allowed-attr",
            "data": ["aria-expanded=\"false\""],
            "relatedNodes": [],
            "impact": "critical",
            "message": "ARIA attribute is not allowed: aria-expanded=\"false\""
          }
        ],
        "all": [],
        "none": [],
        "impact": "critical",
        "html": "<input class=\"govuk-radios__input\" id=\"contact-3\" name=\"contact\" type=\"radio\" value=\"text\" aria-controls=\"conditional-contact-3\" aria-expanded=\"false\">",
        "target": ["#iFrameResizer9", "#contact-3"],
        "failureSummary": "Fix any of the following:\n  ARIA attribute is not allowed: aria-expanded=\"false\""
      }
    ]
  },
  {
    "id": "heading-order",
    "impact": "moderate",
    "tags": ["cat.semantics", "best-practice"],
    "description": "Ensures the order of headings is semantically correct",
    "help": "Heading levels should only increase by one",
    "helpUrl": "https://dequeuniversity.com/rules/axe/4.5/heading-order?application=axe-puppeteer",
    "nodes": [
      {
        "any": [
          {
            "id": "heading-order",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Heading order invalid"
          }
        ],
        "all": [],
        "none": [],
        "impact": "moderate",
        "html": "<h3 id=\"if-youre-asking-more-than-one-question-on-the-page\">If you’re asking more than one question on the page</h3>",
        "target": ["#if-youre-asking-more-than-one-question-on-the-page"],
        "failureSummary": "Fix any of the following:\n  Heading order invalid"
      },
      {
        "any": [
          {
            "id": "heading-order",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Heading order invalid"
          }
        ],
        "all": [],
        "none": [],
        "impact": "moderate",
        "html": "<h3 id=\"radio-items-with-hints\">Radio items with hints</h3>",
        "target": ["#radio-items-with-hints"],
        "failureSummary": "Fix any of the following:\n  Heading order invalid"
      },
      {
        "any": [
          {
            "id": "heading-order",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Heading order invalid"
          }
        ],
        "all": [],
        "none": [],
        "impact": "moderate",
        "html": "<h3 id=\"radio-items-with-a-text-divider\">Radio items with a text divider</h3>",
        "target": ["#radio-items-with-a-text-divider"],
        "failureSummary": "Fix any of the following:\n  Heading order invalid"
      },
      {
        "any": [
          {
            "id": "heading-order",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Heading order invalid"
          }
        ],
        "all": [],
        "none": [],
        "impact": "moderate",
        "html": "<h3 id=\"conditionally-revealing-a-related-question\">Conditionally revealing a related question</h3>",
        "target": ["#conditionally-revealing-a-related-question"],
        "failureSummary": "Fix any of the following:\n  Heading order invalid"
      },
      {
        "any": [
          {
            "id": "heading-order",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Heading order invalid"
          }
        ],
        "all": [],
        "none": [],
        "impact": "moderate",
        "html": "<h4 id=\"known-issues\">Known issues</h4>",
        "target": ["#known-issues"],
        "failureSummary": "Fix any of the following:\n  Heading order invalid"
      },
      {
        "any": [
          {
            "id": "heading-order",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Heading order invalid"
          }
        ],
        "all": [],
        "none": [],
        "impact": "moderate",
        "html": "<h3 id=\"error-messages\">Error messages</h3>",
        "target": ["#error-messages"],
        "failureSummary": "Fix any of the following:\n  Heading order invalid"
      },
      {
        "any": [
          {
            "id": "heading-order",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Heading order invalid"
          }
        ],
        "all": [],
        "none": [],
        "impact": "moderate",
        "html": "<h4 id=\"if-its-a-yes-or-no-question\">If it’s a ‘yes’ or ‘no’ question</h4>",
        "target": ["#if-its-a-yes-or-no-question"],
        "failureSummary": "Fix any of the following:\n  Heading order invalid"
      }
    ]
  },
  {
    "id": "landmark-one-main",
    "impact": "moderate",
    "tags": ["cat.semantics", "best-practice"],
    "description": "Ensures the document has a main landmark",
    "help": "Document should have one main landmark",
    "helpUrl": "https://dequeuniversity.com/rules/axe/4.5/landmark-one-main?application=axe-puppeteer",
    "nodes": [
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer0", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer1", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer2", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer3", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer4", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer5", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer6", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer7", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer8", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      },
      {
        "any": [],
        "all": [
          {
            "id": "page-has-main",
            "data": null,
            "relatedNodes": [],
            "impact": "moderate",
            "message": "Document does not have a main landmark"
          }
        ],
        "none": [],
        "impact": "moderate",
        "html": "<html lang=\"en\" class=\"govuk-template app-example-page__wrapper flexbox no-flexboxtweener\">",
        "target": ["#iFrameResizer9", "html"],
        "failureSummary": "Fix all of the following:\n  Document does not have a main landmark"
      }
    ]
  }
]

[colinrotherham]

This is fab @domoscargin

Do we want to weigh up running Axe without a browser via jest-axe:

• Run axe in jest, with failing builds govuk-frontend#522

Perhaps the reporting is nicer? Or equally, switching to @axe-core/* packages on govuk-frontend and sharing/exporting any "known violations" for other teams