feat(a11y): add jest-axe CI gate for Lit component ARIA patterns

[mekarpeles]

Summary

Adds automated testing of the ARIA semantics of Open Library's Lit web components using jest-axe. This is Phase 1 of OL's accessibility tooling roadmap: a CI gate that catches ARIA regressions before they ship.

Each test renders the real component and runs axe over its actual shadow DOM, so a change to a component's markup is what makes the test fail.

• OlPopover — role="dialog", aria-label, aria-haspopup / aria-expanded / aria-controls on the slotted trigger, non-modal panel, and the mobile tray
• OlOptionsPopover — trigger wiring, role="radiogroup", radio inputs taking their name from a wrapping <label>, aria-hidden group heading
• OlToast — role="status" / role="alert" and aria-live per type, close button accessible name

Each test file includes a regression guard that confirms axe catches violations when an accessible name is absent, so we know the rules are active rather than silently passing.

What this does NOT cover

These tests run in jsdom, which parses markup but never lays out or paints it. Anything that depends on rendered pixels is invisible to them:

• Color contrast is not tested. jest-axe automatically disables axe's entire cat.color category in a jsdom environment, because the computed colors it would need don't exist. That's color-contrast (WCAG 1.4.3), color-contrast-enhanced (1.4.6), and link-in-text-block (1.4.1). A component can pass every test here and still ship unreadable text. Contrast has to stay a manual or browser-based check.
• Three document-level rules are disabled explicitly in test-utils/a11y.js: region, landmark-one-main, and page-has-heading-one. They check whole-page structure, which a mounted component fragment can never satisfy.
• Nothing visual or temporal: focus rings, animation, reflow, z-index and stacking, or how a real screen reader actually announces any of this.

So this gate is a floor, not a ceiling. It catches ARIA wiring that regresses in the markup. It does not certify a component as WCAG 2.1 AA conformant.

Making this work needed a Jest transform entry that transpiles lit out of node_modules. Nothing in tests/unit/js/ imports from 'lit' today, so this is what unlocks testing real Lit components at all. It bypasses the root .babelrc deliberately: that config uses useBuiltIns: "usage" with core-js, and injecting polyfill imports into lit's ESM would be wrong.

Testing

npm run test:a11y

 PASS  openlibrary/components/__tests__/OlPopover.a11y.test.js
 PASS  openlibrary/components/__tests__/OlOptionsPopover.a11y.test.js
 PASS  openlibrary/components/__tests__/OlToast.a11y.test.js

Test Suites: 3 passed, 3 total
Tests:       15 passed, 15 total

npm run test (full suite)

Test Suites: 29 passed, 29 total
Tests:       510 passed, 510 total

These run in CI already: openlibrary/components/__tests__/ is in jest.roots, so npm run test picks them up. test:a11y is a convenience for running just this group locally.

package-lock.json regenerated inside Docker (node v24) to match CI.

Changes

• openlibrary/components/__tests__/OlPopover.a11y.test.js — 5 tests
• openlibrary/components/__tests__/OlOptionsPopover.a11y.test.js — 5 tests
• openlibrary/components/__tests__/OlToast.a11y.test.js — 5 tests
• openlibrary/components/test-utils/a11y.js — shared mount / axe / jsdom-stub helpers
• package.json — adds jest-axe devDependency, test:a11y script, openlibrary/components/__tests__/ to Jest roots, and the lit transform
• eslint.config.cjs — Jest globals for the new test and test-utils directories

Related

Closes #13005

[lokesh]

Blocking

• openlibrary/components/__tests__/*.a11y.test.js — These tests don't actually exercise the components. Each one sets document.body.innerHTML to a hand-transcribed copy of what the component supposedly renders, then runs axe on that string. No component is imported or instantiated, so this validates our understanding of the correct ARIA markup, not the component output. If OlToast.render() drops the close button's aria-label, or the markup shifts, these stay green. As a "CI gate" that's the one failure mode we can't afford.
• The file comments cite tests/unit/js/focusableHostMixin.test.js as precedent, but that test imports the real FocusableHostMixin, defines real custom elements, and drives real DOM. The renderHTML there is fixture content fed to code that actually runs. These tests run no component code at all, so it's not the same approach.
• Concrete drift already exists: OlToast renders <span class="toast__icon"><svg aria-hidden="true">, but the fixture writes <span aria-hidden="true"><!-- info icon --></span>. The test and the component have diverged on day one.

The fix is to render the real components. Lit works in jsdom: createElement, await el.updateComplete, then run axe against el.shadowRoot, the same way focusableHostMixin.test.js drives real elements. That deletes the transcription-drift risk and turns this into a real gate. The regression-guard tests (asserting violations.length > 0 on broken markup) can go once that's in place, since they only prove axe is wired up.

Non-blocking

• The intent is good and the ARIA patterns in the fixtures are sound. If you'd rather not block on the refactor, an option is to land this as smoke scaffolding rather than a "CI gate," with a follow-up issue to render real components. It's test-only, so it can't break production.

---

Tested locally before merging? If yes and CI is green, this looks ready.

[lokesh]

Following up on the review with a concrete path for rendering the real components.

The reason the tests hand-write HTML today is a real constraint: lit ships as untranspiled ESM, and our jest transform ignores node_modules (and .babelrc doesn't apply across the package boundary anyway), so import '../lit/OlToast.js' throws Cannot use import statement outside a module. So it's not that the author skipped rendering out of laziness, jest fights you here.

There are two ways out. We can force jest to transform Lit (promote .babelrc to a root babel.config.js plus a transformIgnorePatterns whitelist), which is a small diff and reuses jest-axe. That works for structural ARIA, but jsdom does no layout or style computation, so color-contrast and visibility rules are silently skipped there.

The path I'd lean toward for real component testing is @web/test-runner + @open-wc/testing. It's what the Lit and Open WC teams recommend: tests run in a real browser (native ESM, so the transform problem disappears entirely), and @open-wc/testing gives us a real-browser axe check behind a clean matcher:

import { fixture, html, expect } from '@open-wc/testing';
import '../lit/OlToast.js';

it('error toast is accessible', async () => {
  const el = await fixture(html`<ol-toast type="error" message="Could not save."></ol-toast>`);
  await expect(el).to.be.accessible();   // real shadow DOM, contrast included
});

Real shadow DOM, ::slotted, focus order, and contrast all behave truthfully, which is exactly the stuff jsdom fakes. The tradeoff is a second test runner and CI job alongside jest. Given we're standing up an a11y gate we'll want to grow, I think that's the right foundation rather than fighting jest's ESM handling. Happy to pick the runner question up in the next Frontend meeting if we want to talk through the CI cost before committing.

[lokesh]

I pushed a change to this branch. Happy to talk it through or back it out if you'd rather take it a different direction.

The short version: the tests were checking a hand-typed copy of each component's HTML rather than the components themselves. I pointed them at the real thing.

Why it matters. Because the copy and the component have no connection to each other, the tests pass whether or not the components are actually accessible. I checked this rather than assumed it: I deleted the close button's label from the toast component — a genuine accessibility bug — and the tests here still passed. Green check, broken button.

How I noticed. The copies had already drifted. The popover copy describes its dialog as "modal", and master has since deliberately made it non-modal so keyboard users can Tab out of it. Worth saying plainly: that wasn't a mistake on your part, it was accurate when you wrote it. That's exactly the trouble with copies — they go stale quietly and nothing tells you.

What's different now. The tests load the actual components, render them, and check what really comes out. Getting there needed a bit of plumbing: the test runner didn't know how to load our component library, and the simulated browser we run tests in is missing two small features the components rely on. That's all tucked into one shared helper file.

The advantages:

• The tests can fail now. That's the entire point of a gate, and it wasn't true before.
• I proved it rather than claiming it. I broke four things on purpose — removed the toast's close-button label, removed the popover dialog's name, added the modal flag back, and un-hid a heading from screen readers. All four were caught. All four passed under the old approach.
• No more duplicate HTML to maintain by hand. It had already fallen out of date within a few weeks, with nothing to flag it.
• The tests now describe how the components genuinely behave, so they double as documentation you can trust when you're wiring up a new one.

Your regression-guard idea carried over — I kept it, and made each guard name the specific rule it expects so it can't pass for the wrong reason.

Also in this push: merged master in, since the branch was a fair way behind and the popover test needs the current version. And I dropped the separate accessibility step in CI — the tests now sit inside the normal test run, so that step was just running them a second time. Failures still fail CI.

What this still doesn't do: colour contrast. The simulated browser can't compute layout or colour, so it skips those checks entirely — that was equally true before, it's just worth being explicit that "WCAG AA" here means structure and labelling, not contrast. Real contrast checking means running in an actual browser, which is a bigger conversation. There's a wrinkle to know about before anyone starts: our components take their colours from a shared stylesheet that wouldn't be loaded in an isolated test, so contrast results would look authoritative and be wrong until that's handled. I'd keep it out of this PR.

One unrelated thing I bumped into. While writing the options-popover test I found that <ol-options-popover>, used exactly the way our design docs page documents it, renders no button to open it. It used to supply its own; that was removed from master and the docs page wasn't updated. Nothing to do with this PR, but someone should decide whether the component or the docs is the one that's wrong — if anything in production uses it that way, it's currently unopenable.

[lokesh]

...

[lokesh]

LGTM - I made some changes so we load the actual components for testing rather than a hand-typed one-off stamp of the component which could easily drift from the actual component. The testing happens in Node, and not in browser, so there are still some limitations. For example, color contrast testing requires more context, specifically our CSS custom properties/variables and their cascade, and that is something that is heard to recreate in a narrow test. Even if tested in browser, we would need to do some work to get color contrast testing to work. We'll revisit in the future.