Skip to content
wcagc
←Back to blog
accessibilityCItestinghow-to

Accessibility testing in CI: a practical guide

How to run accessibility testing in CI with axe-core, Playwright and pa11y, what a check catches, and how to gate a pull request on violations.

P

Pavel Charkasau

Accessibility testing in CI means running an automated engine like axe-core against your pages on every pull request and failing the build when it finds violations. The usual setup is one of three shapes: axe-core driven through your existing browser tests (Playwright, Cypress, or Jest), the pa11y-ci command-line runner against a list of URLs, or Lighthouse CI if you already use it for performance. Each maps its findings to WCAG success criteria and exits non-zero when something breaks, which is what turns a red check into a merge blocker. You wire it into a GitHub Actions job (or GitLab CI, or whatever you run), point it at a built preview of the app, and decide which severities stop a merge. The honest limit is the same one every automated tool has: axe-core finds about 57% of WCAG issues on average (Deque, 2021), so a green check means "no machine-detectable regressions", not "accessible". This guide covers the setup, the gating decisions, and where the line sits.

What does accessibility testing in CI actually do?

It runs the same rules a browser extension like axe DevTools runs, but on a schedule you don't control by hand, against a headless browser, on every push. The value is timing. A missing form label caught the moment a developer opens the pull request costs a two-line fix. The same label caught six months later in a manual audit costs a ticket, a sprint slot, and a regression hunt through code nobody remembers writing.

axe-core is the engine most of these tools share underneath. Its design goal is worth knowing before you gate anything on it: the README states plainly that "it returns zero false positives (bugs notwithstanding)" (axe-core). That is a deliberate trade. axe would rather stay silent on something it can't be certain about than cry wolf, because a checker that flags false positives gets switched off within a week. So when axe reports a violation in CI, it is almost always a real one. What it doesn't report is a much longer list, and I'll get to that.

Which tools should I use for accessibility testing in CI?

Pick the one that matches how you already test. Adding a whole new test runner just for accessibility is how these checks end up unmaintained.

  • You have Playwright or Cypress tests. Use the axe integration for that runner: @axe-core/playwright or cypress-axe. You get to reuse your existing page setup, logins, and fixtures, which matters because most real barriers live behind a login, not on the marketing homepage (axe-core projects).
  • You have Jest and jsdom component tests. jest-axe runs axe against rendered component output. It won't catch layout or focus problems that only appear in a real browser, but it flags missing names and roles at the unit level, before a component is even wired into a page.
  • You have neither, and want a URL-based smoke test. pa11y-ci reads a list of URLs (or a sitemap) and runs axe or HTML_CodeSniffer against each, with a JSON config for thresholds. It's the quickest thing to bolt onto a static site.
  • You already run Lighthouse CI. Its accessibility category is axe-core under the hood, scored 0–100. Fine as a trend signal; weaker as a gate, because a score hides which criterion failed.

Here is the axe-core-in-Playwright case, which is the one I reach for most:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('checkout page has no automatically detectable a11y violations', async ({ page }) => {
  await page.goto('/checkout');
  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
    .analyze();
  expect(results.violations).toEqual([]);
});

The withTags call is doing real work. It scopes the run to the WCAG 2.2 A and AA rules, which is the conformance target behind both EN 301 549 in the EU and Section 508 in the US, so you are testing against the thing regulators actually reference rather than axe's full rule grab-bag.

How do I fail a build on accessibility violations?

The test above already does the mechanical part: expect(results.violations).toEqual([]) throws when the array isn't empty, the test fails, and the runner exits non-zero. CI reads that exit code and marks the check red. The only extra step is running it in a workflow against a real build of the app.

A minimal GitHub Actions job:

name: a11y
on: [pull_request]
jobs:
  axe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run build && npm run preview &
      - run: npx wait-on http://localhost:4173
      - run: npx playwright test a11y.spec.ts

Then make the check required. In the repository's branch protection settings, mark the axe job as a required status check for the default branch. Now a pull request with a new violation can't be merged until someone fixes it or a reviewer explicitly overrides. That branch-protection step is the part people skip, and without it the check is just a suggestion a busy team learns to ignore.

Should I gate on every violation or just new ones?

Here is the one strong opinion I'll put my name to: gate on new violations, not the total count. If you switch on a strict "zero violations" gate over an existing app, the first run turns up 400 issues from code written years ago, every pull request stays red, and within a week someone adds continue-on-error: true and the whole thing is theatre.

The workable pattern is a baseline. Record today's known violations, commit that snapshot, and fail the build only when a pull request adds something not in it. pa11y-ci supports this directly through its threshold and per-URL config; with axe you can diff the results against a stored fixture. New code stays clean, the legacy backlog becomes a separate, planned cleanup, and the check keeps meaning something. Pair that with severity scoping. Many teams block only on axe's critical and serious impacts and log moderate and minor as warnings, so the gate fires on a broken form, not on a redundant ARIA attribute.

What does a CI check catch, and what does it miss?

This is the part a lot of "add a11y to your pipeline" posts skip, and it's the part that decides whether the check earns trust or quietly misleads you.

Deque's 2021 study, drawn from over 2,000 audits across more than 13,000 pages and nearly 300,000 issues, found automated testing covered 57% of accessibility issues by volume — above the older 20–30% industry benchmark, but still a little over half (Deque, 2021). So a realistic figure for what CI catches is roughly 30 to 57% depending on your pages and how you count. W3C's own guidance is blunter about the ceiling: "Web accessibility evaluation tools can not determine accessibility, they can only assist in doing so" (W3C WAI).

What falls in the missing half is not obscure. axe can tell you an image has no alt attribute (1.1.1), but not whether alt="image123.jpg" describes the picture. It can confirm a heading exists, not whether 2.4.6 is met because the heading actually labels its section. It flags a control with no accessible name, not whether the tab order makes sense, whether a custom dropdown is operable by keyboard, whether an error message is genuinely helpful, or whether a video's captions are correct. Those need a person with a keyboard and a screen reader. axe marks some of them as "incomplete" results, its way of saying a human needs to look here, and a good CI report surfaces those rather than hiding them.

That's why the check should be framed as a regression guard, not a compliance stamp. It stops machine-detectable failures from shipping. It does not make your product conformant, and no honest tool claims otherwise. Full conformance with WCAG, or with EN 301 549, still needs manual review by someone who tests the way a disabled user would. If you want the map of what still needs a human, our WCAG checklist marks which criteria are automatable and which aren't.

Frequently asked questions

What is the best tool for accessibility testing in CI?

axe-core is the engine most teams standardise on, run through whatever browser-test tool you already have — @axe-core/playwright or cypress-axe if you have end-to-end tests, pa11y-ci for a URL-based smoke test. There is no single best tool; the best one is the one your team will keep running, which usually means the one closest to your existing setup.

Can automated accessibility testing make my site compliant?

No. Automated tools find roughly 30 to 57% of issues by volume (Deque, 2021), and W3C is clear that tools "can not determine accessibility". A green CI check means no machine-detectable regressions shipped, not that your site conforms to WCAG. Conformance needs manual review.

Should a failing accessibility check block a merge?

For new violations, yes — that's the point of running it in CI. Make the job a required status check in branch protection so a pull request introducing a violation can't merge. For a large existing backlog, baseline the known issues and gate only on new ones, or the check becomes noise the team turns off.

Does axe-core produce false positives?

Rarely, by design. Its manifesto states it "returns zero false positives (bugs notwithstanding)" (axe-core). It achieves that by staying silent on anything it can't verify, which is also why its coverage tops out around half of all issues.

How do I test pages behind a login in CI?

Drive them through a browser-test tool that already handles authentication, then run axe on the resulting page. @axe-core/playwright and cypress-axe reuse your existing login fixtures, so you can point the check at checkout, onboarding, or account pages (where most real barriers live) instead of only the public homepage.

Start with a real scan

Before you wire up a pipeline, it helps to see what an automated engine actually flags on your pages, and how the results map to WCAG criteria. Run a free scan against a few of your real pages, including one behind a login. It uses the same class of engine you'd put in CI, and it shows both the violations and the criteria that still need a human — which is exactly the split you'll be designing your gate around.

Written by Pavel Charkasau, founder of WCAG Compliance. I read the standards so your team can ship against them.

Last updated: 11 August 2026.

Sources