Skip to content
wcagc
←Back to blog
scannerbuying guideWCAGEN 301 549procurement

How to run an accessibility scanner trial before you buy

How to plan an accessibility scanner trial that proves something: which pages to sample, how to compare two tools, and where automation stops.

P

Pavel Charkasau

An accessibility scanner trial is only worth running if you point it at the pages that will decide the purchase, and those are almost never the marketing homepage. Pick a sample of 10 to 20 views using the logic in the W3C's evaluation methodology: your common templates, the pages carrying essential functionality, and every step of at least one complete process such as signup or checkout (W3C WAI). Put that same sample through two tools rather than one, so you can see where they disagree. Then judge each on four things a demo will not show you: whether it reaches content behind a login, whether findings map to the standard you are held to, whether the output survives being handed to someone else, and whether the vendor is straight about what the scanner cannot decide. Two weeks and one real product area is enough. You are buying coverage and evidence, not a longer rule list, and no scanner at any price makes a site conformant on its own.

What should an accessibility scanner trial actually prove?

That the tool works on your product, not on a demo site. Those are different questions, and vendors show you the second one.

Start with reach. Can the scanner get to the views where your barriers actually live? The W3C names this as a selection criterion in plain terms: "Some tools check a single page, while others check entire groups of related pages", and separately, "some can also access password-restricted content" (W3C WAI). Most tools stop at the login screen. If your product is an application, a tool that cannot sign in has failed the trial before it starts, whatever its rule coverage looks like.

Then ask whether the output is usable by the person who has to act on it. A finding that reads "form element has no label, 3.3.2" with the CSS selector and the failing markup is work a developer can pick up before lunch. A score out of 100 is not.

The last thing to prove is that you can defend the result months later. If you are working toward the EAA or a Section 508 procurement, someone will eventually ask which requirement you tested and what the answer was. The trial is where you find out whether the tool can answer that.

Which pages should you scan during the trial?

Use a structured sample, not whatever the crawler finds first. WCAG-EM, the W3C's evaluation methodology, sets out five steps, and step 3 is selecting a representative sample. It asks you to cover the common views, the essential functionality, the different types of pages, and the technologies the site relies on (W3C WAI). Then it adds the rule most trials skip: "Include all samples that are part of a complete process in the selected sample set." A process is a sequence you cannot skip a step in. Checkout. Account signup. A booking flow. Password reset.

That last rule matters commercially, because a broken step four means the process fails even if steps one through three are clean. So a trial that scans twenty product pages and no checkout has told you almost nothing about your risk.

A workable sample for a two-week trial:

  • One page per distinct template. The homepage, a listing page, a detail page, a long-form content page, a form-heavy page.
  • Every step of one complete process, from the first screen to the confirmation.
  • Two or three pages behind the login, including whatever your users see the moment they land.
  • Anything built with a component library or an embedded third party, since that is where scanners and hand-testing disagree most.
  • A random handful on top. WCAG-EM suggests roughly 10% of the structured sample, picked at random, as a check on your own selection bias.

Twelve to eighteen views is usually enough to expose a tool's limits. You can start on a single page with a free scan to see the shape of the output before you build the full sample.

How do you compare two scanners fairly?

Run the identical sample through both, on the same day, and read the differences rather than the totals.

Comparing issue counts is the trap. One tool reporting 340 findings and another reporting 190 tells you nothing about which is better, because they may be deduplicating differently or checking different things. The useful artifact is the disagreement list. Take twenty findings only one tool reported and check them by hand. You learn which tool catches real problems the other misses, and how noisy each one is, from the same twenty.

The W3C publishes a reference that helps here. ACT Rules are formally approved test procedures for WCAG, and the implementation pages "show how many ACT Rules different accessibility test tools and methodologies have 'consistently implemented'" (W3C WAI). Support for those rules is itself one of the criteria the W3C names for tool selection (W3C WAI), so it is a fair thing to ask a vendor about.

Expect the bulk of what both tools find to be the same small set of problems. WebAIM's February 2026 analysis of a million home pages found detected WCAG 2 failures on 95.9% of them, averaging 56.1 errors per page, and six error types accounted for 96% of all errors: low contrast text, missing image alternatives, missing form labels, empty links, empty buttons, and missing document language (WebAIM). If your trial results look like that list, the scanner is working normally. The interesting part of a comparison sits in the long tail.

What will the demo not show you?

The parts that cost you time in month three. Four are worth testing deliberately.

Signing in. Configure authentication yourself rather than watching the vendor do it. Session handling, multi-factor prompts, and a login form that renders client-side are where these setups break. If it takes a support ticket to scan your own dashboard, that is your answer.

The re-scan. Fix three findings, run it again, and see whether the tool shows you the delta or a fresh unrelated list. A trial that only ever produces first scans hides the thing you will use daily.

Then hand the export to a developer who was not in the meeting. If they come back asking where the problem is, the reporting is not good enough, and no amount of dashboard polish fixes that.

Last, check the mapping. Findings should tie to named success criteria and, if the EU matters to you, to EN 301 549 clauses. Annex C of that standard is normative and resolves each requirement to pass, fail, or not applicable (Accessibility Standards Canada, reproducing EN 301 549). That is the structure a reviewer expects, and a violation list without it becomes your manual work later rather than the tool's. Our WCAG checklist shows which rows a scan can even touch.

How does the vendor talk about what the scanner cannot do?

This is the criterion I would weight highest, and it costs nothing to test. Ask any vendor in your trial one question: what does your product not catch?

A straight answer exists. The W3C states it flatly: evaluation tools "can not determine accessibility, they can only assist in doing so" (W3C WAI). Deque's analysis of more than 2,000 audits across roughly 13,000 pages and nearly 300,000 issues found automated testing completely covered about 57% of issues by volume (Deque). Counted by success criteria rather than issue volume the machine-decidable share is lower, which is where the familiar 30 to 57% range comes from. Either way, a scan is a partial method. Whether your focus order makes sense, whether the alt text says the right thing, whether an error is announced at a useful moment: a person with a keyboard and a screen reader decides all of it.

There is a documented cost to the other kind of answer. On 22 April 2025 the US Federal Trade Commission approved a final order requiring the overlay vendor accessiBe to pay $1 million, over claims that its accessWidget could make any website WCAG compliant, which the FTC found false, misleading, or unsubstantiated (FTC). I will state my bias openly, since it shaped what we built: a vendor who will not name the limit in a sales call will not name it in your accessibility statement either, and that is the document with your company's name on it. Our scanner does not tell you whether your checkout is usable with a screen reader. Budget for the manual pass regardless of which tool you pick.

What does a finished trial look like?

A one-page write-up, and it should fit on one page. For each tool: which of your sample views it reached and which it could not, the count of findings it uniquely reported and how many of those held up when you checked them by hand, what happened on the re-scan, what the export looked like in a developer's hands, and the vendor's own answer about coverage limits.

Then decide on reach and evidence rather than rule count. If you are documenting conformance under the ADA, the EAA, or a public-sector procurement, the tool that gets into your authenticated flows and produces a mapped, dated record is worth more than the one that finds a few extra contrast errors on the homepage.

Frequently asked questions

How long should an accessibility scanner trial run?

Two weeks is usually enough if you have your sample ready on day one. That gives you time for a first scan, a round of fixes, and a re-scan, which is the sequence that reveals how the tool behaves in normal use.

Should I trial more than one scanner at the same time?

Yes. Running the same sample through two tools is the only cheap way to see what either one misses, since a single tool's output looks complete by definition. The W3C's ACT Rules implementation reports also let you compare published rule coverage before you start.

Can a scanner trial tell me whether my site is conformant?

No. Automated testing covers part of the picture, around 57% of issues by volume in Deque's analysis and less when counted by success criteria, and the W3C is explicit that tools assist with determining accessibility rather than determining it. A trial tells you how much of the work a tool can take off your plate.

What should I ask a vendor during the trial?

Ask what the scanner does not catch, whether it can scan pages behind your login, which standard clauses findings map to, and how long results are retained. The answers to those four questions separate tools faster than any feature list.

Try it on your own pages

Pick the ugliest page in your product, not the homepage, and run a free scan on it. You will see the exact output format, the selectors, and the criteria mapping in about a minute, which is the fastest way to judge whether a longer trial is worth your team's time. When the results are ready, the accessibility statement generator will draft an honest statement from what the scan found and what still needs a human.

Pavel Charkasau, founder, wcagc.com. Last updated 6 September 2026.

Sources