Think Tank QA

Accessibility Testing: What Automated Scanners Catch and What Only Humans Find

Automated accessibility scanners are fast, inexpensive to rerun, and genuinely useful. The difficult question is not whether to use one. It is what decisions its results can safely support, because that boundary affects your tools, testing budget, release plan, and understanding of risk.

A scanner can determine whether an image has an alternative-text attribute, whether a computable color ratio clears a threshold, or whether a form control exposes a name. It cannot determine whether the alternative text communicates the image’s purpose, whether a screen reader user can complete checkout, or whether an error message helps someone recover.

At Think Tank QA we deliver accessibility testing as a service engagement rather than a software product, combining automated scans with human assistive-technology sessions. This guide explains what automated accessibility testing can reliably find, what still needs human judgment, and how to choose the right combination for your situation.

The short answer

What Is Automated Accessibility Testing?

Automated accessibility testing uses software to evaluate pages or code against accessibility rules, primarily the Web Content Accessibility Guidelines (WCAG), and flag failures a program can detect. Common examples include missing alternative-text attributes, low text contrast, invalid accessibility markup, and form controls without names a screen reader can announce. Most accessibility testing tools deliver some version of this same rule-based layer.

It can run in seconds across many pages, which makes it the right first layer and the wrong final answer.

Automated and human testing answer different questions

QuestionAutomated testingHuman testing
What does it inspect?Code, rendered properties, and configured page statesMeaning, interaction, sequence, and task completion
Where is it strongest?Fast, repeated checks with a computable yes-or-no resultChecks that require context, judgment, or assistive technology
How far does it scale?Hundreds or thousands of pages, if the tool can reach themRepresentative pages, components, and important user journeys
What is its main limitation?It cannot understand the user’s goal or experienceIt takes skilled time and covers only the agreed scope and technology combinations
When should it run?During development, in builds, and through scheduled monitoringDuring feature testing, significant releases, redesigns, and formal audits
What does it produce?Potential or confirmed rule findings that still need reviewVerified barriers, user impact, and evidence tied to real use

What does it inspect?

Automated testing
Code, rendered properties, and configured page states

Human testing
Meaning, interaction, sequence, and task completion

Where is it strongest?

Automated testing
Fast, repeated checks with a computable yes-or-no result

Human testing
Checks that require context, judgment, or assistive technology

How far does it scale?

Automated testing
Hundreds or thousands of pages, if the tool can reach them

Human testing
Representative pages, components, and important user journeys

What is its main limitation?

Automated testing
It cannot understand the user’s goal or experience

Human testing
It takes skilled time and covers only the agreed scope and technology combinations

When should it run?

Automated testing
During development, in builds, and through scheduled monitoring

Human testing
During feature testing, significant releases, redesigns, and formal audits

What does it produce?

Automated testing
Potential or confirmed rule findings that still need review

Human testing
Verified barriers, user impact, and evidence tied to real use

How to choose accessibility testing tools

Choose accessibility testing tools by the job you need done, not by the length of a vendor’s feature list.

Your immediate needTool categoryCommon examplesWhat it does not replace
Check one page while designing, developing, or testingBrowser-based checkerWAVE, axe DevTools, LighthouseTesting dynamic states and complete tasks
Stop known violations from entering a buildRule engine in continuous integrationaxe-core with Playwright or Cypress, Pa11yHuman review of meaning and usability
Watch a large production site for recurring problemsCrawler or monitoring platformScheduled site scanners and monitoring servicesA scoped manual audit
Work systematically through checks that need input from a personGuided evaluation toolAccessibility Insights and similar guided toolsExperienced screen reader and usability testing

Check one page while designing, developing, or testing

Tool category
Browser-based checker

Common examples
WAVE, axe DevTools, Lighthouse

What it does not replace
Testing dynamic states and complete tasks

Stop known violations from entering a build

Tool category
Rule engine in continuous integration

Common examples
axe-core with Playwright or Cypress, Pa11y

What it does not replace
Human review of meaning and usability

Watch a large production site for recurring problems

Tool category
Crawler or monitoring platform

Common examples
Scheduled site scanners and monitoring services

What it does not replace
A scoped manual audit

Work systematically through checks that need input from a person

Tool category
Guided evaluation tool

Common examples
Accessibility Insights and similar guided tools

What it does not replace
Experienced screen reader and usability testing

These are examples, not endorsements. Many products use the same underlying rule engines, so a larger feature list does not automatically mean better accessibility coverage. The W3C evaluation tools list can help you compare products by platform, standard, scope, and testing method. Developers using Playwright can also see a concrete implementation in its accessibility testing documentation.

For current WCAG testing, WCAG 2.2 Level AA is the usual forward-looking baseline. WCAG 2.2 is the current W3C Recommendation, but laws, contracts, settlements, and procurement requirements may still name WCAG 2.0 or 2.1. Confirm the obligation that applies to your organization, then document which version and level your testing used. The WCAG 2.2 specification explains how the versions relate.

How Much Accessibility Testing Can Be Automated?

There is no universal percentage. Coverage changes according to what is counted: success criteria, problem types, individual issue occurrences, pages visited, or rules supported by a tool. A repeated machine-detectable component defect can dominate an issue count while several judgment-heavy requirements remain untouched.

Two boundaries are more useful for planning. First, ask what the configured rules can decide. Automated checks are strongest where a program can inspect a property and compare it with a defined condition, such as whether an attribute exists or a computable contrast ratio clears a threshold. Even a passing rule may settle only part of a WCAG success criterion.

Second, ask which pages and states the tool reached. Authenticated areas, open dialogs, validation errors, role-specific views, mid-workflow steps, and responsive layouts may all sit outside a basic crawl. A clean scan means that no configured rule reported a failure on the states reached at that time. It does not mean every WCAG requirement passed or that complete user journeys work with assistive technology.

What a clean scan covers A grid. Columns: the pages and states the scan reached, and those it did not reach. Top row, checks the rules can make: the rules ran only in the reached column. Bottom row, checks that need judgment: a person has to confirm the outcome. Reached Not reached RULES CAN CHECK Rules ran here Not checked NEEDS JUDGMENT A person has to confirm
A clean scan means that no configured rule reported a failure on the states reached at that time.

The Department of Justice makes the same practical point: a clean automated report does not necessarily mean everything is accessible, and automated checking should be paired with manual review. See its Guidance on Web Accessibility and the ADA.

That limitation does not make automation optional. It makes the result easier to interpret. Use it for fast, repeatable checks at scale, then plan human testing around the criteria, product states, and critical journeys automation cannot settle.

What Automated Accessibility Scans Reliably Catch

Automated scans reliably catch failure patterns with a machine-decidable condition. They are especially valuable when the same check needs to run across many components, pages, and releases.

The table below does not claim that a scanner fully tests each WCAG criterion. It shows the part a high-confidence rule can decide and the part that remains.

Detectable checkWCAG 2.2 connectionWhat automation can decideWhat it cannot decide
An image has no alt attribute1.1.1 Non-text ContentWhether the attribute existsWhether the text communicates the image’s purpose or whether the image should be ignored
The page has no declared language or an invalid language code3.1.1 Language of PageWhether a valid language value is presentWhether that value matches the language actually used
A form control has no associated label or accessible name1.3.1 Info and Relationships; 4.1.2 Name, Role, ValueWhether code exposes a nameWhether the name is clear and useful in context
An ARIA role, state, or property is invalid4.1.2 Name, Role, ValueWhether the accessibility markup follows defined rulesWhether the exposed role matches what the control really does
A page has no document title2.4.2 Page TitledWhether a title existsWhether it describes the page accurately
Computable text colors fall below the required ratio1.4.3 Contrast (Minimum)The ratio when foreground and background colors can be resolvedText over images, gradients, video, and whether an exception applies
A code reference points to a missing elementPotential effect on 1.3.1 and 4.1.2Whether identifiers and references resolveThe actual user impact and whether the intended relationship is correct

An image has no alt attribute

WCAG 2.2 connection
1.1.1 Non-text Content

What automation can decide
Whether the attribute exists

What it cannot decide
Whether the text communicates the image’s purpose or whether the image should be ignored

The page has no declared language or an invalid language code

WCAG 2.2 connection
3.1.1 Language of Page

What automation can decide
Whether a valid language value is present

What it cannot decide
Whether that value matches the language actually used

A form control has no associated label or accessible name

WCAG 2.2 connection
1.3.1 Info and Relationships; 4.1.2 Name, Role, Value

What automation can decide
Whether code exposes a name

What it cannot decide
Whether the name is clear and useful in context

An ARIA role, state, or property is invalid

WCAG 2.2 connection
4.1.2 Name, Role, Value

What automation can decide
Whether the accessibility markup follows defined rules

What it cannot decide
Whether the exposed role matches what the control really does

A page has no document title

WCAG 2.2 connection
2.4.2 Page Titled

What automation can decide
Whether a title exists

What it cannot decide
Whether it describes the page accurately

Computable text colors fall below the required ratio

WCAG 2.2 connection
1.4.3 Contrast (Minimum)

What automation can decide
The ratio when foreground and background colors can be resolved

What it cannot decide
Text over images, gradients, video, and whether an exception applies

A code reference points to a missing element

WCAG 2.2 connection
Potential effect on 1.3.1 and 4.1.2

What automation can decide
Whether identifiers and references resolve

What it cannot decide
The actual user impact and whether the intended relationship is correct

Automation earns a permanent place in the development process because it is:

Your scan output still needs triage. Tools can return false positives, best-practice warnings that are not WCAG failures, and hundreds of duplicate instances created by one broken component. A trustworthy process verifies the result, groups repeated instances by their root cause, and preserves the affected locations so you can retest them.

What Requires Human Accessibility Testing

Human testing is required when the answer depends on meaning, context, sequence, interaction, recovery, or task completion. Software can assist with these checks, but a person has to confirm the outcome.

Human judgment checkWCAG 2.2 connectionWhat the tester must determine
Alternative text communicates the image’s purpose1.1.1 Non-text ContentWhether the text provides equivalent information in this context
Content and keyboard focus follow a meaningful order1.3.2 Meaningful Sequence; 2.4.3 Focus OrderWhether the reading and interaction sequence makes sense during the task
Every action works from a keyboard without trapping the user2.1.1 Keyboard; 2.1.2 No Keyboard TrapWhether a person can reach, operate, and leave every control throughout the workflow
Keyboard focus remains visible and unobscured2.4.7 Focus Visible; 2.4.11 Focus Not ObscuredWhether the user can see where keyboard input will go as layers and components change
Headings, labels, links, and instructions are understandable2.4.4 Link Purpose; 2.4.6 Headings and Labels; 3.3.2 Labels or InstructionsWhether the wording provides enough context to navigate and decide what to do
Form errors reach the user and support recovery3.3.1 Error Identification; 3.3.3 Error SuggestionWhether the message identifies the problem and explains how to correct it
Dynamic updates are announced at the right time4.1.3 Status MessagesWhether important feedback reaches a screen reader without an unexpected focus change
Content remains usable when enlarged or rearranged at narrow widths1.4.4 Resize Text; 1.4.10 ReflowWhether information and functionality survive without overlap, loss, or prohibited scrolling
A drag interaction has a practical alternative2.5.7 Dragging MovementsWhether the same result can be achieved without requiring a drag
A complete sign-in or purchase flow worksMultiple criteria across a complete processWhether someone using the supported assistive technology can understand, recover, and finish

Alternative text communicates the image's purpose

WCAG 2.2 connection
1.1.1 Non-text Content

What the tester must determine
Whether the text provides equivalent information in this context

Content and keyboard focus follow a meaningful order

WCAG 2.2 connection
1.3.2 Meaningful Sequence; 2.4.3 Focus Order

What the tester must determine
Whether the reading and interaction sequence makes sense during the task

Every action works from a keyboard without trapping the user

WCAG 2.2 connection
2.1.1 Keyboard; 2.1.2 No Keyboard Trap

What the tester must determine
Whether a person can reach, operate, and leave every control throughout the workflow

Keyboard focus remains visible and unobscured

WCAG 2.2 connection
2.4.7 Focus Visible; 2.4.11 Focus Not Obscured

What the tester must determine
Whether the user can see where keyboard input will go as layers and components change

Headings, labels, links, and instructions are understandable

WCAG 2.2 connection
2.4.4 Link Purpose; 2.4.6 Headings and Labels; 3.3.2 Labels or Instructions

What the tester must determine
Whether the wording provides enough context to navigate and decide what to do

Form errors reach the user and support recovery

WCAG 2.2 connection
3.3.1 Error Identification; 3.3.3 Error Suggestion

What the tester must determine
Whether the message identifies the problem and explains how to correct it

Dynamic updates are announced at the right time

WCAG 2.2 connection
4.1.3 Status Messages

What the tester must determine
Whether important feedback reaches a screen reader without an unexpected focus change

Content remains usable when enlarged or rearranged at narrow widths

WCAG 2.2 connection
1.4.4 Resize Text; 1.4.10 Reflow

What the tester must determine
Whether information and functionality survive without overlap, loss, or prohibited scrolling

A drag interaction has a practical alternative

WCAG 2.2 connection
2.5.7 Dragging Movements

What the tester must determine
Whether the same result can be achieved without requiring a drag

A complete sign-in or purchase flow works

WCAG 2.2 connection
Multiple criteria across a complete process

What the tester must determine
Whether someone using the supported assistive technology can understand, recover, and finish

What screen reader testing actually involves

A screen reader is software that presents the interface through speech or braille and lets a user navigate using keyboard commands. A structured test session is not someone turning on speech and listening to the home page.

The tester works through agreed tasks using the browser, operating system, and screen reader combinations you put in scope. Examples include signing in, searching, filtering, opening a dialog, submitting a form, correcting an error, or completing checkout. The tester checks:

A keyboard-only pass is also necessary. Keyboard and screen reader use overlap, but they are not interchangeable. Zoom and reflow checks cover another set of barriers. Broader browser, device, and operating-system coverage can be coordinated with web and software platform testing.

Two common "passed the scan, failed the person" patterns

A dialog exists but does not receive focus. Its code may expose a valid role and name. When it opens, keyboard focus can remain on the page behind it. A user continues through background content, cannot find the intended action, and loses the context needed to finish.

A dialog that does not receive focus A page with a dialog open on top of it. The dialog's code exposes a role and a name. Keyboard focus is still on a link in the page behind the dialog. Page Link Dialog Button Focus is still here Code: a role and a name are present Focus: on the page behind it
When it opens, keyboard focus can remain on the page behind it.

An error is announced but says almost nothing. A form can contain a correctly marked error region that a screen reader announces. If the message says only “There was a problem,” the user still does not know which field failed or how to correct it.

What this looked like in a real audit

Case Study

In one accessibility engagement, the audit covered four websites and two online ebook readers. We combined Axe DevTools and Google Lighthouse with manual testing using JAWS, NVDA, and VoiceOver.

The work then continued into developer consultation, a rebuild, and final verification.

Those findings illustrate why issue count alone is a poor priority system. One keyboard trap in the only purchase or reading path can matter more than dozens of repeated low-impact warnings. See the accessibility audit case study for the full engagement.

The Result

The combined evaluation identified critical and high-severity barriers, including keyboard traps in modal dialogs, missing accessible names, meaningful-sequence problems for screen reader users, and reflow and text-resize failures that left pages unnavigable at 200 percent zoom.

Choosing the Right Mix for the Situation

The right testing plan depends on what changed, how important the affected journey is, and what evidence the organization needs at the end.

SituationWhat to do
For routine code and content changesRun automated component and page checks continuously and add human review when a shared component, interaction, or important workflow changes.
For a new form, dialog, checkout, sign-in flow, or custom controlEvaluate every meaningful state with automation, keyboard navigation, screen readers, zoom, reflow, and error recovery before release.
For a major release or redesignScan the agreed surface broadly and complete structured human testing across representative pages and critical end-to-end journeys.
When procurement requests an ACR or VPATDefine the applicable standard, scope, product version, and reporting requirement before testing. The final report must rest on evaluation evidence, not a scanner export.
When time or budget is limitedPrioritize blocked or high-consequence journeys instead of spreading every method thinly across random pages. Record what was not tested so a partial evaluation is not mistaken for a complete conformance picture.

For routine code and content changes

What to do
Run automated component and page checks continuously and add human review when a shared component, interaction, or important workflow changes.

For a new form, dialog, checkout, sign-in flow, or custom control

What to do
Evaluate every meaningful state with automation, keyboard navigation, screen readers, zoom, reflow, and error recovery before release.

For a major release or redesign

What to do
Scan the agreed surface broadly and complete structured human testing across representative pages and critical end-to-end journeys.

When procurement requests an ACR or VPAT

What to do
Define the applicable standard, scope, product version, and reporting requirement before testing. The final report must rest on evaluation evidence, not a scanner export.

When time or budget is limited

What to do
Prioritize blocked or high-consequence journeys instead of spreading every method thinly across random pages. Record what was not tested so a partial evaluation is not mistaken for a complete conformance picture.

The plan should leave a useful result at every stage: fast regression feedback, verified barriers, a remediation path, or a formal point-in-time audit. The required outcome determines the necessary mix.

Where AI Accessibility Testing Fits, and Where It Does Not

AI accessibility testing can increase the speed of the process. It can help run and triage the mechanical first pass, cluster repeated findings, identify patterns across many pages, draft image descriptions, and prepare initial remediation notes.

It does not remove the central boundary. A requirement that depends on meaning, sequence, recovery, or task completion still needs judgment, regardless of what powers the tool.

An AI-generated image description can sound polished while naming the wrong subject, missing why the image is present, or describing an image that should be ignored by a screen reader. An AI-generated severity rating can miss that an ordinary-looking control is the only way to complete a purchase. Fluent output creates a risk of under-review, so AI output should meet the same evidence and verification standard as any other finding.

Overlays are remediation layers, not audit results

An accessibility overlay or widget changes some part of the delivered experience at runtime. It may change presentation, roles, names, or interaction behavior. That makes it another product change to test, not evidence that the complete page or process conforms.

In 2025, the Federal Trade Commission finalized an order involving claims that an AI-powered accessibility product could make any website compliant. The case reinforces the practical rule that automated technology alone cannot establish conformance. See the FTC case summary.

How Automated and Human Testing Combine in a Real Accessibility Audit

A real accessibility audit uses automation for breadth and people for depth. It begins by defining what the evaluation covers and ends with findings a development team can reproduce and act on.

The current WCAG Evaluation Methodology provides a useful backbone: define the scope, explore the product, select a representative sample, evaluate it, and report the findings. In practice, the work looks like this:

01

Define the product boundary and conformance target

Document the pages, applications, third-party content, languages, and product surfaces in scope. Name the WCAG version and level being evaluated.

02

Agree on the browser and assistive-technology baseline

Specify the combinations the product is expected to support. “Tested with a screen reader” is incomplete unless the report says which reader, operating system, browser, and product states were tested.

03

Explore the product and select representative coverage

Include shared templates and components, unusual content types, and every step of critical processes. Make authenticated areas, dialogs, errors, and responsive states explicit.

04

Run and triage the automated layer

Scan the agreed pages and reachable states. Verify findings, remove false positives, and group repeated instances by their underlying component or template problem.

05

Perform structured human testing

Complete important workflows with keyboard-only navigation and the screen readers in scope. Add zoom, reflow, content structure, focus behavior, error recovery, and dynamic-update checks.

06

Map, prioritize, report, and retest

Tie each verified finding to the applicable WCAG criterion, affected component or page, reproduction steps, evidence, user impact, severity, and remediation direction. Prioritize blocked journeys and shared root causes, then verify the fixes.

Our fixed-scope accessibility audit produces four documents for different stakeholders:

  • An audit report with findings, evidence, severity, and recommended fixes.
  • An Accessibility Conformance Report, commonly delivered using a VPAT template.
  • A prioritized remediation roadmap.
  • An executive summary for leadership and legal or procurement stakeholders.

The audit is a point-in-time assessment. Remediation is separate work, and the next product change falls outside the original evidence until it is tested. An audit documents conformance status and gaps within its defined scope. It does not guarantee legal immunity, and whether your organization has met a legal obligation is a determination for your own counsel.

Accessibility audit

If you need a documented baseline for a launch, procurement request, complaint, or rebuild, see our accessibility audit service. If the baseline already exists and the team needs help planning, fixing, and retesting, accessibility consulting is the more appropriate next step.

Common Accessibility Testing Mistakes

Treating a clean scan as proof of accessibility. It reports on configured rules and reached states, not the complete experience.

Testing only the home page. Important failures often appear after sign-in, inside dialogs, during validation, or midway through a transaction.

Running more scanners instead of adding judgment. Multiple engines may expand rule coverage, but they do not turn a question about meaning into a computable one.

Handing raw scanner output to developers. Repeated instances should be grouped by root cause, verified, prioritized, and tied to affected journeys.

Using an overlay as the compliance strategy. A runtime layer still needs complete testing, and it does not replace fixing components, content, and workflows.

Writing for the checker instead of the user. An alt value containing a filename can silence a missing-attribute rule while providing no useful information.

Prioritizing by issue count alone. A single barrier that blocks checkout or sign-in may deserve attention before hundreds of minor repeated findings.

Testing once and filing the report. Content, components, third-party code, and releases introduce new risk. Accessibility needs both ongoing automation and recurring human evaluation.

How Think Tank QA Approaches Accessibility Testing

At Think Tank QA, we deliver accessibility testing as a service engagement rather than a software product.

Our audits combine automated scanning with manual and assistive-technology testing led by experienced accessibility engineers. The automated layer handles repeatable mechanical work. We evaluate keyboard behavior, focus, meaning, interaction, screen reader output, zoom, reflow, and complete tasks by hand, and our senior engineers govern the findings and the conformance judgment.

The result is designed for action: verified findings mapped to applicable WCAG 2.1 and 2.2 success criteria, affected instances grouped by root cause, severity tied to user impact, and documentation for development, leadership, procurement, and remediation planning.

A fixed-scope audit is appropriate when the organization needs a point-in-time baseline and formal documentation. Consulting is appropriate when the team needs ongoing help turning findings into fixes and building accessibility into normal delivery. Neither service replaces the organization’s counsel, and neither promises that a product will remain conformant after it changes.

The Operating Rule

Automate what a machine can decide. Put people on meaning, context, interaction, and complete journeys. Read every result according to the rules that ran and the states that were reached.

The question is not whether automated accessibility testing is worthwhile. It is whether you stop where your scanner stops. If a tool claims to do the whole job, look closely at what it actually tested, what it never reached, and who made the final judgment.

The next step

If your team needs a documented baseline, the next step may be a formal audit. If the baseline already exists and the team needs help fixing and retesting findings, accessibility consulting is the better fit.

Frequently Asked Questions About Automated Accessibility Testing

What is automated accessibility testing?

Automated accessibility testing uses software to evaluate websites or applications against defined accessibility rules, primarily WCAG, and report failures a program can detect. Common findings include missing alternative-text attributes, low contrast, invalid ARIA, and controls without accessible names. It is fast and repeatable, but it covers only the rules configured and the product states reached.

What does automated accessibility testing miss?

It misses or only partially evaluates issues that depend on meaning, context, interaction, or task completion. Examples include incorrect alternative text, confusing focus order, keyboard traps that appear during a workflow, unclear error messages, and dynamic updates that a screen reader does not announce properly.

Does a zero-error scan mean a site is accessible?

No. It means that no configured rule reported a failure on the pages and states the tool reached at that time. It does not establish that every WCAG requirement passed or that complete user processes work with assistive technology.

What is manual accessibility testing?

Manual accessibility testing is evaluation performed by a person across realistic tasks. It commonly includes keyboard-only navigation, screen reader sessions, zoom and reflow checks, and review of meaning, instructions, focus, errors, and dynamic behavior. The tester confirms whether the experience works, not merely whether expected code properties exist.

Do accessibility overlay widgets make a site compliant?

Not by themselves. An overlay may change parts of the delivered experience, but it does not establish that complete pages and processes conform. Treat it as another product layer that requires testing, not as a substitute for evaluation and source-level remediation.