AI agent visual regression testing for design systems

On October 21, Trail of Bits published a review of its own merged patches. Across more than 130 merged pull requests, it confirmed 21 regressions: 16 functional, 1 performance and 4 in build or tests. A first agent pass flagged 58 candidates, and only 21 were real, about 36% precision. The lesson is not that agents write bad patches. It is that a change is judged against the bug it fixes, and nobody checks everything the diff touched.

That is a security team looking at server code. Interfaces have it worse. A functional test asks whether the checkout button submits the form. It does not ask whether the button is still the same green as the one on the pricing page.

Augment Code's guide to testing AI-generated UIs describes the failure plainly. Agent edits can pass linting, type checks and functional tests while still shipping layout shifts, broken CSS and styling drift. It is a vendor guide, so treat its benchmark numbers with care. The mechanism it describes matches what any team with a shared component library has seen.

My view is that this is a review problem, not a model problem. Better models will drift less, but they will also be given bigger tasks. The check has to live outside the author.

A reviewer reading a diff sees --primary: #1a7f4b changing to --primary: #1b8250. Both are valid. Neither looks wrong. The reviewer cannot see that the token feeds hover states, disabled states, focus rings and the dark theme.

In shadcn/ui style setups the component source is copied into your repo, which means an agent can edit it directly. Augment notes that a change to --primary reaches every component that references it. One token, many consumers, one screen checked by the person who wrote the prompt.

Agents also vary run to run. Similar prompts can produce different spacing, different token choices and different responsive behavior. A spacing value of 14px is numerically fine and still off a 4px grid. A linter will not object. A person scanning a pull request will not either.

The fix is to stop asking humans to compute rendered output in their heads. Render it, compare it with the last approved render, and let the human judge only the differences.

The mechanism is simple. Playwright's snapshot testing saves a reference screenshot on the first run with expect(page).toHaveScreenshot(), then compares every later run against it. Storybook's visual testing does the same per story. Once enabled, every story becomes a test, and the first run creates the baselines.

Two details decide whether this helps or annoys your team.

First, thresholds. Playwright's per-pixel threshold defaults to 0.2 on a 0 to 1 scale, and maxDiffPixels is unset by default. That default will pass a subtle color shift on a large button. Tighten it for token-level components, loosen it for pages with photography.

Second, environment. Playwright's docs warn that rendering varies with the operating system, browser, hardware and headless mode, and baselines are named per browser and platform. A baseline made on a designer's laptop will fail in CI. Generate baselines in the same container image that runs the checks, or every run opens with false alarms and people learn to ignore the check.

Pixel diffing has a blind spot too. It tells you something changed, not whether the change was intended. That judgment is the part worth spending human attention on.

Teams usually start by screenshotting whole pages. I would start one level down. Shared components carry the blast radius, so cover each state of each one: default, hover, focus, disabled, loading, error. Storybook makes that cheap because the states already exist as stories.

Then add a short list of flows where a visual break costs revenue. For a SaaS product that is sign-up, the first-run screen and billing. For a store it is the product page, the cart and checkout. Five or six screens at two viewport widths is a better start than sixty screens nobody maintains.

Keep the noise out. Disable animations, wait for fonts, and mask regions that change by design, such as timestamps, rotating banners and ads. Playwright lets you move the mouse off interactive elements so hover styles do not leak into the shot, and a custom stylesheet can hide volatile content. Each of these removes a reason for someone to mute the check.

Where agents come in, give them the screenshots too. An agent that can see the rendered result can catch its own mistake before a human does. It should still not be the one to approve a baseline.

The weak point of every visual test suite is the update command. In Playwright it is npx playwright test --update-snapshots. In Storybook you accept the change in the panel. Both are one click, and an agent that sees a red check will happily click it for you.

That turns the test into a rubber stamp. So write one rule down: an agent may propose a baseline change, a person who owns that component approves it. Make the UI check a required status on pull requests, which Storybook's Chromatic integration supports, and put the component owner on review for any file that changes a baseline.

The second half of the rule is about intent. Augment calls the failure "intent drift", where a change is visually clean but diverges from what the component is for. No screenshot diff finds that. Only someone who knows why the button is that size can, which is where a designer earns the review.

There is a catch worth admitting. Visual tests also report your own design lag. If the design file was not updated after a scope change, the test will flag the code as a regression when the design is stale. Fix the source of truth first, then the baseline.

You do not need a platform purchase to begin. Here is a sequence that fits in a week.

Day one, list the ten components and five flows where a visual mistake would cost the most. Day two, write screenshot tests for them in Playwright, or turn on Storybook visual tests if you already keep stories. Day three, pin the browser and container image, and generate baselines there.

Day four, set thresholds. Start strict on tokens and buttons, looser on image-heavy pages, and review the first week of failures to see which are noise. Day five, make the check required, name an owner per component, and write the rule that agents propose baselines while people accept them.

After that, add one habit. When an agent task touches a token file or a shared component, the pull request description must list the screens it expects to change. If the diff shows others, the reviewer has a concrete question to ask.

None of this makes agents safer on its own. It makes their side effects visible, which is what Trail of Bits was trying to do for code. The cost is a few hours of setup and a review step that sometimes says no.