The AI Visibility Holdout Test: Fix Half Your Pages, Then Check What Changed

A vendor’s AI-visibility chart goes up two weeks after a schema rewrite. I want to see the pages they didn’t touch. Without a holdout, that rising line could just as easily belong to an engine update, a fresh crawl, or a change in how the vendor calculates its score.

I would not buy a bid-management tool because conversions rose during Black Week after someone switched it on. I would ask what happened to comparable campaigns left alone. AI-visibility fixes deserve the same test.

The question: did the fix earn citations?

This protocol asks whether a technical change increases verified crawler retrieval and citations in live answers relative to comparable, untouched pages. That is a narrower and more useful question than whether a dashboard’s visibility score went up.

I use a six-week window: two weeks to establish a baseline, then four weeks with one fix applied to half the pages. The other half stays untouched. At the end, compare changes in verified bot fetches and citations against the holdout, not against a screenshot of where the treated pages started.

That distinction matters when evaluating which tools help businesses fix technical SEO issues that hurt AI visibility. A tool should get credit for a change that beats the holdout, not for being installed when Google next refreshes its index.

Why a before-and-after chart cannot answer the question

Search engines change while your test is running

The familiar vendor sequence goes like this: crawl a site, flag 400 “AI-readability errors,” change the schema, and present a citation chart two weeks later. The chart may be accurate. The explanation may not be.

Search engines do not hold their retrieval behavior steady for your campaign calendar. In an analysis of Google citation behavior, Ahrefs reported that 37.1% of URLs cited in Google AI Overviews came from the top 10 organic results, down from 76% in an earlier analysis. Query fan-out can bring different sources into an answer as an engine pursues background sub-queries. A change in that process can move citations even when you change nothing.

Borrow the PPC control, not the PPC vocabulary

Paid search operators know this problem. Switch bidding strategies in November, see conversions rise in December, and you have not yet separated your work from the shopping calendar. You need a simultaneous baseline.

The same logic applies here. As SearchPilot explains in its SEO testing framework, comparable URLs can be grouped using historical traffic, volatility, and topical intent. When both groups face the same external shift, the difference between them becomes more informative than either group’s before-and-after chart.

AI retrieval adds noise of its own. A page can be fetched without being cited; a citation can appear or disappear across repeated answers. That is why this test records both events separately. A fetch is evidence of retrieval, not a citation. Keep that distinction intact all the way to the decision.

Setup: match pages before touching them

Choose 10–20 URLs, then make lookalike pairs

Take 10–20 pages with similar topical depth, template structure, and publication age. Pull the last 60 days of organic clicks from Google Search Console and pair pages whose traffic falls within roughly 15% of each other. Check intent as well as volume. Two pages with similar click totals but entirely different jobs are not a useful pair.

For example, pair a page about “best CRM for commercial contractors” drawing 800 monthly visits with a page about “best ERP for electrical contractors” drawing 750, rather than with your homepage. Do not put five revenue-driving pages on one side and five forgotten posts on the other and call it an experiment. The spreadsheet will look tidy. The comparison will not.

Assign one page in each pair to the holdout

Once the pairs are set, flip a coin for each one. Put one URL in the treated group and the other in the holdout. Record the assignment before making changes, and do not reshuffle pages after you see an early result. SearchPilot’s explanation of counterfactual modeling describes the purpose of matched controls: estimating what the treated group might have done without the intervention.

Freeze holdout pages for the full test. That means no copy or metadata edits, template changes, internal-link additions, or quiet CMS cleanups. Tell the people who can publish changes which URLs are off limits. An accidental edit in week four is not a footnote you can wave away later; it changes the comparison you set out to make.

Test one fix category, not an entire wishlist

Pick one hypothesis before deployment. These are three separate tests, not a package to install on the same afternoon:

• Structural clarity: Rewrite the first 120 words into a direct answer to the page’s primary query, without a long preamble.
• Entity and schema depth: Add precise JSON-LD, such as TechArticle, ItemList, or relevant about declarations, to clarify the page’s topic.
• DOM and rendering simplicity: Replace client-side accordions or heavy tabs with semantic HTML that exposes the relevant text on initial load.

If you rewrite the introduction, add schema, remove tabs, and rebuild the template together, a positive result cannot tell you which change deserves the next rollout. Nor can a negative result tell you what to remove. Choose one intervention and document exactly which treated URLs receive it.

Measurements: keep fetches and citations separate

Count verified bots, not claimed user agents

Filtering server logs for GPTBot or PerplexityBot user-agent strings is not enough. A scraper can claim either name. A crawler-traffic analysis citing HUMAN Security reports spoofed AI crawler requests, and our 30-day bot-traffic log found 3,482 genuine requests among 13,774 claimed bot hits. The exact mix on your site may differ. The practical point is that a raw header is not verification.

Set up the checks before the baseline begins. As Centinel Analytica’s crawler-verification breakdown explains, Googlebot and Applebot use forward-confirmed reverse DNS, while OpenAI and Perplexity publish IP-range files for checking their crawlers. Exclude requests that fail the relevant check. Otherwise, a noisy or spoofed request stream can masquerade as the result of your fix.

Separate background crawling from live-answer fetches

Not every verified fetch has the same meaning. OpenAI distinguishes its bots: GPTBot crawls for model-related purposes, OAI-SearchBot serves search discovery, and ChatGPT-User fetches content in response to a user interaction. Record them separately rather than pouring every request into one “AI traffic” total.

A verified ChatGPT-User request is a useful leading indicator of live retrieval. It still does not show that your URL appeared as a citation in the answer. Presenc’s guide to the bot explains why these fetches matter and why ordinary pageview analytics may miss them. Use server access logs to measure the requests; use answer checks to measure citations. One cannot stand in for the other.

Check citations against a fixed prompt set

Before you change any page, write five specific user-intent prompts for each pair. That gives you 25–50 prompts across a 5–10-pair test. Keep the wording and page-pair assignment fixed throughout.

Every Tuesday at 10:00 AM UTC, run that same battery across Google AI Overviews, Perplexity Pro, and ChatGPT Search. For each prompt and engine, record 1 if the specific URL appears as an inline reference, card, or footnote; otherwise record 0. Keep the answer or a screenshot with the entry so you can inspect a surprising result later.

Fixed prompts, timing, and sequence do not eliminate variation between answers. They do stop you from moving the goalposts by choosing new prompts after one group starts looking good. Ignore the proprietary visibility score for this decision. The outcome is the citation you can see, not the score a dashboard assigns.

Run the six-week protocol

Weeks 1–2: establish the baseline

Do not deploy the fix when you set up the groups. For 14 continuous days, collect daily verified fetches by URL and bot type for both buckets. Run the scheduled citation checks with the fixed prompts. If verification fails or a page changes during baseline, fix the setup before treating anything.

At the end of day 14, divide treated-bucket fetches by holdout-bucket fetches to get a baseline retrieval ratio. Treat roughly 0.85–1.15 as a practical pairing check, not a certificate that the pages are identical. If the prospective treated pages receive twice the verified crawler volume before any intervention, revisit the pairs. A lopsided starting point will make a later lift hard to read.

Keep the baseline citation records too. Fetches and citations can start from different positions, and you will need both starting points when you compare the groups after deployment.

Day 15: change treated pages only

Deploy the one chosen fix to the treated URLs. For an answer-block test, edit only those openings. For a schema test, place the validated JSON-LD on the treated pages only. Purge cached versions of those URLs so crawlers can reach the changed HTML. Record what you deployed and when.

Then freeze both buckets for the remainder of the test. Do not slip in a metadata improvement, replace an image, or add links from a newly published post because it seems harmless. Some of those changes might be good work. They are not part of this test.

Weeks 3–6: watch for drift and contamination

Leave the fix in place for the remaining four weeks. Continue daily verified-log collection and the weekly prompt checks. The post-deployment run gives retrieval and answer systems time to revisit pages; it does not guarantee that every URL will be reconsidered on your schedule.

Keep a simple change log for the bucket pages. If someone edits a holdout page to repair a pricing table in week four, repair the table if it needs repairing, then remove that entire pair from the evaluation. Do not try to explain away the edit with a spreadsheet adjustment. If you began with 10 pairs and lose two, assess the eight uncontaminated pairs and say that is what you did.

Apply the same rule when a treated page receives an unrelated change. A smaller clean comparison is more useful than a larger one whose pages no longer reflect the protocol.

Read the result against the holdout

At the end of week six, compare each group’s change from its own baseline. Look at verified bot fetches by type and at the fixed-prompt citation records. Do not treat a rise in one as proof of a rise in the other.

A fetch advantage without a citation advantage is its own finding: machines may be reaching the treated pages more often without choosing them for answers. Sparse citations across both groups are also hard to interpret. In either case, do not promote a crawler chart into a claim about earned citations.

I would rather get a null result from a clean test than a beautiful graph I cannot explain. If nested schema changes nothing for the treated pages relative to the holdout, that is a reason to stop paying to roll out that fix on similar pages. It is not proof that schema never matters anywhere. The decision is smaller, and much more useful.

Make the tool earn its place

This is the standard I would put in front of any tool claiming to “fix” AI visibility. A platform that tracks prompts, graphs a synthetic index, and exports a monthly PDF is an observer. It may show you something worth investigating. It has not demonstrated that its recommended fix caused a citation.

The work is operational: spot a retrieval problem, make a defined change, verify which bots reach the page, check whether live answers cite it, and compare all of that with pages left alone. That is the standard behind groas, which executes search and content work continuously with a named human strategist setting direction and accountability. But I would apply the same holdout rule to its changes. A brand name does not get to be the control group.

If the treated pages win on verified retrieval and citations, extend the fix to comparable pages and keep checking. If they do not, leave the holdout alone, retire the claim, and spend the next test on a better hypothesis. The slide can wait.