Different assistants end up with different pictures of a site because they arrive as different agents, obey the rules written for those agents, and take away whatever the page contained at the moment they fetched it. One may have read a post on the day it went up; another may never have requested it; a third may have been refused by a robots rule written years ago for a crawler that no longer exists. The only record that covers all of them is the server log of the site itself.
Whether an AI agent fetched a particular page is a fact only the server that answered the request can hold. No third-party tool can produce it, because the evidence is a request that arrived at one origin and was answered by it. That is why Market4 records which AI agents fetched which posts: not as a ranking signal, but because the alternative is inferring from an assistant's answer what it had read, which is guesswork built on a summary.
The measurement it produces is bounded in one direction worth naming. "No AI agent has ever fetched this post" means no fetch inside the window the log retains, not since the beginning of time. A report that says otherwise is claiming knowledge of a period it has no rows for.
AI agents divide into two kinds that deserve opposite decisions. A retrieval agent fetches a page to answer somebody's question right now, and the answer it produces can cite the page, so blocking one is self-inflicted invisibility. A training agent takes the text into a model that ships months later with no link back to where it came from, so blocking one costs no citations at all. Both may arrive from the same company, under different names, on the same afternoon.
Some agents belong to neither category cleanly. When an operator documents an agent as doing both jobs, the honest classification is unclear, and a tool that guesses in that situation is inventing a policy on the site owner's behalf. Unclear is a real answer, and the decision it calls for is a human one.
| What differs between assistants | Why | What you control |
|---|---|---|
| Which agent name arrived | Each assistant fetches under its own user agent, and often several | The robots rules you write, per name |
| Whether the fetch was retrieval or training | One answers a question now and can cite you; the other feeds a model that ships later | Which of the two kinds you allow |
| What the page contained at fetch time | A fetcher that does not execute scripts sees the server-rendered markup only | Whether the answer is in the HTML rather than assembled in the browser |
| Whether the page was fetched at all | A page nothing links to is reachable only from a sitemap | Internal links, and which pages are actually linked |
No setting anywhere grants a citation. What a small site controls is the shape of its passages and its own server logs. What makes a paragraph cheap to quote, how to tell whether assistants are fetching your pages, and why a cited page can still show zero referrals.
Market4 turns one release note into a changelog page, a blog post, a mail-out and a week of social posts — and then tells you which of them brought anyone back.
A user agent is a string the client chooses, so any script anywhere can announce itself as a well-known AI crawler and be logged as one. Nothing in a page render does a reverse lookup to check, because that would put a DNS round trip in front of every response. The practical consequence is a wording rule: report these numbers as claimed identities, not as confirmed visits from a named company, and treat the remote address as the only field that was actually observed.
This matters most when the numbers are going into a decision. "Three assistants read our pricing page" is a claim about three strings. "Three requests arrived declaring those names" is what happened, and it is still useful — it is just not the same sentence.
The single largest lever is what is in the HTML the server sends. An assistant that does not execute JavaScript reads the markup it was handed, so an answer assembled in the browser is invisible to it however good the page looks to a person. Putting the answer in the server-rendered markup is the one change that helps every fetcher at once, regardless of which company sent it.
The broader point is that optimising for a particular assistant by name is a poor use of effort, because the names move and the classifications change. What does not move is the underlying requirement: a page that can be fetched, that contains its answer in what the server sends, that is linked from somewhere, and whose access rules deliberately allow the agents whose answers can cite it.
From your own server log, or a tool that reads it. The record of a fetch exists only at the origin that answered the request, so no external service can reconstruct it. What you get is which agent names arrived, which pages they requested, how often, and — just as usefully — which published pages no agent has ever requested inside the retained window.
It depends which kind. A retrieval agent fetches a page to answer a question being asked right now and the answer can cite the page, so blocking one removes you from answers you might have appeared in. A training agent takes text into a model that ships later with no link back, so blocking one costs no citations. Decide per kind, not per company.
Treat it as claimed rather than confirmed. A user agent is a string the client picks, so anything can announce itself as a well-known crawler, and verifying each one would mean a reverse DNS lookup on every page render. The remote address is the only field actually observed, so report agent counts as claimed identities and keep decisions robust to a proportion being wrong.
Because they may have read different pages, at different times, in different states. One may have fetched a post the week it was published, another may never have requested it, and a third may have been refused by a robots rule. Add differences in what each one executes when it fetches, and two assistants can hold genuinely different pictures of the same site.
The bandwidth you save by blocking an AI crawler shows up in a log. The citations you lose do not show up anywhere, because nothing reports an answer that did not mention you.
A mention with no hyperlink cannot appear in a referral report, because attribution needs a click that carries a referrer. The measurement that does work is a branded-query baseline in Search Console, taken before the mention runs.
