Blocking an AI crawler is a decision you cannot measure afterwards
The bandwidth you save by blocking an AI crawler shows up in a log. The citations you lose do not show up anywhere, because nothing reports an answer that did not mention you.
Blocking an AI crawler is a decision you cannot evaluate after the fact, because the two sides of it are not equally measurable. The bandwidth and the request count you save appear in a server log the same day. The citations you lose appear nowhere at all: nothing reports an answer that did not mention you, and no assistant sends a message explaining that it could not read your page. So the decision has to be made in advance, on what the fetch is for, and written down next to the rule.
When somebody proposes blocking AI crawlers, the case is usually built from things that can be counted: requests per hour, bandwidth, the share of traffic that is not human. Those are real and they are cheap to produce. The other side of the ledger has no equivalent, because a citation that never happened leaves no trace on your side of the connection. An argument between a measured number and an unmeasurable one is not a fair argument, and the measured number wins by default.
The asymmetry gets worse over time. Once a retrieval crawler is blocked, its fetch count goes to zero, which looks like the block working. The pages it would have cited are still not cited, and that continues to look like nothing at all. There is no experiment that recovers the counterfactual, because you cannot run your site both ways at once for the same question asked by the same person.
A report saying that an AI crawler has never fetched your pages has at least four readings, and they call for different actions. Market4 records these fetches because the customer's blog and feedback pages are reverse-proxied from our own server, so the requests land where we can see them; analytics never runs for a crawler, because bots execute no JavaScript, and Search Console only knows about Google.
| What the report shows | What it can mean | How to tell |
|---|---|---|
| No fetches from a named agent | A robots rule is blocking it | Read the live robots.txt for that user-agent token |
| No fetches from a named agent | It has simply never come | Check whether other agents fetched the same pages in the same period |
| No fetches at all | The agent is not in the catalogue, so nothing was recorded | Look at the raw request log for user-agent strings nobody classified |
| No fetches at all | The pages are not reachable on the path the crawler would try | Fetch the page yourself from outside, as an anonymous client |
The third row is a deliberate design choice worth knowing about. An agent that is not in the catalogue is not recorded, rather than recorded as unknown, because a row saying that something crawled you is advice nobody can act on. It also means the catalogue goes stale by construction: new agents appear and existing ones are renamed, and the maintenance procedure is to re-read the upstream list whenever somebody reports a bot we do not name.
A page can be named in an AI answer without ranking for the question, but the case is narrow: the assistant answered from its own weights instead of running a search. Retrieval produces a link. Memory produces a name with nothing behind it.
Market4 turns one release note into a changelog page, a blog post, a mail-out and a week of social posts — and then tells you which of them brought anyone back.
The question is not whether to block AI bots but which half to block. A retrieval agent fetches a page in order to answer somebody's question now, and its answer can cite the page, so blocking it removes you from answers nobody will tell you about. A training crawler copies the words into a model released much later, with nothing pointing home, so blocking that one costs no citations. Those are two decisions, and a robots file that treats them as one makes the expensive half of the trade by accident.
Some agents genuinely sit between the two, and pretending otherwise is worse than saying so. Amazonbot both improves Amazon's own services and answers questions put to Alexa; GoogleOther is a general-purpose fetcher. Forcing those into one category invents the exact advice somebody then acts on. The safe default when an agent is unclear is to treat it as retrieval, because the downside of allowing a training crawler you did not mean to allow is nothing.
Because the outcome cannot be measured, the durable record of why a crawler was blocked is the sentence somebody wrote at the time. A robots.txt line with no comment is a decision with no author: six months later nobody knows whether that agent was blocked for bandwidth, for a licensing position, or because it was on a list someone copied. Write what the block is for, and what would make you reverse it.
Not directly. You can measure fetches before and after, which tells you the block worked, and you can watch referrals from assistants, which are sparse and arrive for other reasons too. What you cannot recover is the set of answers that would have cited you and did not, because nothing on your side of the connection records an answer you were left out of. Treat the decision as one to reason about in advance rather than to evaluate later.
Only after separating the two kinds. Blocking training crawlers costs no citations, so if bandwidth is the concern that is where to start. Blocking retrieval crawlers removes the fetch that a citation depends on, which is a real cost that will not appear in any report. If the traffic is genuinely a problem, rate limiting or caching addresses the load without removing the page from the answers.
Either it has not visited, or its user-agent is not in the classification list and so nothing was recorded for it. Recording an unrecognised agent as unknown was deliberately avoided, because a row saying something crawled you cannot be acted on. If you suspect a missing agent, look at the raw request log for user-agent strings that no report mentions, and add the ones that matter.
No, because one company often runs several agents with different purposes, and their user-agent strings share a documentation URL that would match all of them at once. A rule written against the company name blocks the retrieval agent alongside the training one, which is the outcome the separation exists to avoid. Write rules against the exact agent tokens.
Each assistant arrives under its own name, obeys the rules written for that name, and takes away whatever the page contained at fetch time. Your own log is the only place all three can be compared.
Publishing a post sends two independent announcements. One IndexNow request reaches Bing, Yandex, Seznam, Naver and Yep. Google takes no part in IndexNow, so it is told by re-submitting the sitemap.