Preflighting a URL before you announce it anywhere
•6 min read
Publishing writes to a database and tells search engines about an address. Preflighting is the step that asks the server answering that address whether it serves a page. What to check, which findings block an announcement, and why the check must never fail a publish.
Preflighting a URL means fetching it yourself before a sitemap or an IndexNow submission tells a search engine it is there. Everything up to that point writes rows to a database and announces an address; nothing has asked the server that answers that address whether it serves a page. It can answer 404, sit behind a deploy that has not shipped, carry a staging noindex, name a different canonical, or render entirely in the browser — and without a preflight all five get announced identically.
What does a preflight actually check?
The HTTP status, and the address that answered after every redirect hop rather than the one that was requested.
The robots directives, all three of them: the meta robots tag, the googlebot-specific meta tag that overrides it for Google alone, and the X-Robots-Tag response header, which is where a stale CDN rule hides where no page source shows it.
The canonical the page declares, and whether it points at the URL that was announced. A page that canonicalises somewhere else is asking not to be indexed at the address you submitted.
Whether the content is present in the server-rendered HTML or only appears once JavaScript has run.
The title and description the server actually served, rather than the ones stored in your database. Those two disagree more often than anyone expects, and the served version is the one that matters.
Which findings block an announcement, and which are warnings?
A preflight is only useful if it separates "this page cannot be indexed at this address" from "this page is worse than it could be". Seven findings are blocking and seven are not, and mixing the two produces a check nobody acts on because it is always complaining about something.
Fourteen findings, split by whether one alone makes a URL not worth announcing.
Finding
Blocking
What it means
Refused address
Yes
The address was refused before any request: off-host, a bare IP, or a name resolving to a private address
Unreachable
Yes
Nothing answered at all
Redirect loop
Yes
The redirects never settle on a final address
HTTP error
Yes
The server answered with an error status
Noindex
Yes
The page asks robots not to index it
Canonical elsewhere
Yes
The page names a different URL as the canonical one
Empty server HTML
Yes
The server-rendered document carries no real content
Redirected
No
A different address answered; worth knowing, not disqualifying
Not HTML
No
The response was not an HTML document
Nofollow
No
Links on the page are marked not to be followed
Canonical missing
No
The page declares no canonical at all
Expected text missing
No
The title we published was not found in the served HTML
A sitemap and an IndexNow submission answer the same question about a site. Building them from two pieces of code guarantees they will disagree. Here are the three rules both have to share, and the four places that read the shared list.
Market4 turns one release note into a changelog page, a blog post, a mail-out and a week of social posts — and then tells you which of them brought anyone back.
No card to start. Cancel from the settings screen, not from an email.
A clean result is not a promise. It means the obvious ways to be un-indexable were checked and none was found, which is a different and much smaller claim than saying the page is or will be indexed. Anything that reports a preflight as an indexing guarantee is over-reading it, and the gap between the two is where a lot of misplaced confidence lives.
What stops a preflight becoming a crawler?
Every bound on a preflight answers the same question: what stops this being a way to hang a request or pull an unbounded amount of somebody else's site? Twenty-five fetches against one origin is a batch; a hundred is a small crawl, and a publishing tool has no business being a crawler. Anything past the ceiling is returned by name in a skipped list rather than dropped, because an unchecked URL that looks like a checked one is worse than no check at all.
25 URLsMaximum URLs one explicit preflight call checks; anything past it comes back named in a skipped list.
5 URLsURLs the publish path preflights on its own: the blog index plus the four most recently changed posts.
Two more bounds do the rest. Three fetches run at once, which is polite against an origin that belongs to the customer and would otherwise rate-limit a burst of its own traffic, and the whole batch runs under a sixty-second wall clock so a site answering every request in fourteen seconds cannot turn twenty-five URLs into six minutes of held job. The publish-path list is sorted by content change date rather than publish date, because the post edited this morning is the one most likely to be sitting behind a stale deploy.
Why must a preflight never fail a publish?
A customer's origin being slow, firewalled against you or briefly returning 502 must not be able to stop their post going out or their URLs being announced. That would hand a third party a veto over your own publishing path, and the third party in question is often the customer's own reverse proxy in the middle of a deploy. So the check runs after the announcements rather than in front of them, it produces a warning and a record of its own, and a failure inside it is a log line rather than an outage.
The same reasoning explains why the preflight checks exactly the set the sitemap lists and IndexNow announces, built from the same list rather than from a second query. A warning about a URL nobody announced is noise, and a URL announced that nothing warned about is the failure the whole check exists to catch.
The same-host rule is a security boundary
A tool that takes a URL and fetches it is an outbound fetcher pointed at whatever the caller types, which is a server-side request forgery primitive wearing a helpful name. Restricting a preflight to the app's own configured site host is what turns it back into a diagnostic: bare IP addresses and local-only names are refused rather than fetched, and any hostname resolving to a private or link-local address is refused on the first request and on every redirect hop, because a redirect is a second chance to point somewhere else.
What should I check before submitting a URL to a search engine?
Fetch it and look at five things: the status after redirects, the robots directives including the X-Robots-Tag header, the canonical and whether it points at this address, whether the content is in the server-rendered HTML, and the title and description the server actually served. Those five cover the failures that make an announced URL worthless, and all five are invisible from your own database.
Does a clean preflight mean the page will be indexed?
No. It means nothing blocking was found: the page answered, it does not ask robots to skip it, it canonicalises to itself, and its content is in the HTML the server sent. Indexing is a decision a search engine makes for its own reasons, and no check you run on your own site can predict it. Treat a clean preflight as the removal of known obstacles rather than as a forecast.
Where does an X-Robots-Tag hide?
In the response headers, which means it never appears in the page source and no amount of reading the template will reveal it. It is usually set by a CDN rule or a reverse proxy configuration written for a staging environment and never removed. A page can look completely correct in a browser, serve a perfect canonical and a full document, and still carry a noindex directive in a header nobody has read since it was added.
Should a failed preflight stop a publish?
No. The customer's origin being briefly unreachable would then be able to block their own publishing, which puts a veto in the hands of whatever is between you and their server. Run the check after the announcements, record what it found, and surface it as a warning. If the page really is broken, the fix is a deploy on their side, and holding the publish would not have produced it any sooner.
The submit call is a PUT that returns an empty body. Whether Google fetched the file, whether it parsed and how many URLs it holds all come back from a different call, on Google's schedule rather than yours.
Inspecting a URL costs quota. Explaining what the inspection meant does not. The two are separate calls so that the expensive one happens when somebody asked for it.