How AI Citations Work
When an AI answer engine answers a question, it does not consult a model's memory alone — it retrieves live pages and cites some of them. That retrieval step is the whole game, and it runs on ordinary web crawlers with ordinary user agents. Which means the first question is not "is my content good enough to cite?" but "is the crawler that does the citing allowed in?"
Those turn out to be different agents at every major provider — and more than one per platform. Conflating them is the most consequential mistake in this entire topic.
The Distinction Almost Everyone Gets Wrong
Every major AI provider runs different user agents for model training and for search retrieval, and the ones that matter for citation are not the ones most SEO advice names. The widely repeated instruction to "block GPTBot to protect your content" targets a training crawler, while the agents that actually fetch your page when someone asks a question often go unexamined.
Two cautions before the table. First, more than one agent is involved in a citation: for ChatGPT, both OAI-SearchBot and ChatGPT-User play a part, and blocking either can cost you. Second — and this is the part almost every guide gets wrong, including earlier versions of this one — no provider has ever stated that the split between training and retrieval is airtight. The documentation tells you what each agent is for. It does not tell you what determines whether you get cited, or whether the systems are as separate in practice as the labels suggest.
| Provider | Involved in retrieval & citation blocking any of these can cost you |
Documented as training-only |
|---|---|---|
| OpenAI / ChatGPT | OAI-SearchBot, ChatGPT-User |
GPTBot |
| Anthropic / Claude | Claude-SearchBot, Claude-User |
ClaudeBot |
| Perplexity | PerplexityBot, Perplexity-User |
— (states it has none) |
| Google — Search & AI Overviews | Googlebot |
— (Google-Extended does not affect these) |
| Google — Gemini | Googlebot, then Google-Extended * |
Google-Extended * |
* Google-Extended appears in both Google-Gemini columns on purpose. It is a single robots.txt token governing training and grounding, so unlike every other provider here, Gemini gives you no way to decline one without losing the other. The section below sets out why.
Each of those assignments comes from the provider's own documentation, not from inference:
"OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." … "GPTBot is used to crawl content that may be used in training our generative AI foundation models."
— OpenAI, Bots and crawlers documentation
Anthropic draws the same line: its documentation describes Claude-SearchBot as navigating the web "to improve search result quality for users," while ClaudeBot collects "web content that could potentially contribute to their training." Perplexity is unusual in having no training crawler at all — it states that PerplexityBot is "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."
The Google case is the most misunderstood
Google AI Overviews are built on the ordinary Search index, so the crawler that matters there is plain Googlebot. Google-Extended is not a crawler at all — it has no separate user agent, the fetching is done by ordinary Googlebot, and the robots.txt token is a permission control layered on top. Blocking it does not remove you from AI Overviews:
"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
— Google Search Central, Google common crawlers
But it is not a training-only switch, and treating it as one is the mistake this section exists to prevent. The same documentation states that Google-Extended governs content use for "training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini" and for grounding in those products. Grounding is the retrieval step — it is how Gemini reaches a live page and cites it.
So blocking Google-Extended does two things at once, and you cannot separate them: it declines Gemini training and removes you from Gemini's grounded answers. Every other provider on the table lets you decline training and stay citable. Google does not. If Gemini visibility matters to you, Google-Extended has to stay unblocked — your content entering Gemini training is the price of it.
What this means in practice: the documented purposes suggest you could decline training while staying citable — but nobody has verified that the separation holds, and the risk is one-sided. Allowing the training agents costs you nothing you can measure. Blocking them buys a benefit you cannot measure either, against a downside — a quieter presence in AI answers — that you would not notice for months and could not attribute if you did. Unless you are protecting paywalled or licensed content, allow all of them. If you do have content worth withholding, block the training agents deliberately, write down why, and watch your mention volume afterwards rather than assuming the labels held.
Check your own robots.txt in two minutes
The failure mode to look for is a catch-all rule that predates any of this, or a blanket "block the AI bots" snippet copied from a blog post. Open your /robots.txt and check whether any Disallow: / reaches the agents in the first column above — either named directly or via User-agent: * with no more specific override. That accidental catch-all is far more common than a deliberate policy.
For most commercial sites the right configuration is the boring one — no AI-specific blocks at all, so every agent in the table can reach you:
User-agent: *
Allow: /Only if you have paywalled or licensed content worth withholding does it make sense to decline training specifically. That looks like this — and note two caveats: it is a bet on a separation no provider has confirmed, and the Google-Extended line costs you Gemini citations as well as Gemini training, because one token covers both:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /Googlebot needs no entry either way — leave it unblocked, as you would for ordinary search. Our free robots.txt tester checks these agents against your live file if you would rather not read the rules by hand.
The path robots.txt cannot govern at all
Everything above assumes the AI system is crawling you. There is a second way your page reaches an answer engine, and robots.txt has no say in it: someone pastes your URL into the chat and asks it to read that page.
Google documents this class separately as user-triggered fetchers — Google-Agent, "used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request", and Google-GeminiNotebook for Gemini Notebook. The property that matters is stated plainly in that documentation: user-triggered fetchers "generally ignore robots.txt rules", because a person asked for the fetch rather than a crawler deciding on its own.
The consequence cuts both ways. If your goal was to keep your content out of AI products entirely, robots.txt was never going to achieve it — a reader can hand your URL to an assistant and the fetch happens regardless. And if your goal is the opposite, a WAF rule that challenges unfamiliar user agents can silently break this path even on a site whose robots.txt allows everything. Only a server or CDN rule governs it, so only a live request can test it.
The same holds elsewhere: ChatGPT-User, Claude-User and Perplexity-User exist for the equivalent user-initiated fetch at their platforms, which is why they sit in the retrieval column of the table above rather than being filed away as incidental.
What Makes a Source Citable
Crawler access is necessary but not sufficient. Once retrieval is possible, the question becomes whether your page is easy to lift a defensible sentence out of.
| Signal | Why it matters |
|---|---|
| Direct one-sentence answers | An answer engine needs a clean, quotable statement it can attribute — buried conclusions get skipped |
| Question-shaped headings | Headings that match how people ask give retrieval an obvious anchor |
| Named sources for every claim | A page that cites primary sources is safer for an AI to repeat than one asserting figures with no provenance |
| Visible author credentials | Attribution and expertise are what separate a citable source from an anonymous page |
| Server-rendered content | Retrieval crawlers are not guaranteed to execute JavaScript — content that only exists after hydration may never be seen |
| A specific, checkable fact | Pages offering a number, a definition, or a procedure get cited; pages offering opinion rarely do |
Note what is absent from that list: keyword density, word count, and any AI-specific markup. Google is explicit that no special schema or AI text file is required — but read that statement for what it is. It is scoped to Google's own AI features, and Google says so: the wording is "to appear in these features." OpenAI, Anthropic and Perplexity publish no equivalent guidance at all. Their crawler docs say what each agent is for; none of them says what earns a citation.
So the honest position is that nobody outside those companies knows where the bar is. That asymmetry argues for doing more than Google's minimum rather than less: clean machine-readable summaries, unambiguous product and entity data, server-rendered content, and an llms.txt are all cheap, none of them harm your Google performance, and they are the only hedge available against platforms that publish no rulebook. The work that makes a page citable is largely the work that has always made a page good — done with an extra eye on whether a machine can extract a single sentence and stand behind it.
Track Whether You're Actually Being Cited
Being citable is only half the equation — you also need to know whether it is working, and citation data is not in Search Console. SearchVitals AI Visibility measures your brand's mention volume across ChatGPT and Google AI Overviews, the keywords triggering those mentions, and how your coverage compares to named competitors.
If you want the mechanics of the Google side specifically — what the data says about clicks, and what Google itself tells publishers to do — see our guide to Google AI Overviews.