Crawling & Indexing: The Complete Guide

How Crawling Works

Search engines discover new pages in two main ways: by following links from already-known pages, and by processing submitted sitemaps. Googlebot starts from a seed set of known URLs and follows every link it encounters, building a map of the web over time.

The crawl process for a single page looks like this:

Googlebot requests the URL. If robots.txt disallows the path, it stops here.
The server responds with HTML. Googlebot notes the status code (200, 301, 404, etc.).
Googlebot parses the HTML and extracts links, adding any new URLs to its crawl queue.
If the page uses JavaScript to render content, Googlebot may queue the page for a second-wave render using a headless browser — but this can take days or weeks.
The page (and its rendered version, if applicable) is sent to the indexing pipeline.

AI crawlers like GPTBot, ClaudeBot, and PerplexityBot follow a similar pattern — they're HTTP bots that request pages, follow links, and respect robots.txt. The difference is in what they do with the content: where Googlebot feeds a ranking algorithm, AI crawlers feed retrieval and generation models.

Crawl Budget

Google defines crawl budget precisely:

"Google defines a site's crawl budget as the set of URLs that Google can and wants to crawl."
— Google Search Central, Crawl Budget Management for Large Sites

It's determined by two factors:

Crawl rate limit: How fast Googlebot is willing to crawl without overloading your server. This is influenced by server response speed and any crawl-rate settings in Search Console.
Crawl demand: How often Google wants to recrawl your pages. Popular, frequently updated pages have high crawl demand; thin or rarely updated pages have low crawl demand.

For most small and medium sites (under 10,000 pages), crawl budget is not a practical concern — Googlebot will discover all your pages in reasonable time. Crawl budget matters when:

Your site has millions of URLs (e-commerce with faceted navigation is the classic case)
Your server is slow to respond, causing Googlebot to throttle crawl rate
A large proportion of your crawlable URLs are low-value (duplicates, thin pages, parameter variants)

Optimizing Crawl Budget

Use robots.txt to block sections of the site that should never be crawled (e.g., /admin/, /cart/, internal search results).
Eliminate or consolidate URL parameter duplicates. Use ? parameter consolidation and canonical tags.
Improve server TTFB — faster responses allow Googlebot to crawl more pages per hour.
Remove faceted navigation from the crawlable URL space if it generates millions of near-duplicate URLs.
Submit an XML sitemap with <priority> and <changefreq> hints for important pages.

How Indexing Works

After a page is crawled, Google's indexing pipeline analyzes it: extracting text, understanding the page's topic, computing quality signals, and deciding whether to store it. Not every crawled page makes it into the index.

Google may decide not to index a page because:

The page has a noindex directive
The page is thin, duplicate, or low-quality
The page has no inbound links from indexed pages
The page is blocked by robots.txt (Google can't see its content)
The page returned a non-200 status code on crawl

robots.txt

robots.txt is a text file at the root of your domain (e.g., https://example.com/robots.txt) that tells crawlers which paths they're allowed to visit. It follows the Robots Exclusion Standard.

Example robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

User-agent: GPTBot
Allow: /

Sitemap: https://example.com/sitemap.xml
llms.txt: https://example.com/llms.txt

Key points about robots.txt:

Blocking a path in robots.txt prevents crawlers from visiting it — but does not prevent the page from appearing in search results if it has inbound links. Only noindex removes a page from the index.
If a page is blocked in robots.txt, Google cannot process a noindex meta tag on that page, because it can't see the page's content. This is a common mistake that results in pages appearing in search when you intended to exclude them.
AI crawlers respect robots.txt. Blocking them here is the fastest way to prevent AI systems from accessing your content — but also means your content won't be cited in AI answers.

noindex Directive

The noindex directive tells search engines not to include a page in their index — Google's own documentation puts it plainly:

"Do not show this page, media, or resource in search results."
— Google Search Central, Robots Meta Tag documentation

It can be set two ways:

As an HTML meta tag:

<meta name="robots" content="noindex">
<!-- or more specific: -->
<meta name="robots" content="noindex, nofollow">

As an HTTP response header:

X-Robots-Tag: noindex

The noindex directive must be set on pages you want excluded from Google's index but that you still want crawled. Common appropriate uses:

Thank-you pages and order confirmations
Staging environments accessible to the public
Paginated pages beyond page 2 (with care)
Internal search results pages

warning

Common noindex mistake

If you block a page in robots.txt AND add noindex, Google cannot process the noindex — it never sees the page's HTML. The page may still appear in search results via inbound links. Fix: remove the robots.txt block if you want noindex to be respected. Only use robots.txt blocking for pages you don't want crawled at all.

XML Sitemaps

An XML sitemap is a file that lists the URLs on your site you want search engines to discover and index. It doesn't guarantee indexing, but it helps crawlers find pages that might otherwise be missed — especially pages with few inbound links.

A well-structured sitemap:

Lists only canonical, indexable URLs (don't include noindex pages or URLs blocked by robots.txt)
Sets accurate <lastmod> dates (don't set all pages to today if they weren't updated)
Uses <priority> to signal relative importance to your site (Google treats it as a hint, not a directive)
Submits news and video content via appropriate sitemap extensions

Submit your sitemap via Google Search Console (Sitemaps report) and include the URL in your robots.txt file.

AI Crawlers and the llms.txt Standard

In 2024 and 2025, a new category of crawlers has become relevant: AI training and retrieval bots. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and others now crawl the web to power AI answer systems.

These crawlers respect robots.txt, and many site owners have inadvertently blocked them by setting Disallow: / for all user-agents. If you want your content to appear in AI-generated answers, you need to explicitly allow these bots.

Beyond robots.txt access, a new standard — llms.txt — has emerged to help AI systems understand your site's structure and content hierarchy. An llms.txt file at your site's root describes your site's key content, tools, and sections in Markdown format optimized for AI retrieval.

auto_stories

arrow_forward

How to Check Whether Your Pages Are Indexed

Google Search Console → Pages report: The most accurate view. Shows which pages are indexed, which are excluded, and the reason for exclusion. Start here.
site: search operator: site:yourdomain.com returns an estimate of indexed pages. Useful for a quick sanity check but not precise enough for diagnosis.
URL Inspection tool (GSC): Inspect a specific URL to see its crawl status, last crawl date, indexing status, and any detected issues.
SearchVitals audit: Checks for common indexing blockers: noindex tags, robots.txt blocks on indexable content, missing canonical tags, and redirect chains that confuse crawlers.

Common Index Coverage Issues (and What They Mean)

The Google Search Console Pages report labels every excluded URL with a specific reason. Not every exclusion is a problem to fix — some are Google working exactly as intended, and treating every excluded URL as a bug to chase is a common way SEOs waste time on pages that were never meant to be indexed in the first place.

GSC label	What it means
Crawled — currently not indexed	Google fetched the page but decided against indexing it, usually a content-quality or duplication judgment. The most common label for thin or low-value pages — worth reviewing whether the page adds anything a similar, already-indexed page doesn't.
Discovered — currently not indexed	Google knows the URL exists (usually from a sitemap or internal link) but hasn't crawled it yet. Often a crawl-budget or priority issue rather than a content problem — internal links from already-indexed pages help.
Duplicate without user-selected canonical	Google found near-identical content across multiple URLs and picked its own canonical — not necessarily the one you'd have chosen. Add an explicit canonical tag to regain control.
Alternate page with proper canonical tag	Not an error. This page correctly points to a canonical elsewhere via a canonical tag, and Google is honoring it as intended.
Excluded by 'noindex' tag	The page explicitly told Google not to index it. Check whether that was intentional — noindex tags left over from staging, or applied by a plugin default, are a common accidental cause.
Soft 404	The page returns HTTP 200 but Google's content analysis concluded it looks like a "not found" page — common on empty search-results pages, or on sites where a missing-page fallback (e.g. a single-page app's catch-all route) returns 200 instead of a real 404 status.
Blocked by robots.txt	Google can't crawl the page at all, so it can't be evaluated for indexing regardless of quality. Check whether the block was deliberate.

Frequently Asked Questions

How long does it take for a new page to get indexed? expand_more

Indexing time varies significantly. A new page on a well-established site with good crawl frequency may appear in Google within hours to days. A new site with no inbound links and no sitemap submitted might take weeks or months. Submitting your URL via the URL Inspection tool in Google Search Console can speed up discovery. Having internal links pointing to the new page from already-indexed pages is the most reliable way to accelerate indexing.

What is the difference between crawling and indexing? expand_more

Crawling is when a bot downloads and reads your page. Indexing is when Google analyzes that content and decides to store it in its database. Every indexed page was crawled first, but not every crawled page is indexed — Google may decide a page is too thin, duplicative, or low-quality to include. You can't rank for a query unless your page is indexed.

Why is my page crawled but not indexed? expand_more

The most common reasons include: the page is thin or duplicate content; the page has very few or no inbound links; the page has a soft 404 (returns 200 but displays "page not found"); the page is included in robots.txt block which prevents noindex processing; or Google's quality assessment found the page low-value. Check the Google Search Console Pages report for the specific exclusion reason.

Can robots.txt block AI crawlers? expand_more

Yes. AI crawlers like GPTBot, ClaudeBot, and PerplexityBot respect robots.txt. A Disallow rule for these bots will prevent them from crawling your pages, which means your content won't be used in AI-generated answers from those systems. Many sites accidentally block all AI crawlers with a catch-all Disallow: / rule. Check your robots.txt if you want your content to be citeable in AI answers.

Does submitting a sitemap guarantee indexing? expand_more

No. A sitemap only tells Google a URL exists and roughly how important you consider it — it's a discovery signal, not a directive. Google still independently evaluates each page for quality, duplication, and relevance before deciding to index it, exactly the same way it would for a URL found through a normal link.