Crawling & Indexing: The Complete Guide

How Crawling Works

Search engines discover new pages in two main ways: by following links from already-known pages, and by processing submitted sitemaps. Googlebot starts from a seed set of known URLs and follows every link it encounters, building a map of the web over time.

The crawl process for a single page looks like this:

1. Googlebot requests the URL. If robots.txt disallows the path, it stops here.
2. The server responds with HTML. Googlebot notes the status code (200, 301, 404, etc.).
3. Googlebot parses the HTML and extracts links, adding any new URLs to its crawl queue.
4. If the page uses JavaScript to render content, Googlebot may queue the page for a second-wave render using a headless browser — but this can take days or weeks.
5. The page (and its rendered version, if applicable) is sent to the indexing pipeline.

AI crawlers like GPTBot, ClaudeBot, and PerplexityBot follow a similar pattern — they're HTTP bots that request pages, follow links, and respect robots.txt. The difference is in what they do with the content: where Googlebot feeds a ranking algorithm, AI crawlers feed retrieval and generation models.

Crawl Budget

Crawl budget is the number of pages Googlebot will crawl on your site within a given time period. It's determined by two factors:

• Crawl rate limit: How fast Googlebot is willing to crawl without overloading your server. This is influenced by server response speed and any crawl-rate settings in Search Console.
• Crawl demand: How often Google wants to recrawl your pages. Popular, frequently updated pages have high crawl demand; thin or rarely updated pages have low crawl demand.

For most small and medium sites (under 10,000 pages), crawl budget is not a practical concern — Googlebot will discover all your pages in reasonable time. Crawl budget matters when:

• Your site has millions of URLs (e-commerce with faceted navigation is the classic case)
• Your server is slow to respond, causing Googlebot to throttle crawl rate
• A large proportion of your crawlable URLs are low-value (duplicates, thin pages, parameter variants)

Optimizing Crawl Budget

• Use robots.txt to block sections of the site that should never be crawled (e.g., /admin/, /cart/, internal search results).
• Eliminate or consolidate URL parameter duplicates. Use ? parameter consolidation and canonical tags.
• Improve server TTFB — faster responses allow Googlebot to crawl more pages per hour.
• Remove faceted navigation from the crawlable URL space if it generates millions of near-duplicate URLs.
• Submit an XML sitemap with <priority> and <changefreq> hints for important pages.

How Indexing Works

After a page is crawled, Google's indexing pipeline analyzes it: extracting text, understanding the page's topic, computing quality signals, and deciding whether to store it. Not every crawled page makes it into the index.

Google may decide not to index a page because:

• The page has a noindex directive
• The page is thin, duplicate, or low-quality
• The page has no inbound links from indexed pages
• The page is blocked by robots.txt (Google can't see its content)
• The page returned a non-200 status code on crawl

robots.txt

robots.txt is a text file at the root of your domain (e.g., https://example.com/robots.txt) that tells crawlers which paths they're allowed to visit. It follows the Robots Exclusion Standard.

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

User-agent: GPTBot
Allow: /

Sitemap: https://example.com/sitemap.xml
llms.txt: https://example.com/llms.txt

Key points about robots.txt:

• Blocking a path in robots.txt prevents crawlers from visiting it — but does not prevent the page from appearing in search results if it has inbound links. Only noindex removes a page from the index.
• If a page is blocked in robots.txt, Google cannot process a noindex meta tag on that page, because it can't see the page's content. This is a common mistake that results in pages appearing in search when you intended to exclude them.
• AI crawlers respect robots.txt. Blocking them here is the fastest way to prevent AI systems from accessing your content — but also means your content won't be cited in AI answers.

noindex Directive

The noindex directive tells search engines not to include a page in their index. It can be set two ways:

As an HTML meta tag:

<meta name="robots" content="noindex">
<!-- or more specific: -->
<meta name="robots" content="noindex, nofollow">

As an HTTP response header:

X-Robots-Tag: noindex

The noindex directive must be set on pages you want excluded from Google's index but that you still want crawled. Common appropriate uses:

• Thank-you pages and order confirmations
• Staging environments accessible to the public
• Paginated pages beyond page 2 (with care)
• Internal search results pages

XML Sitemaps

An XML sitemap is a file that lists the URLs on your site you want search engines to discover and index. It doesn't guarantee indexing, but it helps crawlers find pages that might otherwise be missed — especially pages with few inbound links.

A well-structured sitemap:

• Lists only canonical, indexable URLs (don't include noindex pages or URLs blocked by robots.txt)
• Sets accurate <lastmod> dates (don't set all pages to today if they weren't updated)
• Uses <priority> to signal relative importance to your site (Google treats it as a hint, not a directive)
• Submits news and video content via appropriate sitemap extensions

Submit your sitemap via Google Search Console (Sitemaps report) and include the URL in your robots.txt file.

AI Crawlers and the llms.txt Standard

In 2024 and 2025, a new category of crawlers has become relevant: AI training and retrieval bots. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and others now crawl the web to power AI answer systems.

These crawlers respect robots.txt, and many site owners have inadvertently blocked them by setting Disallow: / for all user-agents. If you want your content to appear in AI-generated answers, you need to explicitly allow these bots.

Beyond robots.txt access, a new standard — llms.txt — has emerged to help AI systems understand your site's structure and content hierarchy. An llms.txt file at your site's root describes your site's key content, tools, and sections in Markdown format optimized for AI retrieval.

How to Check Whether Your Pages Are Indexed

• Google Search Console → Pages report: The most accurate view. Shows which pages are indexed, which are excluded, and the reason for exclusion. Start here.
• site: search operator: site:yourdomain.com returns an estimate of indexed pages. Useful for a quick sanity check but not precise enough for diagnosis.
• URL Inspection tool (GSC): Inspect a specific URL to see its crawl status, last crawl date, indexing status, and any detected issues.
• SearchVitals audit: Checks for common indexing blockers: noindex tags, robots.txt blocks on indexable content, missing canonical tags, and redirect chains that confuse crawlers.

Frequently Asked Questions