do_not_disturb_on Indexability & Header Diagnostic

Noindex Checker

Check up to 20 URLs at once for X-Robots-Tag headers and <meta name="robots"> directives — including none, per-crawler rules and redirect chains. No signup, no email.

link Enter URLs to Check (Max 20)

0 / 20

Paste 1 to 20 URLs (one URL per line). Both HTTP response headers and HTML body will be fetched.

How Noindex Detection Works

Search engines check two distinct places for indexing directives. This tool inspects both simultaneously.

http

1. HTTP Response Header (X-Robots-Tag)

Server-level header directive sent before any HTML content is rendered. Commonly used for non-HTML files (PDFs, images) or global web application routes (e.g. X-Robots-Tag: noindex, nofollow).

HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: noindex

code

2. HTML Meta Robots Tag

HTML element placed in the page <head> block instructing crawlers like Googlebot, Bingbot, or Baiduspider not to index the specific page.

Where the directive is actually coming from

Knowing a page is noindexed is the easy half. The directive can be written by a CMS setting, an SEO plugin, a hosting platform, or a line in a server config, and the page source only shows you the ones written into the HTML. If the result above says the directive came from the HTTP header, nothing in your CMS will show it — start from the server and CDN rows.

One cause is worth checking before all the others: a staging configuration that reached production. Staging sites are routinely noindexed on purpose, and when that config, theme, or environment variable is promoted along with a release, the live site goes silently out of the index. If a site dropped out shortly after a deploy, look there first.

Platform	Written as	Where to look
WordPress	Meta tag	Settings → Reading → "Discourage search engines from indexing this site". This one checkbox noindexes the entire site and is the most common cause of a whole WordPress site vanishing.
Yoast / Rank Math	Meta tag	The post or term editor's Advanced panel ("Allow search engines to show this post?"), plus the global rules for archives, tags and author pages, which are set once and then forgotten.
Shopify	Meta tag	theme.liquid and the template files, where conditional blocks noindex search results and filtered collections. A store still under password protection is also noindexed.
Webflow	Meta tag	Page Settings → SEO for a single page, and Site Settings → SEO for the "Disable subdomain indexing" switch that governs the .webflow.io staging domain.
Wix / Squarespace	Meta tag	The per-page SEO panel, in the option that hides the page from search results.
Next.js / Nuxt	Either	The route's metadata export (robots: { index: false }) or head config for the tag; the headers() rules in next.config.js / server middleware for the header.
Vercel / Netlify	Header	Preview and branch deployments are given an X-Robots-Tag: noindex by the platform. This is intended, and only a problem when a preview URL gets linked or submitted as if it were production.
Nginx	Header	add_header X-Robots-Tag in the server or location block. Note that an add_header inside a location block replaces the ones inherited from the server block rather than adding to them, so the directive can appear and disappear by path.
Apache	Header	Header set X-Robots-Tag in .htaccess or the vhost, frequently inside a FilesMatch block aimed at PDFs that ends up matching more than intended.
Cloudflare	Header	Transform Rules → Modify Response Header, and any Worker that rewrites responses. Headers added here are invisible to the origin server, so nothing on the server explains them.

After you remove a directive, the page will not come back until Google recrawls it and sees that it is gone. Re-run the check here to confirm the directive is really no longer being served — a cached CDN response can keep sending the old header for some time — then request indexing in Search Console for the URLs that matter.

Every robots directive, and what it actually does

Only two values keep a page out of the index. The rest change how a page is presented in results, and are routinely mistaken for indexing controls. Any of these can appear in a meta robots tag or an X-Robots-Tag header, and the checker reports the ones it finds.

noindex Blocks indexing

Keeps the page out of search results. The page is still crawled — that is the only way the directive can be read.

none Blocks indexing

Shorthand for noindex, nofollow. Blocks indexing without containing the word "noindex", which is why naive checkers miss it.

nofollow Presentation only

Do not follow the links on this page. Says nothing about whether the page itself is indexed.

noarchive Presentation only

No cached copy is offered in results. The page still ranks normally.

nosnippet Presentation only

No text snippet or video preview is shown. The result becomes title and URL only, which usually costs clicks.

max-snippet:[n] Presentation only

Caps the snippet at n characters. max-snippet:0 is equivalent to nosnippet; -1 means no limit.

noimageindex Presentation only

Images on the page are not indexed. The page itself is unaffected.

notranslate Presentation only

No translation of this result is offered to users reading in another language.

unavailable_after:[date] Blocks indexing

Drops the page from results after the given date. It is a scheduled noindex, and being dated it fails quietly and long after it was written.

noindex vs nofollow vs robots.txt disallow

These three get swapped for one another constantly, and the mistake is expensive in one specific direction: disallowing a page in robots.txt does not remove it from the index. A blocked page is never fetched, so its noindex is never read, and the URL can sit in results indefinitely with no description.

	noindex	nofollow	robots.txt Disallow
Where it lives	Meta tag or HTTP header	Meta tag, HTTP header, or a link attribute	The robots.txt file
Page gets crawled	Yes — required	Yes	No
Page appears in results	No	Yes	Possibly, with no description
Links on it followed	Yes, unless nofollow is also set	No	No — the page is never read
Crawl budget spent	Yes	Yes	No
Reach for it when	A page should not appear in results	You do not want to endorse the links on a page	A section should not be crawled at all, and is already out of the index

How this check is performed, and what it cannot see

Each URL is requested once over HTTP, and the response headers and raw HTML are parsed. Directives are read from the X-Robots-Tag header and from meta robots tags in the document <head>; a robots meta tag placed in the body is ignored, matching how search engines treat it. The origin's robots.txt and sitemap.xml are fetched once and reused across every URL on that host.

User-agent: SearchVitals-Bot/1.0. Directives aimed at a named crawler are matched against Google's agent names, so a rule written for googlebot is reported as applying.
Redirects: followed manually, up to 3 hops. The verdict describes the final URL, and the chain is shown so a directive is never attributed to the wrong page.
Timeouts: 4 seconds to connect, 6 seconds in total per URL. A slow origin is reported as a timeout rather than guessed at.
Concurrency: at most 3 requests in flight per host, to avoid tripping rate limiting and reading a challenge page as though it were your content.
Limits: 20 URLs per run.

The main limitation: JavaScript is not executed. This tool reads the HTML your server returns, which is what a crawler reads on its first pass. If a directive is injected into the DOM by client-side JavaScript, it will not appear here — and if Search Console reports a noindex you cannot find in the page source, that is a likely explanation. Certificate validation is enforced, so a site with an invalid certificate is reported as an error rather than silently fetched.

Frequently Asked Questions

What is the difference between HTML meta noindex and HTTP X-Robots-Tag?

expand_more

The HTML tag works inside HTML documents. The HTTP response header X-Robots-Tag: noindex works for non-HTML files like PDFs, images and API responses as well as web pages. This tool checks both, because a header set at the server or CDN never appears in the page source people look at.

Why are my pages dropped from Google index despite no meta tag?

expand_more

A common hidden cause is an X-Robots-Tag: noindex HTTP header sent by a web server (Nginx, Apache) or CDN middleware (Cloudflare). Another is a robots.txt rule: if crawling is disallowed, Google never fetches the page, so a noindex on it is never read — and the page can linger in the index without a snippet instead of being removed.

Does content="none" mean the same as noindex?

expand_more

Yes. Google treats content="none" as equivalent to noindex, nofollow. It is easy to miss because the word "noindex" does not appear in the tag at all, and a tool that simply searches the page for that string will report the page as indexable. This checker parses directives individually, so none is caught.

How long does it take to get re-indexed after removing noindex?

expand_more

There is no fixed timeline. Google has to recrawl the page before it can see the directive is gone, so the wait depends on how often the page is crawled — days for a frequently updated site, weeks for a rarely crawled one. Requesting indexing in Search Console for an important URL usually speeds it up.

What is the difference between noindex and nofollow?

expand_more

noindex keeps a page out of search results. nofollow tells crawlers not to follow the links on it. They are independent: noindex, follow keeps a page out of results while still letting link equity flow onward, which is the usual choice for pages you want crawled but not listed.

Should I use noindex or a robots.txt disallow?

expand_more

Use noindex to keep a page out of results, and robots.txt to stop a page being crawled at all. Do not use both on the same URL: a disallowed page is never fetched, so its noindex is never seen, and the page can stay in the index with no description. If a page is already indexed and you want it gone, allow crawling and use noindex.

Why does Search Console say "Submitted URL marked noindex"?

expand_more

The URL is listed in your sitemap while also carrying a noindex directive — the site is asking Google to index a page it has been told not to index. Either remove the noindex or remove the URL from the sitemap. This checker flags that combination directly.

Should paginated, archive or tag pages be noindexed?

expand_more

Usually not by default. Google handles pagination reasonably well, and noindexing archive pages can cut off crawl paths to the content they link to. Reach for noindex when a page has no standalone value to a searcher — internal search results, thin filter combinations, staging duplicates — rather than by page type.

Does noindex waste crawl budget?

expand_more

A noindexed page still gets crawled, since the directive can only be read by fetching the page. On a large site that is real crawl budget spent on pages that will never rank. If a section should not be crawled at all, robots.txt is the right tool — but only once those pages are already out of the index.

Do links on a noindexed page still pass value?

expand_more

While the page is still being crawled, yes, if the directive is noindex, follow. Google has said that over the long term it tends to treat a persistently noindexed page as nofollow as well, because it crawls the page less and less. Do not rely on a noindexed page as a lasting part of your internal linking.

Does blocking AI crawlers affect whether my page is indexed?

expand_more

No. Google indexing is governed by Googlebot and your robots directives. Blocking GPTBot or ClaudeBot declines model training and has no effect on Google indexing — or, despite what is often claimed, on whether ChatGPT or Claude can cite you. The agents that decide citation are the retrieval ones: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. Google-Extended is the exception, covering Gemini training and grounding under one token. How to become the source AI answers cite covers what to do about it.

Guidance on this page last reviewed August 29, 2026.