robots.txt vs noindex: What Is the Difference and When to Use Each

Robots.txt and noindex are the two main tools for controlling what Google can and cannot include in its search results. They look similar on the surface because both can stop a page from appearing in Google. But they work at completely different stages of how Google processes your site.

Using the wrong one is one of the most common and damaging technical SEO mistakes. In 2018, a well-known SEO agency accidentally added Disallow: / to a client’s robots.txt during a site migration. The entire site vanished from Google overnight. It took three weeks to recover because the client had not noticed until traffic had already dropped.

This guide explains exactly how each directive works, when to use which, and the specific scenario where combining them can backfire.

The core difference

How robots.txt works

Your robots.txt file sits at yourdomain.com/robots.txt. It tells search engine crawlers which URLs they are allowed to visit before they ever request the page.

User-agent: Googlebot
Disallow: /admin/
Disallow: /checkout/
Allow: /

When Googlebot sees a Disallow rule matching a URL, it does not visit that URL at all. It has no idea what content is on the page.

The critical implication: If Google has already indexed a URL from a previous crawl before you added the rule, blocking it in robots.txt will not remove it from Google’s index. The page stays indexed. Google just cannot recrawl it to update or remove it.

Robots.txt controls the crawler. It does not control the index.

How noindex works

A noindex directive tells Google: you can visit this page, but do not include it in search results.

It lives either in the page’s HTML head:

<meta name="robots" content="noindex">

Or in the HTTP response header:

X-Robots-Tag: noindex

Google visits the page, reads and processes its content, finds the noindex tag, and then excludes the page from its index. The page is crawled but not shown in search results.

The critical implication: Google still uses the crawl to discover links on the page. A noindexed page can still pass PageRank through its outbound links. A robots.txt-blocked page cannot.

The dangerous combination to avoid

This is where most people go wrong.

Do not put a noindex tag on a page that is also blocked by robots.txt.

If robots.txt blocks a URL, Google cannot read the page at all, which means it cannot read the noindex tag either. The result: the page may stay indexed indefinitely because Google cannot see the instruction to remove it.

John Mueller from Google has confirmed this directly in multiple Google Search Central hangouts. He described it as one of the most frequent errors he sees on large sites during migrations.

If you want to completely remove a page from Google’s index:

1. Remove it from robots.txt so Googlebot can visit the page
2. Add noindex to the page
3. Wait for Google to recrawl and process the noindex tag
4. After you confirm the page is deindexed in Search Console, optionally re-add the robots.txt block if you also want to save crawl budget

The Google Search Console URL Inspection tool will tell you if a page is “blocked by robots.txt” and still indexed. This is a common issue on large sites where robots.txt rules were added after pages were already indexed.

When to use robots.txt

Use robots.txt to save crawl budget and prevent Google from spending time on pages that have no SEO value and should never be indexed:

• Admin and login pages such as /admin/, /login/, /wp-admin/
• Internal search result pages such as /search?q=
• Staging or development environments
• User account pages
• Duplicate content generated by URL parameters when you cannot use canonical tags
• Large volumes of files that generate noise such as JavaScript files or CSS files if not needed

The key rule: only use robots.txt on pages that are genuinely not useful to index and that you do not need Google to read for link discovery purposes.

When to use noindex

Use noindex when Google can crawl the page but you do not want it to appear in search results:

• Thank-you and confirmation pages
• Thin or duplicate content pages that have internal links worth following
• Tag and category archive pages on blogs that add no unique value
• Paginated pages beyond page 1 in some cases
• Privacy policy and legal pages (optional, some prefer to index these)
• Out-of-stock product pages with no alternative content

The key rule: use noindex when crawlability is fine but indexability is the problem.

Quick reference guide

How to check which directive is affecting a page

Use the URL Inspection tool in Google Search Console:

1. Enter the URL you want to check
2. Look at Coverage. It will tell you if the page is Indexed, Blocked by robots.txt, Excluded by noindex, or another status
3. Click “View crawled page” to see what Google actually saw on its last visit

If the status says “Blocked by robots.txt” but the page still appears in search results, that is the dangerous combination described above.

For a broader look at crawl-related issues in Search Console, see how to fix crawl errors in Google Search Console.

Frequently asked questions