Noindex Tag and X-Robots-Tag: When and How to Use Them

The noindex tag tells search engines that a page or file should not appear in search results. It is simple, and easy to get wrong: on the wrong template it quietly removes important pages from Google.

This guide covers how the meta robots tag and the X-Robots-Tag header work, why robots.txt is not a substitute, which page and file types usually belong outside the index, and how to track down an accidental noindex. It ends with a decision matrix you can adapt to your own site.

What the Noindex Tag Does and When You Need It

Noindex means "you may crawl this, but do not show it in search results". Google's robots meta tag documentation describes the rule as: do not show this page, media or resource in search results. It applies to the URL that carries it, not to the whole site or directory.

The cases where noindex is the right tool share one trait: the page exists for users, but arriving at it from a search result would not help anyone. Typical examples:

• Thank-you pages, order confirmations and post-submission screens.
• Internal search results and user-specific lists.
• Profile, tag or category pages that do not yet have meaningful content.
• Campaign landing pages you do not want competing in organic search.
• Staging or preview pages, although password protection is the safer layer there.

Two misconceptions are worth clearing up early. First, noindex is not a privacy control. A noindexed page is still publicly accessible to anyone with the URL; content that must stay private needs authentication, not an indexing rule. Second, noindex is not a shortcut for hiding every page you consider weak. Improving or consolidating that page is often the better decision.

Meta Robots Syntax and Supported Rules

The meta robots tag lives in the head section of an HTML document. Its most common form is a meta element with name set to "robots" and content set to "noindex". A name of "robots" addresses all crawlers; you can target Google specifically by using "googlebot" as the name instead.

Google's documentation on robots meta tags and X-Robots-Tag (Opens in a new tab) lists the supported rules, including:

• all: no restrictions on indexing or serving; this is the default.
• noindex: do not show the page in search results.
• nofollow: do not follow the links on the page.
• none: equivalent to noindex and nofollow together.
• nosnippet: do not show a text snippet or video preview.
• noimageindex: do not index images on the page.
• unavailable_after: stop showing the page after a specified date and time.

Rules can be combined in a comma-separated list or split across several meta tags. When rules conflict, the documentation is explicit: the more restrictive rule applies. If an SEO plugin outputs "index, follow" while the theme template adds "noindex", the page ends up noindexed. So when you check a page, read every robots directive it emits, not just the one you expect to find.

X-Robots-Tag: Index Control Through HTTP Headers

X-Robots-Tag delivers the same rules through an HTTP response header instead of HTML. The meta tag has an obvious limit: it only works in HTML documents. PDFs, images, videos and plain text files have no head section, so they cannot carry a meta tag. The header fills that gap.

The header is sent as "X-Robots-Tag: noindex" in the server response. According to Google's documentation, it accepts the same rules as the meta tag and can optionally name a specific crawler before the rules, so one header line can apply nofollow to googlebot while another applies noindex and nofollow to a different bot. Google's guide to blocking search indexing with noindex (Opens in a new tab) confirms that you can return an X-Robots-Tag header with a value of noindex or none instead of using a meta tag, and points out that this suits non-HTML resources such as PDFs and images.

Headers can be set by directory or file extension, so one rule covers many files without touching templates. The downside is visibility: the page source never shows a header. Make response-header checks a standard part of every audit, or a noindex added by a CDN rule or reverse proxy can sit unnoticed for a long time.

Why robots.txt and Noindex Do Not Work Together

The most common mistake is blocking a page in robots.txt and adding noindex to it at the same time. The two rules cancel each other out.

robots.txt controls crawling; noindex controls indexing. To see a noindex rule, Google has to crawl the page. If robots.txt blocks crawling, Google never reads the page, so it never sees the tag. The robots meta documentation says that in this situation indexing and serving rules will not be found and will be ignored. The blocking-indexing guide adds that a blocked page can still appear in search results, because the noindex was never seen. If other sites link to the blocked URL, it can be listed by its address alone, without its content.

The same guide also notes that putting noindex inside robots.txt is not supported by Google. The order that works is:

1. If you want a page out of the index, allow crawling and add noindex.
2. Let Google recrawl the page and process the rule.
3. Only then, if crawl load is a real concern, consider adding a robots.txt block.

For a structured way to find crawl blocks on the robots.txt side, our robots.txt audit guide for finding crawl blocks complements this article.

Page Types That Usually Belong Outside the Index

A useful rule of thumb: if landing on a page straight from a search result would not help the searcher, the page is a candidate for noindex. Common candidates include:

• Transactional screens: cart, checkout steps, order confirmation, account settings.
• Internal search results: pages that spawn a new URL for every query and often repeat content.
• Low-demand filter and sort combinations: unlimited crossings of colour, size and price filters.
• Empty or half-finished profiles: user, seller or author pages with nothing filled in.
• Pages nobody searches for: some sites also keep utility pages such as cookie settings out of the index.

Keep two cases apart. For filter URLs, noindex is one of the options Google's ecommerce documentation mentions for filtered or re-sorted versions of a listing; the full policy is covered in our guide to ecommerce filter URLs and faceted navigation. Paginated category pages are different. Page two and beyond are the path to different products, and Google's pagination guidance recommends giving each page its own canonical URL rather than hiding it. The details are in our article on SEO for paginated category pages.

Noindex for PDFs, Images and Other Files

Non-HTML files are the most neglected part of most indexing policies. Old price lists, archived catalogues, internal documents and draft PDFs can surface in search the moment something links to them. For these files, the X-Robots-Tag header is the only option.

Three patterns cover most needs:

1. Directory rule: keep files that should stay out of search in one folder and add a noindex header to every response from that folder in your server configuration.
2. Extension rule: keep an entire file type out of the index. Because the scope is broad, define exceptions for any PDF that should remain searchable.
3. Application-level header: if files are served through a script, set the header per file in application code.

Images come with two distinct tools that are easy to confuse. An HTML page that wants its embedded images kept out of the index can use the noimageindex rule; that rule does not remove the page itself. To keep a specific image file out, add X-Robots-Tag to the response for that file. Choose based on whether the problem is the page or the file.

Excluded files that are still linked will keep being crawled; noindex removes them from results, not from the crawl.

The Risk of Adding Noindex with JavaScript

On modern sites, the robots meta tag is sometimes added or changed by JavaScript rather than sent in the server's initial HTML. That creates risk in both directions.

The first risk is trying to switch a page from noindex to indexable with JavaScript. Google's JavaScript SEO basics documentation (Opens in a new tab) warns that when Google encounters a noindex tag, it may skip rendering and JavaScript execution, so the script meant to remove the tag may never run. The same page states that using JavaScript to change or remove a noindex may not work as expected, and that if you want the page indexed, you should not include noindex in the original page code.

The second risk runs the other way. Adding noindex with JavaScript to error or empty-state views is one of the methods Google suggests for avoiding soft 404s in client-rendered apps. That is a legitimate use, but if the condition logic is wrong, noindex can leak onto real content pages.

On JavaScript-heavy sites, set indexing rules on the server wherever possible, either in the initial HTML or in the response header. For a broader look at how rendering works and where it tends to break, read our guide to JavaScript SEO, rendering and indexing. When auditing, check the raw HTML and the rendered HTML separately.

How to Find an Accidental Noindex

An accidental noindex is costly but quick to fix once found. It usually follows a release, theme update or plugin change.

Imagine a fictional travel publisher that moves its site behind a new CDN. To keep the preview hostname out of search, the team adds a rule that sends "X-Robots-Tag: noindex" on preview traffic. A misconfigured hostname match applies the rule to the production domain too. Nothing changes in the page source, so the content team sees nothing wrong. Weeks later, organic traffic to recently updated guides starts to slide, and only a check of the raw response headers reveals the cause.

To catch this kind of error early, build these checks into your routine:

• Watch the "URL marked 'noindex'" reason in the Page indexing report in Search Console, and look for page types you did not expect to see there.
• In the URL Inspection tool, check the "Indexing allowed?" field for your critical page templates.
• Configure your crawler to collect both meta robots tags and X-Robots-Tag headers.
• After every release, test robots rules automatically on the homepage, category templates and key content templates.

To separate noindex from the other reasons a page might be missing from Google, follow the flow in our guide to diagnosing pages Google is not indexing.

What Happens After You Remove Noindex

Removing noindex does not put a page back in the index automatically or instantly. Google has to recrawl the page to see the change, and then the page is evaluated for indexing like any other URL.

Google's blocking-indexing guide notes that it can take a long time for Googlebot to revisit a page. Rather than waiting for a specific date, work through these steps:

1. Confirm the page no longer carries noindex in the raw HTML or the response headers.
2. Run "Test live URL" in the URL Inspection tool to confirm the page now appears indexable.
3. Use "Request indexing" for a few important URLs. Google notes that submitting a request does not ensure the page will be indexed and that requests are subject to a daily limit.
4. Make sure the page is in your sitemap and reachable through internal links.
5. Track the relevant reason in the Page indexing report over the following weeks.

If noindex sat on a page for a long time, the page may not regain its previous visibility right away. That is not a penalty; Google is rediscovering and reassessing the page.

Checking Noindex Before Choosing a Backlink Target

Before you invest in links to a page, confirm that the page can actually be indexed. A noindexed landing page will not appear in search results, so most of the visibility those links could support has nowhere to go. Whatever happens to links pointing at noindexed pages, an indexable target is the sound starting point.

Run these checks on every link target:

• The page returns a 200 status and does not redirect.
• There is no robots meta tag containing noindex or none in the head.
• The HTTP response has no X-Robots-Tag header containing noindex.
• The page is not blocked by robots.txt.
• The canonical tag points to the page itself or to a deliberately chosen target.
• A post-release check is scheduled so a template or plugin change cannot slip noindex in during the campaign.

To assess the target page for content, speed and conversion as well, combine these checks with the checklist in our article on landing page readiness before building backlinks.

Sample Template: Noindex and Crawl Control Decision Matrix

The sample template below summarises which tool to use for crawl and index control across common page and file types. Treat the rows as general recommendations and adapt them to how your site is built.

A simple routine keeps the matrix useful: list every page and file type with its URL pattern, decide whether a searcher should ever land on it, choose the index control (meta tag for HTML, header for everything else), and only then fill in the crawling column. Revisit the table whenever a new plugin, theme or CDN rule goes live, because those are the changes most likely to alter your policy without anyone noticing.

Frequently Asked Questions

Can noindex and nofollow be used on the same page?

Yes. You can combine them in a comma-separated list or use the "none" rule, which Google documents as equivalent to noindex plus nofollow. Keep in mind that nofollow tells Google not to follow any link on the page; in most cases noindex alone is what you actually need.

Does adding noindex help if the page is blocked in robots.txt?

No. When robots.txt blocks crawling, Google cannot read the page and never sees the noindex rule. If you want the page removed, lift the robots.txt block first, let the noindex be processed, and only then decide whether a block is still needed.

Do noindexed pages still pass value through internal links?

There is no safe absolute answer to this, because the outcome depends on the page and on other signals. The practical approach is to keep important pages that you want to pass value indexable, and reserve noindex for pages that genuinely have no place in search results.

Will a page return to the index as soon as noindex is removed?

No. Google first has to recrawl the page and then evaluate whether it is worth indexing, and there is no fixed timeline for that. Running a live test in URL Inspection and requesting indexing for key URLs makes the process easier to monitor.

Should paginated category pages be set to noindex?

Usually not. Later pages in a series are the path to different products, and Google's pagination guidance recommends giving each page its own canonical URL. Noindex is more often considered for low-demand filter and sort variations.

Should I use noindex or a canonical tag?

Use noindex when a page should not appear in search at all, and a canonical tag when you want one of several near-duplicate pages chosen as the representative URL. Google does not recommend noindex for steering canonical selection within a site, because noindex removes the page from Search entirely. Combining both on one page sends mixed signals.

Conclusion and Next Step

Used well, the noindex tag keeps your index clean and purposeful; used carelessly, it hides your most valuable pages without any visible warning. The core rules are short: control indexing with meta robots or X-Robots-Tag, control crawling with robots.txt, and never stack the two on the same URL. Use headers for non-HTML files, prefer server-side rules on JavaScript sites, and test robots directives automatically after every release.

Once you have a set of indexable, fast and content-ready pages, you can support their authority with links that match your topics and target URLs. Browse our backlink packages and run each chosen target page through the checklist above before you place an order.