Two mechanisms tell search engines what to do with a site, and they are constantly mistaken for each other. robots.txt decides what a crawler may fetch. The robots meta tag, and its HTTP twin X-Robots-Tag, decide what happens to a page once it has been fetched. Mixing them up is how staging sites end up in Google and how live pages quietly drop out of it.
This article covers what each one does, the syntax worth knowing, and how to choose between them.
Crawling and indexing are separate steps
A search engine first crawls a URL (downloads it) and then decides whether to index it (store it and show it in results). Each mechanism acts on one step only:
| robots.txt | Meta robots / X-Robots-Tag | |
|---|---|---|
| Controls | Crawling | Indexing and how the result is displayed |
| Scope | URL patterns across one host | One page or file at a time |
| Lives in | A text file at the site root | The HTML <head> or the HTTP response headers |
| Keeps a URL out of results? | No | Yes, with noindex |
That last row is the one that surprises people. A URL blocked in robots.txt can still be indexed if other pages link to it. The search engine has never read the page, so it lists the bare URL, usually without a description. Search Console reports this as "Indexed, though blocked by robots.txt".
Which leads to the rule that resolves most confusion on this topic:
A crawler can't obey a tag it isn't allowed to read. If a page must stay out of search results, it has to remain crawlable so the noindex can be seen.
robots.txt: the essentials
The file must sit at the root of the host (https://example.com/robots.txt) and applies only to that exact protocol, host and port. A subdomain needs its own file.
Google supports four fields: user-agent, allow, disallow and sitemap. A typical small-site file looks like this:
User-agent: *
Disallow: /cart/
Disallow: /*?sort=
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlMatching rules that matter
- Paths are prefixes and are case-sensitive.
Disallow: /cartalso blocks/cartography/. Add the trailing slash when a directory is meant. - Two wildcards are available.
*matches any sequence of characters, and$anchors the end of the URL, as inDisallow: /*.pdf$. - The most specific rule wins. When an
allowand adisallowboth match, the longer path takes precedence, regardless of the order in the file. - A crawler follows one group only. If a group names a crawler specifically, that crawler ignores the
User-agent: *group entirely. Rules are not inherited.
That last point causes real mistakes:
User-agent: *
Disallow: /private/
User-agent: Googlebot
Disallow: /drafts/Here Googlebot may crawl /private/, because it reads only its own group. Any shared rules have to be repeated in each named group.
What robots.txt doesn't do
- It doesn't support
noindex. Google stopped honouring that unofficial rule in 2019. - It doesn't throttle Googlebot. Google ignores
crawl-delay, although some other crawlers, Bing's among them, respect it. - It doesn't protect anything. The file is public and compliance is voluntary. Listing
/secret-area/in it mostly advertises that the area exists. Private content needs authentication.
Two operational details are also worth knowing. A missing file (a 404) means "crawl everything", which is fine. A file that returns server errors is treated far more cautiously, and Google may stop crawling the site until it recovers. And Google caches the file, generally for up to 24 hours, so changes are not instant.
Meta robots and X-Robots-Tag
The meta tag goes in the page's <head>:
<meta name="robots" content="noindex">Non-HTML files such as PDFs have no <head>, so the same rules can be sent as a response header instead. On Apache:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>And on nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}The directives worth remembering
| Directive | Effect |
|---|---|
noindex | Keep this page out of search results. |
nofollow | Don't follow the links on this page. |
nosnippet | Show no text snippet or video preview for this result. |
max-snippet:[n] | Limit the text snippet to n characters. |
max-image-preview:[none|standard|large] | Set the maximum size of the image preview. |
noimageindex | Don't index the images on this page. |
unavailable_after:[date] | Stop showing the page after a given date. Useful for events and offers. |
A few practical notes:
index, followis the default. Declaring it does no harm, but it does nothing either.- A specific crawler can be addressed by name, for example
<meta name="googlebot" content="noindex">. - When directives conflict, including a meta tag that disagrees with a header, the most restrictive one applies.
- For a single passage rather than a whole page, the
data-nosnippetHTML attribute excludes just that element from snippets. - Put
noindexin the HTML the server sends. If Google finds it there, it may skip rendering, so JavaScript that removes the tag later is unlikely to be seen.
Which one to use
| Goal | Use |
|---|---|
| Keep a page out of search results | noindex, with the page left crawlable |
| Keep a PDF or image out of search results | X-Robots-Tag: noindex |
| Stop crawlers wasting requests on filters, internal search or cart URLs | robots.txt disallow |
| Remove a page that is already indexed | noindex or a 404/410, then wait for a recrawl. Search Console's Removals tool hides it faster, but only temporarily. |
| Keep content private | A password. Neither mechanism is access control. |
| Keep a staging site out of search | HTTP authentication, ideally with noindex as a second layer |
Five mistakes that cause most of the damage
- Blocking and noindexing the same URL. The
disallowstops the crawler from ever seeing thenoindex, so the page can stay indexed indefinitely. - Shipping staging settings to production. A leftover
Disallow: /or a site-widenoindexis the classic post-launch disaster. In WordPress, check that "Discourage search engines from indexing this site" is unticked under Settings › Reading. - Blocking CSS and JavaScript. Google renders pages much as a browser does. If it can't fetch the assets, it can't see the page properly.
- Expecting named crawlers to inherit the
*rules. They don't, as shown above. - Using
robots.txtto fix duplicate content. Blocked duplicates can't pass their signals anywhere. Canonical tags are the right tool for that job.
A note on AI crawlers
AI companies now publish their own user-agent tokens, and they are controlled in robots.txt like any other crawler. Most operators separate crawlers that collect training data from those that power search-style answers, so the two can be treated differently:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Google-Extended is a control token rather than a separate crawler. It governs use of content for Google's AI models and has no effect on inclusion or ranking in Google Search. Token names change often, so check each operator's documentation before copying a list, and remember that compliance is voluntary here too.
How to check your setup
- Read the file. Open
/robots.txtin a browser. It should be short enough to understand in a minute. - Use Search Console. The robots.txt report (under Settings) shows the version Google last fetched and any parsing problems. URL Inspection shows whether a specific page is blocked or excluded by
noindex. - Inspect the headers.
curl -I https://example.com/file.pdfreveals anyX-Robots-Tag, which is invisible in the page source. - Re-check after every launch, migration or plugin change. These settings rarely break on their own. They break when something else changes.
In short
robots.txt manages where crawlers spend their time. Meta robots and X-Robots-Tag manage what appears in search results. Most small sites need very little of either: a short robots.txt with a sitemap line, and noindex on the handful of pages that shouldn't be found.
When in doubt, go back to the rule: a crawler can't obey a tag it isn't allowed to read.
Further reading
- Technical SEO
- How Google interprets the robots.txt specification (Google Search Central)
- Robots meta tag, data-nosnippet and X-Robots-Tag specifications (Google Search Central)
- RFC 9309: Robots Exclusion Protocol