robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in Google search results if other sites link to it, because Google honors the block and so it can't see the noindex tag you put inside the page.
That single sentence resolves about 70% of robots.txt mistakes I see on client audits.
The file itself is decades old. The format hasn't changed materially since 1994, per the Robots Refresher series introduction. What changed is that almost every robots.txt mistake now costs you AI Mode visibility on top of regular search visibility, because AI crawlers and Search share crawl infrastructure. The stakes are higher than they were five years ago.
This post is the practical reference: what the file does, what it doesn't, the patterns that don't work, and the 2026 additions worth knowing about.
What robots.txt is for
A robots.txt file lives at the root of your domain, at https://example.com/robots.txt, exactly there, with no subdirectories and no subdomains-of-subdomains. It tells crawlers which URLs on your site they may fetch. Per Google's introduction to robots.txt, it is used "primarily to manage crawler traffic to your site, and usually to keep a file off Google."
The protocol behind it is the Robots Exclusion Protocol, now an IETF standard (RFC 9309, ratified 2022). It's a proposed standard, which means it's stable. The basic syntax is four fields: user-agent, disallow, allow, and sitemap. That's all Google supports. Other directives like crawl-delay and clean-param exist in other engines but Google ignores them, per the Robots Refresher post on future-proofing the REP.
A minimal example:
User-agent: Googlebot
Disallow: /admin/User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Three blocks: block Googlebot from /admin/, allow everything else for every other crawler, and tell crawlers where to find the sitemap. That's it, and most sites don't need anything more elaborate.
> robots.txt only controls crawling. To keep a page out of the Google index, use a noindex meta tag or password protection, and don't also block the URL in robots.txt, or Google can't see the noindex.
The crawl-versus-index distinction
Here's where most teams shoot themselves in the foot.
If you put a page behind Disallow: /private/ in robots.txt, Googlebot won't crawl that URL. But Google can still index it. Per Google's docs: "a page that's disallowed in robots.txt can still be indexed if linked to from other sites." The URL appears in results without a snippet, because Google never fetched the page to extract a description. You end up with an ugly "we have no information for this page" search listing for a URL you thought was hidden.
The fix is counterintuitive: to deindex a page, you have to let Google crawl it so it can see a tag (or an X-Robots-Tag: noindex HTTP header). Once Google sees the noindex, the URL drops out of the index. Then, and only then, you can disallow it in robots.txt if you want to stop further crawling.
Per the Robots Refresher post on page-level granularity: "the most restrictive, valid directives apply," but only if Google can see them. Block the crawl and you blind Google to your own deindex instruction.
Three real outcomes, three different tools:
- Manage crawl traffic (server overload, faceted-nav explosion): robots.txt
Disallow - Keep a page out of the index entirely: noindex meta tag or password protection (and crawl must be allowed for Google to see the tag)
- Keep media or non-HTML files out of search:
X-Robots-Tag: noindexHTTP header, which Google supports for PDFs, images, and other non-HTML formats
Pick the wrong tool and you'll spend weeks debugging an outcome the docs explain in two sentences.
CMS-generated robots.txt is usually fine
If your site runs on WordPress, Webflow, Shopify, Wix, Squarespace, Ghost, or any modern CMS, your robots.txt is generated for you. It's almost always fine. Most CMS-default files are short, conservative, and allow full crawling with a sitemap reference, and you usually don't need to touch it.
There are three exceptions worth checking. The first is a faceted-navigation explosion, like an ecommerce filter combinatoric or a calendar that generates infinite future URLs; block the crawl traps with Disallow: /*?filter= or similar pattern rules. The second is search-results pages that are crawlable and ranking poorly; block /search?q= to stop Google chewing on your internal search. The third is a staging environment leaking into production crawlers; block the staging subdomain at the subdomain's own robots.txt, since rules at example.com/robots.txt do not apply to staging.example.com/robots.txt (they're separate files).
If you don't have those specific problems, leave the default alone. Per Gary Illyes and Martin Splitt on Search Off the Record episode 108, about 84.9% of robots.txt files in the wild return HTTP 200 and most are under 100 KB. The file format is designed to be small and simple, and Google enforces a 500 KiB size limit anyway, so content past that is ignored.
Common patterns that don't work
These are the patterns I see fail on audits. All are debunked in Google's docs.
1. Disallow-as-deindex. Covered above. Blocking a URL with robots.txt does not remove it from the index. If you want it out, allow the crawl, add noindex, wait, then optionally block.
2. Trying to block image URLs by disallowing the page they're on. The page is blocked from crawling, but if other pages link directly to the image, the image is still crawlable and indexable. To suppress an image, either use X-Robots-Tag: noindex on the image file itself or disallow the image's URL directly.
3. Crawl-delay or clean-param to slow Googlebot. Google ignores both. If you need to reduce Googlebot's crawl rate, use Search Console's crawl rate report, return HTTP 500 or 503 responses temporarily (Google backs off), or contact the Crawling team.
4. Subdomain confusion. A robots.txt at example.com/robots.txt does not apply to shop.example.com or m.example.com. Each subdomain needs its own file, each protocol (HTTP vs HTTPS) needs its own, and each non-standard port needs its own. This is a real source of accidentally exposed staging environments.
5. Trailing-slash and case mistakes. Disallow: /admin blocks /admin, /admin/, /administrator, and /admin-tools (anything starting with /admin). Disallow: /admin/ only blocks paths inside the folder. Paths are case-sensitive, so disallow: /file.asp does not block /FILE.asp, which trips up Windows-derived URL paths.
6. Putting noindex in robots.txt. Some old tutorials show Noindex: /path/. Google removed support for that directive in 2019, and it does nothing now. Put noindex in the page's meta tag or HTTP header, not the robots.txt.
7. Treating robots.txt as security. It isn't. Anyone can read your robots.txt, so listing Disallow: /admin-secrets/ tells everyone where your sensitive paths are. Use authentication for security, not robots.txt.
Googlebot-Extended and AI training opt-out
This is the one new thing in robots.txt worth knowing about.
Google supports a user-agent token called Googlebot-Extended that lets sites opt out of having their content used to improve Google's generative AI models (Gemini, AI Overviews training, etc.) without blocking regular Search indexing. To use it:
User-agent: Googlebot-Extended
Disallow: /That blocks AI-training use. It does not block Googlebot itself, so the page can still be crawled, indexed, and ranked in Search. The separation matters. Most sites should think carefully before opting out, because being in training data is one of the few signals that improves your visibility in AI Mode and AI Overviews. We cover the tradeoff in How to actually get cited in an AI Overview.
Other AI bot tokens worth knowing: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and CCBot (Common Crawl). Each is a separate user-agent, and each takes its own rules in robots.txt. Per the Robots Refresher post on flexible control, robots.txt is "the Swiss Army knife of expressing what you want different robots to do." Use that flexibility carefully, because being absent from AI training data is increasingly equivalent to being invisible in AI search.
How Google handles errors
Useful to know because it explains some weird behavior. If your robots.txt returns one of these statuses, here's what Google does:
- 2xx success: Google uses the file as provided
- 3xx redirect: Google follows up to five hops, then treats it as a 404
- 4xx client error (except 429): Google assumes no crawl restrictions, so it crawls everything
- 5xx server error: Google stops crawling the site for 12 hours, then uses the last good cached version for up to 30 days while retrying. If the file stays broken for 30+ days with the site otherwise available, Google assumes no restrictions
The 5xx behavior is important. A server hiccup that returns 503 on robots.txt for a few hours pauses your crawl, and a persistent 5xx for weeks effectively disables your robots.txt. Monitor it.
Per Search Console, you can also see whether Google is parsing your file correctly. If you've written directives Google doesn't support, Search Console explicitly flags them as "recognized but unsupported." Useful debugging.
A clean 2026 robots.txt for a small business site
Here's what a sensible default looks like for a dentist, lawyer, contractor, or small ecommerce site:
# robots.txt for example.com
# Updated 2026-05-26 by [name]User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search?
Allow: /
# Optional: opt out of AI training while staying in Search
# User-agent: Googlebot-Extended
# Disallow: /
Sitemap: https://example.com/sitemap.xml
Six lines of actual rules, one sitemap declaration, and comments for the next person who edits the file. The opt-out line is commented because most sites should not use it.
Don't over-engineer. Per the Search Off the Record episode 108 analysis, the median robots.txt is short for a reason, because there isn't much to control. If you find yourself writing 200 rules, you probably have a site architecture problem, not a robots.txt problem.
Sources
- Introduction to robots.txt (Google Search Central)
- How Google Interprets the robots.txt Specification (Google Search Central)
- How to Write and Submit a robots.txt File (Google Search Central)
- Robots Refresher: introducing a new series (Google Search Central blog, John Mueller, Feb 2025)
- Robots Refresher: page-level granularity (Google Search Central blog, John Mueller, Mar 2025)
- Robots Refresher: a flexible way to control how machines explore your website (Google Search Central blog, Splitt & Mueller, Mar 2025)
- Robots Refresher: Future-proof Robots Exclusion Protocol (Google Search Central blog, Gary Illyes, Mar 2025)
- Analysing Robots.txt at Scale with HTTP Archive and BigQuery (Search Off the Record episode 108)
- RFC 9309: Robots Exclusion Protocol (IETF)