robots.txt for SEO and GEO: the small file that sends bots to your best pages

Here's what robots.txt can actually do in 2026, what it can't, and how to write one that pulls its weight.

A 30-year-old file that just got more important

Martijn Koster proposed the Robots Exclusion Protocol in 1994, and for almost three decades it ran on a handshake. In 2022 it finally became a formal internet standard, RFC 9309. The syntax is still refreshingly simple: groups of rules, each starting with a User-agent line, followed by Allow and Disallow paths.

What has changed is who's knocking. Next to Googlebot and Bingbot, your server logs now show crawlers from OpenAI, Anthropic, Perplexity, Apple and others. Some collect training data, some build search indexes for AI answers, and some fetch a page because a person just asked an assistant about it. robots.txt is the one place where you can talk to all of them at once, in a format every one of them already knows how to read.

Power one: SEO, by spending crawl budget on pages that matter

Search engines don't crawl forever. Each site gets a share of crawler attention, shaped by how quickly your server responds and how valuable the engine thinks your content is. Google is candid that most small sites never hit the ceiling. But the moment you have faceted navigation, internal search, calendars or tracking parameters, you can accidentally generate thousands of near-duplicate URLs, and every one of them competes with your real pages for a visit.

A few lines clear the clutter:

User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /*?sort=
Disallow: /*?sessionid=
Allow: /

The * wildcard matches any run of characters and $ anchors a rule to the end of a URL. Both are part of RFC 9309 and supported by Google and Bing. When rules overlap, the most specific (longest) match wins, so you can close a folder and still open a single page inside it with an Allow line.

Two SEO habits worth breaking

• Blocking CSS and JavaScript. Google renders pages much like a browser does. If it can't load your styles and scripts, it can't see the page your visitors see, and that can hurt how the page is understood and ranked.
• Using robots.txt to hide pages from search. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, just without a description, because the crawler was never allowed to read it. To keep a page out of the index, let it be crawled and add a noindex tag instead.

Power two: GEO, by choosing which AI systems can find and cite you

Generative engine optimization is about showing up inside AI answers: the sources an assistant links to when someone asks a question your site is good at answering. The single biggest mistake here is treating "AI bots" as one thing. The major vendors now publish separate user agents for separate jobs, and robots.txt lets you treat each job differently.

The search-index bots are the ones that make citations possible. Block them and you're asking those assistants not to surface your pages as sources. Training is a separate decision, and a perfectly reasonable one to make either way. The good news is that you no longer have to choose all or nothing:

# Welcome AI search, so answers can cite us
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search
Disallow: /cart
Allow: /

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

A note on those last two: Google-Extended and Applebot-Extended aren't crawlers at all. They're tokens that Googlebot and Applebot check before using your content for AI models. Disallowing Google-Extended doesn't remove you from Google Search, so your regular SEO is untouched.

Two honest caveats. First, robots.txt is a request, not a lock. Reputable crawlers honour it; bad actors don't, so anything that truly must stay private needs authentication or server-level rules. Second, some vendors say user-triggered fetchers may not follow robots.txt at all, on the reasoning that a person, not a crawler, asked for the page. Check each vendor's documentation for the current behaviour, since these policies are still evolving.

Power three: handing bots a map instead of a maze

This is the part people forget. robots.txt isn't only for saying no. It's also the first file most well-behaved bots request, which makes it the perfect place to say "the good stuff is over here."

The Sitemap directive points crawlers at your XML sitemap. It takes an absolute URL, you can list as many as you like, and it sits outside any user-agent group, so every bot that reads the file sees it:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-blog.xml

A crawler that finds your sitemap on its first request doesn't have to discover your pages by following links one at a time. New articles get found sooner, deep pages get found at all, and the crawler spends its visit on content instead of navigation.

You can do the same for AI agents. There's no official llms.txt directive, and parsers ignore lines they don't understand, so a comment is the cleanest signpost:

# Plain-language site guide for AI agents: https://example.com/llms.txt
# Full content in one file: https://example.com/llms-full.txt

Robots.txt parsers skip it. Agents and people who open the file see exactly where to go next. It costs two lines and saves every visitor the guesswork.

Small file, handle with care

A few details decide whether your rules actually get read:

• It lives at the root of each host. shop.example.com and example.com each need their own file, and rules on one don't apply to the other.
• Keep it under 500 KiB. Google stops reading beyond that, so anything after the cutoff is simply ignored.
• Status codes matter. A 404 tells crawlers there are no rules, so they crawl freely. A server error is worse: Google may pause crawling your site until it can read the file again.
• Changes aren't instant. Google generally caches robots.txt for up to a day, so give edits time before judging the results.
• Crawl-delay is patchy. Bing honours it; Google ignores it and sets its own pace based on how your server responds.
• Never list secrets. The file is public. A line like Disallow: /old-admin-backup/ doesn't hide that folder, it advertises it. We cover this and other risky patterns in our piece on dangerous content in agent-readable files.

Putting it together

A compact robots.txt that does all three jobs looks like this:

User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /*?sort=
Allow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search
Disallow: /cart
Disallow: /*?sort=
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

# Plain-language site guide for AI agents: https://example.com/llms.txt

Swap the training block to Allow: / if you're happy for your content to be used that way. Either way, you've made a decision on purpose rather than by accident, and that's the whole point.