robots.txt generator: decide which AI crawlers can read you.
A robots.txt builder that separates the crawlers fetching to train a model from the ones fetching to answer a question — because blocking the second group is what removes you from the answers.
By Mike Cecconello · Updated September 4, 2026
robots.txt generator
Runs in your browser. Nothing you paste is sent anywhere.
robots.txt
Blocking an AI crawler removes you from the answers it generates. That is a legitimate choice, but it is a choice: a page no assistant may read cannot be a page any assistant cites. robots.txt is a request, not a lock; anything private needs authentication, not a Disallow line.
What you can do with it
- Write a robots.txt without hand-editing syntax
- Allow answer engines while blocking training crawlers
- Audit whether you are accidentally invisible to ChatGPT and Perplexity
- Give a client a defensible AI-crawler policy rather than a default
How it works, step by step
- 1
Separate training from retrieval
GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended fetch pages to train models. OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot fetch pages to answer a question someone is asking right now. They are different decisions with different consequences.
- 2
Decide what you are actually protecting
Blocking training crawlers is a reasonable position on how your work is used. Blocking retrieval crawlers is a decision to not appear in AI answers, which for most commercial sites is a decision to be absent from a growing share of discovery.
- 3
Write the rules explicitly
A missing user-agent block means the crawler falls back to User-agent: *. If your wildcard rule is permissive, you are allowing every AI crawler by default, whether or not you decided to.
- 4
Keep the sitemap line
It is unrelated to the bot rules and it is the cheapest discovery win in the file. One line, absolute URL.
- 5
Verify rather than assume
Fetch your own /robots.txt after deploying. A staging file that shipped with Disallow: / is one of the most expensive one-line mistakes in SEO, and it is invisible from inside the CMS.
The distinction almost every robots.txt gets wrong
There are two kinds of AI crawler and they are usually treated as one.
Training crawlers fetch pages to build a model: GPTBot, ClaudeBot, Google-Extended, CCBot,
Applebot-Extended. Blocking them is a statement about how your work is used. It has no effect on
whether you appear in answers today.
Retrieval crawlers fetch a page because a person just asked a question it might answer:
OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot. Blocking these removes you from AI
answers. That is the whole consequence, and it is usually applied by accident, by someone pasting a
“block AI bots” snippet from a forum.
The builder above offers the split explicitly because it is the configuration most publishers actually want and almost nobody writes by hand.
What a permissive wildcard already decided for you
If your file ends with User-agent: * and a short Disallow list, you have allowed every AI crawler
that exists and every one that ships next year. That may be exactly right. It should still be a
decision rather than an inheritance, which is why writing the user-agent blocks explicitly is worth
the extra lines.
The one line people forget
Sitemap: has nothing to do with bot policy and belongs in the file anyway. It is an absolute URL,
it costs one line, and it is the most direct way to tell every crawler what exists on your site.
Robots.txt is a prerequisite, not a strategy
Getting this file right does not make a page citable. It makes a page reachable, which is the step before. Of 274 agency sites we checked against the nine technical signals an assistant needs to read a page, 6 passed every one — and crawl access is only the first of the nine.
Once the file is written, check whether AI crawlers can actually reach your site. The free
check reads your live robots.txt and tests it against eight named crawlers one at a time (GPTBot,
OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Applebot-Extended),
then reports the other eight signals alongside it. No account, and it works on a domain you do not own.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
Not from live answers. GPTBot is the training crawler; OAI-SearchBot and ChatGPT-User are the ones that fetch pages to answer a question in the moment. If you want to be cited in ChatGPT while declining to be training data, allow those and block GPTBot — which is the third option in the builder above.
Should I allow AI crawlers at all?
If you publish content to be found, allowing the retrieval crawlers is close to a necessity: a page no assistant may fetch cannot be a page any assistant cites. Training crawlers are a genuine judgement call and reasonable publishers land on both sides.
Is robots.txt legally binding?
No. It is a convention that well-behaved crawlers respect voluntarily. Major AI companies document their user-agents and honour the file, but it is not access control. If content must not be fetched, it needs authentication rather than a polite request.
Does Disallow stop a page being indexed?
No, and this is the most common robots.txt misunderstanding. Disallow stops crawling. A URL that is linked from elsewhere can still be indexed without its content, which produces a result with no snippet. To keep a page out of the index, allow the crawl and use a noindex meta tag.
Where does the file have to live?
At the root of the host, exactly /robots.txt. It applies per host and per protocol, so a subdomain needs its own file, and the file at the root of your CDN domain does not govern your main site.
More free tools
- Free Internal Link Checker: Audit Anchors on Any Page
- Free Keyword Clustering Tool: Group Keywords by Topic
- Free Keyword Density Checker: A Diagnostic, Not a Target
- Free Keyword Gap Analysis Tool: Compare Two Keyword Lists
- Free Meta Description Generator With SERP Preview
- Free SEO Content Brief Template and Generator
- Free Word Counter and Readability Checker
- SERP Analysis: How to Read a Search Result Page
Be the site the assistant names.
Add a domain and the first draft is written while you watch. Every publish is your decision, a click on the draft or a rule you set and can hold, and you can read the source or self-host it free.
Add a domain, it sets up your workspace