robots.txt tester: find out whether a crawler may fetch this URL.

Enter a page URL and choose a crawler. The tool fetches the site's live robots.txt, shows it, and tells you whether that crawler is allowed or blocked for the page's path, and which rule decided it.

By · Updated September 25, 2026

Run this for every client with AltoRank

robots.txt tester

Runs on our server, which fetches the public URL you enter. We count runs and never save what you enter; an identical request may be answered from memory for up to five minutes.

What you can do with it

  • Check whether a new Disallow rule blocks more pages than you meant it to
  • Confirm that ChatGPT search or Perplexity can fetch your pricing page
  • See which rule applies when a crawler has its own group and the * group also matches
  • Verify a robots.txt change after deploy, against the live file

How it works, step by step

  1. 1

    Find the group that applies

    robots.txt is split into groups, each starting with one or more User-agent lines. A crawler follows only the group that names it most specifically. If no group names it, it follows the User-agent: * group. It does not combine its own group with the * group.

  2. 2

    Collect the rules in that group that match the path

    Each Allow and Disallow rule is a path prefix. A rule matches if the URL's path starts with it. The * wildcard matches any run of characters, and $ anchors the rule to the end of the URL.

  3. 3

    Let the longest match win

    Under Google's documented behavior, the matching rule with the longest path wins, whether it is Allow or Disallow. If an Allow and a Disallow are the same length, Google uses the less restrictive one, which is the Allow.

  4. 4

    No matching rule means allowed

    If nothing in the group matches the path, the crawler may fetch it. A missing robots.txt (a 404) also means everything is allowed.

  5. 5

    Test against the live file

    Run the URL and crawler through the tester after every change. It fetches the robots.txt your server returns now, which is the only version crawlers see.

Two tools, two jobs

If you need to write or rewrite a robots.txt, start with the robots.txt generator. It builds the file for you and separates the crawlers that fetch to train models from the ones that fetch to answer a question.

This tester does the other half. It reads the file the site is actually serving, then answers one question for one crawler and one URL: may it fetch this page? That is the check to run after a deploy, after a CMS or plugin update touched the file, or when a page drops out of search or AI answers and you want to rule robots.txt in or out.

How matching works

Crawlers that follow the Robots Exclusion Protocol, which Google documents and which is published as RFC 9309, resolve a URL in two steps.

First, pick the group. A crawler looks for the group whose User-agent line names it. It obeys that group only. If there is none, it uses the User-agent: * group. This catches people out: a site with a strict * group and a separate User-agent: GPTBot group containing one rule gives GPTBot only that one rule. The * rules no longer apply to it.

Then, pick the rule. Within the group, every Allow and Disallow whose path matches the start of the URL’s path is a candidate. The longest one wins. So with Disallow: /blog/ and Allow: /blog/guides/, a page at /blog/guides/setup is allowed. If two matching rules are the same length, Google picks the less restrictive, the Allow.

Two special characters change what a rule matches. * stands for any sequence of characters, so Disallow: /*?sort= blocks any URL with that parameter. $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDFs without blocking /pdf-guides/. Paths are case-sensitive.

What robots.txt is not

  • Not access control. It is a request. Crawlers that respect it comply, and anything else can ignore it. Never use it to hide a URL, since the file itself is public and lists the paths.
  • Not noindex. Disallow stops a crawler from fetching the page. Google can still index the URL from links elsewhere, just without the content. A noindex tag only works on a page the crawler is allowed to fetch.
  • Not instant. Google generally caches robots.txt for up to a day, so a change can take that long to apply.

What the tester does not cover

The tester answers for the one crawler you pick and the one URL you enter. It does not check whether the crawler is also blocked by a firewall, whether the page returns a 200, or what text the page contains. The AI crawler simulator covers those, and the free AI-readiness check tests robots.txt for eight named AI crawlers across a whole domain, alongside eight other signals.

Letting crawlers in is the start. AltoRank plans and drafts the content worth fetching every week, and a person approves each piece before it publishes.

Frequently asked questions

Which crawlers can I test?

Googlebot, Bingbot, the AI search and on-demand fetchers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), and * for any crawler without its own group.

How is this different from the robots.txt generator?

The tester reads the robots.txt a site is serving right now and tells you what it means for one URL and one crawler. The robots.txt generator writes a new file for you, with AI crawler rules split by what each crawler does. Use the generator to write the file and the tester to check it once it is live.

If robots.txt blocks a page, is it out of Google?

Not necessarily. Disallow stops crawling, not indexing. Google can still index a blocked URL, without its content, if other pages link to it. To keep a page out of search results, allow crawling and use a noindex meta tag or header, because Google has to fetch the page to see the noindex.

Does robots.txt stop scrapers?

No. It is a request that well-behaved crawlers follow. Anything that ignores it can still fetch the page. If content must not be accessed, put it behind authentication, or block the traffic at your server or CDN.

Why does the tester say allowed when the crawler still gets an error?

robots.txt is only one gate. A firewall or CDN bot rule can refuse a crawler that robots.txt allows. The AI crawler simulator requests the page as each crawler and shows the status code that comes back, which catches that case.

Does it store the URL I test?

No. Our server fetches the site's public robots.txt, returns the result to your browser and keeps only a count of how many times the tool ran that day. There is a daily usage limit so it can stay free without a signup.

More free tools

Free robots.txt Generator With AI Crawler Rules Build a robots.txt with AI crawler rules. Which bots train models, which answer questions, and why it matters. Free AI Crawler Simulator: See What GPTBot Sees Fetch any URL as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Googlebot. See each robots.txt verdict, status code and the text in the raw HTML. Free XML Sitemap Checker and Validator Enter a site or sitemap URL. The tool finds the sitemap, checks that it parses, counts its URLs and child sitemaps, and flags obvious problems in a sample. Free Website Metadata Checker: Title, OG, Schema Enter a URL and see the page's title, meta description, canonical, robots tag, Open Graph and Twitter tags, JSON-LD types, h1 and language.
Free growth plan for any domain Compare AltoRank against the alternatives More on SEO and GEO on the blog

Be the site the assistant names.

Add a domain and the first draft is written while you watch. Every publish is your decision, a click on the draft or a rule you set and can hold, and you can read the source or self-host it free.

Add your domain, free Self-host it free

Set up your site and read your first drafts before you enter a card.