Robots.txt Checker — Audit Crawl Rules Instantly
Fetch and analyze any domain's robots.txt file in seconds. See disallow rules, crawl
directives, sitemap declarations, and syntax errors — exactly as Googlebot reads them.
Free, no signup, live data every time.
How to Use the Robots.txt Checker
Audit any domain's robots.txt file in three steps — no account, no install, no waiting.
Step 1 — Enter a Domain
Type the domain you want to check — your own site, a client's domain, or a competitor's. You don't need the full robots.txt URL, just the domain itself.
Step 2 — Click Scan
Prafero fetches the live robots.txt file directly from the domain root and parses it immediately. No cache, no approximation — this is the exact file Googlebot reads right now.
Step 3 — Review Rules, Sitemaps, and Errors
Results appear instantly: every user-agent group, every Allow and Disallow rule, any declared sitemap, and any syntax errors or warnings — all labeled in plain English with an explanation of what each one means for crawling.
What Is a Robots.txt File?
A robots.txt file is a plain text file placed at the root of a domain — yourdomain.com/robots.txt — that tells web crawlers which parts of a site they may or may not access. It is the first file most well-behaved crawlers, including Googlebot, fetch when visiting a site, before requesting a single page.
The file consists of one or more groups, each beginning with a User-agent line identifying which crawler the group applies to, followed by Allow and Disallow rules specifying which paths that crawler may or may not request:
User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /
Sitemap: https://example.com/sitemap.xml
If no robots.txt file exists at all, crawlers assume the entire site is open to crawling and proceed normally — a missing file is not, by itself, an error, though declaring an explicit file with a sitemap reference is still considered best practice.
How Rule Precedence Actually Works
Google and Bing do not process robots.txt rules in the order they're written — the rule with the longest matching path always wins, regardless of position in the file.
This specificity-based model is formalized in RFC 9309, the Robots Exclusion Protocol standard both Google and Bing follow. If two rules match a URL with exactly the same path length, the less restrictive rule — Allow — takes precedence.
User-agent: *
Disallow: /account/
Allow: /account/login/
In this example, everything under /account/ is blocked except /account/login/, because the Allow rule's path is more specific — longer — than the Disallow rule it overlaps with. This works regardless of which line comes first in the file.
Some older or simpler crawlers instead use a "first match" model, processing rules top to bottom and stopping at the first one that applies — which is why writing Allow rules before broader Disallow rules remains good practice for maximum compatibility, even though it makes no difference to Google or Bing.
Wildcards add one more layer of nuance: a trailing * is implicit at the end of any path that doesn't end in $, but it still counts toward path length when determining specificity — a detail that has tripped up even experienced SEOs comparing two rules that look deceptively similar in length.
Robots.txt vs Noindex — The Distinction That Trips Up Almost Everyone
Disallow blocks from crawling. It does not remove a page from Google's index.
If a URL is disallowed in robots.txt, Googlebot will not request that page's content. But if external sites link to that URL, Google can still become aware of it and may display the bare URL in search results — with no title and no description, since it was never crawled to generate either. You may have seen this exact result before: "A description for this result is not available because of this site's robots.txt." This is the worst outcome for a page you actually wanted hidden — it still appears, just with no useful content shown.
To truly remove a page from search results, the page must be crawlable and carry a noindex directive — either in a meta robots tag or an X-Robots-Tag HTTP header. But here is the trap: if that same URL is also disallowed in robots.txt, Googlebot never crawls it, which means it never sees the noindex directive sitting inside it. The two mechanisms actively work against each other when used on the same URL.
The correct approach: choose one mechanism per URL, not both. To exclude a page from search results, allow it to be crawled and rely on noindex. To prevent crawling for reasons like server load or crawl budget — accepting that the URL may still surface via backlinks with no snippet — use robots.txt disallow instead.
Use Prafero's Noindex Checker alongside this tool to check whether a URL has a noindex directive, and confirm with this Robots.txt Checker that the same URL isn't simultaneously blocked from being crawled — the combination that silently prevents the noindex directive from ever working.
Robots.txt for AI Crawlers in 2026
Robots.txt now controls more than search indexing. As of 2026, it is also the primary mechanism for deciding whether AI platforms like ChatGPT, Claude, and Perplexity can access, cite, or train on a site's content — and the major AI providers each ship their own documented, distinct user agent.
The four most consequential AI crawler user-agents to address: GPTBot (OpenAI training), ClaudeBot (Anthropic training and retrieval), PerplexityBot (Perplexity AI search and retrieval), and Google-Extended (controls Gemini and AI Overviews training — separate from standard Googlebot). GPTBot is currently the most-blocked AI crawler on the web, appearing in roughly 5.5% of all Disallow rules tracked across sites in early 2026, narrowly ahead of ClaudeBot.
Blocking an AI training crawler has no effect on Google Search rankings. GPTBot, ClaudeBot, and Google-Extended are entirely separate systems from Googlebot, which handles standard search indexing. You can block any or all of them without any impact on your organic search visibility.
The distinction that matters is between crawlers that train models and crawlers that retrieve content for live AI search answers. Blocking a training crawler protects content from being used in future model training. Blocking a retrieval crawler removes a site from that platform's real-time search citations entirely. A common 2026 posture — allow retrieval, restrict training — looks like this:
# Allow AI search and retrieval bots (citation visibility)
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Restrict AI training-only crawlers
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This is a starting point, not a universal rule — the right configuration depends on whether a site's priority is AI citation visibility, content protection, or both. Use this Robots.txt Checker to confirm exactly which AI crawler directives — if any — are currently set on any domain, including your own.
Common Robots.txt Mistakes This Tool Detects
Disallow: / on the entire site A single Disallow: / under User-agent: * blocks every crawler from the entire domain. This is the most catastrophic robots.txt mistake — often left over from a staging environment that was never updated for production, silently preventing the whole site from being crawled.
Missing or unreachable sitemap declaration No Sitemap: line in the file, or a sitemap URL that returns an error when fetched. While not strictly required, declaring your sitemap in robots.txt is one of the most reliable ways to help crawlers discover it.
File not located at the domain root The robots.txt file must live exactly at yourdomain.com/robots.txt. A file placed in a subdirectory is not recognized by crawlers and is treated as if no robots.txt exists at all.
Conflicting Allow and Disallow rules with unclear precedence Rules that overlap in ways that are easy to misread by eye — where the actual winning rule, based on path specificity, isn't the one a human would assume from reading top to bottom.
Blocking CSS or JavaScript needed for rendering Disallowing folders like /assets/ or /scripts/ can prevent Googlebot from rendering the page correctly, which can affect how content and layout are understood during indexing — even though the page itself isn't blocked.
Using unsupported directives Fields like Crawl-delay are not supported by Google's crawler at all, and rules other than user-agent, allow, disallow, and sitemap are simply ignored by Google's parser — a common case of webmasters assuming a directive works when Google silently disregards it.
Case-sensitivity mismatches Disallow paths are case-sensitive. Disallow: /Page/ does not block /page/. A mismatch between the casing used in robots.txt and the actual URL casing on the site leaves the intended path completely unblocked.
Related SEO Tools
A robots.txt file is one layer of a site's crawlability and indexability. These tools inside SEO Space check the layers that work alongside it:
Noindex Checker: Check whether a URL has a noindex directive — the companion check to this tool, since a robots.txt block can prevent a noindex directive from ever being discovered or honored.
Sitemap Checker: Validate that the sitemap declared in robots.txt actually exists, is accessible, and is correctly formatted — the natural next check in the same crawl-audit workflow.
Meta Tag Analyzer: The robots meta tag is one of several meta tags controlling indexing and display. Get a full audit of every meta tag on any URL, not just crawl-level directives.
SEO Score Checker: Want the complete on-page picture beyond crawl rules? The SEO Score Checker covers meta tags, headings, internal links, canonical tags, and content length in a single health score.