Real-time · 100% Free · No signup · Live data
Your robots.txt file is the most dangerous file on your website. One stray character — a single Disallow: / left over from a staging environment — can wipe your entire site out of Google within days. And yet most people edit it blind, save it, and hope. The fix is simple: test before you trust. Paste any domain into the Robots Txt Checker and see exactly which rules are live, what they block, and whether your most important pages are reachable — instantly, with no signup.
This guide covers everything: what robots.txt actually does, how to read every directive, how to fix the two Search Console errors everyone runs into, how to control AI crawlers like GPTBot and ClaudeBot, and the mistakes that quietly tank traffic.
What Is Robots.txt?
A robots.txt file is a plain text file that lives at the root of your domain — always at https://example.com/robots.txt, never in a subfolder. It tells web crawlers which parts of your site they may or may not request. It's the first thing a well-behaved crawler looks for before it touches anything else on your site.
The file is governed by the Robots Exclusion Protocol, which existed as an informal convention from 1994 until it was formally standardized in September 2022 as the official Robots Exclusion Protocol standard (RFC 9309) by the IETF. That standardization matters for one reason: it means there's now an authoritative reference for how every compliant crawler should parse your file, so you can diagnose problems against a real spec instead of guesswork.
Here's the single most important thing to understand, and the thing almost everyone gets wrong: robots.txt controls crawling, not indexing. Telling Google not to crawl a page does not reliably keep it out of search results. If other sites link to a blocked URL, Google can still index it — it just won't be able to read what's on it. More on that critical distinction below.
One more caveat the standard itself spells out: robots.txt is not a security tool. The file is public — anyone can read yours by visiting the URL — so listing a "secret" admin path in it actually advertises that path to the curious. Real protection means authentication at the server level, not a Disallow line.
How to Check and Test Your Robots.txt
Testing takes three steps:
Enter your domain. Paste any URL into the Robots Txt Checker — it fetches the live file in real time, never a cached copy.
Read the parsed rules. The tool breaks the file into its directives — every User-agent, Disallow, Allow, and Sitemap line — and flags syntax problems.
Confirm your key pages are reachable. Check that nothing important is caught by an overly broad rule.
If you've used Google Search Console in the past, you may remember it had a built-in robots.txt Tester. Google removed that tool in late 2023. It was replaced by a read-only robots.txt report that shows you which files Google fetched and any parse errors — but it no longer lets you interactively test whether a specific URL is blocked. Google's own help docs now tell users to "search for a robots.txt validator" instead. That's the gap this tool fills: a live, interactive checker that does what the deprecated GSC tester used to do, free and without an account.
Robots.txt Syntax Explained
Every robots.txt file is built from groups. Each group starts with one or more User-agent lines (who the rule applies to) followed by the rules themselves. Here's a standard example:
User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xml
User-agent names the crawler. The asterisk (*) is a wildcard meaning "all bots." You can also target specific crawlers — Googlebot, Bingbot, GPTBot — each in its own group. Crawlers pick the single most specific group that matches their name and ignore the rest, so if you have both a * group and a Googlebot group, Googlebot follows only the Googlebot rules.
Disallow tells the bot what not to crawl. The path is everything after your domain. Disallow: /private/ blocks every URL starting with /private/. An empty Disallow: means "block nothing" — i.e., allow everything.
Allow punches an exception through a Disallow. The classic case: block a whole folder but permit one file inside it. Google and Bing apply the longest, most specific matching rule, so Allow: /admin/public/ beats Disallow: /admin/.
Sitemap points crawlers to your XML sitemap. It must be a fully-qualified URL (with https://), and you can list more than one. It's the one directive not tied to any user-agent.
Three rules that trip people up constantly, all confirmed in Google's robots.txt specification:
Paths are case-sensitive. Disallow: /Page/ does not block /page/. The directive name (disallow) is case-insensitive, but the path value is not.
Matching is "starts-with." Disallow: /p blocks /page.html, /products/, and anything else beginning with /p. Use a trailing slash (/p/) to scope it to a folder.
Wildcards are limited. matches any sequence of characters and $ marks the end of a URL. Disallow: /.pdf$ blocks all PDFs. But not every crawler supports these, so test rather than assume.
To confirm your sitemap is correctly declared and actually valid, pair this with the sitemap checker — a broken sitemap reference in robots.txt is a silent discovery killer.
Robots.txt vs Noindex: The Distinction That Breaks Sites
This is the concept that causes more accidental SEO damage than any other, so read it twice.
Disallow in robots.txt = "don't crawl this page." It blocks the bot from opening the page.
noindex meta tag = "don't show this page in search results." It keeps the page out of the index.
These are not interchangeable, and here's the trap: if you Disallow a page in robots.txt, Googlebot can't crawl it — which means it can never see a noindex tag on that page. So if you're trying to remove a page from Google by blocking it in robots.txt and adding a noindex tag, the block prevents Google from ever reading the noindex. The page can stay indexed indefinitely, showing up in results with no description.
The correct approach to keep a page out of search: allow crawling, and add a noindex tag. Google needs to crawl the page to see the instruction telling it not to index. Only after Google has processed the noindex should you consider blocking it (most of the time, you just leave it crawlable).
To audit whether your pages carry the right noindex/nofollow directives — and catch conflicts between your robots.txt and your meta tags — run them through the noindex checker. This is the most common place where robots.txt and meta directives fight each other, and it's worth checking explicitly.
How to Fix "Blocked by robots.txt" and "Indexed, Though Blocked"
These are the two Search Console messages that send people scrambling. They sound similar but mean different things.
"Blocked by robots.txt"
Google found a URL but your robots.txt told it not to crawl. Sometimes that's intentional (you meant to block it). Sometimes it's an accident blocking a page you want ranked.
The fix:
In Search Console, use URL Inspection on the affected page. If the Page Indexing section says "Blocked by robots.txt," you've confirmed it.
Run the URL through the Robots Txt Checker to see which exact rule is blocking it.
If the page should be crawlable, edit your robots.txt to remove or narrow the offending Disallow, then request re-indexing.
"Indexed, Though Blocked by robots.txt"
This one's trickier. It means Google indexed the URL anyway — despite your block — because it found enough links pointing to it to decide it was worth listing. The page shows up in results, usually with no meta description, because Google was never allowed to read the content. Search Console marks these "Valid with warning."
The fix depends on intent:
If the page should be indexed: remove the Disallow so Google can crawl it properly and generate a real snippet.
If the page should not be indexed: this is the classic trap. You must unblock it in robots.txt so Google can crawl it, then add a noindex tag so Google sees the instruction and drops it. Blocking alone won't work because Google can't read a noindex on a page it can't crawl.
A frequent cause of these warnings is internal links pointing at pages you've blocked. Google follows those links, finds the URL, and indexes it. Auditing where your internal links point — and pruning links to pages you want excluded — often clears the warning at the source. Pair that with a check of your redirect paths using the redirect checker, since a blocked URL sitting at the end of a redirect chain can compound the problem.
How to Block (or Allow) AI Crawlers
Since 2023, robots.txt has taken on a second job: controlling whether AI companies can use your content. The same file that manages Googlebot now manages GPTBot, ClaudeBot, PerplexityBot, and dozens of others.
There are two categories of AI crawler, and the difference changes how you should treat them:
Training crawlers collect content to train large language models. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Google's Gemini training).
Retrieval crawlers fetch your page in real time when a user asks an AI a question, and often cite you with a link back. Examples: ChatGPT-User, OAI-SearchBot, Perplexity-User.
The strategic implication: blocking training crawlers protects your content from being absorbed into models. Blocking retrieval crawlers removes you from AI-generated answers and the referral traffic they bring. Many sites now adopt a middle path — block training, allow retrieval — to stay cited in AI answers without donating their content to model training.
A blanket block looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Two facts that resolve the most common fear: blocking these AI bots does not hurt your Google Search rankings. GPTBot, ClaudeBot, and the rest are completely separate from Googlebot, which handles search indexing. And Google-Extended is safe to block too — it controls only Gemini AI training and has zero effect on your Google Search ranking. The one bot you must never block if you want to appear in Google is Googlebot itself.
Give each AI bot its own named block rather than relying on User-agent: *, since some honor the wildcard and some don't. After editing, scan your robots.txt file to confirm each block parses correctly — and remember that compliance is voluntary. A few aggressive scrapers ignore robots.txt entirely, and those need to be stopped at the server or CDN level.
Common Robots.txt Mistakes and How to Fix Them
1. Disallow: / on a live site. The single most catastrophic edit possible. Under User-agent: *, this blocks every crawler from your entire site, and within weeks your pages drop out of Google. It usually happens when a staging file (where blocking everything is correct) gets deployed to production by mistake. Always verify your live robots.txt immediately after any migration or deployment.
2. Blocking CSS and JavaScript. Older advice told people to block /assets/ or .js files. Modern Google renders pages before ranking them, so blocking these resources prevents Google from seeing your layout and can hurt mobile-friendliness and rankings. Leave rendering resources crawlable.
3. Confusing Disallow with noindex. Covered above — the costliest conceptual error. Blocking doesn't deindex; it can do the opposite.
4. Wrong file location or casing. The file must sit at the root and be named in all lowercase: robots.txt, never Robots.TXT. In a subfolder, crawlers won't find it.
5. Trying to hide sensitive paths. Listing /admin/ or /backups/ in robots.txt publicly advertises them. Protect sensitive areas with authentication instead.
6. Forgetting subdomains need their own file. blog.example.com and shop.example.com each need their own robots.txt at their own root. One file does not cover all subdomains.
7. Set-and-forget. Your site changes; your robots.txt should be reviewed every quarter, or after any structural change, as part of a standing technical-SEO checklist.
Related Tools
Explore the full SEO Space toolkit, or jump straight to the most relevant siblings:
Sitemap Checker — validate the XML sitemap your robots.txt declares
Noindex Checker — catch noindex/nofollow conflicts with your crawl rules
Meta Tag Analyzer — inspect every meta tag, including meta robots
Redirect Checker — trace redirect chains affecting crawl behavior
Schema Markup Checker — validate structured data for rich results