Free tool
Robots.txt Generator and Tester
Build a robots.txt file, then test whether a crawler may fetch a URL, using the same matching rules Google documents.
1. Generate a robots.txt
⚠ Every path in "paths to block" must start with / or *.
Your robots.txt
2. Test a URL against a robots.txt
How to use the robots.txt generator and tester
- Generate: choose allow all or block all, list the paths no crawler should fetch (admin areas, carts, internal search results), add your sitemap URLs, and optionally block AI training crawlers. Copy or download the file and upload it as
/robots.txtat the root of your site. - Test: the file goes to the tester with one click, or paste any robots.txt there. Enter a crawler token and a URL or path, and the tester shows whether that crawler may fetch it, which group it used and which line decided.
The rules the tester follows
The tester applies Google’s documented interpretation of robots.txt:
- One group per crawler. The group with the most specific matching user agent applies; groups that name the same crawler are combined; the
*group is used only if no group names the crawler. For Google crawlers such asGooglebot-Image, the tester falls back to theGooglebotgroup before*. - The most specific rule wins: the matching rule with the longest path. On a conflict, the least restrictive rule (allow) wins.
- Wildcards:
*matches any sequence of characters and$marks the end of the URL, soDisallow: /*.pdf$blocks URLs ending in.pdf. - Case: paths are case-sensitive; user-agent names and field names are not.
- Supported fields: user-agent, allow, disallow and sitemap. The tester flags anything else (such as crawl-delay) as ignored, and warns about rules before any user-agent line and files over 500 KiB.
Before you block anything
Blocking crawling is not the same as keeping a page out of Google: a blocked URL can still be indexed without its content. Never block the CSS, JavaScript and images Google needs to render your pages. Our old WordPress site did exactly that, as the technical SEO audit checklist shows. If pages aren’t being indexed, the crawled - currently not indexed and discovered - currently not indexed guides explain the usual causes.
Frequently asked questions
Does Disallow in robots.txt remove a page from Google?
No. Robots.txt controls crawling, not indexing. Google can't read a blocked page, but it may still index its URL and show it without a snippet. To keep a page out of search results, allow crawling and use a noindex robots meta tag.
How does Google decide which robots.txt rule applies?
Google uses one group per crawler: the group with the most specific user agent that matches, or the * group if none does. Within that group the most specific rule (the longest path) wins, and on a conflict between allow and disallow, Google uses the least restrictive rule.
Does Google support crawl-delay?
No. Google supports only the user-agent, allow, disallow and sitemap fields. Other fields such as crawl-delay are ignored.
Are robots.txt paths case-sensitive?
Yes. Paths are case-sensitive, so Disallow: /Fish doesn't block /fish. User-agent names and field names are not case-sensitive.
Will blocking AI crawlers in robots.txt stop AI training?
It works for crawlers that respect robots.txt and publish their user-agent token, such as GPTBot, ClaudeBot and CCBot, and for tokens like Google-Extended and Applebot-Extended that control how content is used. Following robots.txt is voluntary, so it doesn't stop crawlers that ignore it.
Where does the robots.txt file go?
At the root of the host it applies to, for example https://example.com/robots.txt. A subdomain needs its own file. Google ignores content after the first 500 KiB.