Robots.txt Generator Pro
Build a robots.txt file by filling in rules for each crawler, add sitemap addresses, and tick off the AI bots you want to keep out. A built-in checker tells you whether a given URL would be allowed or blocked before you publish. Free preview · Pro required to export. Build robots.txt with up to 2 groups and test URLs free. AI crawler blocks, more groups and the download require Pro.
Free preview · Pro required to export. Build robots.txt with up to 2 groups and test URLs free. AI crawler blocks, more groups and the download require Pro.
Why use this tool?
- A single wrong line in robots.txt can hide a whole site from search engines. Testing paths against the rules first is much safer than finding out after publishing.
- The syntax is easy to slip on: wildcards, the end-of-path dollar sign and the rule that the longest match wins are not obvious. The tester applies them the way crawlers do.
- Many site owners now want to decide whether AI companies may use their content. The list of AI crawler names changes, and the tool keeps a set of them ready to block with a tick.
- The starting presets save writing the common cases from scratch, such as a store that should hide the cart and checkout.
How to use Robots.txt Generator
- Pick a preset under "Start from a preset": Allow everything, Block everything, WordPress, Store or Staging site. Or edit the rules directly.
- For each group, fill in "User-agent" (* means all crawlers), then list paths to block under Disallow and paths to permit under Allow, one per line.
- Press "Add a user-agent group" to give a specific crawler, such as Googlebot, its own rules.
- Tick any bots you want to block under "Block AI crawlers", or use "Select all". Add your sitemap addresses, one per line.
- In "Test a URL", choose a crawler, type a path such as /blog/hello-world and read the Allowed or Blocked answer and the rule that decided it. Then "Copy" or download the finished robots.txt.
Features
- Five presets: Allow everything, Block everything, WordPress, Store and Staging site
- Multiple user-agent groups, each with Disallow, Allow and an optional Crawl-delay
- 12 AI crawlers you can block individually or all at once, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot
- A URL tester that understands the * wildcard and the $ end marker, with longest-match precedence and Allow winning a tie
- A "things to check" list covering missing or relative sitemap addresses, paths without a leading slash and a full-site block
- Sitemap lines and an optional Host line
- Copy to the clipboard or download as robots.txt
Supported formats
Plain text output only, saved as robots.txt. The Host line is for Yandex and is the one you will most likely leave blank.
Tips
- Upload the file to the root of the domain so it is reachable at /robots.txt. A copy in a subfolder is ignored.
- A crawler that has its own group follows only that group and ignores the * group. Repeat any shared rules inside the specific one.
- Blocking Google-Extended keeps your content away from Gemini training and does not remove you from Google Search results.
- The "Block everything" preset is right for a staging site and disastrous on a live one. The tool warns you when a * group disallows the whole site.
Frequently asked questions
Does robots.txt keep a page private?
No. It is a request that well-behaved crawlers follow, and anyone can still open the URL. Anything private needs a login or other real access control.
Will blocking a URL remove it from Google?
Not reliably. Blocking stops crawling, but a blocked page can still appear in results if other pages link to it. To keep a page out of results, use a noindex tag on a page that crawlers are allowed to fetch.
Do AI crawlers have to obey it?
Only if they choose to. The AI bots listed here say they respect robots.txt, but a crawler that ignores it is not stopped by the file.
How does the tester decide between Allow and Disallow?
It finds every rule in the matching group that fits the path, and the longest rule wins. If an Allow and a Disallow are the same length, Allow wins.
Which group does the tester use?
The group named for that exact crawler, if there is one, otherwise the * group. If neither exists, everything is allowed.
Is my robots.txt uploaded anywhere?
No. The file is built in your browser and goes nowhere until you upload it to your own site.
Can I try this Pro tool without an account?
Yes. Build robots.txt with up to 2 groups and test URLs free. AI crawler blocks, more groups and the download require Pro. Your own input is processed locally in your browser.
Will upgrading keep my work?
Checkout and login open in a separate tab. Keep the workspace open to retain inputs, settings and previews. Files stay in that tab only; keep the originals in case you close or reload it. Export access is checked with the server after payment.
Your files are processed directly in your browser and are not uploaded to HubToolify's servers.