robots.txt Generator
Allow and disallow rules you can copy.
Loading tool
Add rules per user-agent and copy the file. A bad Disallow can hide the whole site. This helper does not crawl your host.
How to use this tool
- Fill in the page details. Add the title, URL, or tags you want to preview or generate.
- Review the snippet. Check the preview or generated file in this tab.
- Copy into your site. Paste the tags or robots rules into your project.
Limits
Syntax is the common robots.txt subset. Crawl-delay is non-standard. AI crawlers use many product names; keep the list current yourself.
Frequently asked
Does this submit the file to Google?
No. Place it at https://your-domain/robots.txt.
Will Disallow: / stop ads from loading?
No. robots.txt is for cooperating crawlers, not browsers.
Should I block all AI bots?
That is a policy choice. The page can add common names you select.
Is Sitemap required?
No, but a Sitemap line helps engines that read it.
Can this test my live robots.txt?
Paste the live file to review rules. Path matching is a simple prefix model.