← All Tools
Theme
Robots.txt Generator
Generate robots.txt file for search engine crawlers
Configuration
Default Policy:
Allow All
Disallow All
Custom Rules
Sitemap URL:
Crawl Delay (seconds):
User-Agent Rules:
* (All crawlers)
Googlebot
Bingbot
Slurp (Yahoo)
DuckDuckBot
Baiduspider
Custom
Allow
Disallow
×
Add Rule
Generate
Load Example
Clear
Generated robots.txt
Copy
Download