robots.txt generator
Generate a robots.txt file with crawl rules and the sitemap reference.
Processed on your device. Nothing is sent to any server.
User-agent: * Allow: / Disallow: /admin Disallow: /search
Generate a robots.txt file with crawl rules for search engines and a reference to your sitemap.
Generates the standard format Googlebot and other crawlers recognize to know which paths they can or can't visit on your site, along with the sitemap location.
Use cases
- Blocking crawling of admin or internal search pages.
- Pointing search engines to your sitemap location.
- Preparing the robots.txt for a new site.
How it works
- 1
Set the user-agent
Use * to apply the rule to all crawlers.
- 2
Add allowed and blocked paths
One path per line in each field.
- 3
Add your sitemap
Download the resulting robots.txt file.
Frequently asked questions
- Does a Disallow prevent a page from being indexed?
- Not necessarily: Disallow prevents the content from being crawled, but a blocked URL can still appear in search results with no description if links point to it; to prevent indexing, use a noindex meta robots tag on the page itself.
Related tools
- Favicon GeneratorGenerates a favicon pack in several sizes.
- Meta Tags GeneratorGenerates meta tags for the head.
- Open Graph GeneratorGenerates Open Graph and Twitter Card tags.
- UTM Link GeneratorAdds UTM parameters to a URL.
- Character CounterCount characters and check common limits.
- CSS MinifierReduces a CSS file's size.