Skip to content
QuickToolio
esEspañolenEnglishptPortuguêsfrFrançaisdeDeutschitItaliano

robots.txt generator

Generate a robots.txt file with crawl rules for search engines and a reference to your sitemap.

Processed on your device. Nothing is sent to any server.
User-agent: *
Allow: /
Disallow: /admin
Disallow: /search

Generates the standard format Googlebot and other crawlers recognize to know which paths they can or can't visit on your site, along with the sitemap location.

Use cases

  • Blocking crawling of admin or internal search pages.
  • Pointing search engines to your sitemap location.
  • Preparing the robots.txt for a new site.

How it works

  1. 1

    Set the user-agent

    Use * to apply the rule to all crawlers.

  2. 2

    Add allowed and blocked paths

    One path per line in each field.

  3. 3

    Add your sitemap

    Download the resulting robots.txt file.

Technical details

  • It generates a single rule group for the user-agent you enter (* for all). For different rules per bot, generate each group and combine them in the same file.
  • Put each path on its own line, starting with /; empty lines are ignored. The tool doesn't check that the path exists.
  • The Sitemap line needs the sitemap's absolute URL. The file downloads as robots.txt and must live at the domain root (https://yourdomain.com/robots.txt).

Frequently asked questions

Does a Disallow prevent a page from being indexed?

Not necessarily: Disallow prevents the content from being crawled, but a blocked URL can still appear in search results with no description if links point to it; to prevent indexing, use a noindex meta robots tag on the page itself.

Related tools

← Web & SEO