Robots.txt Generator

Choose your platform, add your sitemap and any folders to keep out of search, and get a ready-to-upload robots.txt file. You can also choose which AI crawlers may read your site.

How to use the robots.txt generator

1. Keep Allow all crawlers selected for any live website.
2. Choose your platform. WordPress and WooCommerce presets add the usual admin and cart rules.
3. Add extra paths you don’t want crawled, such as thank-you pages or internal search results.
4. Paste your sitemap URL. On WordPress with Rank Math or Yoast it is usually /sitemap_index.xml.
5. Tick any AI crawlers you want to block, then copy or download the file.
6. Upload it to your domain root and open yourdomain.com/robots.txt to check it.

Understanding the rules

• User-agent names the crawler a group of rules is for. An asterisk (*) means every crawler.
• Disallow lists paths the crawler should not visit. An empty Disallow means nothing is blocked.
• Allow makes an exception inside a blocked folder, like admin-ajax.php inside /wp-admin/.
• Sitemap points crawlers to your XML sitemap. It must be a full URL.
• Paths are case-sensitive, and * matches any characters.

Mistakes that hurt rankings

• Leaving Disallow: / on after moving a site from staging to live.
• Blocking CSS, JavaScript or image folders. Google needs them to render your pages properly.
• Using robots.txt to hide pages you want removed from Google. Use noindex for that.
• Blocking URL parameters that your important pages depend on.

Not sure which pages Google sees? A technical website review covers crawling, indexing and speed together.

Frequently asked questions

What is a robots.txt file?

It is a plain text file at the root of your domain (for example example.com/robots.txt) that tells crawlers which parts of the site they may visit. Well-behaved crawlers like Googlebot follow it.

Does robots.txt remove a page from Google?

No. Blocking a URL in robots.txt stops crawling, but the URL can still appear in search if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.

Where do I upload robots.txt?

It must sit at the root of your domain, like https://example.com/robots.txt. On WordPress, SEO plugins such as Rank Math and Yoast let you edit it from the dashboard. You can also upload the file with your hosting file manager.

Should I block AI crawlers?

It depends on your goals. Blocking training crawlers like GPTBot or Google-Extended keeps your content out of future AI training. Blocking AI search crawlers can stop your site from being shown or cited in those AI answers, which may cost you visibility. Blocking Google-Extended does not affect Google Search rankings.

Do all bots obey robots.txt?

No. It is a voluntary standard. Major search engines and the AI companies listed here say they respect it, but spam bots and scrapers often ignore it. Never use robots.txt to hide private information.

More free tools

See all free marketing tools