October 27, 2026
Custom robots.txt: What It Is, Examples for WordPress & Blogger, and How to Test It
By Rohit GuptaUpdated 16 min read
One line can hide your whole website from Google: Disallow: /
It often gets added while a developer builds a new site on a test server. Then the site goes live, the line comes along, and Google can’t crawl a single page.
Whether you run a Blogger blog, a WordPress site for your clinic or a WooCommerce store, a custom robots.txt is worth five minutes of your time. Get it right and it quietly helps crawlers. Get it wrong and it hides you.
Let’s make sure yours is right.
Quick answer: what is a custom robots.txt?
A custom robots.txt is a plain text file you write yourself and place at yourdomain.com/robots.txt to tell search engine crawlers which pages or folders they may crawl. It controls crawling, not indexing, so it won’t reliably keep a page out of Google.
- Location: the root of your domain only, like
https://example.com/robots.txt - Main rules:
User-agent,Disallow,AllowandSitemap - Safe default for most sites: allow everything and add your sitemap
- Want a page out of Google? Use a noindex tag, not robots.txt
- Blogger: Settings → Crawlers and indexing → Enable custom robots.txt
Google reads up to 500 KiB of the file and generally caches it for up to 24 hours (as of October 2026).
Open the free robots.txt generator
Here’s the thing: most websites need a very short robots.txt. The danger isn’t writing too little. It’s copying a long file from somewhere without knowing what each line does.
In this guide I’ll walk you through:
- What robots.txt does in SEO, and what it can’t do
- The syntax in plain English, with Google’s own matching rules
- Copy-paste examples for WordPress, WooCommerce, Blogger and staging sites
- How to test your file, and whether to block AI crawlers
- The mistakes that quietly kill rankings
What is robots.txt in SEO?
Robots.txt is a text file at the root of your website that tells search engine crawlers which URLs they may request. Google says its main job is to stop crawlers from overloading your site and to skip pages that don’t matter.
Think of it as a signboard at your shop gate: “Staff only beyond this point.” Polite visitors follow it. It isn’t a lock.
The rules come from the Robots Exclusion Protocol, published as internet standard RFC 9309 in September 2022. Google’s crawlers follow it.
For SEO, it matters in three ways. It decides what gets crawled, so one wrong line can block important pages. It keeps crawlers away from junk like internal search results and cart pages. And its Sitemap: line points crawlers to your list of important pages.
Do you even need one? Not strictly. Search Console Help says having no robots.txt file is fine; Google just crawls what it finds. But a short, correct file with your sitemap is a good habit.
Does robots.txt stop a page from showing on Google?
No. Google’s robots.txt guide says it “is not a mechanism for keeping a web page out of Google.” A blocked URL can still appear in search results, usually without a description, if other pages link to it.
This is the biggest robots.txt misunderstanding, so let’s be clear:
- robots.txt says “don’t crawl this.” (Don’t go in.)
- noindex says “don’t show this in search.” (You can look, but don’t list it.)
And here’s the twist. Google’s noindex guide says that for noindex to work, the page must not be blocked by robots.txt. If Google can’t crawl the page, it never sees the tag.
Want a page out of Google? Let it be crawled, and add noindex. Blocking it in robots.txt can leave it stuck in the results.
Two more things robots.txt can’t do. It can’t protect private content: RFC 9309 says the rules “are not a form of access authorization,” and bad bots ignore them. And you can’t put noindex inside robots.txt, because Google doesn’t support it there (it ignores crawl-delay too).
If your site isn’t appearing on Google at all, a robots.txt block is a usual suspect. Our guide on why your website is not showing on Google shows how to check.
How do you write robots.txt rules? The syntax in plain English
A robots.txt file is made of groups. Each group starts with a User-agent line naming the crawler, followed by Disallow and Allow lines listing paths. Sitemap lines can go anywhere in the file.
# Rules for every crawler
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml- User-agent: which crawler the rules are for.
*means every crawler;Googlebotmeans only Google’s main crawler. - Disallow: a path not to visit.
Disallow: /blocks everything. An emptyDisallow:blocks nothing. - Allow: an exception inside a blocked area.
- Sitemap: the full URL of your XML sitemap, starting with https://.
- # starts a comment that crawlers ignore.
Google also supports two wildcards. * matches any characters, so Disallow: /*?s= blocks any URL containing ?s= (internal search results). $ marks the end of a URL, so Disallow: /*.pdf$ blocks URLs ending in .pdf.
When rules conflict, the longest match wins
Google follows the most specific rule, meaning the one with the longest matching path. If an Allow and a Disallow are equally specific, the less restrictive one (Allow) wins. Each crawler also obeys only one group: the one that names it most specifically.
These examples come from Google’s robots.txt specification:
| URL | Rules | Result | Why |
|---|---|---|---|
/page | Allow: /pDisallow: / | Allowed | The Allow path is longer |
/folder/page | Allow: /folderDisallow: /folder | Allowed | Equal length, so the less restrictive rule wins |
/page.htm | Allow: /pageDisallow: /*.htm | Blocked | The Disallow path is longer |
/ | Allow: /$Disallow: / | Allowed | /$ matches only the homepage |
/page.htm | Allow: /$Disallow: / | Blocked | /$ doesn’t cover other pages |
Source: Google, “How Google interprets the robots.txt specification”, checked October 2026.
One more rule to remember: paths are case-sensitive. Disallow: /Blog/ does not block /blog/.
Where does robots.txt go, and what are Google’s limits?
Robots.txt must sit at the top level of your site, like https://example.com/robots.txt. Crawlers don’t look in subfolders, and each file covers only the exact protocol, host and port it’s served from, so shop.example.com needs its own file.
Name it robots.txt in lowercase and save it as plain UTF-8 text (Notepad or a code editor, not Word). Here’s how Google handles it, per its specification as of October 2026:
| What | How Google handles it |
|---|---|
| File size | Reads up to 500 KiB; anything after that is ignored |
| Caching | Generally up to 24 hours, longer if it can’t fetch a fresh copy |
| Not found (4xx) | Treated as no robots.txt, so everything may be crawled |
| Server error (5xx) | Stops crawling for 12 hours, then uses the last good copy for up to 30 days |
| Supported fields | user-agent, allow, disallow, sitemap (no crawl-delay) |
So what does that mean for you? A robots.txt that throws server errors can pause crawling of your whole site. A missing file is far safer than a broken one.
Robots.txt examples you can copy
Most sites need only a few lines: allow everything, block admin or checkout areas, and list the sitemap. Swap example.com for your own domain before using any of these.
1. Allow everything (safe default)
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml2. Block one folder
User-agent: *
Disallow: /thank-you/End folder paths with a slash. That blocks the folder and everything inside it.
3. Robots.txt for WordPress
If there’s no real file on your server, WordPress serves a virtual robots.txt. The WordPress code reference shows it blocks /wp-admin/ and allows admin-ajax.php. Since WordPress 5.5, it also lists the built-in sitemap at /wp-sitemap.xml when your site is public.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlUsing an SEO plugin like Yoast? Some plugins switch off WordPress’s built-in sitemap and use their own address. Copy the sitemap URL from your plugin and open it in a browser to check it works.
4. WooCommerce (cart, checkout, account)
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlThese pages are different for every shopper and don’t need crawling. Note that WooCommerce already adds a no-robots tag to its cart, checkout and My Account pages, and Google can’t see that tag on a blocked page. Either approach is fine for these pages. If your shop uses other slugs (say /basket/), change the paths.
5. Staging or test site
User-agent: *
Disallow: /Use this only on a test copy, ideally behind a password too. When the site goes live, delete it.
6. Shopify
Shopify writes robots.txt for you, and its Help Center says the default “works for most stores.” It already blocks admin, cart, checkout and account URLs. Changes go in a robots.txt.liquid theme template, and Shopify warns that wrong edits can cost you all your traffic.
How to add a custom robots.txt for Blogger
In Blogger, open Settings, scroll to the “Crawlers and indexing” section, turn on “Enable custom robots.txt” and paste your rules. Blogger then serves your text instead of its default file.
But first, look at what Blogger already gives you. Here’s the live robots.txt of Google’s own Blogger blog, checked in October 2026:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /
Sitemap: https://blogger.googleblog.com/sitemap.xmlThe first group lets the AdSense crawler (Mediapartners-Google) see everything. The second keeps other crawlers out of /search, where Blogger’s search and label pages live (label links look like /search/label/...), and points them to the sitemap.
For most blogs, that default is fine. Go custom only if you want to add something, like blocking an AI training crawler:
- Open Settings for your blog in Blogger.
- Scroll to “Crawlers and indexing” and turn on “Enable custom robots.txt”.
- Paste your rules. Start from the default above, with your own blog address in the Sitemap line, then save.
- Check it live at yourblog.blogspot.com/robots.txt in a private browser window.
Our generator has no separate Blogger option. Choose “Other / custom site”, add your extra rules, then paste the result into Blogger, keeping the Disallow: /search line from the default.
What about “Enable custom robots header tags”? That setting, in the same section, controls the robots tags (like all or noindex) Blogger sends to search engines. It’s the noindex side, not the crawling side. Blogger guides commonly suggest all for the home page, posts and pages, and noindex for archive and search pages. Not sure? Leave both toggles off; the defaults work for most blogs.
How to use our free robots.txt generator (step by step)
Our free generator builds a correct robots.txt in under a minute: pick a platform, add extra folders and your sitemap, choose AI crawlers, then copy or download the file.
- Pick the default rule. Keep “Allow all crawlers” for any live website. Choose “Block everything” only for a staging or private site; the tool warns you when you do.
- Choose your “Website platform”. “WordPress” adds the
/wp-admin/rule with theadmin-ajax.phpexception. “WordPress + WooCommerce” also blocks cart, checkout, my-account and add-to-cart URLs. “Shopify (read-only, see note)” reminds you to edit the robots.txt.liquid template instead. For Blogger or a hand-coded site, choose “Other / custom site”. - Add “Extra folders or pages to block”, one per line, like
/thank-you/. Leave it empty if you’re unsure. - Paste your “Sitemap URL(s)”, one per line, each a full https:// address.
- Choose which AI crawlers may read your site. Under “Block AI crawlers”, tick any you want to keep out, such as GPTBot. Unticked bots stay allowed.
- Click “Copy” or “Download”, then upload the file to your site’s root folder or paste it into your platform’s robots.txt setting.
Open the free robots.txt generator
How to create robots.txt by hand
No tool needed. Google’s steps: write your rules in a plain text editor, save the file as robots.txt in UTF-8, upload it to your site’s root folder (hosting file manager or FTP), then test it. On WordPress with Yoast, you can also use Yoast SEO → Tools → File editor.
Not comfortable touching server files? Our website and CRO team handles this as part of a technical check.
How do you test robots.txt?
Test robots.txt three ways: open yourdomain.com/robots.txt in a private browser window, check the robots.txt report in Google Search Console, and run your key URLs through a robots.txt tester to confirm Googlebot is allowed.
Google Search Console’s robots.txt report
Search Console’s robots.txt report shows which robots.txt files Google found, when it last fetched them, and any rules it couldn’t parse. After fixing a serious mistake, open the menu next to the file and choose “Request a recrawl” so Google refreshes it sooner. The report works only for domain-level properties, and it won’t tell you if one specific URL is blocked. Google points you to the URL Inspection tool for that.
Our free robots.txt tester
Our robots.txt tester answers one question: is this exact URL blocked, and by which line?
- Choose Single URL, or Bulk (1,000) to paste up to 1,000 URLs, even from different sites.
- Pick a crawler: Googlebot, Googlebot-Image, Bingbot, GPTBot (OpenAI), OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot or “Any other bot (*)”.
- Click Fetch robots.txt & test to see Allowed or Blocked, plus the rule responsible.
- Planning a change? Paste or edit the rules in the robots.txt box and test again before you publish.
Test your homepage, a product or service page, a blog post and one CSS file. If any come back Blocked for Googlebot, fix it today.
Should you block AI crawlers like GPTBot in robots.txt?
You can block most AI crawlers with robots.txt, but choose carefully. Training crawlers (GPTBot, ClaudeBot) and search crawlers (OAI-SearchBot, PerplexityBot) are separate, and blocking the search ones can stop AI tools from showing or citing your site.
| Token | Company | Used for | If you block it |
|---|---|---|---|
GPTBot | OpenAI | Content that may train OpenAI’s models | Signals your content shouldn’t be used for training |
OAI-SearchBot | OpenAI | ChatGPT search | Not shown in ChatGPT search answers (may still appear as a plain link) |
ClaudeBot | Anthropic | Content that may train Anthropic’s models | Signals future content should be left out of training |
Google-Extended | Gemini training and grounding | No effect on Google Search ranking or inclusion | |
PerplexityBot | Perplexity | Perplexity search results | Your site may not be surfaced in Perplexity search |
CCBot | Common Crawl | An open web dataset anyone can access | CCBot stops crawling your site |
From each company’s crawler documentation, October 2026. OpenAI and Perplexity say changes can take up to about 24 hours to apply.
Want to stay out of AI training but still be found in AI search? Block the training bots only:
User-agent: *
Disallow:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Two caveats. User-triggered fetchers may not follow robots.txt: OpenAI says its rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. And for a local business, visibility usually matters more. If someone asks ChatGPT for “a dentist in Pune,” you probably want to be found.
Common robots.txt mistakes (and how to fix them)
The most common robots.txt mistakes are leaving a staging block live, blocking CSS or JavaScript, using robots.txt to remove pages from Google, putting the file in the wrong place and getting letter case wrong. Each is easy to fix once you spot it.
| Mistake | What happens | Fix |
|---|---|---|
Disallow: / left from staging | No crawler can read any page on your live site | Delete the line, then “Request a recrawl” in Search Console |
| Blocking CSS or JS files | Google may not understand how your pages look and work | Don’t block resources your pages need to display properly |
| Using robots.txt to hide a page | The URL can still show in Google, without a description | Allow crawling and add noindex, or password-protect it |
| File in the wrong place | example.com/blog/robots.txt is ignored | Put it at the root of each host: example.com/robots.txt |
| Wrong letter case | Disallow: /Admin/ doesn’t block /admin/ | Copy paths exactly as they appear in your URLs |
| Relative sitemap URL | Sitemap: /sitemap.xml isn’t valid | Use the full URL, starting with https:// |
After any change, check your sitemap too. Our free sitemap checker confirms the URL in your Sitemap line works and lists the pages you want crawled.
Custom robots.txt: frequently asked questions
What is robots.txt?
Robots.txt is a plain text file at the root of a website (example.com/robots.txt) that tells search engine crawlers which URLs they may crawl. It follows the Robots Exclusion Protocol, RFC 9309. It controls crawling, not whether a page appears in search results.
What is a custom robots.txt in Blogger?
It’s your own set of rules that replaces Blogger’s default robots.txt. You add it under Settings → Crawlers and indexing → Enable custom robots.txt. The default already blocks /search and lists your sitemap, so most blogs don’t need to change it.
Is robots.txt good for SEO?
A correct robots.txt helps SEO by keeping crawlers away from useless pages and pointing them to your sitemap. A wrong one can block your whole site. For most small sites, a short file that allows everything and lists the sitemap is best.
How do I create a robots.txt file?
Write your rules in a plain text editor, save the file as robots.txt in UTF-8, and upload it to your website’s root folder. On WordPress you can use an SEO plugin’s file editor, on Blogger the Crawlers and indexing setting, and on Shopify the robots.txt.liquid template.
How do I check if a page is blocked by robots.txt?
Paste the URL into a robots.txt tester and choose Googlebot; it shows Allowed or Blocked with the matching rule. In Google Search Console, the URL Inspection tool also tells you if a URL is blocked.
How long does Google take to see robots.txt changes?
Google generally caches robots.txt for up to 24 hours. After an important fix, use “Request a recrawl” in Search Console’s robots.txt report. That refreshes the file, but it doesn’t guarantee newly unblocked pages get crawled right away.
Can I use noindex in robots.txt?
No. Google doesn’t support noindex in robots.txt. Add a noindex robots meta tag to the page, or send an X-Robots-Tag HTTP header, and make sure the page isn’t blocked by robots.txt so Google can see it.
Does blocking GPTBot affect my Google rankings?
No. GPTBot is OpenAI’s crawler and Google Search doesn’t use it. Blocking Google-Extended doesn’t affect Google Search either, since Google says it isn’t a ranking signal. Just never block Googlebot itself.
Sources and how we researched this
We researched this guide in October 2026. We read Google’s robots.txt documentation (moved to Google’s crawling documentation site in late 2025), RFC 9309, Search Console Help, Blogger Help, WordPress, Yoast and Shopify documentation, and each AI company’s own crawler page. We also checked a live Blogger robots.txt to show the real default. AI crawler rules change often, so check the vendor pages before relying on a token.
- Google Search Central: Introduction to robots.txt (what robots.txt is for, and why it doesn’t keep pages out of Google)
- Google: How Google interprets the robots.txt specification (location, 500 KiB limit, 24-hour caching, status codes, wildcards, precedence)
- Google: Create and submit a robots.txt file (format, upload and testing)
- Google: Useful robots.txt rules (block-all, folder and crawler-specific examples)
- Google Search Central: Block indexing with noindex (noindex needs a crawlable page; not supported in robots.txt)
- Search Console Help: robots.txt report (what it shows, “Request a recrawl”)
- RFC 9309: Robots Exclusion Protocol (the standard; rules aren’t access authorization)
- Blogger Help: Manage your blog’s settings (Crawlers and indexing settings)
- Blogger’s official blog: live robots.txt (Blogger’s default file, checked October 2026)
- Auki Henry: Blogger SEO and AI indexing guide (commonly suggested robots header tag settings)
- WordPress code reference: do_robots() (WordPress’s default virtual robots.txt)
- Make WordPress Core: XML sitemaps in WordPress 5.5 (wp-sitemap.xml listed in robots.txt)
- Yoast: Edit robots.txt through Yoast SEO (Tools → File editor)
- wp-kama: WooCommerce wc_page_no_robots() (no-robots tag on cart, checkout and My Account)
- Shopify Help Center: Editing robots.txt.liquid (default file, editing, warnings)
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)
- Anthropic: Does Anthropic crawl data from the web? (ClaudeBot)
- Google: Common crawlers (Google-Extended, Googlebot-Image)
- Perplexity: Perplexity crawlers (PerplexityBot, Perplexity-User)
- Common Crawl: CCBot (crawler token and opt-out rule)
Spotted a rule that’s changed? Tell us and we’ll update this guide.
The bottom line
A custom robots.txt is a short file with a big job: telling crawlers where they’re welcome.
For most small business sites, the best version is boring. Allow everything, block admin and checkout areas, add your sitemap. Done.
Remember the rule that trips everyone up: robots.txt controls crawling, not indexing. To keep a page out of Google, let it be crawled and use noindex.
Build yours with the free robots.txt generator, then run your key pages through the robots.txt tester before you publish.
Not sure whether crawling, indexing or something else is holding your site back? Get a free growth audit and our SEO team will check your robots.txt, sitemap and indexing for you.
Founder of 93Digital. Rohit has worked in SEO and digital marketing since 2007, through every major Google update, and runs paid ads with one rule: a campaign has to pay for itself.