Robots.txt Tester
Enter a page URL to fetch its site’s robots.txt and see whether Googlebot, Bingbot or AI crawlers are allowed to crawl it. You can also paste and edit rules to test changes before you publish them. Switch to Bulk to test up to 1,000 URLs at once, even from different websites.
Robots.txt tester
| URL | Result | Matching rule |
|---|
How to test a URL against robots.txt
- Enter the page URL you want to test, for example a product or blog page.
- Pick the crawler: Googlebot for search, or an AI crawler such as GPTBot.
- Click Fetch robots.txt & test. You’ll see Allowed or Blocked and the exact rule responsible.
- To try a fix, edit the rules in the box and click Test path again.
Reading the result
- Allowed with no matching rule means nothing in the file applies to that path.
- Blocked by Disallow shows the line number, so you can find and change it.
- Allowed by Allow means a more specific Allow rule overrides a broader Disallow.
Examples of how rules match
| Rule | Effect |
|---|---|
| Disallow: /admin/ | Blocks /admin/ and everything inside it, such as /admin/users |
| Disallow: /*?s= | Blocks any URL with ?s= in it, such as internal search results |
| Disallow: /*.pdf$ | Blocks URLs that end in .pdf |
| Allow: /wp-admin/admin-ajax.php | Re-allows one file inside a blocked folder (the longer rule wins) |
| Disallow: | An empty Disallow blocks nothing |
Want to create a new file? Use the robots.txt generator.
Frequently asked questions
How does the tester decide if a URL is blocked?
It follows Google’s documented rules: the crawler uses the most specific User-agent group that matches it, the longest matching rule wins, and if an Allow and a Disallow rule are equally long, Allow wins. * matches any characters and $ marks the end of the URL.
Can I test many URLs at once?
Yes. Choose Bulk and paste up to 1,000 URLs, one per line. The tool fetches each site’s robots.txt once (up to 50 sites per run) and shows Allowed or Blocked for every URL, with the matching rule.
Google retired its robots.txt Tester. What should I use?
Search Console now has a robots.txt report that shows which robots.txt files Google found and any errors. This tool fills the gap for testing individual URLs and trying rule changes.
If robots.txt blocks a page, will it disappear from Google?
Not always. A blocked page can still be indexed without its content if other pages link to it. To keep a page out of search results, allow crawling and use a noindex tag.
What happens if there is no robots.txt?
If robots.txt returns 404, crawlers assume everything is allowed. If it returns a server error (5xx), Google may pause crawling the site until it can read the file.
Can I test changes before publishing?
Yes. Edit the rules in the box and click Test path. Nothing is saved or published.
More free tools
Free growth audit
Pages not showing up on Google?
Crawl blocks, noindex tags and canonical mistakes are common reasons pages never rank. Our technical SEO review finds them. Book a free 30-minute call.