Website SEO Crawler

Enter your website and the crawler follows your internal links like a search engine, up to 500 pages. You get a prioritised issue list (broken pages, redirects, duplicate titles and content, missing tags, noindex and canonical problems, thin pages, orphan pages from your sitemap) and a full page table you can export.

How to crawl your website

1. Enter your homepage (or any start URL) and choose how many pages to crawl: up to 500 while you wait, or 750 to 1,000 as an email report.
2. Keep “Respect robots.txt” and “Compare with XML sitemap” on for a search-engine view of your site.
3. Click Start crawl. The crawler follows internal links three pages at a time and shows the time left. Don’t want to wait? Tick Email me the report, close the page, and open the link from your inbox.
4. Work through the Issues list from the top: errors first, then warnings, then notices. Each issue shows the affected URLs and which pages link to them.
5. Use the Pages table to filter by status or search, and tap a page to see its details and inlinks.
6. Export the pages, issues and links as CSV, or download a ready sitemap.xml of your indexable pages.

What to fix first

• Broken pages and server errors. Fix them or 301-redirect them, then update the links.
• Accidental noindex, canonical or robots.txt blocks on pages that should rank.
• Duplicate titles and content. Give every page its own title and purpose, or merge duplicates.
• Internal links to redirects. Point links straight to the final URL.
• Orphan and weakly linked pages. Link to important pages from related content and menus.
• Thin pages, missing H1s and images without alt text on pages that matter.

Good to know

• Only internal links on the same domain are crawled. Subdomains are treated as separate sites.
• Links in content loaded only by JavaScript may not be found, because the crawler reads the HTML your server sends.
• Response times are measured from our server, so they show server speed, not full page load. Use Google PageSpeed Insights for Core Web Vitals.
• Crawling a big site in sections? Start from a folder such as /blog/ and tick “Only crawl inside this folder”.

Need just one page in depth? Use the on-page SEO checker. For a sitemap only, use the XML sitemap generator.

Frequently asked questions

How is this different from Screaming Frog?

Screaming Frog is a desktop program that crawls up to 500 URLs free and more with a paid licence. This crawler needs nothing installed: it crawls up to 500 pages while you keep the page open, or up to 1,000 pages on our server with the report link sent to your email. For very large sites a desktop crawler is still the right tool.

What does the crawler check?

Status codes (4xx, 5xx, redirects and chains), robots.txt blocks, noindex and canonical tags, missing, duplicate, long or short titles and meta descriptions, missing or multiple H1s, thin content, exact and near-duplicate content, images without alt text, crawl depth, weakly linked pages, slow or heavy pages, URL format, hreflang return links, structured data and, if you have a sitemap, orphan pages and sitemap problems.

How does duplicate content detection work?

For each page the crawler reads the main content (ignoring header, footer and menus). Exact duplicates have identical text. Near-duplicates are found by comparing overlapping three-word phrases, and pages that are 90% or more similar are flagged.

Why did the crawl stop before finishing my site?

A live crawl covers up to 500 URLs, and an email report up to 1,000. If your site is bigger, crawl key sections one at a time by starting from a folder URL and ticking Only crawl inside this folder.

How long does a crawl take?

Usually about 2 to 4 minutes for 500 pages, depending on how fast the website responds. The tool shows an estimate before you start and the time left while it runs. Email reports wait in a short queue if others are running, and the status box shows your place and the expected time.

Does crawling slow down my website?

A live crawl fetches three pages at a time, and email reports fetch four at a time with short pauses, which is lighter than a few people browsing your site. Both respect robots.txt by default.

Is my data stored?

Live crawls are analysed in your browser and nothing is saved. For an email report, we store the crawled pages on our server for 7 days so your link works, and keep your email address as described in our privacy policy.

More free tools

See all free marketing tools