Log File Analyser
Drop in your server access log and see exactly what search engine and AI bots did on your site: which URLs Googlebot crawled, how often, what status codes it got, and where crawl budget is wasted on redirects, errors and parameter URLs. Everything runs in your browser, so the log never leaves your device.
Log file analyser
Or paste log lines
How to analyse your log files
- Download your access logs from your hosting panel. A few weeks of data gives the clearest picture.
- Choose the files (several at once is fine, .gz included) or drop them on the box.
- Tick Verify to separate real Googlebot and Bingbot from fake bots.
- Click Analyse logs, then use the tabs: Bots, Googlebot URLs, Errors & redirects, By day, By folder and Top bot IPs.
- Download any table as CSV, or the full list of URLs crawled by bots.
What to look for
- Important pages Googlebot rarely visits: add internal links and make sure they are in your sitemap.
- Crawl budget spent on junk: parameter URLs, filters, old redirects and 404s. Fix links and consider robots.txt rules.
- 5xx errors served to bots: repeated server errors can make Google slow down crawling.
- Sudden drops in Googlebot visits: check the By day tab against deploy dates and Search Console.
- Heavy AI crawler traffic: decide which AI bots you allow, then set it with the robots.txt generator.
Compare what bots crawl with what your site links to by running the website SEO crawler, and test single URLs with the robots.txt tester.
Frequently asked questions
What is log file analysis in SEO?
Every request to your website, including from Googlebot, is recorded in your server’s access log. Analysing it shows what search engines actually crawl, instead of guessing. It reveals pages Google ignores, crawl budget spent on junk URLs, and errors that only bots see.
Where do I get my log files?
Look in your hosting control panel for Raw Access Logs or Access Logs, or ask your host. On a VPS they are usually at /var/log/nginx/access.log or /var/log/apache2/access.log. If your site is behind Cloudflare or another CDN, its logs may be more complete.
Which log formats work?
The standard Apache / Nginx / LiteSpeed combined format, which most hosts use. Plain .log or .txt files and .gz compressed files are supported. Lines in other formats are skipped and counted, and the first one is shown so you can check.
Are my logs uploaded anywhere?
No. The file is read and analysed in your browser. If you tick Verify, only the IP addresses of bots claiming to be Googlebot or Bingbot are looked up on Google Public DNS.
How does Googlebot verification work?
Anyone can fake a Googlebot user agent. A real Googlebot IP has reverse DNS ending in googlebot.com or google.com, and that hostname resolves back to the same IP. Bingbot uses search.msn.com. This is the method Google and Bing recommend.
How big a log can I analyse?
Logs of several hundred MB usually work on a laptop because the file is read line by line. Up to 200,000 unique URLs are listed; any extra are still counted in the totals.
More free tools
Free growth audit
Googlebot crawling the wrong pages?
We turn log file findings into fixes: internal links, redirects, robots.txt and sitemaps that point Google at the pages that earn you money. Book a free 30-minute call.