Log File Analysis
Reveals exactly which URLs search engine bots request from your server, giving raw evidence of crawl behaviour that analytics tools cannot see.
If you have ever relied on crawl estimates from a tool instead of server logs, you already know the difference between what a bot actually did and what you guessed it did.
Key points
- Collect at least 30 days of logs to spot crawl trends and sitewide issues.
- Filter by Googlebot user agent, but also check other bots and non-HTML resources.
- Parse logs into a tabular format before analysis; raw logs are hard to filter.
- Cross-reference log data with status codes to find wasted crawl budget.
A 50,000-page site finds 80% of crawl budget wasted
A large e-commerce site collected 60 days of server logs and parsed them by URL pattern. The data showed that Googlebot spent 80% of requests on parameter-heavy filter pages that returned 200 status codes but thin content. Product pages with full descriptions were crawled only once every two weeks. The logs also revealed a spike in 404 errors after a site migration that analytics had missed. The team used this evidence to block filter URLs in robots.txt and prioritise product pages in the XML sitemap, applying a distilled SEO approach by focusing only on the most impactful signals.
Three things log analysis reveals that analytics cannot
- Actual bot behaviour Server logs record every request from every user agent, so you see exactly what Googlebot crawled, when, and how often — not an estimate from a crawler tool.
- Crawl budget waste Logs show which URLs consume the most requests. If Googlebot hits thin pages or redirect chains, you can block those paths in robots.txt to preserve crawl budget for important content.
- Status code patterns Every HTTP status code is logged. A sudden jump in 404s or 500s after a site change is visible immediately, unlike analytics which may not capture server errors.
Three common mistakes in log file analysis
- Too short a window A few days of logs miss crawl trends. SEO guides recommend at least 30 days, and about two months for crawler-specific patterns.
- Ignoring other bots Focusing only on Googlebot misses crawling by Bingbot or others, which can affect server load or signal security issues.
- Skipping log parsing Raw logs are unstructured. Without parsing into a tabular format by date, URL, and status code, you cannot filter or visualise effectively.
Common questions
How do I access your server logs?
Check the hosting dashboard, raw access logs, file manager, or FTP. The method varies by host, so look for 'access logs' or 'raw logs'.
What fields in a log file matter for SEO?
IP address, user agent, URL, timestamp, HTTP status code, and referrer are the key fields to parse and analyse.
Sources
- Google Search Central Documentation Official guidance on crawl budget and server logs from Google.
- SEMrush Log File Analysis Guide Practical workflow for collecting, parsing, and analysing server logs for SEO.
- Botify Log File Analysis Detailed explanation of log fields and how to interpret crawl patterns.