RankpediaA plain-English encyclopedia of search

Crawl Budget

Determines how many pages Google will crawl on your site in a given period, balancing its interest in your content with your server's capacity.

If you run a small blog, you probably never need to think about crawl budget—but on large e‑commerce sites, wasted crawling can keep new products out of the index.

Key points

  • Consolidate duplicate content so crawlers focus on unique pages rather than multiple identical URLs.
  • Fix soft 404s and long redirect chains before you invest in faster hosting.
  • Keep your XML sitemap up to date and include only canonical URLs you want crawled.
  • Use robots.txt to block low-value sections like faceted navigation filters, not to hide thin content.
  • Monitor crawl stats in Search Console after site migrations or template changes to spot new inefficiencies.

A product catalogue that grew too fast

A mid‑sized e‑commerce site added 50,000 new product pages in a month. Google’s crawlers spent 80% of their daily budget on duplicate descriptions and soft 404s from discontinued items. The new products took three weeks to appear in search results. After consolidating duplicates and updating the sitemap, crawl budget shifted to the new pages, and they were indexed within days.

Three ways crawl budget gets wasted

  • Duplicate content When the same product description appears at multiple URLs, Google may crawl each one, spending budget on pages that add no value. Consolidate with canonical tags or redirects to focus crawling on the original.
  • Soft 404 pages Pages that return a 200 status but show a 'not found' message keep getting crawled repeatedly. Use a true 404 or 410 status to tell Google to stop wasting budget on them.
  • Long redirect chains A chain of three or more redirects forces Google to follow each hop, consuming crawl budget without reaching the final page. Keep redirects to one hop where possible.

Three signals that increase crawl demand

  • Content freshness Google is more likely to recrawl pages that change frequently, such as news articles or product stock updates. Regular updates signal that the page deserves more of the budget.
  • Popularity and uniqueness Pages that attract links, shares, and user engagement tell Google the content is valuable. Unique, high‑quality content earns higher crawl demand than thin or syndicated material.
  • User value signals High dwell time, low bounce rate, and strong click‑through rates suggest the page satisfies searchers. Google may increase crawl demand for pages that demonstrate real user value.

Three checks before you optimise crawl budget

  • Audit your sitemap Ensure your XML sitemap lists only canonical, indexable URLs. Remove low‑value pages like parameter‑heavy faceted navigation URLs that waste crawl budget.
  • Review robots.txt Check that you are not accidentally blocking important pages. Use robots.txt to exclude sections like search results or infinite scroll archives that add no value.
  • Analyse crawl stats Use Search Console’s crawl stats report to see how many pages Google crawls per day and which status codes dominate. Spikes in 404s or redirects indicate wasted budget.

Common questions

What exactly is crawl budget?

It is the number of pages Google will crawl on your site per day, based on your server’s capacity and the perceived value of your content.

How to check crawl budget?

Use the Crawl Stats report in Google Search Console to see daily crawl counts, response times, and status code distributions.

Sources

  1. Google Search Central: Crawl budget management Official Google documentation defining crawl budget and how to manage it.
  2. Google Search Central: Crawl stats report Guide to interpreting crawl statistics in Search Console.
  3. Google Search Central: Manage crawling and indexing of site URLs Overview of crawling, indexing, and sitemap best practices.
↑