Crawling SEO
Discovers and fetches web pages by following links from known URLs so search engines can analyse them for indexing.
If you treat crawling and indexing as the same thing, you will assume every crawled page appears in search results — it does not.
Key points
- Internal links are the main path crawlers use to find deeper pages on your site.
- An XML sitemap helps Googlebot discover URLs more efficiently, but does not guarantee crawling.
- Orphan pages with no internal links may never be crawled at all.
- Crawling is a separate step from indexing; a page can be crawled and still not be stored in the search index.
- Blocking important resources with robots.txt can prevent crawlers from seeing your full page content.
A new blog post that never got crawled
You publish a blog post on your site but link to it only from a press release page that has no other internal links. Googlebot crawls your homepage and your top-level category pages every few days, but never visits the press release page. Six weeks later, the post still has zero impressions in Search Console. You add a link from a category page, and within 48 hours Googlebot fetches the post. The page is crawled, but still not indexed — that is a separate step in the seo indexing process.
Three situations where crawling changes your decision
- New site launch Without a sitemap submission or strong internal links, Googlebot may take weeks to discover your pages. Submit a sitemap via Search Console to speed discovery.
- JavaScript-heavy pages Googlebot may render JavaScript, but if critical content or links are loaded only after user interaction, the bot might not see them. Ensure key resources are in the initial HTML.
- Server errors spike If your server returns 5xx errors during a crawl cycle, Googlebot reduces its crawl rate. Monitor crawl errors in Search Console to avoid losing crawl budget.
Four ways it goes wrong
- Blocked by robots.txt A disallow directive for a page or resource stops Googlebot from fetching it entirely. Use robots.txt only for non-essential paths like admin areas.
- Orphan pages Pages with no internal links are invisible to crawlers unless submitted via a sitemap. Even then, a sitemap is a hint, not a guarantee.
- Slow server response If your server takes more than a few seconds to respond, Googlebot may abandon the crawl. Aim for under 200ms time to first byte.
- Redirect chains Multiple redirects between the starting URL and the final page waste crawl budget. Keep redirects to one hop where possible.
What to check before acting on crawl data
- Search Console reports The Crawl Stats report shows how many pages Googlebot requests per day and any errors. Use it to spot sudden drops or spikes.
- robots.txt testing Use the robots.txt Tester in Search Console to verify that important pages are not accidentally blocked. Test after every change.
- Sitemap coverage Check the Sitemaps report to see how many submitted URLs were actually crawled. A low ratio indicates discoverability issues.
Common questions
What is the difference between crawling and indexing?
Crawling is discovery and fetching of a page; indexing is storing and organising it for search results. A page can be crawled without being indexed.
How does a search engine work step by step?
It crawls URLs by following links, fetches page content, renders it if needed, then indexes the processed content for ranking in search results.
Sources
- Google Search Central: Crawling and indexing overview Official Google documentation on the crawling and indexing process
- Moz Beginner's Guide to SEO Explains how search engines discover and process web pages