RankpediaA plain-English encyclopedia of search

Screaming Frog

Crawls a website locally to collect page-level data for technical SEO audits, surfacing issues like broken links, missing metadata, and redirect chains.

If you have ever run a crawl without configuring JavaScript rendering or exclusions, you already know the data can mislead as easily as it informs.

Key points

  • Set up JavaScript rendering before crawling any site that loads content dynamically.
  • Use the free version only for sites under 500 URLs; anything larger needs a paid licence.
  • Prioritise issues by impact on crawlability and indexability, not by count.
  • Check that the site does not block bots or require authentication before you trust the crawl results.
  • Export crawl data to a spreadsheet for sorting and filtering beyond what the tool offers.

A 10,000-page site with 300 duplicate title tags

An ecommerce site with 10,000 product pages ran a Screaming Frog crawl and found 300 duplicate title tags. Most were on paginated category pages that shared a template. The team fixed the template, added canonical tags, and re-crawled. The duplicate count dropped to 12. The crawl data made the pattern visible in minutes instead of hours.

Three situations where this changes your decision

  • JavaScript-heavy sites Without enabling JavaScript rendering, the crawler sees only the raw HTML. You miss content loaded by scripts, which can hide broken links or missing metadata.
  • Large-scale audits The free 500-URL cap forces you to sample. For a full site audit, you need the paid licence to crawl every page and get a complete picture of issues like duplicate content or crawl budget waste.
  • Custom extraction needs When you need data not in the default reports — prices, schema values, or specific text — use CSS Path, XPath, or regex to extract it. This turns the crawl into a tailored dataset.

Four ways it goes wrong

  • Unconfigured exclusions Crawl session URLs, filter parameters, or staging environments can inflate your URL count and distort reports. Set exclusions before you start.
  • Ignoring authentication Password-protected sections or login walls block the crawler. If you do not provide credentials, you miss entire parts of the site.
  • Treating every issue equally A missing meta description on a low-traffic page matters less than a broken link on your homepage. Prioritise by template impact and user experience.
  • Stale crawl data A crawl from last month may not reflect current site structure. Re-crawl after significant changes to keep your technical SEO audit accurate.

What it costs you to get wrong

  • Wasted developer time Flagging false positives from an unconfigured crawl sends developers chasing issues that do not exist, eroding trust in the tool.
  • Missed critical issues Failing to enable JavaScript rendering can hide a broken checkout flow or missing structured data, directly affecting revenue.
  • Incorrect prioritisation Without sorting by impact, you might fix 50 low-value issues while a redirect chain on your top landing page stays broken.

Common questions

Is Screaming Frog safe?

Yes, it is a read-only crawler that does not modify your site. It requests pages like a browser and stores data locally.

How much does a Screaming Frog licence cost?

A paid licence starts at $279 per year for 1–4 licences, removing the 500-URL limit and unlocking advanced features.

What is the URL limit for the free version?

The free version crawls up to 500 URLs, suitable for small sites or testing before committing to a paid licence.

Sources

  1. Screaming Frog SEO Tools Official site with product overview and feature list.
  2. Screaming Frog Pricing Current pricing and licence options.
  3. Screaming Frog Blog: 11 Little-Known Features in the SEO Spider Details on custom extraction, JavaScript rendering, and integrations.
↑