Robots.txt
Tells compliant search-engine crawlers which URLs or directories on your site they may access or should avoid, helping manage crawl traffic and server load effectively.
If you have used robots.txt to keep a page out of Google's index, you already know it is not a reliable indexing control and can cause more harm than good when misapplied.
Key points
- Place robots.txt only in your site's root directory; any other location is ignored by compliant crawlers.
- Use robots.txt to reduce crawl waste on low-value pages, not to block indexing or hide content.
- Test robots.txt changes in Search Console before assuming they work correctly for Googlebot.
- Remember that not all bots respect robots.txt; it is a guideline for compliant bots, not a barrier.
- Small sites may not need a robots.txt file at all; omitting it is often fine and simpler.
- Allow CSS and JavaScript files that crawlers need for rendering; blocking them can hurt indexing and rankings.
Three things robots.txt is not
- Not a security measure robots.txt is a public file that any user or bot can read. It does not prevent access; it only requests compliant bots to stay out of specified areas.
- Not an indexing control Google explicitly states robots.txt is not a mechanism for keeping a page out of Google. Use noindex or password protection instead for indexing control.
- Not a way to hide pages Blocked URLs can still be indexed if linked from elsewhere. Robots.txt only stops crawling, not indexing, so it cannot hide content from search results.
Four common robots.txt mistakes
- Blocking resources needed for rendering Disallowing CSS or JavaScript can cause Googlebot to see broken pages, hurting rankings. Always allow assets needed for proper rendering and indexing.
- Using robots.txt to block indexing This is the most common error. Blocked URLs may still appear in search results if they have external links. Use meta robots noindex instead for reliable blocking.
- Placing the file in the wrong directory robots.txt must be at the root of the host. A file in a subdirectory is ignored by crawlers, so verify the location carefully.
- Assuming all bots obey Malicious crawlers ignore robots.txt entirely, so it is a guideline for compliant bots rather than a security control. Never rely on it to protect sensitive content.
Three situations where robots.txt changes your crawl strategy
- Large sites with crawl budget issues Blocking low-value sections (e.g., search results, pagination) can help Googlebot focus on important pages. Use log file analysis to identify wasted crawl requests and adjust accordingly.
- Sites with faceted navigation Faceted navigation seo often generates thousands of parameter-heavy URLs. Blocking them in robots.txt preserves crawl budget for product pages and improves crawl efficiency.
- Sites using Next.js or similar frameworks Modern crawlers need JavaScript and CSS to render pages. Blocking these resources can break next seo performance; always allow them in robots.txt for proper rendering.
Common questions
Can robots.txt keep a page out of Google?
No, robots.txt only prevents crawling, not indexing. A blocked page can still appear if linked from elsewhere.
Where should robots.txt be placed?
In the root directory of your site. Any other location is ignored by compliant crawlers.
Do all crawlers follow robots.txt?
No, only compliant bots like Googlebot and Bingbot. Malicious crawlers ignore it, so it is not a security measure.
Does my small site need a robots.txt file?
Probably not. Google says small sites may not need one; omitting it is fine and simpler to manage.
Sources
- Google Search Central — robots.txt Introduction and Guide Official Google documentation on robots.txt purpose and usage for crawl management.
- Google Search Console Help — robots.txt Troubleshooting and testing robots.txt in Search Console for correct implementation.
- MDN Web Docs — Robots.txt glossary General web developer reference for the Robots Exclusion Protocol and its syntax.