robots.txt tester
When a page does not get indexed, robots.txt is the first place to look. Enter the address and we read the file and show the Googlebot result.
How it works
It downloads /robots.txt from the domain of the address you enter, extracts the rules that apply to Googlebot and shows whether the address is allowed or blocked. The longest matching rule wins; at equal length Allow beats Disallow.
Common robots.txt mistakes
- A "Disallow: /" line moved to production: Blocked by robots.txt
- Using robots.txt alone to hide a page: Indexed, though blocked by robots.txt
- Sitemap URLs closed by robots.txt: How to build a sitemap and submit it to Search Console
robots.txt manages crawling, it does not remove from the index. If you want a page to stay out of the index use noindex. For the term itself see robots.txt.
Frequently asked questions
What if I have no robots.txt?
If the file is missing (404) crawlers are free to crawl everything.
What if the result says "blocked"?
Narrow or delete the rule shown in the result and resubmit the page; the Search Console robots.txt report also helps.