Free tools

robots.txt tester

When a page does not get indexed, robots.txt is the first place to look. Enter the address and we read the file and show the Googlebot result.

How it works

It downloads /robots.txt from the domain of the address you enter, extracts the rules that apply to Googlebot and shows whether the address is allowed or blocked. The longest matching rule wins; at equal length Allow beats Disallow.

Common robots.txt mistakes

robots.txt manages crawling, it does not remove from the index. If you want a page to stay out of the index use noindex. For the term itself see robots.txt.

Frequently asked questions

What if I have no robots.txt?

If the file is missing (404) crawlers are free to crawl everything.

What if the result says "blocked"?

Narrow or delete the rule shown in the result and resubmit the page; the Search Console robots.txt report also helps.