What does “Blocked by robots.txt” mean and how do you fix it?
robots.txt stops Googlebot from fetching the URL. A Disallow rule in your robots.txt forbids crawling this URL. Google cannot read the content; if other sites link to it, it can still appear in the index without content.
Contents
What does this status mean?
A Disallow rule in your robots.txt forbids crawling this URL. Google cannot read the content; if other sites link to it, it can still appear in the index without content.
Possible causes
- A "Disallow: /" rule added during development was forgotten live.
- A broad pattern (for example /*?) also catches pages you want.
- The CMS or a plugin rewrote robots.txt on its own.
- Sitemap URLs sit in folders closed by robots.txt.
How to fix it
- Use the Search Console robots.txt report or a robots tester to find which rule closes the URL.
- Narrow or delete the rule; close only folders that really should not be crawled.
- Verify that sitemap URLs do not conflict with robots.txt.
- After fixing, give Google a day to refresh its robots.txt cache or request a recrawl in the report.
- Send the URL for indexing and follow progress with URL inspection.
Steps for your platform
Open the matching fix page for menu names and setting locations on your platform:
- WordPress
- WooCommerce
- Shopify
- Wix
- Blogger
- Webflow
- Squarespace
- Joomla
- Drupal
- Magento
- OpenCart
- PrestaShop
- Ticimax
- IdeaSoft
- Next.js
- React SPA
- Laravel
- Ghost
After fixing
In Search Console start "Validate fix" for the status, resubmit the page with URL inspection and confirm the blocker is gone with the free Free indexability checker tool. For many URLs see Bulk URL indexing: how to send thousands of pages to Google. Validation can take days to weeks.
Related terms
Frequently asked questions
Does robots.txt remove a page from the index?
No. It blocks crawling, it does not remove from the index. To remove, allow crawling and use noindex.
How do I test robots.txt?
Enter the file and a URL in our free robots.txt tester to see the Googlebot allow status.