Google Is Indexing Parameters You Blocked in Robots.txt

Robots.txt controls crawling, not indexing, and that one distinction explains why parameter URLs you blocked keep turning up in search. Google can list a URL it has never fetched, learning that it exists from links. An easy reflex, adding more Disallow lines, preserves the problem: a noindex only works if Google can crawl the URL and see it, and the block guarantees it can’t. To clean parameter URLs out of the index, you have to let Google back in to read the instruction, then decide whether to block again.

A crawl barrier isn’t an index barrier

Google’s introduction to robots.txt says robots.txt is used mainly to avoid overloading your site with requests and is not a mechanism for keeping a web page out of Google. A blocked URL can still appear in search results, without a description, because Google knows the address from links even though it never read the page.

The help for the Page indexing report says the same about URLs blocked by robots.txt: the block doesn’t guarantee the page won’t be indexed through some other means. And it gives the fix in one line: to make sure a page isn’t indexed by Google, remove the robots.txt block and use a noindex directive.

The paradox, stated precisely

noindex has to be crawled to be seen, so a blocked URL hides its own noindex. Google’s guide to blocking indexing with noindex says that for the rule to be effective, the page must not be blocked by robots.txt and must be accessible to the crawler. If you add noindex to parameter URLs and also disallow them, Google never fetches them and never reads the rule.

The same applies to everything else you might add to a blocked URL. A canonical tag on a blocked page is never read, so it consolidates nothing. Listing a blocked URL in a sitemap doesn’t grant permission to crawl it. The block that was meant to clean up the index is what keeps the removal instruction from reaching them.

Deliver noindex by pattern with X-Robots-Tag

Parameter URLs follow patterns, which makes the X-Robots-Tag response header a practical delivery path: the server can add X-Robots-Tag: noindex to any response whose URL matches the parameter pattern, without touching each template. It has the same effect as the meta tag, with the same condition: Google has to fetch the response to read the header. On a URL blocked in robots.txt, the header is as invisible as a meta tag.

The cleanup sequence

1. Inventory what is indexed. Use a site: search with the parameter and the Page indexing report to find which parameter patterns have indexed URLs. Treat site: as a spot check, not a count.
2. Serve noindex on those patterns, through X-Robots-Tag or a meta tag.
3. Remove the robots.txt block for the patterns that have indexed URLs. Patterns with nothing indexed can stay blocked.
4. Wait for Google to recrawl the unblocked URLs, read the noindex and drop them. Watch the indexed URLs for those patterns fall in the Page indexing report and in spot checks.
5. Decide whether to block again. Once the indexed URLs are gone, you can restore the Disallow to stop the crawling. From then on, new URLs in the pattern can’t show Google their noindex, so the lasting fix includes no longer linking to parameter URLs you don’t want found.

The trade-off is temporary crawling on URLs you’d rather not have crawled, in exchange for being able to deliver the removal instruction at all. The blanket block made that impossible.

Removals: a temporary hide

When a parameter URL has to disappear from results quickly, the Removals tool can hide it, but not permanently. Google’s help for the Removals tool offers two options:

• Temporarily remove URL blocks the URL from Google Search results for about six months. Google says to make the removal permanent, or the page could appear again.
• Clear snippet in search wipes the page description in results until the page is indexed again. The page can still appear. It suits a page where you removed sensitive text, not an over-indexed parameter set.

Use the temporary removal for an urgent case, and set up the noindex sequence at the same time so something permanent is in place before the removal expires.

What to do

• Stop adding Disallow lines for parameter URLs that are already indexed.
• Serve noindex on the indexed patterns, preferably with X-Robots-Tag set by URL pattern.
• Unblock those patterns until Google has recrawled them and dropped them.
• Block again only once they are gone, and keep parameter URLs you don’t want found out of your internal links.
• Reserve the Removals tool for urgent, temporary hiding.

Frequently asked questions

Why are my blocked parameter URLs showing up in Google?

Google can learn the URLs from links and list them without ever fetching them. That is why such results show without a description, and why the block can’t remove them.

Can I add noindex and keep the robots.txt block?

No. Google’s noindex guide says the page must not be blocked by robots.txt for the rule to be effective. Unblock the URLs you need removed, let Google read the noindex, then block again if you want the crawl savings.

Does the URL Removals tool fix this?

Not permanently. A temporary removal lasts about six months, and the URL can return afterward unless a permanent fix, such as a noindex Google can read, is in place.