Dev here, posting because I’m the one who broke it.
We have staging and production behind the same CDN. Staging sends X-Robots-Tag: noindex. Last month a config change made the cache key ignore the hostname for a subset of paths, so for about 3 days roughly 1,800 production product pages were served with the staging header to anyone, including Googlebot.
What happened after:
– day 4: GSC showed ~600 pages as “Excluded by noindex tag”
– impressions on those pages dropped about 70% within a week
– we fixed it, requested reindexing for the top 100, left the rest
– 3 weeks later about 85% are back, long tail still trickling in
What I added since: a synthetic check that fetches 20 production URLs every 15 min and alerts if any robots header or meta robots appears.
Questions for the SEO folks: is there anything faster than waiting for recrawl for the remaining few hundred? And what other header-level checks would you put in a monitor like this?
.
