In a webmaster hangout, a publisher inquired about the speed at which Google removes pages from its index after adding a noindex nofollow to them. The publisher noted that they had applied a noindex, but the page still appeared in Google’s index. Google’s John Mueller provided insight into how frequently some pages are indexed.
John Mueller explained that URLs are crawled at varying rates, which is commonly known. However, of particular interest was his comment that some URLs might be crawled as infrequently as once every six months.
The publisher stated:
“We’re noticing instances from a long time ago where we’ve changed the noindex nofollow, but it’s still in the index. This is happening several months after making the change.”
John Mueller responded:
“I think the challenging aspect is that we don’t crawl URLs with a uniform frequency. Some URLs are crawled daily, others weekly, and some potentially every few months or even once every six months.
We aim to strike a balance to avoid overloading your server. If you’ve made significant changes across your website, many will be recognized quickly, but some will linger.
When conducting site queries, you might find URLs that only get crawled every six months still appearing after a few months. This is part of our standard reprocessing/re-crawling timeframe. It doesn’t necessarily indicate a technical issue.
If you believe certain URLs shouldn’t be indexed, consider supporting this with a sitemap file that includes the last modification date, prompting Google to recheck these more swiftly.”
Use The Site Map to Trigger Updated Crawling
John Mueller suggested using an updated sitemap to let Googlebot identify the last modified date, prompting it to crawl old web pages.
Google URL Inspection Tool
While John Mueller didn’t mention it, Google’s URL Inspection tool can be useful. According to Google’s Webmaster Help page, a submission might take one to two weeks to process.
The URL Inspection tool works well for re-crawling a few individual URLs. For a large number of pages, Google recommends submitting a sitemap instead.
Watch the Webmaster Hangout
Images by Shutterstock, Modified by Author
