In Google’s latest Q&A video, John Mueller addresses a site owner’s concern about their website not being crawled frequently enough.
“How often does Google re-index a website? It seems like it’s much less often than it used to be. We add or remove pages from our site, and it’s weeks before those changes are reflected in Google Search.”
Mueller clarifies that it takes time to crawl entire websites. If Googlebot were to crawl an entire site quickly, it would cause unnecessary strain on the server. Therefore, Googlebot has a strict daily limit on the number of pages it can crawl (though the exact number is not specified).
Given that Googlebot can only crawl a limited number of pages, it prioritizes pages that should be crawled more frequently. According to Google, the most “important” pages are the homepage and high-level category pages. These pages are crawled most often, so changes will appear in Google Search faster than, for example, an old blog post.
Googlebot will re-crawl important pages at least every few days, possibly even more frequently in some cases. Mueller did not specify how often other types of pages are crawled. Considering this, it appears that patience is crucial for waiting for changes on other pages to be reflected in Google Search.
John Mueller’s full response is quoted below.
“Looking at the whole website all at once, or even within a short period of time, can cause a significant load on a website. Googlebot tries to be polite and is limited to a certain number of pages every day. This number is automatically adjusted as we better recognize the limits of a website. Looking at portions of a website means that we have to prioritize how we crawl.
So how does this work? In general, Googlebot tries to crawl important pages more frequently to make sure that most critical pages are covered. Often this will be a website’s home page or maybe higher-level category pages. New content is often mentioned and linked from there, so it’s a great place for us to start. We’ll re-crawl these pages frequently, maybe every few days, maybe even much more frequently depending on the website.”
