Certainly! Here is a revised version of your article:
In the Settings section of Google Search Console, there exists a report that isn’t commonly discussed among SEO professionals, but I find it valuable to monitor.
These are referred to as Crawl Stats.
This section provides a fascinating array of metrics concerning Googlebot’s crawl activity, particularly useful for websites with extensive page quantities.
Googlebot ‘Crawl Stats’
Historically, Google Search Console readily displayed metrics on Googlebot crawl activity. Over time, these seemed to have vanished but are now hidden within the Settings area.
How to access the crawl stats reports:
- Select Settings at the bottom of the left navigation panel.
- Navigate to the Crawl stats section.
- Click Open report.
About the Crawl Stats Data
As Googlebot explores your site, Google logs various aspects of its activity, presenting them in the crawl stats section. This is where you find foundational statistics about Google’s crawling behavior on your site.
Google says this data is for advanced users
The data on Googlebot Crawl Stats isn’t suited for those new to technical SEO. Google labels this data as intended for "advanced users" with substantial page counts on their sites, possibly explaining its discrete location in the interface, unnoticed by many in the SEO community.
Factors such as network issues and cloud services like Akamai can impact these metrics, which may account for Google’s classification of this report as advanced.
Who Will Find Crawl Stats Most Useful?
In my opinion, Crawl Stats are not strictly "advanced" reports. Instead, they offer utility to enterprise SEOs without access to crawler monitoring tools like Lumar and Botify.
In enterprise SEO, handling a website with thousands or millions of pages, optimizing crawler efficiency is crucial. The activity metrics here guide essential steps for advancement.
For smaller sites, crawler activity isn’t generally a pressing issue since they are likely allotted sufficient crawl budget to ensure proper crawling speed.
However, enterprise websites often have far more pages than Googlebot might crawl daily, necessitating optimization for crawler activity to guide future actions.
What to Look for in This Data
After years of data evaluation across various sites, my main guideline is:
- Do not invest excessive time in these reports unless you notice patterns and connections.
Many times, these reports are intriguing but not actionable. Examples warranting investigation include:
- HTML request changes coinciding with JavaScript data fluctuations.
- Response time changes correlating with HTML request adjustments.
- Total download size variations without changes in HTML requests.
- Increased discovery requests without a corresponding URL launch on the site.
When to Examine Crawl Stats
Monthly examinations of crawl stats are advisable. They are especially useful after substantial updates like platform migrations or designs to understand Google’s reactions to such changes.
Remember, if you possess a bot monitoring tool like Lumar or Botify, these metrics might not hold as much importance as those provided by such tools.
Cautions about the Crawl Stats Data
Numerous factors beyond a standard development update can sway Google’s crawl stats metrics.
It’s crucial for the teams involved—SEO, product managers, and developers—to adopt a broader perspective when assessing fluctuations, considering potential influencers both within site updates and the broader network and tech configuration.
Alterations in systems like Akamai could impact this data.
Log the Crawl Stats Data in a Spreadsheet
I prefer archiving this data since Google’s reports only reflect a brief timeframe.
An ongoing challenge with a client illustrates this: current reports suggest improvement, yet my archive shows metrics are still 40% higher than they were six months ago.
This de-identified analysis reveals ongoing issues that can perplex development, and addressing those is beyond this article’s scope.
Screenshots may seem handy, but they complicate long-term comparisons since charts lack clear axis labeling, underscoring the importance of spreadsheet documentation for clear, comparable records over time.
Precise historical data can provide crucial context during discussions about metrics with development teams.
Keep in Mind that the Reports’ Sections are Paginated
Key metrics are likely in view by default, yet many report sections have additional pages, easily overlooked.
Which Metrics to Monitor and Why
Let’s delve into the key metrics to examine each month, along with tips for extracting actionable insights.
Total Crawl Requests
- Review this in Google Search Console (found in the top chart).
- Google defines it as “the total number of crawl requests issued for URLs on your site, whether successful or not.”
- If fluctuations occur, compare with average response time and total download size.
An increase might be tied to code or page changes, but that is not the sole reason.
Total Download Size (Byte)
- Review this in Google Search Console (found in the top chart).
- Google’s definition: “Total bytes downloaded from your site during crawling for the specified period.”
- If fluctuations occur, compare with average response time.
Such increases may result from large-scale code additions or page launches.
Average Response Time (ms)
- Google Search Console report (in the upper chart).
- Google’s description: “Average response time for resources fetched during the specified period.”
- Fluctuations should be compared against total crawl requests and total download size.
Crawl Requests Breakdown by Response
- View this in Google Search Console (positioned below the top chart).
- Google describes this as “the responses Google received when crawling, grouped by response type, as a percentage of all crawl responses…”
- Typical responses include: OK (200), Moved permanently (302), Server error (5xx), Other client error (4xx), Not found (404), Not modified (304), Page timeout, Robots.txt unavailable, Redirect error.
Crawl Requests Breakdown by File Type
- Review this in Google Search Console.
- Google explanation: “The file type returned by the request. This reflects the percentage of responses of each type, rather than the bytes retrieved.”
- Common responses include: JSON, HTML, JavaScript, Image, PDF, CSS, Syndication, Other XML, Video, Other file types, and Unknown (failed requests).
Crawl Requests Breakdown by Crawl Purpose
- Review this in Google Search Console.
- This metric is intriguing for presentations, though practical use cases are limited. Examples include notable shifts in Discovery or Refresh activities without corresponding changes on the site.
Crawl Requests Breakdown by Googlebot Type
- View this in Google Search Console.
- Google says this reveals “the type of user agent used for crawling. Google employs several user agents for varied reasons and behaviors.”
- This metric is of minor interest, showcasing continued use of Google’s desktop crawler. I often disregard these metrics.
You can dive into each metric for more information
When presenting SEO concerns to teams, they often demand sample URLs. You can click any metric to view example URLs.
An interesting element to review is “other file types” for its ambiguity, often comprising font files.
In example reports, each row indicates one crawl request, thus a page indexed multiple times might appear multiple times under “examples.”
Keep in mind Google Search Console provides a data sample, not a comprehensive collection of requests from Googlebot.
Do You Share These Metrics with Developers and Product Managers?
These metrics usually prompt two reactions:
- “Nothing noteworthy here.”
- “What might have caused that?”
Often, addressing "what caused that" requires insight from product managers and/or developers.
When presenting data or pursuing potential issue causes, be sure to clarify that these metrics are about Googlebot’s activity, not user interactions with the site.
Developers and Product managers may become confused if these don’t match other metrics or site knowledge—this frequently happens during Google Search Console data discussions.
If Crawl Stats show no concerning fluctuations or correlations, avoid involving developers or product management; it becomes disruptive noise hindering their focus on critical metrics.
What’s Next?
Review your crawl stats to ensure no alarming spikes or correlations.
Then, decide the frequency of your reviews and establish methods for systematic, analytical monthly evaluations of these and other Google Search Console metrics.
While you examine Googlebot Crawl Stats, I’ll proceed to Part 4 of this series, discussing how to identify which URLs warrant technical SEO improvements and focusing on Core Web Vitals metrics.
Dig Deeper
This is the third article in a series recapping my SMX Advanced presentation on transforming SEO metrics into actionable items. Previously released articles in this series can provide additional insights.
Note: Contributing authors are invited to create content for aspiring SEO professionals, selected for their expertise and impact within the search community. Our contributors operate under an editorial oversight ensuring quality and relevance to our audience. The views they express are theirs alone.
