The Crawl Stats report in Search Console shows how often Google requested your site, how much it downloaded, how quickly your server answered and which response codes came back. The short answer on how to read it: treat it as a warning panel, not a verdict. Check host status first, then the response split, then the movement in request volume. One odd day means little. A direction that holds for a couple of weeks means something.
Everything below follows what Google's Crawl Stats report help page (Opens in a new tab) says. You will see what each section counts, which rows deserve action and which rows are just context. Raw access-log work is a different job, and this guide stays with the report Search Console already builds for you. It is general information, not a diagnosis of any particular site.
Where the Report Lives and Who Can See It
The report is not available in every property. According to the help page it exists only for root-level properties, meaning a Domain property or a root-level URL-prefix property. If you verified only a subfolder, you will not find it. The page also describes the report as aimed at advanced users and says sites with fewer than a thousand pages generally do not need it.
Finding it in Search Console
Open property settings with the gear icon and choose Crawl stats, or use the direct address that the help page gives. Because there is no prominent shortcut, write the path into your team's onboarding notes. An agency that manages many clients saves real time by keeping that path next to each property name.
What the numbers leave out
Reading the report without knowing its boundaries leads to wrong conclusions. The help page sets these limits:
- Data covers the selected domain only. Requests to other domains, including page resources hosted elsewhere, do not appear.
- Sibling subdomains do not show up in each other's reports, but requests between subdomains do appear in the parent domain's report.
- For URL-prefix properties the data includes both http and https, while example URLs are limited to the protocol of the property.
- Client-side redirects are not counted. Each server-side redirect hop counts as a separate request.
That last point matters in daily use. A page that sits behind a two-step redirect chain costs two requests, so the total is not a count of pages crawled.
Report Versus Raw Server Logs
The report and your access logs measure different things. Logs keep one line per request that reached your server, and you verify the crawler identity yourself. The report is a summary built from Google's side. The help page says the report may miss some requests and that differences from server logs are expected. It adds that when robots.txt has been unavailable for too long, the report can show more than your logs because it counts crawls that were never actually made.
So when the two disagree, the first reaction should not be to ask which one is wrong. Use the report for trends and classification, and use logs for single requests and verification. The article on server log analysis for SEO covers the log side in depth. The workflow here suggests a quick panel scan first and a trip into the logs only when the scan raises a question.
| Question | Crawl Stats report | Server access log |
|---|---|---|
| Spot a trend quickly | Good, charts are ready | Needs export and cleanup first |
| Full request history of one URL | Weak, example URLs are a sample | Good |
| Confirm crawler identity | Google reports its own crawling | You verify it yourself |
| See each redirect hop | Each server-side hop is a separate request | Each hop is its own line |
| Access needed | Root-level property access | Server or CDN log access |
Requests, Bytes and Response Time
The top of the report holds three charts. Each answers a separate question, and none should be read alone.
Total crawl requests as a workload figure
The help page says this figure includes both successful and failed requests, with duplicate requests counted individually. Failed requests include DNS failures, connectivity failures and redirect loops. A rising number is therefore not automatically good news. Broken redirects or a freshly generated set of parameter URLs can push it up just as easily as new content.
Download size and what changes it
This chart shows bytes downloaded during crawling. According to the page, a cached resource is requested only the first time it is needed. If you replace a large script bundle or reorganize an image folder, expect a bump here. When size climbs while the request count stays flat, the same number of pages is being served with heavier files.
Average response time and its blind spot
The average covers every resource fetched in the period, with each linked resource counted separately. A slowdown in one widely used script can therefore lift the whole average. If response time rises while requests fall, the server may be struggling and crawling may be backing off. The help page says crawl rate should be fairly stable over a week or two, and that sudden drops can come from broad robots.txt rules, slow responses, rising server errors or low-value content.
Example scenario: a docs site after a template change
Imagine a software documentation site that ships a new page template (example scenario, no figures). A few days later total requests rise and average response time rises with them. There are two readings. Either Google is discovering many new address combinations, or the server is generating the new template slowly. The purpose and response breakdowns decide between them. A larger discovery share supports the first reading. A larger share of 5xx responses or timeouts supports the second. That habit of starting with one chart and testing it against the next breakdown is the core skill.
Host Status: The Quiet Alarm
Host status gets the least attention and offers the most direct actions. Per the help page it has three states:
- No significant issues in the past 90 days.
- Issues occurred in the last 90 days, but more than a week ago.
- A recent availability issue in the last week.
Three categories are assessed: robots.txt fetching, DNS resolution and server connectivity. Each chart carries a dotted red threshold line. The page's own example is DNS failing for more than 5% of requests on a given day.
How a robots.txt failure plays out
The help page spells out the robots.txt rules. A 200 response (valid, invalid or empty file) and a 403, 404 or 410 count as successful. A 429 or any 5xx counts as unsuccessful. During the first 12 hours of failure, Google stops crawling. From 12 hours to 30 days it uses the last successfully fetched file. After 30 days, if the home page is available Google crawls without restraint, and if it is not, Google stops crawling.
In practice, a server that answers 5xx on the robots.txt address affects crawling of the whole site, not just that file. Test that single URL after a maintenance window or CDN change. For firewall and CDN blocking questions, the guide on Googlebot access verification and WAF or CDN blocks is the right next stop.
Deciding when to step in
A green status reassures but tells you little. With a yellow or red status, work in this order: find which category crossed its threshold, then the date it crossed, then look for a deployment, DNS change or hosting incident on that date. After you fix the cause, follow the recovery in the report, though the documentation gives no promised timeline for how fast the status recovers.
Reading the Response Split
The by-response section groups the HTTP answers Google received. Shares are based on request counts, not unique URLs. The help page groups them like this:
| Group | Responses included | How to treat it |
|---|---|---|
| Good | 200, 301 and 308, 302 and 307, meta refresh, 304 | Expected; if the redirect share is high, inspect chains |
| Possibly good | 404 | May reflect broken links, often correct for removed pages |
| Bad | robots.txt unavailable, 401 and 407, 5xx, other 4xx, DNS errors, fetch errors, unreachable page, timeouts, redirect errors | Look here first |
Why 404 sits in the middle
The page labels 404 "possibly good" because a permanently removed page should answer 404. The trouble starts when your own site still links to those addresses. A growing 404 share over several periods can mean links to discontinued products were never cleaned up. For the meaning of each code and the redirect options, see the HTTP status codes guide.
What a heavy redirect share says
Every server-side redirect is its own request, so a migration or a rewritten URL structure raises the redirect share for a while. If it is still high months later, suspect internal links that point at old addresses and a sitemap that was never updated. Links that go straight to the final address shorten the path for visitors and for crawlers alike.
Which bad-class row first
Decision rule: server errors and timeouts are on your side, so they come first. Authorization errors (401 and 407) usually mean an access restriction sits in the wrong place. DNS and robots.txt rows should be read together with host status. For each row, open a few example URLs and fetch the address yourself to see the answer with your own eyes.
File Types and Crawl Purpose
The file-type section shows what kinds of resources Google asked for, measured by response count rather than bytes. Categories are HTML, image, video, JavaScript, CSS, PDF, other XML, JSON, syndication, audio, geographic data, other file type (which includes redirects) and unknown (failed).
Reading the file mix
On a content-led site you expect HTML to dominate. A high JavaScript or CSS share suggests pages depend on many resources, though that alone is not an error. After shipping a new front-end bundle, a shift in the JavaScript share is natural. The useful question is whether a sudden change lines up with a deployment date.
Discovery versus refresh
The purpose section splits crawls into two classes. Discovery means a URL Google has never crawled before. Refresh means a recrawl of a known page. The help page advises including rapidly changing pages in a sitemap and requesting recrawls for slower-changing ones.
For a publication that issues new articles, a normal picture is a discovery bump on publishing days followed by a refresh share that settles. A discovery share that stays high while refresh stays low can mean Google keeps finding new addresses but is not returning to the important pages it already knows. Check whether those new addresses are really new pages or copies created by filters and session parameters. For the wider framework on how crawl attention is spent, use the crawl budget optimization guide.
Which Googlebot made the request
The report also separates the crawler type: smartphone, desktop, image, video, page resource load (secondary fetches for images and CSS, counted here and not under image or video), AdsBot, StoreBot and other agent type. According to the page, AdsBot shows up about every two weeks. If your ad landing pages have a problem, it is plausible that the AdsBot row shows it, but that is an inference, so confirm through the example URLs.
Example URLs and Their Limits
Each breakdown lists example URLs. The help page states they are not comprehensive and represent a sample. A URL missing from the list does not mean it was not requested. Examples are weighted by day, so some categories can show more examples than others, and the page says this should balance out over time.
Using samples as leads
Treat example URLs as leads, not proof. If the 5xx breakdown repeats one path pattern, try several addresses from that pattern by hand. If you can reproduce the error, fix it. If you cannot, think about an intermittent server fault or a protection layer, and then inspect the server log.
Crawl rate expectations
According to the help page, Google will not increase crawl rate on request, though you can ask for a lower rate. A robots.txt block can take up to a day to take effect and should not stay in place too long. If you need to return 503 or 429 temporarily, it should not last more than two or three days, or Google may crawl less often over the long term.
The Action Table
The table below turns report sections into decisions. Each row pairs an observation with a likely cause and a first check. The causes are possibilities, not confirmed diagnoses.
| Observation | Possible cause | First check |
|---|---|---|
| Red host status | robots.txt, DNS or server connectivity trouble | Find the category that crossed its threshold and the date |
| Total requests dropped suddenly | Broad robots.txt rule, slow responses, more server errors | Date of robots.txt change, response time, 5xx share |
| Response time climbing | Server load, heavy page generation, lost cache | Deployments and hosting events in the same period |
| 404 share growing | Links still pointing at removed pages | Source of the example URLs: internal link or sitemap |
| High redirect share | Post-migration chains, stale links | Chain length and the target of internal links |
| Very high discovery share | Parameter or filter URL generation | Example URL patterns and canonical rules |
| Very low refresh share | Important pages weak in the sitemap and in internal links | Sitemap lastmod values and internal links |
A weekly review checklist
- Open host status and look for any red category in the last week.
- Read requests, download size and response time side by side for the same date range.
- Compare the bad-class share in the response split with last week.
- Note the balance between discovery and refresh.
- Match any change against your release and deployment calendar.
- Try at least one example URL by hand.
- Log the finding with its date in one line.
A Ten-Minute Triage Order
A fixed order narrows interpretation when you open the report under pressure. This sequence gets you to a first split in minutes.
- Host status. If red, identify the category and the date. Park the rest, this row takes priority.
- Bad-class share in the response split. Did it grow, and which response type drives it: 5xx, timeouts, DNS, robots.txt unavailable.
- Response time chart. Is there a jump on the same date? Line it up with deployments and infrastructure events.
- Request chart. If it fell, rule out robots.txt changes and server errors first, then weigh low-value content.
- Purpose breakdown. Did the discovery share move? If yes, inspect new URL generation.
- File type and crawler type. Note any unexpected shift.
Example scenario: a store replatform
A small online store moves to a new platform and changes its URL structure (example scenario, no figures). In the first weeks a high redirect share and a high discovery share are what you would expect. The real question is whether both settle after a few weeks. If they do not, check whether menus, sitemap and canonical tags still reference the old addresses. The goal is for Google to reach the right page in a single step.
Common Misreadings
Teams that use the report tend to fall into the same traps. Knowing them cuts down on false alarms.
Celebrating a request spike
When requests climb, the first impulse is to celebrate. Yet the total includes failed and duplicate requests, and a redirect chain produces more than one request per page. Look at the response and purpose shares before you decide anything.
Blaming a penalty for a drop
A decline is not always a penalty. The help page lists broad robots.txt rules, slow responses and low-value content as things that can reduce crawl rate. Rule those out first. The documentation does not say a drop should be read as a penalty signal.
Mistaking crawling for ranking
Crawling, indexing and ranking are separate stages. Heavy crawling carries no promise of strong rankings, and light crawling does not mean a page is poor. Use the report for accessibility and crawl health, and go back to Search Console performance reports and your own analytics for visibility.
Over-investing on a small site
The help page says sites with fewer than a thousand pages generally do not need the report. On a small site, glancing at host status and the response split once a month is usually enough. Time spent on content and page quality often pays back more.
Writing the Monthly Note
The report earns its keep when you take notes in the same format every month. A note can have three parts: observation, likely cause, action taken. A line such as "Response time rose for three weeks, a plugin update shipped in the same period, cache rule rewritten" still makes sense six months later.
When you present to a client or manager, lead with direction and decision rather than raw numbers. Write down in advance which rows you watch and at what point you act. Then, when something rises or falls, the discussion starts from a rule and not from a reaction.
A note template
- Date range and property name.
- Host status: green, yellow or red, plus the category.
- Direction of the three main charts: rising, falling, flat.
- The largest bad-class row in the response split.
- Balance of discovery and refresh.
- Work done or planned, who owns it and when to look again.
What to expect after a fix
Do not expect the report to change the next day. Since the help page says crawl rate should be fairly stable over a week or two, judge a fix over a comparable window. In the first days, check only that the direction has not turned the wrong way, and give the result a few weeks. Decision rule: if the bad-class share has not fallen within a week, confirm from the server itself that the fix really reached production.
Pairing the Report With Other Records
Crawl Stats is half a story on its own. Three records complete it: the deployment calendar, server or CDN events and the content publishing calendar. Deployments explain shifts in response time and file types. Server events explain 5xx and timeout spikes. The publishing calendar explains swings in the discovery share.
A shared change log is enough to make this work. Record each notable change with its date: a robots.txt edit, a URL structure change, a new plugin, a DNS update, a large publication. When you spot a shift in the report, checking the log first usually shortens diagnosis.
When to move into the server logs
- You cannot reproduce the problem from the example URLs.
- The report and a third-party tool disagree.
- You doubt that a particular client is really Googlebot.
- You need to know which path patterns drive a rise in response time.
Even then, let the report guide you: rebuild the same response groups in the logs so the two sources can be compared. If the report showed more 5xx, filter the 5xx lines first. If the redirect share rose, look at where the 301 and 302 lines point. The report tells you where to look, and the logs tell you exactly which addresses were affected.
Frequently Asked Questions
Why can't I see the Crawl Stats report in my property?
According to the help page the report exists only for root-level properties, which means a Domain property or a root-level URL-prefix property. A property verified for a subfolder will not show it. The page also says the report is aimed at advanced users and that sites with fewer than a thousand pages generally do not need it.
Does total crawl requests equal the number of pages crawled?
No. The total includes successful and failed requests, duplicate requests are counted individually, and each server-side redirect hop is a separate request. The figure therefore differs from the number of unique pages crawled.
Why do the report and my server logs disagree?
The help page says the report may miss some requests and that differences from server logs are expected. If robots.txt has been unavailable for too long, the report can also count crawls that were never made, so its totals can look higher than your logs.
Is a high 404 share an emergency?
The help page puts 404 in the possibly good group: it may point to broken links, yet it is the correct answer for pages that were really removed. Check where the example URLs come from. If your own site still links to them, update those links.
Can I ask Google to crawl my site faster?
According to the help page Google will not increase crawl rate on request, though you can ask for a lower rate. Adding rapidly changing pages to a sitemap and requesting recrawls for slower-changing pages is the route the page recommends.
What should I do if host status turns red?
First find which category (robots.txt, DNS or server connectivity) crossed its threshold and on which day, then review deployments and infrastructure events from that date. For a lasting fix, work with your hosting and development teams.
Conclusion
The limits and time windows on Google's help pages can change; check the current page before you apply anything.
The Crawl Stats report gives you a ready-made panel for watching how Google sees your site. The order is host status, response split, then request, size and response time trends, and finally purpose and file type shares. Example URLs are leads and do not replace server logs. Watch the direction across several weeks, match changes to your deployment log, and do not load the report with penalty or ranking meaning it does not carry.
The report offers no site-specific diagnosis and no guarantee of results. To support the technical work with visibility efforts, you can look at our backlink packages.
Post a comment
Please log in to leave a comment. Login