How to Use Crawl Stats in Google Search Console

The Crawl Stats report in Google Search Console shows Google’s recent crawling history on your site: how many requests it made, how much data it downloaded, how fast your server responded, and whether it hit availability problems. To open it, click Settings in the left-hand menu, then open Crawl stats.

Two things to know before you dig in. The report only exists for root-level properties, so a Domain property or a root URL-prefix property like https://example.com shows it, but a subfolder property like example.com/blog/ does not. And Google’s Crawl Stats help page describes it as a tool for advanced users: if your site has fewer than a thousand pages, Google says you should not need it. It is still the fastest way to confirm a server or robots.txt problem when crawling suddenly changes.

What the Crawl Stats Report Shows

The top of the report has three over-time charts plus a host status summary. Below them, the Crawl requests breakdown groups the same requests four ways. Here is what each part measures, based on Google’s help page and its 2020 announcement of the redesigned report.

Total Crawl Requests

Every crawl request Google issued for URLs on your site, successful or not. It includes page resources hosted on your site, like images and CSS, and it counts repeat requests for the same URL separately. It also counts requests that never happened because your robots.txt file was unavailable, plus fetches that failed on DNS, connectivity, or redirect loops. Treat it as a measure of activity, not a score.

Total Download Size

The total bytes Google downloaded while crawling during the period. A resource shared by many pages, such as a stylesheet, is only counted the first time Google fetches and caches it. A sharp rise usually means heavier pages or a new batch of large files.

Average Response Time

The average time your server took to respond to every resource Google fetched, with each resource on a page counted as its own response. This is often the most useful line on the page. Google’s help page lists slow responses as one reason Googlebot throttles back, so a climbing response time is worth fixing even before users notice it.

Host Status

Host status tells you whether Google ran into availability problems. It checks three categories, each with its own chart and a dotted red threshold line:

• robots.txt fetching tracks the failure rate when Google requests your robots.txt file.
• DNS resolution tracks times your DNS server did not recognize the hostname or did not respond.
• Server connectivity tracks times your server was unresponsive or sent an incomplete response.

If a category crosses its threshold on a given day (Google’s example is DNS failing for more than 5% of requests), that counts as an issue. The summary then shows one of three states: no significant availability issues in the past 90 days, an issue in the past 90 days that happened more than a week ago, or a recent issue within the last week.

robots.txt deserves extra attention. A 200 response with any file, even an empty or invalid one, counts as success, and so do 403, 404, and 410 (Google treats those as “no robots.txt”). A 429 or 5xx counts as a failure. If Google cannot get a good robots.txt response, it stops crawling for the first 12 hours, then uses the last good copy for up to 30 days.

By Response

This breakdown shows crawl responses as a percentage of all requests, not of URLs, so a URL that returned a 500 once and a 200 once counts as 50% each. Google sorts the responses into three groups:

Group	Responses	What to do
Good	OK (200), Moved permanently (301 or 308), Moved temporarily (302 or 307), Moved (other) for meta refreshes, Not modified (304)	Nothing, as long as redirects are intentional. Most responses should be 200.
Possibly good	Not found (404)	Often correct for pages that are truly gone. Fix the ones caused by your own broken links.
Bad	robots.txt not available, Unauthorized (401/407), Server error (5XX), Other client error (4XX), DNS unresponsive, DNS error, Fetch error, Page could not be reached, Page timeout, Redirect error, Other error	Fix these. Server errors in particular trigger availability warnings.

A redirect chain shows up as several requests: page1 returning a 301, page2 returning a 301, and page3 returning a 200 are three separate rows of data, and the redirects are filed under the “Other file type” file type.

By File Type

The share of responses (not bytes) by file type: HTML, Image, Video, JavaScript, CSS, PDF, Other XML, JSON, Syndication (RSS or Atom feeds), Audio, Geographic data, Other file type, and Unknown (Failed) for requests that failed. If your server is slow, click into a type to see its response time by date. Google suggests checking whether it is spending requests on many small images or on resources hosted somewhere slower.

By Purpose

Only two values here. Discovery means Google requested a URL it had never crawled before. Refresh means it recrawled a known page. After you publish a batch of new content or submit a sitemap, you should see discovery go up. If pages that change often are not being recrawled, Google recommends making sure they are in your sitemap.

By Googlebot Type

The user agent that made the request: Smartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot, or Other agent type. Page resource load covers the images, CSS, and scripts Google fetches to render a page, so an image loaded as part of a page counts there rather than under Image. If crawling spikes, this is the first breakdown to check. A jump in AdsBot usually follows new Dynamic Search Ads targets.

You will not find a Google-Extended row. Google-Extended is a robots.txt token that controls whether Google can use your content for Gemini, not a separate crawler: crawling is done with Google’s existing user agents, and the token has no effect on Search.

Click any row in the breakdowns to see example URLs. Google notes these are a representative sample, so a URL missing from the list may still have been crawled.

How Google Counts the Data

A few details explain most “why doesn’t this match” questions:

• URLs are the actual URLs Google requested, not the canonical versions used in other Search Console reports.
• Only your property’s own domain is counted. Resources hosted on another domain, such as a third-party CDN, do not appear, but a parent domain property includes all of its subdomains, and the Hosts list shows up to 20 child domains that received traffic in the past 90 days.
• Crawl data includes both http and https, even for a URL-prefix property.
• Google says the report does not capture every request yet, so expect small differences from your server logs.
• If the report shows far more requests than your logs, check robots.txt fetching first. Google counts crawls it would have made while robots.txt was unavailable.

How to Read Crawl Stats Without Guessing

Look at weeks, not days. Google says crawl rate should be fairly stable over a week or two, so one odd day rarely means anything. Then work through the report in this order:

1. Check host status. If any category shows a recent issue, fix that before anything else.
2. Compare total requests with average response time. Requests falling while response time rises is the classic sign of a server that Googlebot is backing off from.
3. Open the response breakdown. A rising share of 5xx points at hosting or caching; a rising share of 404s points at broken links or deleted pages.
4. Check Googlebot type if requests spiked, to see which crawler caused it.
5. Click through to example URLs to find the actual pages behind the numbers.

For finer detail than the report gives, such as which exact URL patterns waste requests, use your raw logs. Our guide to finding wasted crawl budget in server logs walks through that.

What Common Crawl Patterns Usually Mean

A sudden drop in crawl requests

Google’s help page lists the usual causes: a new or overly broad robots.txt rule, a server that has started responding slowly, a rising server error rate, or content that changes rarely or is not high quality. In practice, firewall rules and security plugins that block Googlebot produce the same symptom as a server problem, so check them if the drop lines up with a recent change.

A noindex tag will not cause a crawl drop. Google has to crawl a page to see the noindex, so a sudden fall points to access or server problems, not indexing directives.

A spike in crawl requests

Spikes usually follow something you did: unblocking a large section of the site, adding a large new section, or adding many Dynamic Search Ads targets. If the extra load is causing availability problems, Google suggests returning 503 or 429 for a short time (no more than two or three days) or reducing the crawl rate another way. Blocking Googlebot in robots.txt also works, but can take up to a day and should not stay in place long.

More 404 responses

Usually stale internal links, deleted pages without redirects, or old URLs still in menus, sitemaps, or other sites’ links. Google says it is not possible or useful to fix every 404, so focus on URLs that should still exist and on links you control. Our guide to fixing soft 404s covers the related case where a page returns 200 but Google treats it as missing.

More server errors

Fix these quickly. Google says server errors cause availability warnings, and a rising error rate makes Googlebot throttle back. Common causes are hosting limits, missing caching, broken code after a deploy, or traffic spikes that overwhelm the server.

What to Fix After You Spot an Issue

Match the fix to what the report shows:

• If host status shows issues, check DNS with your registrar or DNS provider, confirm robots.txt returns a 200 or 404 quickly, and make sure no firewall rule blocks Googlebot.
• If response times are climbing, improve caching, upgrade hosting, or reduce heavy page resources, then watch whether average response time comes down over the next few weeks.
• If bad responses are piling up, remove dead links from menus and footers, use 301 redirects for pages that moved, and keep your sitemap limited to live, canonical URLs.
• If Google keeps requesting junk URLs, block patterns that should never be crawled, like internal search results, in robots.txt, and make sure important pages are linked from crawlable parts of the site.

If crawl stats look healthy but pages still are not indexed, the problem is probably not crawling. Work through crawled – currently not indexed next, since that usually comes down to content quality, duplication, or weak page purpose. Once a page is fixed, you can request indexing to ask Google to recrawl it.

FAQs About Crawl Stats in GSC

Does a crawl drop mean a penalty?

No. A drop is a signal to check robots.txt, server speed, and error rates. Google also crawls sites less often when content changes rarely, which is normal for many small sites.

Can I ask Google to crawl my site more?

No. Google says you cannot tell it to increase your crawl rate. For small and medium sites, keep your sitemaps updated and make sure you are not blocking pages you want crawled.

How often should I check crawl stats?

For a small blog, check it when something changes: a migration, a new robots.txt rule, a hosting move, or a sudden traffic drop. Larger sites that publish often may want a monthly look at host status and response time.

Why can’t I find the Crawl Stats report?

Most often because the property is a subfolder URL-prefix property (like https://example.com/blog/). Crawl Stats only appears for Domain properties and root-level URL-prefix properties.