The mechanics behind crawl allocation, one section at a time.
A working session reviewing a week of raw access logs.
What crawl budget means when you're under 10,000 pages
A site with three thousand pages does not automatically get every one of them crawled every week. Google still allocates a finite number of fetches to that domain, based on how quickly the server responds and how much new or changed content it expects to find.
The confusion usually starts because most public writing about crawl budget references large marketplaces or news publishers. Those examples make it sound like a problem reserved for sites with a million URLs, which is not accurate.
A small directory site with a search box, a tag archive, and a print-friendly version of each page can multiply its real page count several times over in Google's eyes. None of those extra variants were part of the original content plan, yet each one competes for the same limited crawl allocation.
The practical takeaway is not to panic about numbers. It is to check, using the crawl stats report described next, whether Googlebot's attention is landing on the pages that matter or scattered across variants nobody intended to publish.
How to find crawl stats in Google Search Console
The crawl stats report sits under Settings, not under the more commonly used Pages or Performance tabs. Once opened, it shows total crawl requests over the past ninety days, broken down by response code, file type, purpose, and Googlebot type.
Response code breakdown is usually the fastest place to spot a problem. A large share of 404s or 301 redirects means Googlebot is spending fetches on dead ends instead of live content.
The "by purpose" chart separates refresh crawls from discovery crawls. A site heavy on discovery crawls for a domain that isn't actively publishing new pages often signals that Google is still finding unexpected URLs somewhere, frequently through parameters or pagination.
Host status, a related panel on the same settings page, flags whether Google has recently had trouble fetching the site at all. Robots.txt fetch failures and DNS resolution issues both show up here before they show up anywhere else.
Why duplicate content wastes crawl allocation
Two URLs that render the same content still represent two separate crawl requests from Google's perspective. A print version of an article, a mobile-specific subdomain mirroring desktop content, or a staging path accidentally left indexable all fall into this category.
The waste compounds because Google typically has to crawl a duplicate at least once before it can determine it is, in fact, a duplicate. On a site with thousands of near-identical listing pages, that first-pass crawling adds up to a meaningful share of the available budget.
Canonical tags help Google understand which version should be treated as primary, but they do not stop the initial crawl from happening. A cleaner fix usually involves consolidating the duplicate templates at the source, or blocking the duplicate paths from being generated in the first place.
Session-based URLs are a common overlooked source. A shopping cart or a login-protected area that appends a unique identifier to every URL can generate a functionally infinite number of duplicate pages if crawlers manage to reach them.
How URL parameters create thousands of pages you never intended
A filterable product listing with options for size, color, and sort order can produce hundreds of parameter combinations from a single base URL. Multiply that across several categories and the count climbs into the thousands without anyone deliberately publishing that many pages.
Googlebot generally treats each unique combination of parameters as its own distinct URL unless told otherwise. That means a crawl budget calculation has to account for every combination that is technically reachable through a link, not just the pages a site owner considers real content.
Faceted navigation is the most frequent source, but internal search results pages, calendar-based archive pages, and tracking parameters used for internal campaigns all contribute the same way. Each pattern adds reachable, crawlable variants that were never meant to be treated as separate pages.
Blocking these patterns through robots.txt stops crawling but does not remove already-indexed variants. A combination of consistent internal linking, parameter handling settings, and selective noindex tags tends to address the issue from more than one angle at once.
What log file analysis reveals that no other tool shows
Search Console's crawl stats are aggregated across a rolling window and rounded for presentation. A raw server log records every single request line by line, including the exact timestamp, response code, and user agent string for each visit.
That level of detail makes it possible to see patterns Search Console simply does not surface, such as Googlebot repeatedly hitting a specific broken redirect chain at three in the morning. It also reveals the gap between Googlebot's declared user agent and its actual IP range, which matters when checking for spoofed crawlers.
Parsing logs typically means filtering by user agent for Googlebot variants, then grouping requests by URL path, status code, and day. Spreadsheet tools can handle a few thousand lines, but larger sites benefit from a dedicated log parser or a simple script built around the extended log format.
The output is worth the setup time. A log-based view answers questions that no crawl stats summary can, like exactly which hour Googlebot visited a specific page last, or how many times a redirect loop was hit before anyone noticed.
This site uses cookies to understand basic usage.
Cookie preferences
Choose which categories of cookies you allow. Necessary cookies are required for the site to function and cannot be disabled.
NecessaryRequired for basic site functionality.
AnalyticsHelps understand which pages are read.
PreferencesRemembers display settings between visits.
