Crawl budget is the number of pages Googlebot crawls on your site within a given timeframe, determined by your server's capacity and Google's assessment of how often your content is worth revisiting.
Gary Illyes confirmed in 2025 that the practical threshold is roughly one million unique pages changing weekly or ten thousand changing daily. Below those numbers, Google generally crawls everything and optimising for crawl budget solves a problem you do not have. John Mueller has called it overrated for most websites, and for most websites he is right.
Above those numbers, crawl budget is a real constraint, and the way to find whether it is binding on your site is reading your server logs rather than guessing from Search Console.
Crawl Budget Is Two Things
Google defines it as the combination of crawl rate limit and crawl demand.
Crawl rate limit is the capacity ceiling. How many parallel connections Googlebot opens and how fast it requests pages, bounded by what your server can handle without degrading the experience for real users.
Crawl demand is Google's interest. How much of your site it wants to revisit, driven by perceived content quality, size, freshness, and popularity.
A site can be limited by either. A slow server throttles the rate even when demand is high. A site with low quality content has low demand even when the server is fast.
Finding Your Actual Crawl Pattern From Logs
Search Console's Crawl Stats report gives an overview. Server logs give the ground truth.
#!/bin/bash
# Extract Googlebot requests from an Nginx access log
# Adjust the log format regex to match your configuration
LOG="/var/log/nginx/access.log"
echo "=== Googlebot crawl summary ==="
echo ""
# Total Googlebot requests today
echo "Requests today:"
grep "Googlebot" "$LOG" | wc -l
echo ""
echo "Status code breakdown:"
grep "Googlebot" "$LOG" | \
awk '{print $9}' | sort | uniq -c | sort -rn
echo ""
echo "Top 20 crawled paths:"
grep "Googlebot" "$LOG" | \
awk '{print $7}' | \
sed 's/?.*//' | \
sort | uniq -c | sort -rn | head -20
echo ""
echo "Requests per hour:"
grep "Googlebot" "$LOG" | \
awk -F'[' '{print $2}' | \
awk -F: '{print $2}' | \
sort | uniq -cRun this daily for a week and the crawl pattern becomes visible. Where the top crawled paths are parameter URLs, faceted navigation, or paginated archives, that is where the budget is going.
The Five Biggest Crawl Budget Sinks
Faceted navigation. Filter combinations on ecommerce sites multiply URLs into the tens of thousands. Most show near identical inventory and all consume crawl requests. This is the largest single waste category and the ecommerce SEO guide covers the handling.
Parameter URLs. Tracking parameters, sort orders, session identifiers, and view mode parameters each create a new URL for the same content. Canonicalisation resolves how they index. It does not prevent them being crawled.
# Find the parameter URLs Googlebot is wasting time on
grep "Googlebot" "$LOG" | \
awk '{print $7}' | \
grep '?' | \
awk -F'?' '{print $1}' | \
sort | uniq -c | sort -rn | head -20
# Shows which base URLs are being hit with the most parameter variantsInfinite scroll and JavaScript rendering. JavaScript heavy pages cost more to process. Google queues them for rendering, which consumes additional budget. The rendering budget concern covered in the JavaScript SEO guide directly constrains crawl coverage.
Redirect chains. Every hop in a chain is a separate request consuming budget while producing no indexable content. The redirect guide covers detection and flattening.
Soft 404s and empty pages. Pages returning 200 with no meaningful content still get crawled repeatedly, per the status codes guide. Returning a proper 404 or 410 tells Googlebot to stop visiting.
Measuring Crawl Waste as a Percentage
import re
from collections import Counter
def analyse_crawl_waste(log_path: str, valuable_paths: set):
"""Calculate what percentage of crawl budget hits non-valuable URLs."""
total = 0
wasted = 0
waste_reasons = Counter()
with open(log_path) as f:
for line in f:
if 'Googlebot' not in line:
continue
total += 1
# Extract path
match = re.search(r'"GET (\S+)', line)
if not match:
continue
path = match.group(1).split('?')[0]
# Check status code
status_match = re.search(r'" (\d{3}) ', line)
status = int(status_match.group(1)) if status_match else 0
# Classify
if status in (301, 302, 307, 308):
wasted += 1
waste_reasons['redirects'] += 1
elif status in (404, 410):
wasted += 1
waste_reasons['dead_pages'] += 1
elif '?' in match.group(1):
wasted += 1
waste_reasons['parameter_urls'] += 1
elif path not in valuable_paths:
wasted += 1
waste_reasons['non_valuable_paths'] += 1
print(f"Total Googlebot requests: {total}")
print(f"Wasted: {wasted} ({wasted/total*100:.1f}%)")
print(f"Breakdown: {dict(waste_reasons)}")
# valuable_paths = set of URLs you actually want ranked
# analyse_crawl_waste('/var/log/nginx/access.log', valuable_paths)A waste percentage above roughly 50 percent on a large site is a clear optimisation target. Below 20 percent, the gains from further work are marginal and effort belongs elsewhere.
Server Speed Is Crawl Budget
Because the rate limit responds to how quickly pages respond, server performance directly affects how many pages get crawled per session. A server averaging 200 millisecond responses lets Googlebot fetch substantially more pages than one averaging 800 milliseconds.
That makes server side performance work also crawl budget work, which is the overlap with the Core Web Vitals guide. Faster pages improve user experience metrics and increase crawl coverage simultaneously.
Sitemaps as a Crawl Priority Signal
A sitemap does not increase crawl budget. It tells Google which URLs exist and when they last changed, which influences how demand is allocated across the URLs it already plans to crawl.
Including only pages you want indexed, with accurate lastmod dates, makes the sitemap a genuine priority signal rather than a complete URL dump. The sitemap guide covers this in detail.
A sitemap containing fifty thousand URLs where thirty thousand return noindex sends conflicting signals and wastes the prioritisation the sitemap was supposed to provide.
Robots.txt Is a Blunt Instrument for Crawl Control
Blocking a path in robots.txt prevents crawling of those URLs entirely, which saves requests. The cost is that any canonical tags, noindex directives, or schema on those URLs are never read.
That makes robots.txt appropriate for paths that should never be crawled under any circumstances, such as admin panels, internal search results, and API endpoints. It is inappropriate for URLs that need consolidation, where canonical tags are the tool.
The interaction between robots.txt and canonical tags catches teams regularly. A URL blocked by robots.txt that receives external links has link equity pointing at a URL Google cannot access. It cannot follow the canonical to consolidate that equity elsewhere, because it cannot reach the page to read the canonical. The link value sits unused.
# Good: block paths that should never be crawled
Disallow: /admin/
Disallow: /internal-search/
Disallow: /api/
# Bad: blocking parameter URLs that need canonical handling
# Disallow: /*?sort= # Prevents canonical being read
# Disallow: /*?color= # Same problem at scaleLeave parameterised URLs crawlable and handle them with canonical tags instead, per the canonicalization guide.
When to Actually Worry
Three symptoms that indicate a real crawl budget problem rather than an imagined one.
Search Console's Discovered currently not indexed count growing steadily, meaning Google knows URLs exist and is not getting around to crawling them.
New pages taking weeks to appear in search results despite being in the sitemap and internally linked.
Important pages being crawled less than once a month, visible in log analysis, while parameter URLs and archives are crawled daily.
If none of these apply, crawl budget is not your problem and the time belongs on content or architecture.
AI Crawlers Have Their Own Budget
GPTBot, ClaudeBot, and other AI crawlers each maintain separate crawl patterns with their own rate limits. Optimising for Googlebot does not automatically improve coverage by these crawlers, per the crawlers guide.
The NotioncCue AI Crawler Audit reports what specific AI crawlers receive from your URLs, which is a different test than log analysis of Googlebot behaviour.
Start your free NotioncCue trial and check whether pages you consider important are actually reachable by AI crawlers. Crawl budget optimisation that considers only Googlebot misses the crawlers that feed a growing share of how buyers find products.
Run the log analysis script on one day of access logs. If Googlebot's top crawled paths are parameter URLs or paginated archives rather than your product or content pages, the budget is being spent on the wrong things regardless of whether you have a budget problem.
Common Questions
Does my site have a crawl budget problem?
Almost certainly not if it has fewer than ten thousand pages. Check the three symptoms above before spending time on it.
Does blocking URLs in robots.txt save crawl budget?
It prevents those URLs being crawled, which saves requests. It also prevents their canonical tags being read, which can create unresolved duplicate content. Use canonicals and noindex instead where consolidation matters.
Can I increase my crawl rate limit?
Not directly. Faster server responses let Googlebot request more pages per session. Search Console previously offered a crawl rate setting, which is now deprecated. The practical lever is server performance.