If your site has fewer than ten thousand unique URLs, crawl budget is not your problem. Google's own documentation says so, in language as close to a direct quote as the docs ever get.
I have watched small business owners spend afternoons reading SEO blog posts about crawl budget optimization for sites that have eighty-seven pages. It is the SEO equivalent of optimizing your tax strategy when you make twelve thousand dollars a year. The math does not justify the time.
Crawl budget is about to matter more for small sites, for a reason that has nothing to do with the old advice. I'll get to it under AI fan-out.
Who actually has a crawl budget problem
Per Google's own crawl budget guide, the documentation is "intended primarily" for:
- Sites with 1 million+ unique pages where content changes weekly
- Sites with 10,000+ unique pages where content changes daily
- Sites with a large fraction of URLs in Search Console's "Discovered, currently not indexed" bucket
> If your site does not have a large number of pages that change rapidly, or if pages get crawled the same day they publish, you do not need a crawl budget guide. Keep the sitemap current and check index coverage.
That is the bar. Most small business sites sit well below it, including every site we ship at the studio for dentists, restaurants, lawyers, and accountants. Pages are published. Pages get crawled within days. The crawl budget exists, technically, but it is enormous relative to your URL count.
The sites with actual crawl budget problems share three traits: scale (hundreds of thousands of URLs minimum), velocity (daily or hourly content changes), and architecture that generates URL space combinatorially. That last one is the killer.
The faceted-nav trap
Per Gary Illyes on Search Off the Record episode 104 ("Crawling Challenges: What the 2025 Year-End Report Tells Us"), Google's internal categorization of crawl-issue reports breaks down like this:
- Faceted navigation: ~50% of reports
- Action parameters (add-to-cart, add-to-wishlist as GET URLs): ~24-25% of reports
- Irrelevant parameters and session IDs: ~10%
- Calendar/event date infinite spaces: ~5%
- "Weird stuff" (mostly double-encoded URLs): ~2%
Half of every reported crawl problem is faceted navigation. The pattern: a category page with five filter dimensions (color, size, price, brand, sort), and the implementation makes each filter combination a unique URL. Five dimensions with ten options each is a hundred thousand URL combinations. Most return effectively the same content with different ordering. Googlebot crawls all of them before figuring out they were not worth crawling.
> Per Google's faceted navigation guide, filter-parameter URL combinations generate "infinite URL spaces" that overcrawl the site and slow discovery of legitimately new content. Block them, do not just hope Google figures it out.
The fix, per Google's faceted navigation documentation: block the filter-parameter URLs with robots.txt. Gary on the podcast pointed at google.com/robots.txt as a reference example.
The action-parameter bug
This one is sneaky. Per Gary on SOTR 104, almost a quarter of reported crawl problems are action parameters: GET URLs (the plain links a browser fetches without changing anything) like ?action=save, ?add_to_cart=true, ?add_to_wishlist=true. WordPress plugins are the most common source.
Googlebot follows GET URLs. It does not follow POST or PUT URLs, because those would change server state and Google does not do that. So if your "add to cart" button is implemented as a link with ?add_to_cart=1, Googlebot will click it. Every product on your site now has a duplicate URL. Combine add-to-cart with add-to-wishlist and you have tripled your crawl space.
The fix lives in the form method: actions belong in POST or PUT requests, so add-to-cart should be a form submission, not a link.
Other crawl-budget killers
Session IDs in URLs. Per Gary, "session IDs are very 2000." Crawlers do not persist sessions. If you must keep them, use a named parameter (?sessionid=abc), not a single letter (?s=abc) that Google cannot heuristically classify as a session.
Calendar plugins generating infinite event spaces. Plugins that auto-generate /event/2026/01/01, /event/2026/01/02, ad infinitum. Googlebot has no way to detect an empty event date without crawling it. Block these patterns in robots.txt if you cannot disable the plugin.
Double-encoded URLs. %2520 instead of %20 for a space. A common bug where one layer of code URL-encodes a value, then another layer encodes the already-encoded version. Establish encoding at one layer of your stack and stop.
Long redirect chains. Per the crawl budget guide, each hop in a redirect chain costs crawl budget. Direct redirects from old to new, no intermediate hops.
soft 404 pages. These are 200-status pages that look like errors. Google still crawls them, finds them low-quality, and the crawl was wasted. Per the docs, return real 404s for missing content, even in SPAs.
When to return 503
Per Google's reduce crawl rate documentation: if Google is overloading your server, return 503, 500, or 429 HTTP status codes. The crawler will back off.
This is the under-used emergency lever. Per Gary on SOTR 107 ("Google crawlers behind the scenes"): "If it gets a 503 HTTP response, then it slows down even more because that actually means that the server was most likely overwhelmed in some way." A 4xx response does not slow Googlebot down. Per Gary again: "403, 404, like all those, they don't mean anything, like random client error."
> When your server is overloaded, return 503 with a Retry-After header. Do not return 4xx, because 403 and 404 do not slow Google's crawler down. Google reads 503 as "back off" and reduces requests within minutes.
Use this for one to two days, max. If you keep serving 503s longer than that, Google may start dropping the URLs from the index. It is a circuit breaker, not a permanent fix.
For longer-term crawl rate reduction, you can file a special request through Search Console's Googlebot report form. You cannot request an increase, only a decrease.
Why crawl budget is coming back
Here is the bridge from foundations to the I/O 2026 reality. Crawl budget has been a "large sites only" concern for a decade. That is changing under AI fan-out.
The story so far: Googlebot crawls once, indexes once, serves once. Crawl budget was a function of how many URLs Googlebot pulled. With AI Mode, Information Agents, and the agentic web Google announced at I/O 2026, the crawl pattern flips. A single user query in AI Mode triggers Google fanning out across multiple sources to compose an answer. An Information Agent follows up. Agents acting on behalf of users, what Google calls the "agent traffic class," make requests to your site that look nothing like a traditional Googlebot crawl.
The net effect for sites that previously had no crawl budget concern: your "crawler" traffic just multiplied. Not because Google's central crawl budget changed, but because the surface area of automated requests against your site expanded. The 50% faceted-nav share of crawl complaints from the 2025 year-end report was Googlebot alone. The 2026 picture includes agents, AI Mode fan-out fetches, and third-party LLM crawlers (OAI-SearchBot, PerplexityBot, ClaudeBot) hitting your servers.
For a small business site that has never thought about crawl budget, this is still mostly fine. Your absolute traffic is small. But for contractors running heavy programmatic location-page footprints, for home builders with portfolio catalogs, for any site running pSEO, the old "if you have under 10,000 URLs, ignore crawl budget" advice is starting to need an asterisk.
Three things stay true regardless:
- Robots.txt your filter parameters and session IDs
- Do not put actions in GET URLs
- Return 503, not 4xx, when overloaded
These were good practice when crawl budget meant Googlebot. They are necessary practice in a world where the crawler is plural.
Sources
- Optimize your crawl budget (large sites) (Google Search Central, December 2025)
- Crawl budget overview (Google Search Central, December 2025)
- Managing crawling of faceted navigation URLs (Google Search Central, December 2025)
- Reduce the Google crawl rate (Google Search Central, December 2025)
- Google crawlers behind the scenes, Search Off the Record episode 107 (Gary Illyes and Martin Splitt, March 2026)
- Crawling Challenges: What the 2025 Year-End Report Tells Us, Search Off the Record episode 104 (Gary Illyes and Martin Splitt, February 2026)