Open your crawl stats in Search Console and look at the "By response" breakdown. On a healthy store, the share of requests returning 200 OK should dominate. On most e-commerce sites I audit, it doesn't. Faceted URLs, internal search pages and session parameters eat a third of the requests before Googlebot ever touches a product page.
That's the whole job in one sentence: crawl budget optimization on an e-commerce site is about redirecting bot requests away from URLs that never convert, not about pleading with Google for more. You can't buy crawl rate. You can only stop wasting it.
Key Takeaways
- Crawl budget matters most for stores with tens of thousands of URLs. Small shops rarely hit a ceiling.
- Filter and sort parameters are usually the single biggest source of wasted requests.
- Robots.txt, noindex and canonical tags solve different problems. Using the wrong one backfires.
- Log files give you the truth. Search Console gives you the trend. You need both.
- JavaScript rendering consumes crawl capacity you probably can't afford on a large catalog.
Why crawl budget decides your e-commerce SEO (and when it doesn't)
Google splits crawl capacity into two ideas: crawl rate limit (how fast it can fetch without melting your server) and crawl demand (how much it wants to fetch, driven by popularity and freshness). Crawl budget is the intersection of the two. You influence the second one; the first one belongs to your infrastructure.
Here's the uncomfortable part. Most stores don't have a crawl budget problem. They have an indexing problem dressed up as one.
What is crawl budget?
Crawl budget is the number of URLs Googlebot is willing to fetch on your site within a given period. It's not a fixed daily quota, and it isn't published anywhere. It fluctuates with your site's perceived importance, how often content changes, and how quickly your server responds.
Which means the phrase "increase your crawl budget" is misleading. You don't increase it directly. You make each request count for more.
The size threshold that actually matters
If your store has 400 products and a handful of categories, stop reading about crawl budget and go fix your internal linking. Seriously. Below a few thousand indexable URLs, Google crawls everything it cares about without breaking a sweat, and your time is better spent elsewhere.
The problem starts when your catalog generates combinatorial URLs. A store with 800 products, 6 filter dimensions and 3 sort options doesn't have 800 pages. It has a number with a lot of digits in it.
I audited a mid-market furniture retailer last year: roughly 2,100 products in the sitemap, but the crawl log showed 180,000+ distinct URLs requested in 30 days. Over 70% of those requests went to filter and sort combinations. Product pages were being recrawled every three weeks. For a store with weekly stock changes, that's a slow-motion failure.
Where e-commerce crawl budget actually leaks
Six sources account for nearly every case I've seen. Rank them by request volume in your log file before you touch anything.
- Faceted navigation — color, size, price range, brand. Unbounded combinations, each one a fetchable URL.
- Sort parameters —
?orderby=price,?sort=asc. Pure duplicates, zero search value. - Internal search result pages —
/search?q=can generate infinite URLs. - Session IDs and tracking parameters appended by your own marketing stack.
- Pagination beyond the depth anyone browses.
- Out-of-stock and discontinued products, left indexable for years.
Notice what's not on that list: your actual products. Products are never the problem. The scaffolding around them is.
Robots.txt vs noindex vs canonical: pick correctly
This is where most teams make it worse. The three tools do different jobs and mixing them up creates silent damage.
| Tool | What it does | Use it for | Risk |
|---|---|---|---|
| robots.txt Disallow | Blocks crawling entirely | Sort params, session IDs, internal search, cart, checkout | Google can't see the page, so it can't see a canonical or noindex either |
| noindex meta tag | Allows crawl, drops the page from the index | Thin filter pages that are linked internally and must be seen | Consumes crawl budget permanently — every noindexed URL still gets fetched |
| rel=canonical | Consolidates signals to a preferred URL | Filter combinations with real search demand, paginated series | Google may ignore it if the pages differ too much |
The trap: if you Disallow a URL in robots.txt, Googlebot never fetches it, so it never reads your noindex tag. People disallow a parameter and add noindex to the same URL "for safety." The noindex is invisible. It does nothing.
My rule of thumb: block with robots.txt when the URL has no business being indexed and no one will ever search for it. Use canonical when the page has genuine search demand but duplicates a parent. Use noindex sparingly, because it still costs a fetch.
How to optimize crawl budget without breaking your store
Run this as a sequence. Skipping straight to robots.txt rules is how stores accidentally block their own category pages.
- Pull 30 days of log files. Group requests by URL pattern, not by individual URL. You want the shape of the waste.
- Classify every pattern: product, category, filter, sort, search, cart, static asset. Assign each a "should be crawled" yes or no.
- Set a target ratio. On a well-run large store, I aim for 60%+ of bot requests on product and category URLs. Most stores I open sit near 30%.
- Write robots.txt rules for the obvious waste — sort parameters, session IDs, internal search, cart and account pages.
- Handle facets deliberately. Keep crawlable only the combinations with measurable search demand. Block the rest.
- Re-crawl and compare after two weeks. The ratio should shift. If it doesn't, your rules aren't matching the real URLs — check for parameter encoding differences.
Step three is where teams argue. Someone always wants a bigger number. It's fine: the target is directional, not a certification.
Handling faceted navigation without killing revenue
Facets are the hard case because some of them earn money. A "solid oak" filter page can rank and convert. A "price 200–250, size M, color grey, sort by newest" page will not.
The workable compromise: allow crawling of one-dimension facets with search volume, and block multi-dimension combinations. Then link to the keeper facets from category pages so they get discovered through real internal links rather than through the filter widget.
And check the filter widget itself. On plenty of platforms it emits plain <a href> links for every option — which means every possible combination is one click from every category page. That's your leak, and it's in the template, not the content.
How to increase crawl speed
You can't ask Google for more. You can remove the friction that makes it slow down.
Googlebot throttles when your server responds slowly or errors out. Every 5xx response is a signal to back off. A store I worked with had an origin response time averaging 1.8 seconds under bot load; after moving to cached category pages with a proper CDN, it dropped below 200ms and the daily crawl volume on product URLs roughly doubled within a month. Nothing else changed.
- Serve cached HTML for category and product pages. Caching is the highest-leverage fix and nobody talks about it.
- Kill 5xx errors. They're worse than slow responses because they actively suppress crawl rate.
- Compress and consolidate assets. Bot fetches of CSS and JS consume the same budget as pages.
- Keep response times under half a second for bot requests.
A 429 or 503 storm during a traffic spike can set you back weeks. I made that mistake once by deploying a firewall rule that treated Googlebot as a scraper for six hours. Recovery took eleven days.
JavaScript rendering and crawl capacity
This is the part missing from most crawl budget advice, and it's the one that quietly ruins large stores.
If your product grid only exists after JavaScript runs, Googlebot has to queue the page for rendering. Rendering is a second, separate capacity constraint, and it's tighter than fetching. Pages waiting in the render queue can sit for days.
- Use real
<a href>links for navigation and pagination, not click handlers on divs. - Server-render or pre-render product and category pages so the primary content is in the HTML.
- Check the rendered HTML in Search Console's URL inspection — if the product description is missing there, it's missing for Google.
Sound familiar? A store where filters are JS-driven and links are synthetic has effectively made its entire faceted surface invisible and wasted render capacity on nothing.
How can I optimize my e-commerce website beyond crawl budget?
Crawl budget gets you in the door. Indexing and internal linking decide whether it was worth opening.
Three checks I run on every store, in this order:
- Index coverage. How many of your submitted URLs are actually indexed? A large gap between "submitted" and "indexed" usually points at thin content or duplicate product descriptions.
- Internal link depth. Can a bot reach any product in four clicks from the homepage? Deep catalogs often bury older SKUs behind pagination nobody follows.
- Duplicate descriptions. Manufacturer copy pasted across 300 products is the most common reason product pages get crawled but not indexed.
And the stale-stock question: out-of-stock products with no return date should be noindexed, not deleted, if there's any chance they come back. Deleting them destroys link equity and creates 404s that bots keep re-checking.
Does a sitemap increase crawl budget?
No, but a clean one helps Google prioritize. Submit only canonical, indexable, 200-status URLs. A sitemap stuffed with redirected or noindexed URLs teaches Google to distrust the file, and I've seen stores where the sitemap was 40% dead weight.
How do I check Google's crawl status?
Search Console's crawl stats report shows total requests, response breakdown, and crawl purpose over time. Pair it with raw server logs for URL-level detail — Search Console won't tell you which filter combinations are being fetched.
What is a good SEO budget?
People ask me this expecting a percentage of revenue. The honest answer is that a "good" SEO budget depends on how much technical debt you're carrying, and crawl budget work is one of the cheapest line items in it.
Rough allocation I'd defend for a mid-size store: the majority of effort on content and category page quality, a meaningful chunk on technical fixes like crawl and rendering, and a small slice on tooling. Log analysis tooling is inexpensive relative to what it saves.
What I'd push back on: spending on link building while your filter parameters are still generating 100,000 crawlable URLs. You're paying to send authority into a maze.
The part nobody measures
Here's what stays with me after these audits. The stores with crawl budget problems almost never describe them that way. They say "our new products take forever to rank" or "Google ignores our category pages." The symptom is always visibility. The cause is always plumbing.
So before you write a single robots.txt rule, open the log file and count how many requests went to URLs a customer could actually buy from. That number, more than any tool or tactic, tells you whether you have a crawl problem at all.