Faceted navigation is the single largest source of crawl waste in ecommerce, and it is not a bug. Filters exist because shoppers need them. The problem is that four filters with five options each generate 625 URL combinations per category, and a catalog with forty categories produces twenty-five thousand URLs that nobody planned.
Most of those URLs should never be crawled. A small number of them are among the most valuable commercial pages you could own. Telling them apart is the whole job.
Start by counting what you actually have
Before deciding anything, establish the size of the problem:
- Crawl one category with all facets enabled and let it run. The URL count on a single category, multiplied across your category set, is your worst case.
- Pull Search Console’s index coverage report and look for “Indexed, not submitted in sitemap”. Facet URLs live here.
- Check your server logs for the percentage of Googlebot requests hitting URLs with query parameters. On a site with no facet control, 40–70% is common.
That last number is the one to quote internally. It means the majority of your crawl budget is being spent on URLs that will never earn a click.
The three-bucket decision
Every facet combination belongs in exactly one of three buckets.
Bucket 1 — Index it
A facet combination deserves indexing when people search for it as a phrase and it returns a stable, useful set of products.
The test is concrete: does the combination have real search demand, and would a shopper landing on it find a coherent selection? “Women’s waterproof hiking boots” passes. “Women’s hiking boots, size 7, blue, under $150, in stock” does not — nobody searches that string, and the result set changes daily.
In practice, indexable combinations are almost always single-facet and drawn from a small set of attributes:
- Category + brand — “Nike running shoes”
- Category + primary attribute — “waterproof hiking boots”, “oak dining tables”
- Category + use case — “trail running shoes”
- Category + gender or age segment
These pages need to be treated as real landing pages, not filtered views. That means a unique H1, an intro paragraph that is not templated boilerplate, a self-referencing canonical, and inclusion in the sitemap. A filtered view with an auto-generated title is not going to outrank a competitor’s purpose-built category page.
Bucket 2 — Crawlable but not indexed
Some combinations need to be reachable — for users, and so link equity flows through them — but should not compete in the index.
These get noindex, follow. Typical members: sort orders, two-facet combinations that are useful but not searched, and pagination beyond the first page where you have decided the series should not rank.
The important detail: noindex does not save crawl budget. Google still has to fetch the page to see the directive. It solves the index bloat problem, not the crawl waste problem.
Bucket 3 — Never crawled
Everything else. Three-plus facet combinations, price ranges, in-stock toggles, colour-plus-size permutations, and view-mode parameters.
These should not be crawlable at all. The mechanisms, in order of reliability:
- Render the links as non-crawlable controls. If the filter UI uses buttons with JavaScript handlers and updates the URL via the History API rather than emitting
<a href>for every combination, crawlers never discover the combinations in the first place. This is by far the most robust solution and the one worth asking your developers for. - Block the parameter patterns in robots.txt. Effective and cheap, with one caveat: blocked URLs can still be indexed without content if they receive external links, and any link equity pointing at them is stranded.
- Canonicalise to the unfiltered category. Works, but Google treats canonicals as hints, and it does nothing to reduce crawling.
Use the first where you can and the second where you cannot. Do not rely on the third alone.
The order to implement in
Doing this in the wrong order can remove pages that were earning revenue.
- Export every indexed facet URL with its impressions and clicks from the last 12 months. Some of them are earning money. You need to know which before you touch anything.
- Promote the earners to Bucket 1. Any facet URL with meaningful clicks becomes a real landing page with unique content, not a casualty of cleanup.
- Decide the crawl rules for the rest. Write the rules down as patterns, not as a list of URLs, because the list regenerates every time merchandising adds an attribute.
- Ship robots rules and noindex directives together, and only after step 2 is live.
- Watch index coverage weekly for eight weeks. Expect the indexed URL count to fall substantially. That is the intended outcome, and the traffic line should not follow it down.
That last point is where nerve is required. Seeing “indexed pages” drop by 80% looks alarming on a dashboard. If the pages removed were never earning clicks, the only thing that changes is that the crawler starts spending its time on your real category pages.
Two adjacent problems worth fixing at the same time
Duplicate variants. If every colourway of a product has its own URL with the same description, you have a second duplication problem stacked on the first. Consolidate to one canonical product URL with variant selection on-page, unless individual variants have genuine standalone demand.
Out-of-stock and discontinued URLs. The default behaviour on most platforms is to 404 a product when it is removed, which discards every link pointing at it. A discontinued product with external links should redirect to its category or to a successor product, not disappear. Products that are temporarily out of stock should stay live with availability marked in the product schema, which also keeps the merchant feed consistent.
What good looks like afterwards
On a catalog site that has been through this properly:
- Googlebot requests to parameter URLs drop toward single digits as a percentage of crawl.
- Category and collection pages get crawled several times more often than before.
- New products and collections get indexed in days rather than weeks.
- The indexed URL count is a fraction of what it was, and organic revenue is not.
That last pairing is the point. Crawl budget is not an abstraction on a large catalog — it is the reason a new collection page takes six weeks to appear.
Faceted navigation is the first thing we look at on any catalog site, and it is covered in the free technical audit. See how ecommerce SEO runs here, or send us your domain and the audit will name your worst facet patterns specifically.
