Index bloat is the gap between the number of URLs Google has indexed and the number that could plausibly earn a click. On most sites above a few thousand pages that gap is large, and it is invisible until someone measures it.
Measuring that gap is the first step in every audit run by the team at seothumb.com. We reconcile crawl data with Search Console before touching a single URL, then forecast what a cleanup should return. One senior consultant makes each keep, noindex or remove call, so pages that quietly earn clicks stay where they are.
Key takeaways
- Index bloat is the gap between indexed URLs and pages that could earn a click; above about 1.3 times is worth investigating.
- Reconcile a full crawl, the page indexing report and twelve months of performance data before deciding anything.
- Never noindex or remove a URL that earned impressions or clicks in the last twelve months; improve it instead.
It matters for two reasons. Crawl budget spent on worthless URLs is crawl budget not spent on your money pages, so new content takes longer to appear. And a site where most indexed URLs are thin gives Google a weaker overall signal about what the domain is good at.
Measure the Gap First
Two numbers, five minutes: This is the pattern seothumb.com checks first on any site with more than a few thousand URLs.
- Indexed URLs. Search Console → Indexing → Pages → “Indexed”. Not the sitemap count, the total.
- URLs that deserve to rank. Your real page inventory: products, categories, service pages, genuinely useful articles.
If the first number is more than about 1.3× the second, you have bloat worth investigating. If it is 5× or more, it is probably the highest-leverage technical work available to you.
Where It Comes From
Almost always one of these seven, in roughly descending order of volume:
Faceted and parameter URLs. The biggest source on any ecommerce site by a wide margin. Filter combinations, sort orders, view modes and tracking parameters each multiply the URL space. This is a large enough topic to have its own guide.
Internal search results. ?s= and /search/ URLs get indexed when someone links to them or a sitemap plugin includes them. They are generated on demand, which means the URL space is effectively infinite.
Tag and author archives. A blog with 400 posts and 300 tags produces 300 archive pages, most holding one or two posts. They compete with the posts they list and rarely earn anything.
Pagination. Page 2 onward of a category or blog series, where the intro copy and title are duplicated from page 1. Sometimes these should be indexed; usually the duplicate template copy means they should not be. In every seothumb.com audit, that performance check comes before any removal list is written.
Attachment and media pages. WordPress in particular generates a page per uploaded image by default. On a site with an image library this alone can double the indexed count.
Staging and development leftovers. Old subdomains, /dev/, /v2/, /old/ directories, and test pages that were never removed. These also frequently carry noindex that was removed at launch and not reinstated.
Thin programmatic pages. Location pages for cities you do not serve, category pages with two products, “compare X vs Y” pages generated for every combination. Programmatic content is not inherently bad, but the threshold for “this page is useful” has to be enforced somewhere.
Find It Without Guessing
The reliable method is a reconciliation, not a crawl on its own.
- Crawl the site fully, ignoring robots.txt on the second pass so blocked URLs surface too.
- Export Search Console’s page indexing report, including the “Indexed, not submitted in sitemap” bucket. That bucket is where bloat concentrates.
- Export 12 months of performance data at page level, including zero-click pages. You need impressions as well as clicks.
- Join the three. Any indexed URL with zero clicks and negligible impressions over twelve months is a candidate.
The twelve-month window matters. A page with seasonal demand can look dead in a 90-day export and be valuable in November.
Decide What to Do with Each Group
Sort candidates into patterns, not individual URLs, because the individual list regenerates.
Delete and return 410 when the page has no purpose, no traffic and no external links. A 410 tells Google the removal is intentional and gets the URL dropped faster than a 404. Google’s guide to managing crawl budget on large sites also recommends returning a 404 or 410 for permanently removed pages.
Redirect 301 when the page has no value itself but has external links pointing at it, or when a clearly better equivalent exists. Do not redirect everything to the homepage — an irrelevant redirect target is treated as a soft 404 anyway.
Noindex, follow when the page needs to exist for users but should not compete. Tag archives, filtered views and internal search pages usually land here. Remember this does not save crawl budget; Google still fetches the page to read the directive.
Block in robots.txt when you want to stop the crawling as well, and the URLs have no external links worth preserving. Never combine robots.txt disallow with noindex on the same URL — if the crawler cannot fetch the page, it cannot see the directive. Google’s documentation on blocking indexing with noindex confirms that a robots.txt block stops the crawler from ever seeing the rule.
Improve when the page is thin but the topic is genuinely valuable. A category with two products is not a technical problem, it is a merchandising one.
This decision matrix summarizes which action fits each pattern and whether it reduces crawling.
| Situation | Action | Reduces crawling? |
|---|---|---|
| No purpose, no traffic, no external links | Delete and return 410 | Yes, over time |
| External links or a clearly better equivalent | 301 to the relevant page | Yes, once redirects are processed |
| Needed by users but should not compete | Noindex, follow | No, Google still fetches it |
| Crawling should stop and no links are worth keeping | Disallow in robots.txt | Yes |
| Thin page on a valuable topic | Improve the content or merchandising | Not the goal |
The Mistake That Costs Traffic
The dangerous version of this work is running a bulk noindex based on a crawl alone, without checking performance data first.
Pages that look thin by word count are sometimes earning steadily. Location pages, spec pages and small category pages often have short copy and high commercial intent. Deleting them because a crawler flagged low word count is how a cleanup turns into a traffic loss.
Rule: no URL gets removed or noindexed without first checking twelve months of impressions and clicks. If it earned anything, it goes on the “improve” list rather than the “remove” list.
What to Expect Afterwards
Indexed URL count will fall, often dramatically, and it should. Watch these instead:
- Crawl requests to your money templates should rise in the log files within two to four weeks.
- Time to index for new pages should shorten noticeably.
- Impressions on your real pages should be flat or up. If they drop, something valuable was caught in the cleanup and should be restored.
Give it eight weeks before judging. Deindexing is not instant, and the first few weeks will show a falling page count with no upside yet.
Index bloat is one of the nine checks in the free technical audit, and the reconciliation above is exactly how we run it. See the full audit scope, or send your domain and we will tell you what your real gap is.
On stores, most bloat starts with filters, so pair this with our guide to faceted navigation SEO and the wider ecommerce SEO section. Index bloat is one check among many in our technical SEO audit checklist, and you can find related crawling and indexing guides in the technical SEO category.
Frequently Asked Questions
How Do I Know If My Site Has Index Bloat?
Compare the total indexed count in Search Console's page indexing report with your real inventory of products, categories, service pages and useful articles. If indexed URLs exceed that inventory by about 1.3 times, investigate. At five times or more, cleanup is probably your highest-leverage technical task.
Does Noindex Save Crawl Budget?
No. Google still has to fetch a noindexed page to read the directive, so crawling continues. Noindex controls what appears in results. If you need crawling to stop and the URLs have no external links worth keeping, a robots.txt disallow is the tool, but never on the same URL as noindex.
How Long Does an Index Bloat Cleanup Take to Show Results?
Allow about eight weeks before judging it. Deindexing is gradual, so early weeks show a falling page count with no upside yet. Look for more crawl requests to money templates in the logs, faster indexing of new pages, and flat or rising impressions on your real pages.
