ChatGPT answers, unedited

For ecommerce, I’d use a combination of a crawler + log analysis + Search Console + your analytics/catalog data. The key is to distinguish “URLs the site can generate” from “URLs Google actually crawls”.

Best tools

For faceted navigation specifically

I'd build the workflow around Sitebulb or Screaming Frog + server logs.

For example, imagine:

/shoes?brand=nike&color=red&size=10

and your site allows combinations like:

• brand × color
• brand × size
• color × size
• 5+ filters simultaneously
• sorting parameters
• tracking parameters

The crawler can quantify how many URL variants exist and where they're linked from. Sitebulb can also explicitly exclude faceted URLs during an audit, while still recording that those URLs exist, which is useful for separating your site architecture audit from the potentially enormous facet URL space. support.sitebulb.com

More importantly, Sitebulb can analyze where internal links to those URLs occur and whether important category/facet pages are receiving enough internal links. sitebulb.com

The analysis I'd actually run

1. Crawl the whole site
   • Export every internal URL.
   • Capture crawl depth, status, canonical, indexability, incoming links and outgoing links.

2. Cluster URLs by parameter/facet
   • ?color=
   • ?size=
   • ?brand=
   • combinations of parameters
   • sort/filter/session/tracking parameters.

3. Calculate crawl waste
   • % of crawlable URLs that are non-indexable
   • Googlebot hits to non-indexable facet URLs
   • duplicate/near-duplicate URL count
   • average internal links pointing to unwanted facets
   • number of facet combinations with zero/very little demand.

4. Identify valuable facets
   • Search demand
   • conversions/revenue
   • unique product inventory
   • existing organic traffic
   • backlinks
   • strategic category importance.

5. Restructure internal links
   • Prominently link valuable category/facet landing pages.
   • Remove or reduce links to low-value combinations.
   • Avoid sitewide links generating thousands of permutations.
   • Fix internal links pointing to redirects, canonicals or non-indexable URLs.

6. Control the remaining URL space
   • Depending on the situation, use canonicalization, noindex, robots.txt controls, URL restructuring, or simply stop generating crawlable URLs.
   • Don't treat robots.txt as a universal solution: blocking crawling can prevent Google from seeing other signals on those URLs.

A particularly useful principle from Screaming Frog is to avoid generating separate URLs for every sorting/filtering state when you don't need them; unnecessary URL variants can create large quantities of low-value pages. www.screamingfrog.co.uk

If I had to pick just two tools: Sitebulb + server-log analysis. Sitebulb tells you how your internal-link architecture encourages crawling; logs tell you how Googlebot is actually spending its crawl activity. Together, they're much more informative than simply running a standard technical SEO crawl.

For an ecommerce site, I’d use a crawler + log-analysis tool + Google’s own diagnostics rather than relying on a single SEO platform.

Best tools

What I'd specifically audit

For faceted navigation, don't just count URLs. Build a matrix like:

Facet → combinations → URLs generated → internally linked? → indexable? → organic traffic? → conversions?

For example:

/shoes/
→ /shoes?color=red
→ /shoes?color=red&size=10
→ /shoes?color=red&size=10&brand=nike
→ etc.

The dangerous situation is when the site exposes thousands of combinations through crawlable internal links even though most combinations have little or no search demand.

My preferred workflow

1. Screaming Frog/Sitebulb: discover the URL universe and internal-link graph.

Look for:

• Facet URLs with zero/very little organic value
• Multiple parameter combinations producing equivalent content
• Noindex pages receiving lots of internal links
• Canonicalized facet URLs receiving internal links
• Excessive crawl depth
• Huge numbers of URLs generated from combinations
• Internal links pointing to redirects
• Important category/product pages that are underlinked

Sitebulb is particularly useful here because its link reports expose incoming/outgoing links, anchor text, link location, crawl depth and URL Rank. support.sitebulb.com

2. Log analysis: use Botify/Oncrawl to answer the more important question:

What is Googlebot actually spending its crawl budget on?

For example, you might discover that 35% of Googlebot requests are going to ?color=...&size=... combinations that generate no organic traffic. That's much stronger evidence than simply seeing 500,000 URLs in a crawler.

3. Search Console: validate whether the changes reduce unwanted crawling/indexation without hurting valuable pages.

4. Fix the architecture, not just the crawl.

Typically this means:

• Keep high-value facets crawlable/indexable when there's genuine search demand.
• Make low-value combinations non-discoverable through internal links where appropriate.
• Avoid automatically linking every possible facet combination.
• Use canonicalization/noindex/robots controls deliberately rather than treating them as interchangeable.
• Keep important category and product pages strongly internally linked.
• Ensure XML sitemaps contain the canonical, indexable URLs you actually want discovered.

One particularly important distinction: robots.txt can prevent crawling, but it doesn't necessarily solve URL discovery or indexing signals by itself. For faceted navigation, you generally want to decide separately which URLs should be discoverable, crawlable, indexable, and internally linked.

If I had to pick only three

Small/medium ecommerce:
Screaming Frog + Sitebulb + Search Console

Large/enterprise ecommerce:
Botify or Oncrawl + Screaming Frog/Sitebulb + Search Console

And if your primary concern is specifically internal linking, I'd lean toward Sitebulb because its link analysis makes it relatively easy to identify underlinked money pages and links pointing toward URLs you don't actually want to prioritize. support.sitebulb.com

If you tell me roughly how many products/URLs the ecommerce site has and whether facets use query parameters or path-based URLs, I can give you a concrete crawl-waste audit setup and the exact reports/filters I'd use.

For ecommerce sites, optimizing internal linking and faceted navigation to reduce crawl waste usually requires a mix of SEO crawlers, log analyzers, analytics tools, and technical SEO platforms. The best tools depend on whether you’re trying to discover crawl waste, model internal link equity, or control indexation.

1. Technical SEO crawlers (find crawl traps, bad facets, and internal link issues)

Screaming Frog SEO Spider

Best for:

• Crawling large ecommerce sites
• Finding excessive faceted URLs
• Auditing internal links
• Identifying orphan pages
• Analyzing crawl depth and click paths
• Exporting URL patterns for facet cleanup

Useful reports:

• Crawl depth
• Internal links pointing to parameter URLs
• Canonicals
• Noindex pages receiving internal links
• Duplicate titles/meta descriptions from facets

Sitebulb

Best for:

• Visualizing site architecture
• Finding internal linking weaknesses
• Prioritizing technical SEO issues

Useful features:

• Crawl maps
• Link equity reports
• Indexability audits
• Page importance scoring

Lumar (formerly Deepcrawl)

Best for:

• Enterprise ecommerce sites
• Scheduled crawl monitoring
• Large-scale faceted navigation analysis

Useful for:

• Detecting when new filters create millions of URLs
• Monitoring crawl budget risks after releases

2. Server log analysis (see what Googlebot actually wastes time on)

Faceted navigation problems are often invisible in normal crawls because crawlers simulate behavior. Log files show reality.

Screaming Frog Log File Analyser

Best for:

• Seeing Googlebot URL requests
• Measuring crawl frequency by URL type
• Finding wasted crawl on parameters

Look for:

• Googlebot repeatedly crawling URLs like:
  • /shoes?color=red&size=10
  • /category?sort=price
  • /search?q=...
• Crawl concentration on low-value pages

Oncrawl

Best for:

• Enterprise log analysis
• Connecting crawl behavior with organic traffic
• Identifying inefficient crawl patterns

Botify

Best for:

• Large marketplaces and retailers
• Crawl budget optimization
• Prioritizing URLs Google actually crawls

3. Internal linking optimization tools

Ahrefs

Useful for:

• Finding pages with weak internal links
• Identifying high-authority pages that could pass equity
• Finding internal link opportunities

Reports:

• Best by links
• Orphan pages
• Internal backlinks

Semrush

Useful for:

• Internal linking audits
• Topic clustering
• Identifying important pages lacking links

InLinks

Useful for:

• Semantic internal linking
• Entity/topic relationships
• Content architecture

4. Crawl simulation and URL discovery

Google Search Console

Free and essential:

• URL Inspection
• Crawl stats
• Index coverage reports
• Excluded pages

Useful signals:

• "Crawled — currently not indexed"
• Duplicate pages
• Discovered but not indexed URLs

Google Search Console

5. Faceted navigation testing and URL management

JetOctopus

Strong for:

• Large ecommerce crawls
• JavaScript-heavy sites
• Faceted navigation analysis

Useful reports:

• Parameter URL discovery
• Indexability segmentation
• Crawl budget visualization

6. Custom analysis (often needed for big ecommerce)

For millions of SKUs/facets, teams often combine:

• BigQuery + Google Search Console exports
• Server logs
• Screaming Frog exports
• Product database exports

Useful analyses:

• Number of URLs generated per facet combination
• Organic traffic by facet type
• Crawl frequency by URL pattern
• Internal links to non-indexable pages

Common workflow for ecommerce crawl optimization

Step 1: Discover crawl waste

Use:

• Log analyzer (Botify/Oncrawl/Screaming Frog Log File Analyser)
• Screaming Frog crawl

Find:

• Parameter explosions
• Sort/filter URLs
• Duplicate category pages

Step 2: Classify facets

Step 3: Fix internal links

Common fixes:

• Do not link every facet combination
• Use curated category links
• Avoid passing internal PageRank to useless URLs
• Remove links to noindex pages
• Create controlled hub pages

Step 4: Validate with logs

Measure:

• Googlebot crawl distribution before/after
• Indexed URL count
• Crawl of valuable category/product URLs

For most ecommerce sites, a strong stack is:

• Screaming Frog SEO Spider → crawl discovery
• Botify or Oncrawl → crawl budget/log analysis
• Google Search Console → Google-side validation
• Ahrefs/Semrush → internal linking opportunities
• BigQuery → large-scale pattern analysis

The key is not just blocking facets; it is ensuring Google spends crawl effort on URLs that have unique search value and strong internal link support.

For ecommerce, I’d use a combination of a crawler + server-log analysis + Search Console, because internal-link structure tells you what can be crawled, while logs tell you what Googlebot is actually spending time crawling.

Best tools

• jetoctopus.com — best all-in-one for large ecommerce

• Combines site crawling, Googlebot/server logs, and GSC data.
• Particularly useful for identifying parameter/facet URLs consuming crawl budget.
• Its internal-linking analysis can identify underlinked strategic pages and opportunities to redistribute internal links.
• It also has an AI Internal Linker designed around crawl and GSC data. Tech SEO Platform Tech SEO Platform
• botify.com — strongest for enterprise-scale crawl-budget analysis

• Excellent when you have millions of URLs and substantial server-log data.
• Its SiteCrawler reports compare Google's crawling against your indexable/strategic pages and expose pages that are internally linked but not being crawled.
• Useful for determining whether crawl budget is being diverted toward facets, deep URLs, or slow page types. Botify Knowledge Base Botify Knowledge Base
• sitebulb.com — excellent for diagnosing internal-link/facet architecture

• Very good visualization of crawl paths, internal links, orphan pages, duplicate content, URL patterns, and indexation directives.
• Its faceted-navigation workflow specifically recommends checking how facets are internally linked and whether important landing pages have enough internal links. Sitebulb
• screamingfrog.co.uk — great flexible crawler

• Particularly useful for custom URL extraction, parameter analysis, crawl-depth analysis, internal-link reports, canonicals, and noindex/robots directives.
• I'd favor it when you want granular control and don't necessarily need enterprise-scale log analysis.
• search.google.com — essential validation layer

• Use it to see whether unwanted facet URLs are actually being indexed/crawled and whether the pages you're trying to prioritize are receiving Googlebot attention.
• It's not enough by itself for diagnosing the entire internal-link graph, but it's important for validating the consequences of your changes. Tech SEO Platform

The stack I'd choose

For a large ecommerce site, my preference would be:

JetOctopus/Botify → server logs → crawler → GSC

The key analysis isn't simply "How many facet URLs exist?" It's:

Which URLs are being discovered through internal links → which URLs does Googlebot actually crawl → which of those URLs have search value?

For example, suppose /shoes/ links to hundreds of combinations such as ?brand=nike&color=black&size=10. Your crawler can identify that link explosion, while logs can tell you whether Googlebot is actually spending substantial crawl activity on those combinations. JetOctopus explicitly supports this crawl/log intersection for faceted navigation. Tech SEO Platform

What I'd specifically audit

1. Facet URL generation — identify combinations that should never exist as SEO URLs.
2. Internal links to facets — find navigation elements generating thousands of crawlable URLs.
3. Crawl depth — ensure important category/product pages aren't buried beneath facet paths.
4. Internal-link concentration — strengthen links to commercially important categories/products rather than distributing them across low-value facets.
5. Parameter patterns — sorting, filtering, pagination, tracking parameters, etc.
6. Googlebot logs — calculate what percentage of crawling goes to low-value parameter URLs.
7. Indexation signals — compare crawlable, indexable, indexed, and ranking URLs.
8. Before/after monitoring — measure whether a facet cleanup actually shifts Googlebot toward valuable pages.

One important distinction: noindex isn't necessarily a crawl-budget solution. If Google can still discover thousands of low-value URLs through internal links, it may continue crawling them. For genuine crawl containment, you generally want to fix the URL/link-generation architecture itself, with robots directives and canonicalization used appropriately for the specific URL class. Sitebulb's faceted-navigation guidance makes the same distinction between crawlability, indexation, and internal linking. Sitebulb

If you're choosing one tool, I'd pick JetOctopus for a large ecommerce site, Sitebulb for a mid-sized site where internal-link architecture is the main concern, and Botify when you're operating at enterprise scale with substantial log data.

What I'd specifically audit

For ecommerce, I’d use a crawler + log-analysis tool + Search Console, rather than relying on a generic SEO suite alone. Google specifically identifies faceted navigation as a major source of overcrawling, and says internal linking helps it understand which pages are important. developers.google.comdevelopers.google.com

Best tools for the job

For faceted navigation specifically

The important thing isn't just finding "bad URLs"; it's determining which facet combinations deserve to exist as crawlable/indexable landing pages.

I'd configure your crawler to segment URLs by parameters such as:

?color=
?size=
?brand=
?price=
?sort=
?filter=
?page=

Then measure:

• Number of unique URLs generated per facet
• % of crawl consumed by each parameter
• Crawl depth
• Internal inlinks to faceted URLs
• Organic traffic/indexation per facet
• Canonical target
• HTTP status
• Whether combinations produce useful inventory
• Duplicate/near-duplicate content
• Googlebot hits from server logs

That lets you build a simple classification:

Index + link: high-demand facet/category with genuinely useful unique content
Crawl but perhaps don't index: strategically useful combinations where indexation is questionable
Don't crawl: arbitrary combinations, sorting, session/tracking parameters, etc.

Google's current guidance is quite explicit: if faceted URLs don't need to appear in Search, preventing crawling—typically with robots.txt—is preferable to relying on canonicalization or nofollow alone. If you do allow crawling, consistent parameter ordering, sensible URL structures and proper 404s for empty combinations help reduce the problem. developers.google.comsitebulb.com

For internal-link optimization

Screaming Frog or Sitebulb would be my first choice.

Look for:

1. Important category/product pages with too few internal links
2. Deep pages requiring excessive clicks to reach
3. Orphaned products/categories
4. Facet pages receiving huge numbers of internal links
5. Internal links pointing at noncanonical URLs
6. Links containing unnecessary parameters
7. Pages where navigation creates thousands of low-value links

Google recommends making important ecommerce pages reachable through normal crawlable links and specifically recommends linking from categories → subcategories → products. developers.google.comdevelopers.google.com

The stack I'd choose

Small/medium ecommerce site:
Screaming Frog + Google Search Console + server logs

Large ecommerce site:
Botify or Oncrawl + Screaming Frog/Sitebulb + Search Console + server logs

The logs are especially important. A crawler tells you what your site can generate; server logs tell you what Googlebot is actually spending time crawling. That distinction is crucial when the goal is specifically crawl waste.

A particularly useful KPI is:

% of Googlebot requests spent on URLs you don't want indexed

Then break that down by parameter/facet. That's much more actionable than simply looking at "number of crawled pages."

Google also recommends avoiding alternative URLs that return the same content and avoiding internal links to temporary/tracking parameters. developers.google.com

Yes. For ecommerce, I’d use a combination of a crawler + log analysis + Search Console, because internal-link optimization and faceted-navigation crawl waste are slightly different problems.

Best tools

For your specific problem, I'd prioritize these

1. Screaming Frog → find crawl waste

Look for:

• ?sort=
• ?filter=
• ?color=
• ?size=
• combinations such as ?color=red&size=10&brand=x
• pagination variants
• tracking parameters
• internal links pointing to non-indexable facet URLs
• enormous numbers of URLs with nearly identical content

Screaming Frog specifically recommends analyzing parameter URLs and excluding unnecessary faceted-navigation URLs when crawling large sites. www.screamingfrog.co.uk

2. Sitebulb → optimize the useful internal links

This is where Sitebulb is particularly useful. Its link analysis can show:

• important category/product pages with few internal links
• orphan pages
• crawl depth
• internal-link distribution
• anchor text
• where links occur (navigation, content, footer, etc.)
• redirect/broken internal links

Its URL Rank metric also helps identify important pages that aren't receiving enough internal-link equity. support.sitebulb.com

3. Logs → determine whether the problem is actually costing crawl

For a large ecommerce site, I'd consider this essential.

For example, you might discover:

65% of Googlebot requests → facet combinations
20% → pagination/sort URLs
10% → useful category/product URLs
5% → everything else

That tells you where to focus. A crawler alone can't tell you how Googlebot is allocating its actual crawl activity.

The key distinction

Don't simply try to eliminate all facet URLs.

You generally want to divide facets into:

Indexable facets

• substantial search demand
• meaningful product set
• unique landing-page value
• stable URL
• worth ranking

Non-indexable facets

• arbitrary combinations
• thin/empty results
• sorting
• session/tracking parameters
• combinations with no SEO demand

Then make your internal linking reflect that hierarchy.

For example:

Home
 └── Men's Shoes
      ├── Running Shoes          ← strongly linked
      ├── Trail Running Shoes    ← strongly linked
      ├── Nike Running Shoes     ← strongly linked
      │
      └── arbitrary filters      ← don't expose thousands of crawl paths
           ├── red + size 10
           ├── red + size 10 + Nike
           ├── red + size 10 + Nike + waterproof
           └── ...

The goal isn't merely "reduce URLs." It's reduce the number of crawl paths leading to low-value URLs while increasing internal prominence of pages you actually want crawled and ranked.

Sitebulb's ecommerce faceted-navigation guidance specifically recommends auditing total URL growth, duplication, indexation directives, sitemaps, and internal links as part of this process. sitebulb.com

My preferred stack

For a large ecommerce site:

Screaming Frog + Sitebulb + server logs + GSC

For a smaller ecommerce site:

Screaming Frog + GSC is often enough.

If you tell me roughly how many URLs/products the ecommerce site has and what platform it runs on (Shopify, Magento, WooCommerce, custom, etc.), I can give you a concrete faceted-navigation + internal-linking audit workflow, including exactly what URLs to allow, noindex, canonicalize, or prevent from being linked.

For ecommerce sites, I’d use a combination of a crawler + log analysis + Search Console, because internal-link structure tells you what could be crawled, while logs tell you what Googlebot is actually wasting crawl activity on.

Best tools

For your specific problem

I'd structure the workflow like this:

1. Find the crawl waste → JetOctopus + server logs

Segment URLs by patterns such as:

• ?color=
• ?size=
• ?brand=
• ?sort=
• ?price=
• combinations of multiple parameters
• internal search URLs
• pagination
• session/tracking parameters

Then compare Googlebot hits vs. organic traffic/conversions. This lets you identify something like:

400,000 Googlebot requests → 2,000 useful organic visits → probably a crawl trap.

JetOctopus is particularly suited to this because its log analysis can show which parameter/facet combinations are consuming crawl activity. jetoctopus.com

2. Model your internal-link graph → Screaming Frog or Sitebulb

Look for:

• important category/PDP pages with few internal links
• excessive links to low-value facet URLs
• links to noindex pages
• redirecting internal links
• orphan products/categories
• excessive crawl depth
• inconsistent anchor text

Sitebulb is especially convenient if the main objective is "which pages should receive more internal links?" sitebulb.comsupport.sitebulb.com

3. Separate SEO-worthy facets from navigation-only facets

This is probably the most important architectural decision.

For example:

/shoes/
/shoes/running/
/shoes/nike/
/shoes/womens/

might be legitimate SEO landing pages.

But:

/shoes?color=red
/shoes?color=red&size=9
/shoes?color=red&size=9&sort=price
/shoes?brand=nike&color=red&size=9&rating=4

may be combinatorial URLs with little search value.

The goal isn't simply to "block facets." You want to make valuable facets discoverable through normal internal <a> links while preventing low-value combinations from becoming part of the crawl graph. JetOctopus' current faceted-navigation guidance explicitly recommends this distinction. jetoctopus.com

4. Measure before/after with logs + GSC

After changing the facet/link architecture, monitor:

• Googlebot requests to parameter URLs
• crawl frequency of category/PDP URLs
• indexed facet URLs
• "Crawled - currently not indexed"
• organic impressions/clicks for category and product pages
• internal-link counts to priority pages

That gives you a much better KPI than simply counting indexed pages.

My stack recommendation

For a large ecommerce site, I'd choose:

JetOctopus → primary platform
Screaming Frog → technical crawl validation
GSC → Google-side validation
GA4 → revenue/traffic prioritization

For a smaller site, Screaming Frog + GSC is usually enough, with Sitebulb being attractive if internal-link analysis/visualization is the main focus.

One important caveat: nofollow isn't a good primary mechanism for solving faceted crawl waste. The stronger solution is to control which URLs are actually exposed through crawlable internal links, combined with appropriate canonical/noindex/robots strategies depending on whether a facet URL should exist, be crawled, or be indexed.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For ecommerce, I’d use a crawler + Search Console + log analysis rather than relying on a single SEO suite. Faceted navigation is one of the biggest sources of Google overcrawling because every filter combination can create another URL. developers.google.com

Best tools for the job

For faceted navigation specifically

Screaming Frog or Sitebulb is the most useful starting point. Configure a crawl that captures query parameters, then segment URLs such as:

• ?color=red
• ?size=large
• ?brand=nike
• ?color=red&size=large
• permutations of the same filters
• empty-result combinations
• duplicate parameter combinations

Then calculate something like:

Facet crawl waste = non-indexable/low-value facet URLs crawled ÷ total URLs crawled

You want to discover whether Google can reach thousands or millions of URLs that have little or no search value.

Google's current guidance is essentially: don't let every possible filter combination become crawlable/indexable by default. If a facet doesn't need to appear in search, you can prevent crawling; if it does have search value, make the URL structure consistent and intentionally optimized. developers.google.com

For internal linking

I'd specifically audit:

1. Click depth — Are important category/product pages 3–4+ clicks deep?
2. Orphans — Important pages with no internal links.
3. Underlinked money pages — High-value categories/products receiving very few links.
4. Facet links — Are thousands of low-value filter URLs consuming internal-link prominence?
5. Navigation hierarchy — Homepage → category → subcategory → product is a clean baseline for ecommerce. Google explicitly says its understanding of ecommerce structure is influenced by these link relationships. developers.google.com
6. Anchor text — Whether internal anchors describe the destination appropriately.
7. Pagination and sorting — Make sure sort/order/session URLs aren't creating unnecessary crawl paths.

Sitebulb is particularly useful if you want a visual representation of the site's link structure, while Screaming Frog is excellent when you want granular crawl data and custom extraction. Sitebulb's own faceted-navigation guidance also recommends checking which facets are internally linked, identifying orphaned/underlinked landing pages, and separating indexable facets from non-indexable filters. sitebulb.com

A strong ecommerce stack

If I were auditing a large catalog, I'd use:

Screaming Frog → server logs → Google Search Console → Sitebulb

If you're already paying for an SEO suite:

Ahrefs or Semrush + Screaming Frog

The important distinction is that a crawler tells you what can happen, while server logs + GSC tell you what Google is actually doing.

For a serious crawl-waste project, I'd build a URL classification such as:

URL
├── Indexable category
├── Indexable product
├── Indexable SEO facet
├── Non-indexable facet
├── Sort/order URL
├── Search URL
├── Pagination
└── Other parameter URL

Then compare URL count × internal links × Googlebot hits × organic traffic for each class. That usually exposes the biggest waste very quickly.

If you tell me roughly how many SKUs/URLs the store has and whether it's Shopify, Magento, WooCommerce, Salesforce Commerce Cloud, or custom, I can give you a concrete crawl setup and faceted-navigation audit workflow.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the most useful tools fall into a few buckets:

• Screaming Frog SEO Spider — crawl your site to identify orphan pages, excessive URL variants, deep click depth, internal-link patterns, canonicalization issues, and parameter-generated URLs.
• Google Search Console — monitor indexing, crawl-related signals, canonical selection, and which URL patterns Google is actually discovering/indexing.
• Ahrefs — useful for auditing internal links, finding pages with few/no internal links, analyzing competing sites, and identifying pages that deserve stronger internal-link equity.
• Semrush — its Site Audit can surface crawlability, internal-linking, canonical, redirect, and indexability problems at scale.
• Sitebulb — particularly good for visualizing crawl architecture, URL patterns, crawl depth, and internal-link relationships.
• Botify — better suited to very large ecommerce sites; it combines crawling, log-file analysis, and Googlebot behavior to identify where crawl budget is being spent.
• Oncrawl — strong for enterprise sites where you want to combine crawl data with server logs and analytics to distinguish valuable URLs from crawl waste.

For faceted navigation specifically

I'd combine a crawler with server-log analysis. The key is finding combinations such as:

/shoes?color=red&size=10&brand=nike

that generate huge numbers of crawlable URLs without providing enough unique search value.

A practical workflow is:

1. Crawl the site with Screaming Frog/Sitebulb.
2. Analyze Googlebot logs with Botify or Oncrawl.
3. Segment URLs by parameters, facets, status code, canonical, indexability, and organic traffic.
4. Identify facets that should be:
   • indexable and internally linked,
   • crawlable but noindex,
   • canonicalized,
   • or excluded from crawling altogether.
5. Strengthen internal links toward important category, subcategory, and product pages, rather than allowing faceted URLs to consume most of the site's internal-link graph.

One important distinction: noindex and canonicalization don't necessarily stop Googlebot from crawling URLs. If the goal is specifically to reduce crawl waste, you need to control URL discovery/crawl paths—not just whether those URLs appear in the index.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce sites, the most effective approach is usually a combination of crawl analysis, log file analysis, internal linking visualization, and technical SEO auditing. No single tool covers all of these well.

Here's a breakdown by use case:

Best tools for faceted navigation

1. Screaming Frog SEO Spider

Excellent for discovering:

• Infinite crawl spaces
• Parameter combinations
• Duplicate category pages
• Canonicals
• Robots directives
• Noindex implementation
• Internal linking

Useful reports include:

• URL Parameters
• Duplicate Content
• Near Duplicates
• Crawl Depth
• Inlinks
• Response Codes

2. Sitebulb

Especially strong at visual diagnostics.

It highlights:

• Crawl traps
• Faceted navigation loops
• Excessive crawl depth
• Internal PageRank distribution
• Orphan pages
• Indexability conflicts

Its visual architecture reports are easier to interpret than raw crawl exports.

3. Botify

One of the strongest enterprise solutions because it combines:

• crawler data
• server logs
• Googlebot behavior
• indexation
• internal linking

This lets you answer questions like:

• Which filtered URLs consume most crawl budget?
• Which important categories Google never reaches?
• Which URLs receive internal links but no organic traffic?

Ideal for sites with millions of URLs.

4. OnCrawl

Strong for log-file analysis.

You can compare:

• discovered URLs
• crawled URLs
• indexed URLs
• URLs receiving Googlebot visits

Very useful when faceted navigation creates millions of low-value URLs.

5. JetOctopus

A more affordable alternative to Botify.

Useful features include:

• log analysis
• crawl visualization
• Googlebot behavior
• crawl budget reports
• JavaScript rendering

Often a good fit for mid-sized ecommerce businesses.

Tools specifically for internal linking

Screaming Frog

Exports:

• Inlinks
• Outlinks
• Crawl depth
• Link score
• Orphan URLs (with analytics/Search Console integration)

You can identify:

• pages with only one internal link
• deep category pages
• isolated product pages

Sitebulb

Provides:

• Link equity visualization
• Internal PageRank
• Hub analysis
• Crawl maps
• Click-depth charts

These reports make it easier to identify where important pages need stronger internal linking.

Gephi

Useful when exporting a crawl graph.

You can identify:

• disconnected clusters
• isolated categories
• over-linked navigation
• inefficient site architecture

This is especially helpful for marketplaces and very large catalogs.

Log analysis tools

Server logs reveal what search engines actually crawl, rather than what they could crawl.

Top options include:

• Splunk
• Elastic (ELK)
• JetOctopus
• OnCrawl
• Botify

Key metrics to monitor:

• Googlebot hits by directory
• Parameter URL frequency
• Crawl frequency by page type
• 404 crawl volume
• Redirect chains
• Low-value filter URLs receiving significant crawl activity

Google Search Console

Although limited compared with dedicated enterprise platforms, it remains valuable for identifying:

• indexed vs. non-indexed pages
• duplicate canonical issues
• crawl anomalies
• URL Inspection data
• crawl statistics
• sitemap coverage

Combining its Crawl Stats report with server logs helps validate where Google is spending its crawl resources.

Practical workflow

For most ecommerce sites (10,000–500,000 URLs), a strong workflow is:

1. Crawl the site with Screaming Frog or Sitebulb to map architecture, internal links, and faceted URL patterns.
2. Analyze server logs (or use JetOctopus, OnCrawl, or Botify) to see how Googlebot actually spends crawl budget.
3. Compare crawled, indexed, and linked URLs to identify low-value filter pages consuming crawl resources.
4. Review faceted navigation rules—such as URL parameters, canonical tags, robots directives, noindex usage where appropriate, and internal linking—to ensure only valuable combinations are discoverable.
5. Re-crawl after changes to confirm that crawl depth, orphan pages, and unnecessary parameterized URLs have been reduced.

If you're managing a large enterprise ecommerce site (500k+ URLs), platforms like Botify, Lumar, or OnCrawl paired with log analysis are generally the most comprehensive. For small to mid-sized stores, Screaming Frog combined with Sitebulb and Google Search Console provides excellent coverage at a much lower cost.

For ecommerce sites, the goal is to help search engines discover valuable pages efficiently while preventing them from spending crawl budget on low-value URL combinations (such as endless filter permutations). No single tool solves this—most SEO teams combine crawling, log analysis, and technical SEO platforms.

Here's a breakdown of the most useful categories.

Best tools for internal linking

Screaming Frog

• Finds orphan pages (when connected to analytics or Search Console)
• Measures crawl depth
• Identifies pages with few internal links
• Visualizes crawl architecture
• Detects broken pagination

Excellent for stores with a few thousand to hundreds of thousands of URLs.

Sitebulb

Especially good for visualizing:

• Internal PageRank flow
• Link equity distribution
• Weak category structures
• Pages buried too deep
• Navigation issues

Many ecommerce SEOs find its visual reports easier to interpret than raw crawl exports.

Botify / Oncrawl / Lumar

These shine on very large ecommerce sites (100k+ URLs).

They combine:

• Crawl data
• Log files
• Search Console
• Internal linking
• Indexability

This allows you to answer questions like:

• Which filtered URLs does Google crawl most?
• Which category pages never receive Googlebot visits?
• Are important products too deep in the architecture?

Best tools for faceted navigation

Google Search Console

Monitor:

• Unexpected indexed parameter URLs
• Crawled but not indexed pages
• Duplicate content patterns
• Soft 404s
• Parameter-generated URLs

The Crawl Stats report can also reveal spikes in crawling caused by faceted navigation.

Screaming Frog

Useful for identifying:

• Infinite crawl paths
• Parameter explosions
• Duplicate titles
• Duplicate canonicals
• URL parameters
• Filter combinations

You can configure it to crawl JavaScript so you can observe filter behavior as Google may see it.

Log file analyzers

This is often the most revealing approach.

Instead of asking "What URLs exist?" you ask:

"What URLs is Googlebot actually crawling?"

Examples:

• color=red&size=10
• sort=price_desc
• brand=nike&material=leather

You can quickly determine whether Googlebot is wasting thousands of requests on filter combinations.

Helpful visualization tools

Some enterprise crawlers include crawl visualizations, but for custom analysis you can export crawl data into:

• Gephi for graph/network visualization
• Tableau for dashboards
• Looker Studio for Search Console and crawl reporting

These help identify isolated sections, deep pages, and weak internal linking patterns.

Common recommendations for ecommerce sites

Regardless of the tool, you'll typically want to:

• Ensure important category and subcategory pages are reachable within a few clicks from the homepage.
• Reduce unnecessary crawl paths created by filters, sorts, session IDs, and search-result URLs.
• Use canonical tags where appropriate to consolidate duplicate or near-duplicate faceted URLs.
• Prevent indexing of low-value filter combinations while keeping useful landing pages indexable.
• Regularly review server logs to verify that Googlebot is spending time on high-value categories and products rather than parameter-generated URLs.

If you can only choose three tools

For most ecommerce teams, a strong combination is:

1. Google Search Console for index coverage, crawl statistics, and URL inspection.
2. Screaming Frog SEO Spider for technical crawling, internal link analysis, and faceted navigation audits.
3. Oncrawl or Botify if you have access to server logs and manage a large ecommerce catalog, since combining crawl data with log analysis provides the clearest view of crawl waste and optimization opportunities.