ChatGPT answers, unedited

For ecommerce sites, I’d use a combination of a crawler, log analysis, Search Console, and an SEO research platform. Faceted navigation is one of the biggest sources of crawl waste because combinations of filters can create enormous URL spaces. Google specifically recommends preventing crawling of faceted URLs you don't need indexed, or tightly controlling the URL patterns you do expose. Google for Developers Google for Developers

Tools worth using

• Screaming Frog SEO Spider — Excellent for crawling the actual internal-link graph. Use it to find:

• Faceted/parameter URLs being crawled
• Orphan pages
• Excessive crawl depth
• Internal links pointing to canonicalized or non-indexable URLs
• Pages receiving too few internal links
• Duplicate/near-duplicate URL patterns

Shopify's current ecommerce SEO guidance specifically recommends crawlers such as Screaming Frog or Sitebulb for this type of audit. Shopify

• Sitebulb — Similar role to Screaming Frog, but particularly useful when you want visualizations of crawl paths and site architecture. It's useful for seeing whether important products/categories are buried while thousands of facet URLs are consuming the crawl graph.
• Google Search Console — Essential because it shows what Google is actually discovering/crawling/indexing rather than merely what your crawler can find. Compare indexed/discovered URLs against the URLs you actually want searchable. Shopify
• Server-log analysis / Botify / Lumar — For large ecommerce sites, this is extremely valuable. Your crawler tells you what could be crawled; server logs tell you what Googlebot actually requests. This lets you quantify something like:

• 65% of Googlebot requests → ?color=...
• 20% → sort/order parameters
• 10% → pagination
• 5% → valuable product/category URLs

For very large catalogs, platforms such as Botify and Lumar are designed around crawl and log analysis.

• Ahrefs Site Audit — Useful for combining technical crawling with backlink/internal-link analysis. Its documentation specifically discusses using Site Audit to identify large numbers of non-indexable facet URLs and crawler traps. Ahrefs
• Semrush Site Audit — Another good all-in-one option for identifying internal-link, canonical, crawlability and indexability problems. I'd generally choose either Semrush or Ahrefs rather than buying both solely for this purpose.
• Google Analytics + Search Console data — Useful for deciding which facets deserve to become actual SEO landing pages. Don't automatically index every filter combination; identify facets with meaningful search demand and build deliberate, internally linked landing pages for those combinations.

A practical stack

For a small/medium ecommerce site:

Screaming Frog + Google Search Console + Ahrefs/Semrush

For a large catalog (hundreds of thousands/millions of URLs):

Botify/Lumar + server logs + Google Search Console + Screaming Frog/Sitebulb for targeted crawls

What I'd actually optimize

The tools are only useful if you turn their data into rules. A good faceted-navigation architecture might look like:

Homepage
   ↓
Category
   ↓
Subcategory
   ↓
Product

while treating most filters as UX controls rather than crawlable SEO destinations:

Category
 ├── Color filter       → generally not crawlable
 ├── Size filter        → generally not crawlable
 ├── Price filter       → generally not crawlable
 ├── Sort by price      → not crawlable
 └── "Women's running shoes" → dedicated indexable landing page

Google also recommends using direct HTML <a href> links for important ecommerce navigation and making important categories/products reachable through the site's internal link structure. Google for Developers Google for Developers

The key distinction is crawl optimization vs. indexation optimization. A canonical or noindex can help with indexing, but if Google is still discovering millions of facet URLs, you're still making it spend resources crawling them. Google explicitly notes that crawling faceted URLs can consume substantial computing resources. Google for Developers

If you tell me your platform (Shopify, Magento/Adobe Commerce, WooCommerce, BigCommerce, custom) and approximate catalog size, I can give you a specific tool stack + faceted URL strategy + crawl audit workflow.

For ecommerce, I’d use a combination of a crawler + Search Console + server-log analysis, rather than relying on one SEO suite. Google specifically identifies faceted navigation as a major source of overcrawling and recommends controlling which facet URLs are crawlable. Google for Developers Google for Developers

Tools worth using

• Screaming Frog SEO Spider — Excellent for auditing internal linking at URL level. Use it to find:

• Faceted/parameter URLs being discovered
• Orphaned products/categories
• Excessive crawl depth
• Internal links to canonicalized/noindex URLs
• Duplicate or near-duplicate URLs
• Pages receiving very few internal links

Shopify's current ecommerce audit guidance specifically recommends crawlers such as Screaming Frog for checking priority pages, crawl depth, canonicals and parameterized URLs. Shopify

• Sitebulb — Particularly useful when you want visualizations of site architecture and crawl paths. It can make it easier to see whether important categories/products are buried while facet URLs consume large portions of the crawl.
• Ahrefs Site Audit — Useful for combining crawling with backlink/internal-link analysis. Its documentation specifically covers identifying faceted-navigation crawl waste and finding internal backlinks pointing toward problematic facet URLs. Ahrefs
• Semrush Site Audit — Good if you're already using Semrush for keyword/competitor work and want technical crawling in the same platform.
• Google Search Console — Essential because it shows what Google actually discovers, crawls and indexes, rather than merely what your crawler finds. Compare indexed/discovered URLs with the URL types you actually want searchable. Shopify
• Server-log analysis tools such as Botify or Lumar — Most useful on very large catalogs. Logs can show whether Googlebot is actually spending significant crawl activity on ?color=, ?size=, ?brand=, sorting, pagination, internal search, etc., rather than just showing that those URLs exist.

A practical stack

For a 10k–500k URL ecommerce site, I'd generally structure the workflow like this:

1. Screaming Frog/Sitebulb → discover the architecture

Classify URLs into:

category → subcategory → product

versus:

category → facet → facet → facet → product

Then quantify how many discovered URLs belong to each facet combination.

2. Search Console → compare discovery/indexation

Look for patterns such as:

• Crawled — currently not indexed
• Discovered — currently not indexed
• Duplicate/canonicalized URLs
• Huge numbers of URLs generated by parameters

3. Logs → determine actual crawl waste

This is the key distinction. A crawler can tell you "there are 2 million facet URLs." Logs can tell you "Googlebot actually requested 800,000 of them last month."

4. Build an internal-link map

Your important commercial pages should be strongly connected through normal HTML links. Google says it uses the relationships created by internal links to understand site structure and relative page importance, and recommends linking categories → subcategories → products. Google for Developers

Then identify links going into URLs that you don't want crawled/indexed. Those are often a major source of unnecessary crawling.

The important architectural decision

Don't automatically try to block all facets. First divide them into:

Google's current guidance explicitly recommends preventing crawling of facet URLs when they don't need to appear in Search, while allowing deliberately selected facet pages to remain crawlable. It also recommends consistent parameter ordering and returning 404 for combinations with no results. Google for Developers

One subtle but important point: canonical tags aren't equivalent to crawl prevention. Google notes that canonicalization can reduce crawling of non-canonical variants over time, but blocking unnecessary facet crawling is the more direct approach when those URLs don't need to be discovered. Google for Developers

If you tell me roughly how many products/URLs your store has and what platform it's on (Shopify, Magento, WooCommerce, custom, etc.), I can lay out a specific tool stack and faceted-navigation architecture for it.

For ecommerce sites, the best stack usually combines crawl diagnostics, internal link analysis, log analysis, and controlled faceted-navigation management. The goal is not to remove all facets—it is to make sure crawlers spend time on URLs that can actually rank. Faceted navigation can create huge numbers of low-value URL combinations, causing crawl waste, duplicate content, and diluted internal signals. Shopify Search Engine Land

Crawl analysis and technical SEO tools

• Screaming Frog SEO Spider

• Crawl large ecommerce sites to find: - Faceted URLs being discovered
• Excessive internal links to parameters
• Duplicate titles/meta descriptions
• Canonical inconsistencies
• Orphaned category/product pages
• Useful for comparing crawl maps before and after navigation changes.
• Sitebulb

• Strong visualization features for: - Internal PageRank flow
• Crawl depth
• Link distribution
• Architecture problems
• Oncrawl

• Better suited for very large catalogs.
• Combines crawl data with log files and analytics to identify crawl waste patterns.
• DeepCrawl (now part of Lumar)

• Enterprise audits of large ecommerce inventories and complex navigation systems.

Internal linking optimization tools

• Ahrefs

Useful for:

• Finding pages with too few internal links
• Identifying over-linked faceted URLs
• Reviewing internal backlinks to filter pages
• Finding category pages that need stronger link support
• Semrush

Useful for:

• Internal linking audits
• Topic/category hierarchy checks
• Identifying pages receiving weak internal authority
• Sitebulb

Particularly useful for visualizing whether mega menus and filters are overwhelming important pages with unnecessary links.

Server log analysis tools

These show what Googlebot is actually crawling, which is often more valuable than a simulated crawl.

• Screaming Frog Log File Analyser
• Oncrawl
• Botify

Look for:

• Googlebot repeatedly hitting URLs like: - ?color=
• ?size=
• ?sort=
• ?filter=
• combinations of multiple parameters
• Important product/category URLs receiving fewer crawls than low-value filters

Log analysis is often the fastest way to prove crawl waste because it shows real crawler behavior rather than just possible problems. Search Engine Land

Faceted navigation management tools/features

Google Search Console

Use it to monitor:

• Indexed URLs that should not exist
• Excluded pages
• Crawl statistics
• URL patterns appearing in Google

Platform-specific controls

Many ecommerce platforms provide controls for:

• Which filters generate URLs
• Which filters are indexable
• Whether filters use AJAX instead of crawlable links

Examples:

• Shopify — often requires theme/app-level control because default filtering behavior may expose many combinations.
• Adobe Commerce — provides more granular catalog/navigation controls.

Automation and custom analysis

For large catalogs, teams often build custom scripts to analyze:

• URL parameter combinations
• Crawl frequency
• Conversion value by facet
• Internal link counts
• Indexation status

Common data sources:

• Google Search Console API
• Server logs
• XML sitemaps
• Crawl exports from Screaming Frog/Oncrawl/Botify

A practical workflow

1. Crawl the site

• Identify all parameter URLs and internal links.
• Analyze logs

• Find which URLs Googlebot actually spends time on.
• Classify facets

• Keep indexable: - High search demand
• Unique inventory
• Conversion-driving combinations
• Restrict: - Sort orders
• Empty results
• Thin combinations
• Near-duplicate filters
• Reduce internal links

• Do not expose every possible facet combination through crawlable <a href> links.
• Surface only valuable facet landing pages.
• Apply controls

• Depending on the case: - Canonical tags
• noindex
• robots.txt rules
• AJAX/client-side filtering
• URL normalization

A common mistake is using noindex alone: it can remove pages from search results, but crawlers may still spend resources discovering those URLs. Preventing unnecessary URLs from being generated or heavily internally linked is usually the stronger fix. Shopify Ahrefs

For an enterprise ecommerce site (100k+ URLs), I’d typically pair Screaming Frog or Oncrawl + log analysis + Search Console + internal link visualization rather than rely on a single SEO crawler.

For ecommerce sites, the best tools are usually a combination of crawl auditors, log analyzers, internal link analyzers, and indexation monitors. The goal is to identify where crawlers are wasting time (filter URLs, duplicate paths, orphan pages, unnecessary pagination) and then redesign internal linking so authority flows toward valuable category/product pages. Faceted navigation can create huge numbers of URL combinations, so controlling which ones are crawlable and indexable is a core technical SEO task. Shopify Search Engine Land

Crawl analysis & faceted navigation tools

screamingfrog.co.uk

Best for:

• Finding crawl traps caused by faceted URLs
• Auditing internal links and click depth
• Identifying duplicate titles, canonicals, thin pages, and indexability issues
• Segmenting URLs by parameters (e.g., ?color=, ?size=, ?sort=)

Useful workflow:

1. Crawl the site with JavaScript rendering enabled.
2. Export URLs containing parameters.
3. Group by facet type.
4. Compare crawl frequency vs. organic value.

sitebulb.com

Best for:

• Visual crawl maps
• Internal linking audits
• Finding pages that are too deep
• Explaining technical issues to stakeholders

Good for ecommerce teams that need prioritized recommendations rather than raw crawl exports.

oncrawl.com

Best for:

• Large ecommerce sites (hundreds of thousands or millions of URLs)
• Combining crawl data with server logs and analytics
• Measuring whether Googlebot is spending time on low-value URLs

Server log analysis tools

screamingfrog.co.uk

Best for answering:

• "Is Googlebot actually crawling my filter URLs?"
• "Which sections consume crawl budget?"
• "Are important category pages being ignored?"

Log analysis is especially useful because a crawler shows what can happen, while logs show what search engines are actually doing. Search Engine Land

botify.com

Best for enterprise ecommerce:

• Crawl budget monitoring
• Googlebot behavior analysis
• Linking crawl data to organic performance

Internal linking optimization tools

ahrefs.com

Useful for:

• Finding orphan pages
• Reviewing internal backlinks
• Identifying pages with high authority that could link to key categories
• Finding underlinked products/categories

semrush.com

Useful for:

• Internal linking audits
• Keyword-to-page mapping
• Finding pages that rank but lack internal support

search.google.com

Essential for:

• Checking indexed URL patterns
• Finding excluded pages
• Reviewing crawl/index coverage issues
• Comparing sitemap URLs vs. indexed URLs

Enterprise ecommerce SEO platforms

lumar.io

Good for:

• Large technical SEO audits
• Template-level issues
• Governance across big ecommerce platforms

searchmetrics.com

Useful for:

• Enterprise SEO monitoring
• Content/category performance analysis

Tools for managing URL parameters & crawl waste

search.google.com

Use it to monitor:

• Indexed facet URLs
• Excluded duplicates
• Crawl anomalies

analytics.google.com

Use it to determine:

• Which filtered pages actually generate revenue
• Which facets users rely on
• Which URLs should potentially become SEO landing pages

Practical tool stack by site size

Metrics I would monitor

• % of Googlebot requests hitting faceted URLs
• Number of indexed parameter URLs
• Crawl depth of money pages
• Orphan product/category count
• Internal links pointing to filters vs. canonical categories
• Organic traffic from facet pages
• Number of URLs discovered vs. URLs intended for indexing

A common optimization pattern is:

• Keep filters for users.
• Allow indexing only for high-demand facet combinations.
• Prevent internal links from exposing thousands of low-value variants.
• Build dedicated SEO landing pages for valuable combinations (for example, "black leather boots") rather than letting every filter combination compete. Shopify Search Engine Land

For ecommerce sites, the best tools are usually a combination of crawl auditing, log analysis, internal link analysis, and URL/facet monitoring. Faceted navigation can create huge numbers of near-duplicate URLs, dilute internal link equity, and consume crawl resources if filter combinations are exposed indiscriminately. Shopify Search Engine Land

Crawl auditing tools

• Screaming Frog — one of the most commonly used tools for large ecommerce audits.

• Finds: - crawlable faceted URLs
• parameter explosions
• duplicate titles/meta descriptions
• orphan pages
• excessive internal links
• deep click paths
• Useful reports: - Crawl Path Analysis
• Internal Links
• URL Parameters
• Duplicate Content
• Sitebulb — strong for visualizing site architecture.

• Helpful for: - internal link graphs
• identifying category pages buried too deeply
• spotting crawl traps
• Ahrefs Site Audit — useful for combining crawl data with backlink and ranking data.

• Helps identify: - index bloat patterns
• internal link issues
• competing URLs
• faceted pages receiving external links

Internal linking optimization tools

• Link Whisper

• Better suited for content-heavy ecommerce sites.
• Helps find: - orphan pages
• missing contextual links
• weak anchor text patterns
• InLinks

• Useful when you want to build stronger semantic connections between: - category pages
• buying guides
• product collections
• Custom crawlers (Python, SQL, BigQuery, etc.)

• Often the best option for large catalogs.
• Typical analyses: - "Which category pages receive fewer than X internal links?"
• "Which products are more than 4 clicks from the homepage?"
• "Which facet URLs receive internal links but no organic traffic?"

Log file analysis tools

Crawlers show what can be crawled; logs show what search engines actually crawl.

• Screaming Frog Log File Analyser
• Botify
• Oncrawl

Look for:

• Googlebot spending large percentages of requests on: - ?color=
• ?size=
• ?sort=
• session URLs
• filter combinations
• Important category/product URLs receiving little crawl activity

Log analysis is especially valuable because faceted navigation problems often appear as wasted crawler activity rather than obvious ranking errors. Search Engine Land

Google tools

• Google Search Console - Use: - Page indexing reports
• Crawl stats
• URL inspection
• Search performance by page type

Useful questions:

• Are thousands of filter URLs discovered but not indexed?
• Are category pages losing impressions?
• Are parameter URLs appearing in search?

Enterprise ecommerce SEO platforms

For very large catalogs:

• Botify

• Crawl budget forecasting
• Log analysis
• URL segmentation
• Lumar

• Technical audits
• JavaScript crawling
• large-scale issue monitoring
• Semrush

• Broader SEO monitoring: - rankings
• site audits
• competitors
• internal linking suggestions

Tools specifically for faceted navigation decisions

A good workflow is:

1. Crawl the site

• Find every URL pattern generated by filters.
• Classify facets

• Keep indexable: - high-search-demand combinations
• commercially valuable landing pages
• Control: - random combinations
• sorting URLs
• low-value filters
• Measure demand

Use:

• Search Console queries
• keyword tools
• conversion data
• Implement controls

Common approaches:

• prevent unnecessary facet URLs from being linked internally
• use canonicalization where appropriate
• use robots rules carefully for crawl prevention
• use client-side filtering when URLs do not need to exist for search Shopify Ahrefs

A practical stack by site size

For most ecommerce teams, the highest-impact combination is Screaming Frog + Google Search Console + server log analysis because it reveals both the potential crawl waste and the actual Googlebot behavior. Ahrefs Search Engine Land

For ecommerce, I’d use a crawler + log-analysis + SEO platform rather than relying on one tool. Faceted navigation is a major source of overcrawling because every filter combination can create another discoverable URL. Google Developers Google Developers

Best tools

• Screaming Frog — Probably the best hands-on tool for auditing internal links and faceted URLs. Crawl the site, then segment URLs by parameters, canonical/noindex status, response code, crawl depth, and inlinks. Its custom extraction and JavaScript crawling are particularly useful for modern ecommerce sites.
• Sitebulb — Excellent for visualizing site architecture and finding low-internal-link pages, orphan pages, excessive URL growth, duplicate content, and problematic facets. Sitebulb specifically recommends scheduled crawls to monitor facet-generated URL explosions. Sitebulb
• Ahrefs — Useful for combining crawling with keyword/backlink data. Its Site Audit can expose huge ratios of non-indexable-to-indexable URLs, while Internal Backlinks can help identify internal links pointing to unwanted facet URLs. Ahrefs
• Semrush — Good for large-scale technical auditing plus identifying which category/facet combinations actually have organic search demand. This helps distinguish SEO-worthy landing pages from filters that should remain crawl-controlled.
• Google Search Console — Essential for validating what Google actually discovers, crawls, and indexes. Pair it with your crawler rather than treating it as the sole diagnostic tool.
• Server log analysis tools — For large ecommerce sites, this is arguably the most important addition. Tools such as Screaming Frog Log File Analyser let you see what Googlebot is actually requesting, rather than merely what your site makes crawlable.

A useful workflow

I'd structure the audit like this:

1. Crawl the site with Screaming Frog or Sitebulb.
2. Group URLs by facet/parameter, e.g. ?color=, ?size=, ?brand=, ?sort=, etc.
3. Calculate the ratio of: - indexable URLs
4. canonicalized URLs
5. noindex URLs
6. blocked URLs
7. useless/empty combinations
8. Identify facets receiving internal links but having little/no organic value.
9. Use Ahrefs/Semrush keyword data to determine which facet combinations deserve dedicated, indexable landing pages.
10. Use log files to determine whether Googlebot is actually spending significant crawl activity on the unwanted combinations.
11. Restructure internal linking so your important hierarchy is explicit: home → category → subcategory/facet landing page → products. Google explicitly says internal link relationships help it understand ecommerce site structure and relative importance. Google Developers
12. Keep low-value filter combinations from becoming crawl paths in the first place.

The key distinction

Don't simply try to "block all facets."

A better architecture is usually:

Indexable/search-worthy facets
→ clean URL
→ crawlable internal links
→ self-canonical
→ useful unique content
→ XML sitemap where appropriate

UX-only filters
→ don't create crawlable <a href> paths where possible
→ control crawling/indexation
→ avoid generating combinatorial URL permutations

Google's current guidance explicitly recommends preventing crawling of faceted URLs you don't need in search, while recognizing that potentially useful facet URLs can be optimized instead. Google Developers

If I were choosing a stack:
Screaming Frog + Google Search Console + server logs for the technical crawl problem, then Ahrefs or Semrush to decide which facets are actually worth keeping for SEO.

If you tell me the platform (Shopify, Magento/Adobe Commerce, WooCommerce, Salesforce Commerce Cloud, custom, etc.) and approximate URL count, I can give you a specific faceted-navigation audit setup and the exact reports/filters I'd use.

For ecommerce SEO, I’d split the tooling into crawl diagnostics, internal-link analysis, and faceted-navigation controls.

• screamingfrog.co.uk — Excellent for crawling large catalogs, finding orphan pages, excessive crawl depth, duplicate/faceted URLs, parameter combinations, canonicals, and internal-link patterns. Its custom extraction and JavaScript crawling are particularly useful for ecommerce.
• sitebulb.com — Strong visualization of crawl paths and link architecture. Useful for identifying category pages that are poorly connected and faceted URLs consuming crawl budget.
• search.google.com — Essential for comparing your intended architecture with what Google actually crawls/indexes. Look at indexing, crawl stats, and URL patterns.
• ahrefs.com — Useful for internal-link audits plus discovering which category/product pages have external authority that could be better distributed through internal links.
• semrush.com — Site Audit can surface internal-linking problems, crawlability issues, redirects, canonicals, duplicate URLs, and indexation problems.
• oncrawl.com — Particularly good for large ecommerce sites. It combines crawl data with log-file analysis so you can distinguish URLs that merely exist from URLs Googlebot is actually spending crawl resources on.
• botify.com — Enterprise-oriented alternative, especially valuable when you have millions of URLs and need log analysis, crawl-budget modeling, and large-scale URL segmentation.
• lumar.io — Enterprise technical SEO crawling with strong site-architecture and indexability analysis.

For faceted navigation specifically

The tool matters less than being able to segment URLs by parameter/facet combination. I'd want a crawler/log-analysis setup that lets you answer:

1. Which facet combinations are being discovered?
2. Which combinations are actually being crawled?
3. Which are indexed?
4. Which receive organic traffic?
5. Which generate internal links?
6. How many near-duplicate URLs are created?
7. Are valuable filtered category pages being accidentally blocked or canonicalized away?

For a large store, Screaming Frog + Google Search Console is a good starting point. If crawl waste is substantial, add Oncrawl or Botify with server-log analysis.

The architecture I’d aim for

A common pattern is:

Category → subcategory → product

while treating most faceted combinations as non-indexable discovery URLs, unless a particular facet combination has genuine search demand and deserves to become a landing page.

For example, instead of allowing:

/shoes?brand=nike&color=black&size=10&material=leather&sort=price

to generate another crawlable/indexable URL, you might deliberately expose only high-value combinations such as:

/shoes/nike/black-leather/

Then give those strategic landing pages prominent internal links, while preventing the enormous number of low-value combinations from becoming part of the site's crawlable architecture.

If you tell me roughly how large the ecommerce site is (e.g. 50k, 1M, or 100M URLs) and what platform it uses (Shopify, Magento, Salesforce Commerce Cloud, custom, etc.), I can recommend a specific tool stack and a faceted-navigation architecture.

For ecommerce SEO, I’d split the tooling into crawl diagnostics, internal-link optimization, and faceted-navigation control.

Best tools by job

• Screaming Frog SEO Spider — Excellent for crawling large sites, finding orphan pages, analyzing crawl depth, detecting excessive parameter URLs, and modeling internal-link structures. Particularly useful for comparing “crawlable” vs. “indexable” faceted URLs.
• Sitebulb — Strong visualization and prioritization. Its crawl maps and internal-link reports make it easier to identify category/product pages that are buried too deeply or receive weak internal signals.
• Google Search Console — Useful for validating whether Google is actually discovering/crawling parameterized URLs and for monitoring indexing patterns after navigation changes.
• Google Analytics / server logs — Log-file analysis is especially valuable for crawl waste because it shows what Googlebot actually crawls, rather than merely what your site exposes.
• Botify — Enterprise-focused crawl-budget and log analysis. Good for very large catalogs where you need to quantify Googlebot activity across faceted URLs, templates, categories, and products.
• Oncrawl — Combines crawl and log data and is useful for correlating internal linking, crawl behavior, indexability, and organic performance.
• Lumar (formerly Deepcrawl) — Enterprise technical SEO crawling with large-scale monitoring and segmentation, useful for detecting faceted-navigation and internal-linking problems over time.
• Ahrefs / Semrush — More useful for identifying important pages, competing internal-link structures, backlinks, and keyword/page relationships than for directly managing crawl budget.

For faceted navigation specifically

The important thing is not simply “block facets.” You want to determine which combinations deserve to be discoverable/indexable and which create effectively infinite URL combinations.

A typical strategy is:

1. Identify high-value facets — e.g. brand, category, or a meaningful product attribute with real search demand.
2. Create controlled landing pages for combinations worth indexing.
3. Noindex or otherwise control low-value combinations where appropriate.
4. Prevent internal links from exploding into every possible parameter combination.
5. Canonicalize carefully rather than using canonicals as the only crawl-control mechanism.
6. Use robots.txt selectively when the goal is specifically to prevent crawling—not merely to prevent indexing.
7. Analyze server logs afterward to confirm that Googlebot's crawl behavior actually changed.

For a large ecommerce site, my preferred stack would be Screaming Frog or Lumar + server-log analysis + Google Search Console. For enterprise-scale sites, Botify or Oncrawl becomes particularly compelling because you can connect crawl behavior with actual bot activity.

If you tell me your platform (Shopify, Magento, WooCommerce, custom, etc.) and approximate number of URLs/products, I can suggest a specific faceted-navigation + internal-linking architecture and the tools I'd use to audit it.

For ecommerce, I’d use a combination of a crawler + log analysis + Search Console, because internal-link structure tells you what can be crawled, while logs tell you what Googlebot actually crawls.

Best tools

• Screaming Frog SEO Spider — Excellent for diagnosing internal-link architecture. Use it to find orphan/low-inlink products, excessive crawl depth, links to redirects/noindex pages, parameter URLs, pagination problems, and facet-generated URLs. It’s particularly good when you want granular control over crawl settings. Shopify specifically recommends it for ecommerce crawl audits. Shopify Sitebulb
• Sitebulb — Probably my choice for visualizing internal-link architecture. It identifies orphaned pages, broken internal links, links to redirects, link counts, link locations, anchor text, and deep pages. Its cloud version can crawl up to 10 million URLs. Sitebulb Sitebulb
• JetOctopus — One of the strongest choices for crawl-budget optimization on large ecommerce sites because it combines crawling, server logs, and Google Search Console. You can see which facet/parameter URLs Googlebot actually visits versus which valuable products/categories aren't getting crawled. It also has an internal-linking module for finding weak/orphan pages. Tech SEO for Google & AI Tech SEO for Google & AI
• Botify / OnCrawl — Worth considering at very large enterprise scale, particularly when log-file analysis is central to the project. Current industry comparisons generally position these alongside JetOctopus for large-site crawl-budget work. CrawlBudget
• Google Search Console — Essential companion rather than a replacement for a crawler. Compare indexed URLs and discovered URLs against the URLs you actually want indexed, and investigate parameter/facet URL patterns appearing in coverage. Shopify

For faceted navigation specifically

The most useful workflow is:

1. Crawl the site
Identify every URL pattern generated by filters such as:

/shoes?brand=nike&size=10&color=red

/shoes?brand=nike&size=10&color=red&sort=price

Then classify facets into:

• SEO-worthy combinations → potentially indexable/linked landing pages
• Useful to users but not SEO → generally prevent unnecessary crawling/indexation
• Combinatorial/duplicate facets → aggressively control

2. Analyze server logs

This is where JetOctopus, Botify, or OnCrawl become particularly valuable. You can quantify something like:

18% of Googlebot requests → facet URLs
7% → sorting URLs
3% → internal search
72% → products/categories

That is much more actionable than simply knowing those URLs exist.

3. Analyze internal links

Use Sitebulb/Screaming Frog/JetOctopus to answer:

• Which products have <1–2 meaningful internal links?
• Which categories are >3–4 clicks deep?
• Are category pages linking to valuable products?
• Are navigation links creating thousands of facet combinations?
• Are you internally linking to URLs that are canonicalized, noindexed, or redirected?
• Are important product/category pages competing for internal-link prominence with low-value URLs?

Sitebulb is particularly good here because it exposes where links occur and how strongly pages are connected. Sitebulb

My recommended stack

For a large ecommerce site specifically concerned with faceted navigation and crawl waste, I'd lean toward JetOctopus + Screaming Frog/Sitebulb: JetOctopus tells you where Google's crawl budget is actually going, while the crawler gives you the structural detail needed to redesign internal linking. JetOctopus also now has an AI internal-linking module designed around weak/orphan pages and stronger "donor" pages. Tech SEO for Google & AI

The important distinction is that don't try to solve crawl waste purely by adding noindex. The bigger architectural question is which URLs your internal links allow bots to discover in the first place, and whether those links lead them toward commercially important categories/products or into an enormous facet URL graph. Shopify's current ecommerce audit guidance similarly recommends checking filtered/parameterized URLs and ensuring priority products and collections are reachable through standard HTML links. Shopify

If you tell me roughly how many products/URLs the site has and what platform it's on (Shopify, Magento, Salesforce Commerce Cloud, custom, etc.), I can give you a concrete faceted-navigation + internal-linking audit workflow and the exact reports I'd pull from these tools.

For ecommerce sites, optimizing internal linking and faceted navigation to reduce crawl waste usually requires a mix of SEO crawlers, log analysis, Google tools, and technical SEO platforms. Here are the most useful tools by job:

1. Crawl analysis tools (find crawl waste and internal linking issues)

Screaming Frog SEO Spider

Best for: deep site audits and faceted navigation discovery.

Useful for:

• Finding crawlable parameter URLs (?color=red&size=large)
• Measuring crawl depth and internal link equity
• Identifying orphan pages
• Auditing canonical tags, noindex, robots directives
• Visualizing internal linking structures
• Exporting URL inventories for facet cleanup

Good workflows:

• Crawl with JavaScript rendering enabled for modern ecommerce sites
• Crawl URL parameters separately to see index bloat
• Compare “all discovered URLs” vs “indexable URLs”

Sitebulb

Best for: visual SEO auditing.

Useful for:

• Internal link graphs
• Page importance scoring
• Identifying over-linked filter pages
• Finding pages buried too deep
• Crawl budget reports

Oncrawl

Best for: large ecommerce sites.

Useful for:

• Combining crawl data with server logs
• Understanding Googlebot behavior
• Segmenting URLs by template, category, or parameter
• Measuring crawl efficiency

2. Server log analysis (the most accurate crawl waste signal)

Screaming Frog Log File Analyser

Best for:

• Seeing what Googlebot actually crawls
• Finding wasted crawl requests
• Detecting excessive crawling of filters, sorting URLs, and duplicate pages

Look for:

• High Googlebot activity on non-indexable facet URLs
• Crawl frequency on discontinued products
• Bot hits to session IDs or tracking parameters

Botify

Best for enterprise ecommerce.

Useful for:

• Crawl budget optimization
• Googlebot behavior modeling
• Linking crawl demand to SEO value
• Prioritizing technical fixes

3. Google tools (free essentials)

search.google.com

Use for:

• Checking indexed URL patterns
• Reviewing crawl stats
• Finding duplicate/canonical issues
• Monitoring excluded pages

Useful reports:

• Pages → “Crawled - currently not indexed”
• Pages → “Duplicate without user-selected canonical”
• Settings → Crawl stats

analytics.google.com

Useful for:

• Finding internal links that drive conversions
• Identifying valuable category/filter combinations
• Avoiding removal of navigation paths that users actually need

4. Internal linking optimization platforms

Ahrefs

Useful for:

• Internal link auditing
• Identifying pages with weak internal authority
• Finding important pages receiving few links
• Reviewing “Best by links” reports

Semrush

Useful for:

• Site Audit internal linking checks
• Crawlability issues
• Broken internal links
• Orphan page detection

InLinks

Useful for:

• Entity-based internal linking suggestions
• Content hubs
• Semantic relationships between pages

5. Faceted navigation testing and URL control

Google Search Console URL inspection

Useful for validating:

• Whether canonicalization works
• Whether Google sees a facet URL as indexable
• Whether robots/noindex changes are being processed

JetOctopus

Useful for:

• Large ecommerce crawls
• Parameter URL analysis
• Log analysis
• Crawl budget visualization

Lumar

Useful for:

• Enterprise-scale technical SEO
• Faceted navigation audits
• Finding infinite URL spaces

6. Tools for URL parameter discovery

For ecommerce facets, you often need to identify combinations like:

/shoes?color=black
/shoes?color=black&size=10
/shoes?sort=price
/shoes?brand=nike&material=leather

Helpful tools:

• Screaming Frog (URL extraction + crawl analysis)
• Sitebulb (crawl maps)
• Log analyzers (actual Googlebot behavior)
• BigQuery + exported server logs (large sites)

Recommended workflow for reducing crawl waste

1. Inventory URLs

• Crawl the site with Screaming Frog/Sitebulb
• Export all parameter URLs
• Compare discovered vs indexed

• Use Search Console
• Identify URL patterns creating index bloat
• Analyze Googlebot behavior

• Use log analysis
• Find where crawl budget is spent
• Classify facets

• Keep indexable: - High-demand category combinations
• Pages with search demand and unique value
• Control: - Sorting
• Session parameters
• Near-duplicate filters
• Infinite combinations
• Improve internal linking

• Link strongly to: - Category pages
• High-margin collections
• Search-demand landing pages
• Avoid automatically linking every possible facet combination

For large ecommerce sites (100k+ URLs), a strong stack is usually:

• Screaming Frog or Sitebulb → crawl diagnostics
• Oncrawl/Botify/JetOctopus → logs + crawl budget
• Google Search Console → indexing validation
• Ahrefs/Semrush → internal authority analysis

For smaller stores, Screaming Frog + Search Console is often enough.

For large ecommerce sites (100k+ URLs), a strong stack is usually:

For ecommerce SEO, the most useful tools fall into a few buckets:

• Crawlers: Screaming Frog, Sitebulb, and Lumar — map internal links, crawl depth, orphan pages, parameter URLs, canonicalization, and crawl traps.
• Google Search Console: Useful for spotting indexing/crawling patterns and validating whether changes to faceted URLs improve the situation.
• Log-file analysis: Botify, Oncrawl, and Splunk/ELK-style log analysis can show what Googlebot actually crawls, which is particularly valuable for identifying crawl budget wasted on filters, sort parameters, pagination, and duplicate URLs.
• Internal-link analysis: Screaming Frog and Sitebulb can calculate internal-link counts, PageRank-style internal link equity, click depth, and identify important category/product pages that are poorly linked.
• Faceted-navigation auditing: Screaming Frog's custom extraction/JavaScript crawling, Lumar, and Botify can help discover combinations such as /shoes?color=red&size=10&brand=x and quantify how many crawlable URL permutations your architecture creates.
• Keyword/entity-driven internal linking: Tools such as Semrush and Ahrefs can help identify pages and queries that deserve stronger internal links, although they are less useful than crawlers/logs for diagnosing crawl waste itself.

A practical stack

If you're optimizing a large ecommerce site, I'd typically use:

1. Screaming Frog or Sitebulb → crawl the site and model internal-link/faceted URL structure.
2. Google Search Console → compare indexing and crawling behavior before/after changes.
3. Botify or Oncrawl → analyze server logs to distinguish URLs you've made crawlable from URLs Googlebot actually spends time crawling.
4. Ahrefs/Semrush → determine which category, product, and landing pages deserve additional internal-link authority.

The key is to separate discoverability from crawlability. You generally don't want every possible facet combination to be internally linked and crawlable. A good architecture deliberately gives strong internal links to valuable indexable category/facet pages while limiting or blocking low-value parameter combinations.

If you tell me roughly how large the ecommerce site is and what platform it uses (Shopify, Magento, Salesforce Commerce Cloud, custom, etc.), I can suggest a specific tool stack and a faceted-navigation/internal-linking audit workflow.

For ecommerce sites, I’d use a combination of a crawler + log analysis + Google’s own crawl/indexation data. The strongest options are:

Best tools

• Screaming Frog SEO Spider — probably the best hands-on tool for diagnosing internal-linking architecture. It can surface crawl depth, internal links, redirects, orphan-like patterns, and excessive URL variations. Its crawl-depth analysis is particularly useful for finding important category/product pages buried too deeply. Screaming Frog

screamingfrog.co.uk

• Sitebulb — my top choice if faceted navigation is the main problem. It lets you exclude facet URL patterns or even entire filter-navigation elements from the audit, while still identifying how those links affect architecture. It also reports internal-link distribution, crawl depth, URL Rank, orphan pages, and anchor text. Sitebulb Support Sitebulb Support

sitebulb.com

• Ahrefs Site Audit — useful for large-scale internal-link analysis, broken/redirecting links, indexability problems, and internal-link opportunities. Its Page Explorer and Link Explorer provide extensive filtering for finding specific URL/link patterns. Ahrefs Help Center Ahrefs

ahrefs.com

• Google Search Console — essential as the reality check. Use it alongside crawlers to compare your intended architecture with what Google is actually discovering, indexing, and crawling. In particular, look at Page indexing, Crawl Stats, and URL Inspection.
• Server log analyzers — arguably the most valuable addition for a very large ecommerce site. Logs tell you what Googlebot actually requests, rather than what a third-party crawler thinks it might request. This is especially useful for quantifying Googlebot activity on parameter/facet URLs versus valuable category and product URLs. Sitebulb also recommends log analysis for identifying repeated crawling of low-value URLs. Sitebulb

A practical stack

For a large ecommerce site, I'd use:

Screaming Frog/Sitebulb → Google Search Console → server logs

Then build a URL inventory like:

The key is not simply "block all facets." Some facets can be valuable landing pages. Sitebulb's faceted-navigation guidance recommends distinguishing indexable facets from those that should be excluded, and checking canonical/indexation directives, sitemaps, and internal links accordingly. Sitebulb

What I'd specifically measure

For internal linking:

• Crawl depth of category/product pages
• Internal followed links per important URL
• Orphan pages
• Pages receiving links from low-value/noindex URLs
• Redirecting internal links
• Anchor-text distribution
• Number of clicks/hops from important commercial pages

For faceted navigation:

• Number of parameterized URLs discovered
• Facet combinations per category
• % of Googlebot requests going to facets
• Indexable vs. non-indexable facet URLs
• Duplicate/canonicalized facet URLs
• Facets linked in navigation
• Dead/empty facet combinations
• Crawl depth generated by facet combinations

If your site has millions of URLs, I'd put server-log analysis at the top of the priority list. A crawler tells you how your architecture could be crawled; logs tell you where Googlebot is actually spending its crawl activity.

• Crawl depth of category/product pages
• Internal followed links per important URL
• Orphan pages
• Pages receiving links from low-value/noindex URLs
• Redirecting internal links
• Anchor-text distribution
• Number of clicks/hops from important commercial pages

For faceted navigation:

For ecommerce SEO, I’d split the tooling into crawl diagnostics, internal-link analysis, and faceted-navigation control.

• Screaming Frog SEO Spider — excellent for crawling large ecommerce sites, identifying orphan pages, excessive crawl depth, parameterized URLs, duplicate/near-duplicate pages, canonicalization issues, and discovering where internal links point. Its custom extraction and JavaScript crawling are particularly useful for faceted navigation.
• Sitebulb — strong visualizations for crawl paths, internal-link equity, indexability, and crawl-depth analysis. Useful for figuring out whether filters are creating huge URL combinations or whether important category/product pages are underlinked.
• Google Search Console — use Crawl Stats and indexing reports to see how Google is actually encountering your site. It won't give you a complete faceted-URL inventory, but it's valuable for validating whether changes reduce unnecessary crawling.
• Google Analytics / log-file analysis tools — server logs are especially valuable because they show Googlebot's actual URL requests, rather than merely simulating a crawl. For serious crawl-budget work, tools such as Screaming Frog Log File Analyser can reveal which parameters Googlebot spends time crawling.
• Ahrefs Site Audit — useful for large-scale internal-link and crawlability audits, including broken links, redirect chains, canonical issues, orphan pages, and internal-link opportunities.
• Semrush Site Audit — similar role, with useful internal-linking reports and crawlability/indexability diagnostics.
• Oncrawl — particularly strong for enterprise ecommerce. It combines crawling with log-file data and can segment crawl behavior by URL type, directory, parameter, bot, etc. That's valuable for quantifying faceted-navigation crawl waste.
• Lumar (formerly Deepcrawl) — enterprise-oriented crawling and technical SEO analysis, useful for very large ecommerce sites and complex URL architectures.
• JetOctopus — another strong option for large sites because it combines crawler data with server-log analysis and can visualize crawl behavior at scale.

For faceted navigation specifically

The most useful stack is usually:

Crawler + log analyzer + Search Console

For example:

1. Crawl the site with Screaming Frog/Sitebulb.
2. Export URLs containing parameters such as ?color=, ?size=, ?brand=, ?sort=, etc.
3. Classify facets into: - Indexable landing pages — e.g. /shoes/running/
4. Potential SEO pages — e.g. /shoes/running/nike/
5. Combinatorial/filter URLs — e.g. ?color=red&size=10&sort=price
6. Compare those URL classes against actual Googlebot requests in server logs.
7. Identify parameter combinations consuming crawl activity without generating useful search traffic.
8. Restrict those combinations through your site's architecture, canonicalization, noindex where appropriate, and/or robots directives—being careful because robots.txt can prevent Google from seeing canonical/noindex signals.
9. Strengthen internal links to the relatively small set of facet/category pages you actually want crawled and indexed.

One important distinction: don't try to solve every faceted-navigation problem with robots.txt. If Google can't crawl a URL, it also can't reliably discover the canonical or noindex directive on that page. The right solution depends on whether the goal is prevent indexing, prevent crawling, or consolidate signals.

If you tell me your ecommerce platform (Shopify, Magento/Adobe Commerce, Salesforce Commerce Cloud, custom, etc.) and approximate URL count, I can recommend a specific tool stack and faceted-navigation architecture.

Yes. For ecommerce, I’d use a crawler + log analyzer + Google Search Console, because internal-link problems and faceted-navigation crawl waste are related but not identical.

Best tools

• Screaming Frog SEO Spider — Best all-around crawler for auditing internal linking. It can expose crawl depth, inlinks, link position, orphan/poorly linked pages, redirects, and parameterized URLs. Its crawl-depth analysis is particularly useful for finding important products/categories buried too deeply. Screaming Frog Screaming Frog

screamingfrog.co.uk

• Sitebulb — Excellent for visualizing site architecture and internal-link distribution. It can identify low/zero internal links, crawl depth, anchor-text problems, and links from different site sections. Crucially for ecommerce, you can exclude faceted URLs from the audit crawl so your analysis isn't overwhelmed by millions of filter combinations. Sitebulb Support Sitebulb Support

sitebulb.com

• JetOctopus — Probably the most interesting option for large ecommerce sites. It combines crawling with server-log and GSC data, so you can see not merely which URLs exist, but which URLs Googlebot is actually spending resources crawling. Its Internal Linker is designed specifically around finding weakly linked pages and creating links between stronger "donor" pages and weaker "acceptor" pages. JetOctopus JetOctopus

jetoctopus.com

• Google Search Console — Essential as the reality check. Use it alongside your crawler to compare your intended architecture against Google's crawling/indexing behavior. It's especially useful for spotting indexing problems after changing facet rules, canonicals, or internal links. Google's own faceted-navigation documentation is worth following here. Google for Developers

search.google.com

What I'd use for each problem

A particularly useful workflow

For a large ecommerce site, I'd do this:

1. Crawl the site with Screaming Frog or Sitebulb and classify URLs into products, categories, brands, pagination, facets, search, etc.
2. Calculate inlinks + crawl depth for commercially important URLs.
3. Identify facet combinations producing huge URL expansion—e.g. ?color=red&size=10&brand=x.
4. Pull server logs with JetOctopus and determine whether Googlebot is actually wasting crawl activity on those combinations.
5. Separate facets into: - SEO-worthy facets → allow crawling/indexing and give them intentional internal links.
6. User-only facets → prevent unnecessary crawl paths.
7. Duplicate/low-value combinations → consolidate or otherwise restrict them.
8. Strengthen internal links to revenue-driving category/product pages while reducing links that expose enormous numbers of low-value parameter URLs.
9. Re-crawl and compare crawl depth, inlinks, URL discovery and bot activity before/after.

One important nuance: don't simply block every faceted URL in robots.txt. Some facets can be legitimate search landing pages. Google's current guidance discusses controlling faceted-navigation crawling and notes approaches such as appropriate URL design and, where applicable, nofollow/canonical signals. Google for Developers

If I had to pick one stack:
Screaming Frog + Google Search Console for a small/medium store; JetOctopus + GSC + Screaming Frog/Sitebulb for a large catalog (100k+ URLs). JetOctopus becomes particularly valuable once you need to answer "What is Googlebot actually spending its crawl on?" rather than just "What URLs can my crawler discover?" JetOctopus JetOctopus

For ecommerce, I’d use a combination of crawler + log-analysis + internal-linking tools rather than one SEO platform. The key is to identify which URL patterns are generating crawl demand, then control which facets are actually discoverable.

Best tools

• screamingfrog.co.uk — Best hands-on crawler for diagnosing internal-link architecture. It can surface crawl depth, orphan/poorly linked pages, excessive internal links, redirects, canonicalization problems, and JavaScript-generated links. It’s particularly useful for crawling parameterized/faceted URLs and comparing raw HTML vs rendered links. Screaming Frog Screaming Frog
• botify.com — Best choice for large ecommerce/enterprise sites. Its SiteCrawler and log/crawl data can show Googlebot's actual crawl behavior, internal inlinks/outlinks, PageRank distribution, crawl waste, and problematic URL segments. Its SmartLink product can also recommend internal-link improvements using crawl, Search Console, analytics, and server-log data. Botify Knowledge Base Botify Knowledge Base Botify
• semrush.com — Good all-around option for finding deep pages, orphan pages, internal-link issues, redirects, and crawlability problems. Its Crawled Pages report exposes crawl depth, incoming/outgoing internal links, canonicalization and Internal Link Rank. Semrush Semrush
• lumar.io — Strong enterprise technical-SEO platform for crawling large sites and analyzing faceted-navigation problems, especially when you need repeatable technical audits and monitoring. Lumar
• Google Search Console + server logs — Not a dedicated optimization tool, but essential for validating whether your theoretical crawl strategy matches what Googlebot actually does. For serious crawl-budget work, I'd prioritize log data over crawler estimates.

For faceted navigation specifically

Google's current guidance is particularly relevant here: ecommerce filters can create an enormous URL space, and crawling those combinations can consume substantial computing resources. If a facet URL doesn't need to appear in Google, Google recommends preventing its crawling; if it does need to be indexed, the URL structure should be deliberately designed. Google for Developers

I'd use the tools to build a matrix like:

The important distinction is that robots.txt, noindex, canonicalization, and internal-link suppression solve different problems. For example, if you don't want Google discovering millions of facet combinations, don't simply leave those URLs extensively linked throughout your site's filter UI and expect canonical tags to eliminate the crawl problem.

My recommended stack

For a large ecommerce site:

Botify + Screaming Frog + Google Search Console + server logs

For a mid-sized ecommerce site:

Screaming Frog + Semrush + Search Console

And if your primary problem is "Googlebot is spending huge amounts of time crawling filter combinations", I'd start with server logs + Botify, because you want to measure actual Googlebot crawl waste, not merely count how many URLs your crawler can generate. Botify specifically provides analysis around whether crawl budget is being spent on strategic/indexable pages and whether internal linking is directing crawlers appropriately. Botify Knowledge Base Botify

One especially useful audit is: “Show me every faceted URL that Googlebot crawled in the last 30 days, its source/internal links, whether it is indexable, and how much crawl activity it received.” That dataset usually makes the biggest crawl-waste opportunities obvious.

For ecommerce sites, optimizing internal linking and faceted navigation to reduce crawl waste usually requires a mix of SEO crawlers, log analysis tools, search console data, and technical SEO platforms. Here are the most useful tools by task:

1. Crawling & internal linking analysis

screamingfrog.co.uk

Best for:

• Finding excessive internal links
• Identifying orphan pages
• Analyzing crawl depth
• Auditing faceted URL generation
• Checking canonical tags, noindex directives, pagination, and URL parameters

Useful ecommerce workflows:

• Crawl the site with JavaScript rendering enabled to see dynamically generated filters.
• Export all URLs containing parameters like ?color=, ?size=, ?brand=.
• Compare crawlable faceted URLs against your intended indexable landing pages.

sitebulb.com

Best for:

• Visualizing internal linking structure
• Finding overlinked pages
• Understanding crawl paths
• Highlighting pages that waste crawl budget

Useful reports:

• Crawl depth distribution
• Link equity flow
• Internal PageRank-style analysis

oncrawl.com

Best for:

• Large ecommerce sites
• Combining crawl data with log files
• Measuring crawl efficiency

Useful for answering:

• "Is Googlebot spending time on filter URLs?"
• "Which categories receive crawl attention?"
• "Are important products too deep?"

2. Log file analysis (critical for crawl waste)

screamingfrog.co.uk

Best for:

• Seeing actual Googlebot behavior
• Finding wasted crawling of faceted URLs
• Comparing discovered URLs vs crawled URLs

Key metrics:

• Crawl frequency by URL pattern
• Status codes hit by bots
• Parameter-heavy URLs crawled
• Crawl allocation to non-indexable pages

botify.com

Best for enterprise ecommerce:

• Crawl budget optimization
• Search engine bot behavior analysis
• Large catalog sites

Common use cases:

• Detecting millions of unnecessary filter combinations
• Prioritizing which URL patterns should be crawlable
• Measuring impact after faceted navigation changes

3. Faceted navigation and parameter discovery

search.google.com

Useful for:

• Finding indexed parameter URLs
• Checking crawl/index coverage
• Seeing "Discovered – currently not indexed" patterns
• Monitoring URL removal impact

Look at:

• Pages indexed with unexpected parameters
• Excluded pages caused by canonical/noindex conflicts
• Crawl stats

ahrefs.com

Useful for:

• Finding internal links pointing to unnecessary URLs
• Identifying keyword value of filter pages
• Auditing competing category structures

semrush.com

Useful for:

• Parameter URL discovery
• Duplicate content detection
• Internal linking audits
• Crawlability issues

4. Internal linking optimization & architecture planning

inlinks.net

Best for:

• Semantic internal linking recommendations
• Entity-based content relationships
• Improving contextual links

pageoptimizer.pro

Useful for:

• Content-focused internal linking decisions
• Supporting category pages with relevant topical signals

5. URL pattern and crawl simulation tools

regex101.com

Useful for:

• Creating rules to identify faceted URLs
• Grouping URL parameters for analysis

Example patterns:

/products?color=
/category?size=
/search?filter=

search.google.com

Useful for:

• Testing whether Google can crawl/index preferred URLs
• Confirming canonical selection

Recommended ecommerce workflow

Step 1: Discover crawl waste

Use:

• Screaming Frog
• Botify/Oncrawl (large sites)
• Server logs

Find:

• Thousands of filter combinations
• Duplicate category paths
• Search URLs being crawled
• Sort/order parameters

Step 2: Decide what deserves indexing

Create rules:

Usually index:

• High-demand category pages
• Valuable filtered landing pages (e.g., "red running shoes")
• Pages with search demand and unique content

Usually block or consolidate:

• Sort parameters (?sort=price)
• Tracking parameters (utm_*)
• Empty filters
• Multi-filter combinations with no demand

Step 3: Control discovery

Tools help you implement:

• Better internal linking hierarchy
• Fewer crawlable filter links
• Canonicalization
• noindex where appropriate
• Robots.txt rules (carefully)
• XML sitemaps containing only canonical URLs

Step 4: Validate impact

Monitor:

• Googlebot crawl distribution
• Indexed URL count
• Organic traffic to category pages
• Crawl stats in Search Console

For most ecommerce sites, the strongest combination is:

• Screaming Frog + Google Search Console for small/medium stores
• Botify or Oncrawl + log analysis + Screaming Frog for large catalogs with millions of URLs.

Monitor:

• Googlebot crawl distribution
• Indexed URL count
• Organic traffic to category pages
• Crawl stats in Search Console

For most ecommerce sites, the strongest combination is:

For ecommerce SEO, optimizing internal linking and faceted navigation to reduce crawl waste usually requires a mix of crawling tools, log analysis, SEO platforms, and technical controls. Here are the most useful tools by category:

1. Crawling & Internal Link Analysis Tools

Screaming Frog

Best for: finding internal linking issues and crawl traps

Useful features:

• Crawl your site like a search engine bot
• Identify: - orphan pages
• pages with too many/few internal links
• deep click depth
• duplicate URLs created by filters
• parameterized URLs
• indexability problems
• Visualize crawl paths and internal link distribution

Good for ecommerce tasks:

• Find category pages that need more internal links
• Detect faceted URLs being linked unintentionally
• Audit anchor text patterns

Sitebulb

Best for: visual SEO audits

Useful features:

• Internal linking graphs
• Crawl budget reports
• Duplicate content detection
• URL parameter analysis
• Page importance scoring

It can help identify whether filter URLs are consuming crawl attention compared with revenue-driving pages.

DeepCrawl (now part of Lumar)

Best for: large ecommerce sites

Useful features:

• Enterprise-scale crawling
• JavaScript rendering
• Faceted navigation monitoring
• Historical crawl comparisons
• Technical SEO alerts

Good for catalogs with hundreds of thousands or millions of URLs.

2. Google Tools (Free)

Google Search Console

Best for: seeing how Google actually interacts with your site

Useful reports:

• Crawl Stats: - Googlebot activity
• response codes
• crawl volume trends
• Indexing reports: - pages excluded
• duplicate URLs
• crawled-but-not-indexed pages
• URL Inspection: - test important category/filter URLs

Use it to validate whether faceted URLs are being crawled or indexed unnecessarily.

Google Analytics

Useful for:

• Finding high-value pages that lack internal links
• Comparing organic landing pages against crawl priority

3. Server Log Analysis Tools

These are among the best tools for crawl waste, because they show what search engines actually request.

Screaming Frog Log File Analyser

Useful for:

• Analyzing Googlebot hits
• Finding wasted crawl activity
• Identifying: - excessive filter URL crawling
• old discontinued URLs
• inefficient crawl patterns

Oncrawl

Best for:

• Combining log files + crawl data + analytics
• Crawl budget optimization
• Enterprise ecommerce analysis

Useful insights:

• "Googlebot spends X% of crawl activity on faceted URLs"
• "Category pages receive fewer crawls than low-value filters"

Botify

Best for very large ecommerce sites.

Features:

• Crawl budget monitoring
• Search engine bot behavior analysis
• Internal linking optimization
• Indexation monitoring

4. SEO Platforms With Internal Linking Recommendations

Semrush

Useful for:

• Internal linking audits
• Keyword-based linking suggestions
• Site health monitoring

Ahrefs

Useful for:

• Internal backlink analysis
• Finding pages with strong authority that can pass internal equity
• Identifying weakly linked category pages

Siteimprove

Useful for:

• Large organizations managing content quality and technical SEO

5. Ecommerce-Specific Faceted Navigation Tools & Controls

Tools can diagnose problems, but preventing crawl waste usually requires platform-level controls:

URL parameter management

Common approaches:

• Remove unnecessary parameters from internal links
• Use canonical tags carefully
• Apply noindex, follow where appropriate
• Block low-value crawling paths with robots.txt (carefully)
• Use clean URLs for valuable filtered landing pages

Platform tools

• business.adobe.com — has extensive catalog/filter configuration options
• shopify.com — often requires apps, theme adjustments, and SEO customization for advanced faceted control
• bigcommerce.com — provides built-in SEO controls and URL management options

6. Helpful Technical SEO Crawling Scripts / APIs

For advanced teams:

• Google Search Console API — monitor indexing and URL patterns
• Server log pipelines — analyze bot behavior at scale
• Custom crawlers using Python: - detect parameter combinations
• score internal links
• find crawl traps

Recommended Tool Stack by Site Size

Metrics to Watch for Faceted Navigation

A good audit should measure:

• % of Googlebot hits going to parameter URLs
• Number of crawlable faceted URLs
• Indexed pages vs actual product/category pages
• Internal links pointing to filters
• Average click depth of money pages
• Crawl frequency of important categories
• Duplicate content caused by filters
• Crawl budget spent on URLs with no organic value

For most ecommerce sites, the biggest wins come from stopping accidental internal links to infinite filter combinations, then using log analysis to confirm Googlebot behavior.

Metrics to Watch for Faceted Navigation

A good audit should measure:

For ecommerce SEO, optimizing internal linking and faceted navigation to reduce crawl waste usually requires a mix of crawl-analysis tools, log analysis, SEO platforms, and technical auditing tools. The best tools depend on whether you need to discover crawl issues, model internal link equity, or control indexation.

Crawl analysis tools

• screamingfrog.co.uk

• One of the most widely used tools for auditing ecommerce crawl paths.
• Useful for: - Finding excessive faceted URLs
• Identifying orphan pages
• Analyzing internal link depth
• Seeing which filters generate crawlable URLs
• Reviewing canonical tags, robots directives, and indexability
• sitebulb.com

• Strong visualization and reporting for complex ecommerce sites.
• Helpful for: - Crawl tree analysis
• Internal linking graphs
• Detecting inefficient navigation structures
• Communicating issues to stakeholders
• oncrawl.com

• Designed for large sites with millions of URLs.
• Useful for combining: - Crawl data
• Server logs
• Search Console data
• Internal link metrics

Enterprise SEO platforms

• botify.com

• Built specifically around crawl efficiency and large websites.
• Useful for: - Crawl budget analysis
• Googlebot behavior monitoring
• Identifying wasted crawl paths
• Prioritizing technical fixes
• lumar.io

• Enterprise crawler for large ecommerce environments.
• Useful for: - Finding indexation problems
• Monitoring faceted navigation changes
• Technical SEO governance
• semrush.com and ahrefs.com

• Better for broad SEO monitoring than deep crawl-budget modeling.
• Useful for: - Internal link issues
• Broken links
• Redirect chains
• Crawlability checks

Internal linking optimization tools

• ahrefs.com

• Helps analyze: - Internal backlinks
• Important pages receiving too few links
• Pages with excessive outbound internal links
• semrush.com

• Helps identify: - Weak internal link distribution
• Pages that need more contextual links
• inlinks.net

• Uses semantic analysis to suggest internal links and improve topical relationships.
• linkwhisper.com

• More common for content sites, but can help ecommerce blogs/category content with internal linking suggestions.

Log file analysis tools

For crawl waste, server logs are often the most accurate source because they show what search engines actually crawl.

• screamingfrog.co.uk

• Shows: - Googlebot crawl frequency
• Wasted crawls on filters
• Important URLs being ignored
• Crawl patterns after technical changes
• semrush.com

• Helps visualize bot activity and crawl behavior.

Faceted navigation testing and URL management

Browser-based crawling/testing

• search.google.com

• Essential for: - Excluded pages reports
• Duplicate/canonical issues
• Crawl stats
• URL inspection
• developer.chrome.com

• Useful for checking: - Whether filters create URLs
• JavaScript-generated links
• Network requests triggered by facets

Large-scale URL pattern analysis

• cloud.google.com (with Search Console exports or log data) - Useful when ecommerce sites have millions of parameter combinations.
• Allows analysis of: - Facet combinations
• Crawl frequency
• Indexation patterns
• Wasted URL clusters

Typical ecommerce crawl-waste workflow

1. Crawl the site

• Screaming Frog / Sitebulb / Lumar
• Identify: - Filter URLs
• Parameter variations
• Crawl traps
• Analyze Googlebot behavior

• Log analyzer + Search Console
• Compare: - URLs crawled vs URLs valuable for search
• Classify facets

• Keep crawlable/indexable: - High-demand category combinations
• Search-friendly landing pages
• Restrict: - Sort parameters
• Price sliders
• Color/size combinations with no demand
• Infinite combinations
• Improve internal linking

• Link prominently to: - Core categories
• High-value filtered landing pages
• Seasonal/product collections
• Avoid exposing thousands of low-value filter combinations in crawlable links

Tool stack by site size

For most ecommerce sites, the highest-impact combination is Screaming Frog + server log analysis + Google Search Console + a clear faceted URL strategy (canonicalization, selective indexation, and controlled internal links).

For e-commerce SEO, the most useful tools fall into a few buckets:

• Screaming Frog SEO Spider — crawl your site, identify orphan pages, excessive crawl depth, internal-link opportunities, duplicate URLs, canonical issues, and parameter/facet combinations. Particularly useful for modeling how faceted navigation expands the crawlable URL set.
• Sitebulb — strong visualizations of internal-link architecture, crawl depth, URL discoverability, and indexability. Its crawl-analysis reports can make faceted-navigation problems easier to diagnose.
• Google Search Console — useful for validating whether Google is actually discovering/crawling problematic parameter URLs and monitoring indexing patterns. Pair it with your own crawl data rather than treating it as a complete crawl report.
• Ahrefs — useful for internal-link analysis and identifying pages with few/no internal links. Its Site Audit can also surface canonical, redirect, duplicate, and indexability problems that often accompany faceted navigation.
• Semrush Site Audit — similar use case: internal-linking analysis, crawlability/indexability checks, duplicate URLs, canonicalization, and architecture issues.
• JetOctopus — particularly good for large e-commerce sites because it combines large-scale crawling with log-file analysis. This lets you compare your intended URL architecture with what Googlebot actually crawls.
• Botify — enterprise-focused. Strong for combining crawling, server logs, Googlebot behavior, and indexation data to identify crawl-budget waste at very large scale.
• Oncrawl — another enterprise option for combining crawl data and log analysis, including segmentation by URL types such as facets, filters, categories, and parameters.

For faceted navigation specifically

I'd prioritize Screaming Frog/Sitebulb + server-log analysis for a typical site, or JetOctopus/Botify/Oncrawl for a large catalog.

The important analysis isn't simply "how many URLs exist?" It's:

URLs generated by facets → URLs internally linked → URLs crawled by Googlebot → URLs indexed → URLs generating organic traffic

That lets you find patterns such as:

/shoes?color=red&size=10&brand=nike

being repeatedly crawled even though the combination is low-value or non-indexable.

For internal linking, look for:

• Important category/product pages receiving too few internal links
• Facet URLs receiving disproportionate internal links
• Excessive links to filtered/parameterized URLs
• Very deep product pages
• Orphan products/categories
• Crawl paths containing unnecessary facet combinations
• Inconsistent canonical targets
• Pagination and sorting URLs consuming crawl paths
• Facet combinations that are indexable despite having little/no search demand

A particularly powerful setup is Screaming Frog + Google Search Console + log files. The crawler tells you what can be crawled; logs tell you what Googlebot actually spends resources crawling. That distinction is crucial when diagnosing crawl waste.

For ecommerce, I’d use a combination of crawler + log analysis + Search Console data, rather than relying on one SEO tool. The key is distinguishing valuable faceted URLs that deserve internal links from combinations that should never consume crawl resources.

Best tools

1. JetOctopus — probably the strongest fit

JetOctopus is particularly well suited to this problem because it combines site crawling, server-log analysis, GSC data and internal-link analysis. Its log analyzer lets you see where Googlebot is actually spending its crawl activity, rather than merely what your crawler could discover. Tech SEO Platform Tech SEO Platform

Its internal-linking functionality can identify underlinked/orphan pages and prioritize opportunities, while its faceted-navigation analysis can identify parameter/facet combinations consuming crawl resources. Tech SEO Platform Tech SEO Platform

Especially useful workflow:

Googlebot logs → facet patterns → wasted crawl → valuable facets → internal-link opportunities

That's much more actionable than simply finding pages with few internal links.

2. Sitebulb — excellent for architecture and link graphs

sitebulb.com is excellent if your main problem is "Are we linking to the right pages, and are we accidentally linking to thousands of junk URLs?"

It can identify:

• Orphan pages
• Pages with very few incoming internal links
• Internal links pointing to redirects
• Link distribution by navigation/header/footer/content
• Anchor-text optimization
• Links to noindexed URLs
• Important pages buried deep in the architecture

SitebulbSitebulb also has a dedicated faceted-navigation methodology for separating indexable facets from those that should be excluded, and checking whether valuable facets are actually internally linked. Sitebulb

3. Screaming Frog — best flexible workhorse

screamingfrog.co.uk is great when you want to build your own analysis around URL parameters.

For example, you can crawl the site and segment URLs into:

/category/shoes
/category/shoes?brand=nike
/category/shoes?brand=nike&color=red
/category/shoes?brand=nike&color=red&size=10
/category/shoes?brand=nike&color=red&size=10&sort=price

Then calculate things like:

• Number of facet combinations
• Crawl depth
• Internal inlinks
• Indexability
• Canonical target
• Organic traffic
• Duplicate-content similarity
• Number of products returned
• Whether the URL is in an XML sitemap

It's particularly useful when you need highly customized extraction and segmentation.

4. Botify — strongest enterprise alternative

botify.com is particularly interesting for large ecommerce sites where server logs are essential.

Its faceted-navigation methodology explicitly compares Googlebot crawls with organic visits to identify facets that consume substantial crawl budget without producing search traffic. Botify

That gives you a useful prioritization metric:

High Googlebot crawl + low/no organic value = candidate for crawl containment

Botify also recommends separating facets into categories such as index, don't index, and block crawling, rather than treating every facet identically. Botify

5. Google Search Console — mandatory baseline

Don't overlook GSC. It provides the reality check for what Google is doing.

For faceted ecommerce, I'd specifically examine:

• Crawled – currently not indexed
• Indexed pages containing facet parameters
• URL patterns in Page Indexing
• Sitemap vs. indexed URLs
• Search performance of facet landing pages
• Internal-link discovery of important pages

The important distinction is that noindex isn't necessarily a crawl-budget solution. A noindexed URL can still be crawled. If the objective is specifically to stop Googlebot wasting resources, you need to address how those URLs are generated and linked, not just whether they're indexed. Botify

The setup I'd recommend

For a serious ecommerce site, I'd use:

JetOctopus or Botify + Screaming Frog/Sitebulb + GSC + server logs

Then build a facet classification such as:

The key principle is don't try to make every facet indexable. Instead, deliberately turn the facets with genuine search demand and sufficient inventory into SEO landing pages, and make the rest difficult for crawlers to discover.

For internal linking specifically, I'd prioritize high-value category/facet/product pages with few internal links, while simultaneously removing links that expose enormous numbers of low-value facet combinations. Site architecture and crawl efficiency are therefore being optimized together rather than independently. Tech SEO Platform Sitebulb

If you tell me roughly how many URLs/SKUs the ecommerce site has and whether facets use query parameters or URL paths, I can recommend the best tool stack and an audit workflow for that scale.

For ecommerce SEO, the best tools for optimizing internal linking and faceted navigation depend on whether you need crawling analysis, log analysis, link modeling, or technical controls. A strong stack usually combines several.

Crawl analysis and crawl waste detection

• screamingfrog.co.uk

• Excellent for crawling large ecommerce sites and finding: - Excessive faceted URLs
• Deep click paths
• Orphan pages
• Internal link distribution issues
• Duplicate URLs caused by parameters
• Crawl traps
• sitebulb.com

• Provides more visual audits around: - Crawl depth
• Internal linking opportunities
• Indexability problems
• Duplicate content from filters
• lumar.io

• Useful for enterprise ecommerce sites with millions of URLs.
• Helps monitor: - URL parameter explosions
• Crawl budget waste
• Internal linking changes after releases

Log file analysis (best for seeing real crawl waste)

• oncrawl.com

• Combines crawl data with server logs.
• Useful for identifying: - Googlebot crawling low-value faceted URLs
• Crawl budget spent on filters instead of products/categories
• Which internal links are influencing bot behavior
• screamingfrog.co.uk

• Good for validating whether blocked or canonicalized faceted URLs are still being crawled.

Internal linking optimization

• ahrefs.com

• Helps find: - Important pages with weak internal links
• Over-linked pages
• Pages buried too deeply
• Broken internal links
• semrush.com

• Useful for: - Internal link reports
• Crawlability checks
• Architecture monitoring
• inlinks.net

• Uses semantic analysis to suggest internal linking opportunities based on topical relationships.

Faceted navigation management

For preventing crawl waste from filters (size, color, brand, price, etc.), the biggest wins usually come from controlling URL generation and signals:

• search.google.com

• Monitor: - Indexed filtered URLs
• Duplicate/canonical issues
• Crawl anomalies
• bing.com

• Useful for additional crawl/indexation diagnostics.
• Platform-level tools:

• business.adobe.com
• shopify.com
• salesforce.com

These platforms often need custom rules for:

• Which filters create crawlable URLs
• Which combinations deserve indexation
• Canonical behavior
• Robots directives
• Parameter handling

Advanced enterprise options

• botify.com

• Built specifically around crawl budget optimization.
• Combines: - Log files
• Crawl data
• Search performance
• Internal linking analysis
• conductor.com

• Useful for continuous monitoring of technical SEO changes.

A practical ecommerce stack

For most medium-to-large ecommerce sites:

1. Screaming Frog or Sitebulb → map internal links and discover crawl traps
2. Oncrawl or Botify → confirm what Googlebot actually crawls
3. Ahrefs/Semrush → improve internal link equity flow
4. Search Console → monitor indexation outcomes
5. Platform controls → restrict unnecessary facet combinations

Key metrics to watch:

• % of crawls going to faceted URLs
• Number of indexed parameter URLs
• Average click depth of revenue pages
• Internal PageRank distribution
• Crawl frequency on low-value URLs
• Canonical vs indexed URL mismatch rate

For large ecommerce catalogs, log analysis plus crawl simulation is usually the fastest way to find where crawl budget is being wasted.

Yes. For ecommerce, I’d use a combination of a crawler, log-file analysis, and an internal-linking tool rather than relying on one platform.

Best tools by job

• screamingfrog.co.uk — probably the best starting point. Crawl your site with faceted URLs included, then analyze:

• internal links to filter URLs
• orphan/underlinked category and product pages
• crawl depth
• canonical/noindex directives
• URL parameter patterns
• duplicate pages
• response codes

Its crawl data is particularly useful for finding where your navigation is generating thousands of unnecessary URLs.

• sitebulb.com — excellent for visualizing site architecture and diagnosing faceted-navigation problems. Its faceted-navigation analysis specifically looks at internal links, indexability, duplicates, orphan pages and the distinction between useful facets and redundant filters. Sitebulb
• ahrefs.com — useful for combining technical crawling with organic traffic/search-demand data. Its Site Audit can expose large ratios of non-indexable to indexable URLs, which is a strong signal of faceted crawl waste. Ahrefs
• botify.com — particularly valuable for large ecommerce sites. The important capability is correlating Googlebot crawling with actual organic visits. For example, if URLs with 3+ facets consume a huge proportion of Googlebot activity but produce almost no organic traffic, you've identified genuine crawl waste rather than merely a large URL count. Botify
• oncrawl.com — another strong enterprise option for combining crawl data with server logs and analytics, allowing you to distinguish URLs that search engines crawl heavily from URLs that actually matter.
• Google Search Console — essential for validating the outcome. Look at indexing/crawling patterns and compare Google's discovered URLs against the URL sets you actually want indexed.

A practical stack

For a typical large ecommerce site, I'd choose:

Screaming Frog/Sitebulb → server logs → Search Console → implementation

The workflow is:

1. Crawl the site and classify URLs into:

• indexable category/product pages
• valuable SEO facets
• low-value filters
• parameter combinations
• duplicates
• orphan/underlinked pages.
• Measure internal-link waste.

Find how many internal links point to URLs that you ultimately don't want crawled/indexed. This is important because Google's understanding of ecommerce hierarchy is heavily influenced by internal link relationships. Google for Developers

3. Use logs to measure actual Googlebot behavior.

A URL being generated doesn't necessarily mean it's wasting crawl budget. The strongest evidence is something like:

70% of Googlebot requests → faceted URLs → 0.5% of organic traffic.

4. Create facet rules.

Typically divide facets into:

• Index: genuinely valuable search landing pages
• Crawl but don't index: situations where you need controlled access but don't want search results
• Don't crawl: combinatorial filters with little/no SEO value
• Reduce links to the waste bucket.

Don't just slap noindex on everything. Google explicitly notes that faceted navigation can create enormous URL spaces and overcrawling; if those URLs don't need to appear in search, preventing crawling can be preferable. Google for Developers

6. Strengthen internal links to your important pages.

Your SEO-worthy categories and facets should have intentional links from navigation, category pages and relevant content—not be buried behind thousands of filter combinations. Google specifically recommends linking category → subcategory → product and using ordinary <a href> links. Google for Developers

One important nuance: canonical and noindex aren't substitutes for controlling crawl paths. A noindex directive can still require Googlebot to crawl the URL to see the directive. For actual crawl-waste reduction, controlling which URLs are discoverable/crawlable is generally more important. Google for Developers Botify

If you tell me roughly how large the ecommerce site is (e.g. 50k, 5M, or 100M URLs) and whether it's Shopify, Magento, Salesforce Commerce Cloud, or custom, I can recommend the most appropriate stack and a specific faceted-navigation architecture.

For ecommerce, I’d use a combination of a crawler + log analysis + Search Console + your analytics/catalog data. The key is to distinguish “URLs the site can generate” from “URLs Google actually crawls”.

Best tools

For faceted navigation specifically

I'd build the workflow around Sitebulb or Screaming Frog + server logs.

For example, imagine:

/shoes?brand=nike&color=red&size=10

and your site allows combinations like:

• brand × color
• brand × size
• color × size
• 5+ filters simultaneously
• sorting parameters
• tracking parameters

The crawler can quantify how many URL variants exist and where they're linked from. Sitebulb can also explicitly exclude faceted URLs during an audit, while still recording that those URLs exist, which is useful for separating your site architecture audit from the potentially enormous facet URL space. support.sitebulb.com

More importantly, Sitebulb can analyze where internal links to those URLs occur and whether important category/facet pages are receiving enough internal links. sitebulb.com

The analysis I'd actually run

1. Crawl the whole site
   • Export every internal URL.
   • Capture crawl depth, status, canonical, indexability, incoming links and outgoing links.

2. Cluster URLs by parameter/facet
   • ?color=
   • ?size=
   • ?brand=
   • combinations of parameters
   • sort/filter/session/tracking parameters.

3. Calculate crawl waste
   • % of crawlable URLs that are non-indexable
   • Googlebot hits to non-indexable facet URLs
   • duplicate/near-duplicate URL count
   • average internal links pointing to unwanted facets
   • number of facet combinations with zero/very little demand.

4. Identify valuable facets
   • Search demand
   • conversions/revenue
   • unique product inventory
   • existing organic traffic
   • backlinks
   • strategic category importance.

5. Restructure internal links
   • Prominently link valuable category/facet landing pages.
   • Remove or reduce links to low-value combinations.
   • Avoid sitewide links generating thousands of permutations.
   • Fix internal links pointing to redirects, canonicals or non-indexable URLs.

6. Control the remaining URL space
   • Depending on the situation, use canonicalization, noindex, robots.txt controls, URL restructuring, or simply stop generating crawlable URLs.
   • Don't treat robots.txt as a universal solution: blocking crawling can prevent Google from seeing other signals on those URLs.

A particularly useful principle from Screaming Frog is to avoid generating separate URLs for every sorting/filtering state when you don't need them; unnecessary URL variants can create large quantities of low-value pages. www.screamingfrog.co.uk

If I had to pick just two tools: Sitebulb + server-log analysis. Sitebulb tells you how your internal-link architecture encourages crawling; logs tell you how Googlebot is actually spending its crawl activity. Together, they're much more informative than simply running a standard technical SEO crawl.

For an ecommerce site, I’d use a crawler + log-analysis tool + Google’s own diagnostics rather than relying on a single SEO platform.

Best tools

What I'd specifically audit

For faceted navigation, don't just count URLs. Build a matrix like:

Facet → combinations → URLs generated → internally linked? → indexable? → organic traffic? → conversions?

For example:

/shoes/
→ /shoes?color=red
→ /shoes?color=red&size=10
→ /shoes?color=red&size=10&brand=nike
→ etc.

The dangerous situation is when the site exposes thousands of combinations through crawlable internal links even though most combinations have little or no search demand.

My preferred workflow

1. Screaming Frog/Sitebulb: discover the URL universe and internal-link graph.

Look for:

• Facet URLs with zero/very little organic value
• Multiple parameter combinations producing equivalent content
• Noindex pages receiving lots of internal links
• Canonicalized facet URLs receiving internal links
• Excessive crawl depth
• Huge numbers of URLs generated from combinations
• Internal links pointing to redirects
• Important category/product pages that are underlinked

Sitebulb is particularly useful here because its link reports expose incoming/outgoing links, anchor text, link location, crawl depth and URL Rank. support.sitebulb.com

2. Log analysis: use Botify/Oncrawl to answer the more important question:

What is Googlebot actually spending its crawl budget on?

For example, you might discover that 35% of Googlebot requests are going to ?color=...&size=... combinations that generate no organic traffic. That's much stronger evidence than simply seeing 500,000 URLs in a crawler.

3. Search Console: validate whether the changes reduce unwanted crawling/indexation without hurting valuable pages.

4. Fix the architecture, not just the crawl.

Typically this means:

• Keep high-value facets crawlable/indexable when there's genuine search demand.
• Make low-value combinations non-discoverable through internal links where appropriate.
• Avoid automatically linking every possible facet combination.
• Use canonicalization/noindex/robots controls deliberately rather than treating them as interchangeable.
• Keep important category and product pages strongly internally linked.
• Ensure XML sitemaps contain the canonical, indexable URLs you actually want discovered.

One particularly important distinction: robots.txt can prevent crawling, but it doesn't necessarily solve URL discovery or indexing signals by itself. For faceted navigation, you generally want to decide separately which URLs should be discoverable, crawlable, indexable, and internally linked.

If I had to pick only three

Small/medium ecommerce:
Screaming Frog + Sitebulb + Search Console

Large/enterprise ecommerce:
Botify or Oncrawl + Screaming Frog/Sitebulb + Search Console

And if your primary concern is specifically internal linking, I'd lean toward Sitebulb because its link analysis makes it relatively easy to identify underlinked money pages and links pointing toward URLs you don't actually want to prioritize. support.sitebulb.com

If you tell me roughly how many products/URLs the ecommerce site has and whether facets use query parameters or path-based URLs, I can give you a concrete crawl-waste audit setup and the exact reports/filters I'd use.

For ecommerce sites, optimizing internal linking and faceted navigation to reduce crawl waste usually requires a mix of SEO crawlers, log analyzers, analytics tools, and technical SEO platforms. The best tools depend on whether you’re trying to discover crawl waste, model internal link equity, or control indexation.

1. Technical SEO crawlers (find crawl traps, bad facets, and internal link issues)

Screaming Frog SEO Spider

Best for:

• Crawling large ecommerce sites
• Finding excessive faceted URLs
• Auditing internal links
• Identifying orphan pages
• Analyzing crawl depth and click paths
• Exporting URL patterns for facet cleanup

Useful reports:

• Crawl depth
• Internal links pointing to parameter URLs
• Canonicals
• Noindex pages receiving internal links
• Duplicate titles/meta descriptions from facets

Sitebulb

Best for:

• Visualizing site architecture
• Finding internal linking weaknesses
• Prioritizing technical SEO issues

Useful features:

• Crawl maps
• Link equity reports
• Indexability audits
• Page importance scoring

Lumar (formerly Deepcrawl)

Best for:

• Enterprise ecommerce sites
• Scheduled crawl monitoring
• Large-scale faceted navigation analysis

Useful for:

• Detecting when new filters create millions of URLs
• Monitoring crawl budget risks after releases

2. Server log analysis (see what Googlebot actually wastes time on)

Faceted navigation problems are often invisible in normal crawls because crawlers simulate behavior. Log files show reality.

Screaming Frog Log File Analyser

Best for:

• Seeing Googlebot URL requests
• Measuring crawl frequency by URL type
• Finding wasted crawl on parameters

Look for:

• Googlebot repeatedly crawling URLs like:
  • /shoes?color=red&size=10
  • /category?sort=price
  • /search?q=...
• Crawl concentration on low-value pages

Oncrawl

Best for:

• Enterprise log analysis
• Connecting crawl behavior with organic traffic
• Identifying inefficient crawl patterns

Botify

Best for:

• Large marketplaces and retailers
• Crawl budget optimization
• Prioritizing URLs Google actually crawls

3. Internal linking optimization tools

Ahrefs

Useful for:

• Finding pages with weak internal links
• Identifying high-authority pages that could pass equity
• Finding internal link opportunities

Reports:

• Best by links
• Orphan pages
• Internal backlinks

Semrush

Useful for:

• Internal linking audits
• Topic clustering
• Identifying important pages lacking links

InLinks

Useful for:

• Semantic internal linking
• Entity/topic relationships
• Content architecture

4. Crawl simulation and URL discovery

Google Search Console

Free and essential:

• URL Inspection
• Crawl stats
• Index coverage reports
• Excluded pages

Useful signals:

• "Crawled — currently not indexed"
• Duplicate pages
• Discovered but not indexed URLs

Google Search Console

5. Faceted navigation testing and URL management

JetOctopus

Strong for:

• Large ecommerce crawls
• JavaScript-heavy sites
• Faceted navigation analysis

Useful reports:

• Parameter URL discovery
• Indexability segmentation
• Crawl budget visualization

6. Custom analysis (often needed for big ecommerce)

For millions of SKUs/facets, teams often combine:

• BigQuery + Google Search Console exports
• Server logs
• Screaming Frog exports
• Product database exports

Useful analyses:

• Number of URLs generated per facet combination
• Organic traffic by facet type
• Crawl frequency by URL pattern
• Internal links to non-indexable pages

Common workflow for ecommerce crawl optimization

Step 1: Discover crawl waste

Use:

• Log analyzer (Botify/Oncrawl/Screaming Frog Log File Analyser)
• Screaming Frog crawl

Find:

• Parameter explosions
• Sort/filter URLs
• Duplicate category pages

Step 2: Classify facets

Step 3: Fix internal links

Common fixes:

• Do not link every facet combination
• Use curated category links
• Avoid passing internal PageRank to useless URLs
• Remove links to noindex pages
• Create controlled hub pages

Step 4: Validate with logs

Measure:

• Googlebot crawl distribution before/after
• Indexed URL count
• Crawl of valuable category/product URLs

For most ecommerce sites, a strong stack is:

• Screaming Frog SEO Spider → crawl discovery
• Botify or Oncrawl → crawl budget/log analysis
• Google Search Console → Google-side validation
• Ahrefs/Semrush → internal linking opportunities
• BigQuery → large-scale pattern analysis

The key is not just blocking facets; it is ensuring Google spends crawl effort on URLs that have unique search value and strong internal link support.

For ecommerce, I’d use a combination of a crawler + server-log analysis + Search Console, because internal-link structure tells you what can be crawled, while logs tell you what Googlebot is actually spending time crawling.

Best tools

• jetoctopus.com — best all-in-one for large ecommerce

• Combines site crawling, Googlebot/server logs, and GSC data.
• Particularly useful for identifying parameter/facet URLs consuming crawl budget.
• Its internal-linking analysis can identify underlinked strategic pages and opportunities to redistribute internal links.
• It also has an AI Internal Linker designed around crawl and GSC data. Tech SEO Platform Tech SEO Platform
• botify.com — strongest for enterprise-scale crawl-budget analysis

• Excellent when you have millions of URLs and substantial server-log data.
• Its SiteCrawler reports compare Google's crawling against your indexable/strategic pages and expose pages that are internally linked but not being crawled.
• Useful for determining whether crawl budget is being diverted toward facets, deep URLs, or slow page types. Botify Knowledge Base Botify Knowledge Base
• sitebulb.com — excellent for diagnosing internal-link/facet architecture

• Very good visualization of crawl paths, internal links, orphan pages, duplicate content, URL patterns, and indexation directives.
• Its faceted-navigation workflow specifically recommends checking how facets are internally linked and whether important landing pages have enough internal links. Sitebulb
• screamingfrog.co.uk — great flexible crawler

• Particularly useful for custom URL extraction, parameter analysis, crawl-depth analysis, internal-link reports, canonicals, and noindex/robots directives.
• I'd favor it when you want granular control and don't necessarily need enterprise-scale log analysis.
• search.google.com — essential validation layer

• Use it to see whether unwanted facet URLs are actually being indexed/crawled and whether the pages you're trying to prioritize are receiving Googlebot attention.
• It's not enough by itself for diagnosing the entire internal-link graph, but it's important for validating the consequences of your changes. Tech SEO Platform

The stack I'd choose

For a large ecommerce site, my preference would be:

JetOctopus/Botify → server logs → crawler → GSC

The key analysis isn't simply "How many facet URLs exist?" It's:

Which URLs are being discovered through internal links → which URLs does Googlebot actually crawl → which of those URLs have search value?

For example, suppose /shoes/ links to hundreds of combinations such as ?brand=nike&color=black&size=10. Your crawler can identify that link explosion, while logs can tell you whether Googlebot is actually spending substantial crawl activity on those combinations. JetOctopus explicitly supports this crawl/log intersection for faceted navigation. Tech SEO Platform

What I'd specifically audit

1. Facet URL generation — identify combinations that should never exist as SEO URLs.
2. Internal links to facets — find navigation elements generating thousands of crawlable URLs.
3. Crawl depth — ensure important category/product pages aren't buried beneath facet paths.
4. Internal-link concentration — strengthen links to commercially important categories/products rather than distributing them across low-value facets.
5. Parameter patterns — sorting, filtering, pagination, tracking parameters, etc.
6. Googlebot logs — calculate what percentage of crawling goes to low-value parameter URLs.
7. Indexation signals — compare crawlable, indexable, indexed, and ranking URLs.
8. Before/after monitoring — measure whether a facet cleanup actually shifts Googlebot toward valuable pages.

One important distinction: noindex isn't necessarily a crawl-budget solution. If Google can still discover thousands of low-value URLs through internal links, it may continue crawling them. For genuine crawl containment, you generally want to fix the URL/link-generation architecture itself, with robots directives and canonicalization used appropriately for the specific URL class. Sitebulb's faceted-navigation guidance makes the same distinction between crawlability, indexation, and internal linking. Sitebulb

If you're choosing one tool, I'd pick JetOctopus for a large ecommerce site, Sitebulb for a mid-sized site where internal-link architecture is the main concern, and Botify when you're operating at enterprise scale with substantial log data.

What I'd specifically audit

For ecommerce, I’d use a crawler + log-analysis tool + Search Console, rather than relying on a generic SEO suite alone. Google specifically identifies faceted navigation as a major source of overcrawling, and says internal linking helps it understand which pages are important. developers.google.comdevelopers.google.com

Best tools for the job

For faceted navigation specifically

The important thing isn't just finding "bad URLs"; it's determining which facet combinations deserve to exist as crawlable/indexable landing pages.

I'd configure your crawler to segment URLs by parameters such as:

?color=
?size=
?brand=
?price=
?sort=
?filter=
?page=

Then measure:

• Number of unique URLs generated per facet
• % of crawl consumed by each parameter
• Crawl depth
• Internal inlinks to faceted URLs
• Organic traffic/indexation per facet
• Canonical target
• HTTP status
• Whether combinations produce useful inventory
• Duplicate/near-duplicate content
• Googlebot hits from server logs

That lets you build a simple classification:

Index + link: high-demand facet/category with genuinely useful unique content
Crawl but perhaps don't index: strategically useful combinations where indexation is questionable
Don't crawl: arbitrary combinations, sorting, session/tracking parameters, etc.

Google's current guidance is quite explicit: if faceted URLs don't need to appear in Search, preventing crawling—typically with robots.txt—is preferable to relying on canonicalization or nofollow alone. If you do allow crawling, consistent parameter ordering, sensible URL structures and proper 404s for empty combinations help reduce the problem. developers.google.comsitebulb.com

For internal-link optimization

Screaming Frog or Sitebulb would be my first choice.

Look for:

1. Important category/product pages with too few internal links
2. Deep pages requiring excessive clicks to reach
3. Orphaned products/categories
4. Facet pages receiving huge numbers of internal links
5. Internal links pointing at noncanonical URLs
6. Links containing unnecessary parameters
7. Pages where navigation creates thousands of low-value links

Google recommends making important ecommerce pages reachable through normal crawlable links and specifically recommends linking from categories → subcategories → products. developers.google.comdevelopers.google.com

The stack I'd choose

Small/medium ecommerce site:
Screaming Frog + Google Search Console + server logs

Large ecommerce site:
Botify or Oncrawl + Screaming Frog/Sitebulb + Search Console + server logs

The logs are especially important. A crawler tells you what your site can generate; server logs tell you what Googlebot is actually spending time crawling. That distinction is crucial when the goal is specifically crawl waste.

A particularly useful KPI is:

% of Googlebot requests spent on URLs you don't want indexed

Then break that down by parameter/facet. That's much more actionable than simply looking at "number of crawled pages."

Google also recommends avoiding alternative URLs that return the same content and avoiding internal links to temporary/tracking parameters. developers.google.com

Yes. For ecommerce, I’d use a combination of a crawler + log analysis + Search Console, because internal-link optimization and faceted-navigation crawl waste are slightly different problems.

Best tools

For your specific problem, I'd prioritize these

1. Screaming Frog → find crawl waste

Look for:

• ?sort=
• ?filter=
• ?color=
• ?size=
• combinations such as ?color=red&size=10&brand=x
• pagination variants
• tracking parameters
• internal links pointing to non-indexable facet URLs
• enormous numbers of URLs with nearly identical content

Screaming Frog specifically recommends analyzing parameter URLs and excluding unnecessary faceted-navigation URLs when crawling large sites. www.screamingfrog.co.uk

2. Sitebulb → optimize the useful internal links

This is where Sitebulb is particularly useful. Its link analysis can show:

• important category/product pages with few internal links
• orphan pages
• crawl depth
• internal-link distribution
• anchor text
• where links occur (navigation, content, footer, etc.)
• redirect/broken internal links

Its URL Rank metric also helps identify important pages that aren't receiving enough internal-link equity. support.sitebulb.com

3. Logs → determine whether the problem is actually costing crawl

For a large ecommerce site, I'd consider this essential.

For example, you might discover:

65% of Googlebot requests → facet combinations
20% → pagination/sort URLs
10% → useful category/product URLs
5% → everything else

That tells you where to focus. A crawler alone can't tell you how Googlebot is allocating its actual crawl activity.

The key distinction

Don't simply try to eliminate all facet URLs.

You generally want to divide facets into:

Indexable facets

• substantial search demand
• meaningful product set
• unique landing-page value
• stable URL
• worth ranking

Non-indexable facets

• arbitrary combinations
• thin/empty results
• sorting
• session/tracking parameters
• combinations with no SEO demand

Then make your internal linking reflect that hierarchy.

For example:

Home
 └── Men's Shoes
      ├── Running Shoes          ← strongly linked
      ├── Trail Running Shoes    ← strongly linked
      ├── Nike Running Shoes     ← strongly linked
      │
      └── arbitrary filters      ← don't expose thousands of crawl paths
           ├── red + size 10
           ├── red + size 10 + Nike
           ├── red + size 10 + Nike + waterproof
           └── ...

The goal isn't merely "reduce URLs." It's reduce the number of crawl paths leading to low-value URLs while increasing internal prominence of pages you actually want crawled and ranked.

Sitebulb's ecommerce faceted-navigation guidance specifically recommends auditing total URL growth, duplication, indexation directives, sitemaps, and internal links as part of this process. sitebulb.com

My preferred stack

For a large ecommerce site:

Screaming Frog + Sitebulb + server logs + GSC

For a smaller ecommerce site:

Screaming Frog + GSC is often enough.

If you tell me roughly how many URLs/products the ecommerce site has and what platform it runs on (Shopify, Magento, WooCommerce, custom, etc.), I can give you a concrete faceted-navigation + internal-linking audit workflow, including exactly what URLs to allow, noindex, canonicalize, or prevent from being linked.

For ecommerce sites, I’d use a combination of a crawler + log analysis + Search Console, because internal-link structure tells you what could be crawled, while logs tell you what Googlebot is actually wasting crawl activity on.

Best tools

For your specific problem

I'd structure the workflow like this:

1. Find the crawl waste → JetOctopus + server logs

Segment URLs by patterns such as:

• ?color=
• ?size=
• ?brand=
• ?sort=
• ?price=
• combinations of multiple parameters
• internal search URLs
• pagination
• session/tracking parameters

Then compare Googlebot hits vs. organic traffic/conversions. This lets you identify something like:

400,000 Googlebot requests → 2,000 useful organic visits → probably a crawl trap.

JetOctopus is particularly suited to this because its log analysis can show which parameter/facet combinations are consuming crawl activity. jetoctopus.com

2. Model your internal-link graph → Screaming Frog or Sitebulb

Look for:

• important category/PDP pages with few internal links
• excessive links to low-value facet URLs
• links to noindex pages
• redirecting internal links
• orphan products/categories
• excessive crawl depth
• inconsistent anchor text

Sitebulb is especially convenient if the main objective is "which pages should receive more internal links?" sitebulb.comsupport.sitebulb.com

3. Separate SEO-worthy facets from navigation-only facets

This is probably the most important architectural decision.

For example:

/shoes/
/shoes/running/
/shoes/nike/
/shoes/womens/

might be legitimate SEO landing pages.

But:

/shoes?color=red
/shoes?color=red&size=9
/shoes?color=red&size=9&sort=price
/shoes?brand=nike&color=red&size=9&rating=4

may be combinatorial URLs with little search value.

The goal isn't simply to "block facets." You want to make valuable facets discoverable through normal internal <a> links while preventing low-value combinations from becoming part of the crawl graph. JetOctopus' current faceted-navigation guidance explicitly recommends this distinction. jetoctopus.com

4. Measure before/after with logs + GSC

After changing the facet/link architecture, monitor:

• Googlebot requests to parameter URLs
• crawl frequency of category/PDP URLs
• indexed facet URLs
• "Crawled - currently not indexed"
• organic impressions/clicks for category and product pages
• internal-link counts to priority pages

That gives you a much better KPI than simply counting indexed pages.

My stack recommendation

For a large ecommerce site, I'd choose:

JetOctopus → primary platform
Screaming Frog → technical crawl validation
GSC → Google-side validation
GA4 → revenue/traffic prioritization

For a smaller site, Screaming Frog + GSC is usually enough, with Sitebulb being attractive if internal-link analysis/visualization is the main focus.

One important caveat: nofollow isn't a good primary mechanism for solving faceted crawl waste. The stronger solution is to control which URLs are actually exposed through crawlable internal links, combined with appropriate canonical/noindex/robots strategies depending on whether a facet URL should exist, be crawled, or be indexed.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For large ecommerce sites, the best approach is usually to combine technical crawling tools, log file analysis, and internal linking optimization software. Each solves a different part of crawl efficiency.

Here's a breakdown by use case.

1. Screaming Frog (best value)

This is the go-to desktop crawler.

It helps identify:

• infinite faceted URLs
• crawl depth
• orphan pages
• excessive parameter combinations
• pages with very few internal links
• duplicate URLs caused by filters
• canonical issues

Useful reports:

• Crawl Depth
• URL Parameters
• Inlinks
• Orphan URLs
• Near Duplicates
• Canonicals

Ideal for sites up to several hundred thousand URLs (with database mode).

2. Lumar (formerly Deepcrawl)

Best for enterprise ecommerce.

Strengths:

• scheduled crawls
• faceted navigation reports
• parameter explosion detection
• internal link visualization
• JavaScript rendering
• change tracking over time

Very useful if your catalog has millions of URLs.

3. Botify

Botify combines three datasets:

• crawler
• log files
• search performance

This makes it especially useful because you can answer questions like:

• Google crawled this URL 500 times
• but it gets zero traffic
• and has zero internal links

That immediately highlights crawl waste.

4. Oncrawl

Excellent for internal linking analysis.

It measures:

• PageRank flow
• link equity
• crawl budget allocation
• orphan pages
• pages buried 8–10 clicks deep

It also overlays:

• server logs
• Google Analytics
• Search Console

to prioritize fixes.

5. Log File Analysis

Many ecommerce SEO problems only become obvious in server logs.

Good tools include:

• Splunk
• Elastic (ELK)
• Botify
• Oncrawl

Look for patterns such as:

Googlebot

/category?

/category?color=red

/category?color=blue

/category?color=red&size=small

/category?size=small&sort=price

/category?sort=price&brand=nike

If Google spends 70% of its crawl budget on these combinations, important product pages may be crawled less frequently.

6. Google Search Console

The Crawl Stats report shows:

• crawl requests
• response codes
• crawl purpose
• host status
• average response time

The URL Inspection tool also lets you verify:

• canonical selection
• crawl history
• indexing eligibility

7. Ahrefs Site Audit

While better known for backlinks, its Site Audit can identify:

• orphan pages
• weak internal linking
• redirect chains
• duplicate content
• crawl depth
• excessive indexable filters

It's especially useful if you already use Ahrefs for SEO.

Optimizing faceted navigation

Regardless of tool, check for:

✅ Infinite URL combinations

?color=red

?color=red&size=M

?color=red&size=M&sort=price

?color=red&size=M&sort=price&page=5

✅ Crawlable filters that shouldn't be indexed

Examples:

• sort
• availability
• price ranges
• session parameters

Often handled with:

• robots.txt for crawl control where appropriate
• nofollow on certain filter links (used selectively)
• canonical URLs
• making low-value filters non-crawlable via JavaScript when appropriate

✅ Valuable filter pages

Some combinations deserve indexing:

• /running-shoes/mens/
• /nike-running-shoes/
• /waterproof-hiking-boots/

while low-value combinations generally should not.

Internal linking improvements

The best tools identify pages with:

• only one internal link
• no incoming links
• excessive click depth
• low internal PageRank
• links pointing primarily to parameterized URLs

A good ecommerce structure typically follows:

Homepage
    ↓
Category
    ↓
Subcategory
    ↓
Product

with contextual links such as:

• related products
• popular brands
• top-selling categories
• breadcrumbs
• "Customers also bought"
• editorial buying guides

This helps search engines discover important pages efficiently without wasting crawl budget.

Recommended stack by site size

• Small (<50k URLs): Screaming Frog SEO Spider + Google Search Console
• Medium (50k–500k URLs): Screaming Frog SEO Spider + Ahrefs + log file analysis
• Enterprise (500k+ URLs): Botify or Oncrawl combined with Lumar and server log analysis

This layered approach gives you visibility into both the site's architecture and how search engine crawlers actually interact with it, making it easier to reduce crawl waste while preserving discoverability for high-value category and product pages.

For ecommerce, I’d use a crawler + Search Console + log analysis rather than relying on a single SEO suite. Faceted navigation is one of the biggest sources of Google overcrawling because every filter combination can create another URL. developers.google.com

Best tools for the job

For faceted navigation specifically

Screaming Frog or Sitebulb is the most useful starting point. Configure a crawl that captures query parameters, then segment URLs such as:

• ?color=red
• ?size=large
• ?brand=nike
• ?color=red&size=large
• permutations of the same filters
• empty-result combinations
• duplicate parameter combinations

Then calculate something like:

Facet crawl waste = non-indexable/low-value facet URLs crawled ÷ total URLs crawled

You want to discover whether Google can reach thousands or millions of URLs that have little or no search value.

Google's current guidance is essentially: don't let every possible filter combination become crawlable/indexable by default. If a facet doesn't need to appear in search, you can prevent crawling; if it does have search value, make the URL structure consistent and intentionally optimized. developers.google.com

For internal linking

I'd specifically audit:

1. Click depth — Are important category/product pages 3–4+ clicks deep?
2. Orphans — Important pages with no internal links.
3. Underlinked money pages — High-value categories/products receiving very few links.
4. Facet links — Are thousands of low-value filter URLs consuming internal-link prominence?
5. Navigation hierarchy — Homepage → category → subcategory → product is a clean baseline for ecommerce. Google explicitly says its understanding of ecommerce structure is influenced by these link relationships. developers.google.com
6. Anchor text — Whether internal anchors describe the destination appropriately.
7. Pagination and sorting — Make sure sort/order/session URLs aren't creating unnecessary crawl paths.

Sitebulb is particularly useful if you want a visual representation of the site's link structure, while Screaming Frog is excellent when you want granular crawl data and custom extraction. Sitebulb's own faceted-navigation guidance also recommends checking which facets are internally linked, identifying orphaned/underlinked landing pages, and separating indexable facets from non-indexable filters. sitebulb.com

A strong ecommerce stack

If I were auditing a large catalog, I'd use:

Screaming Frog → server logs → Google Search Console → Sitebulb

If you're already paying for an SEO suite:

Ahrefs or Semrush + Screaming Frog

The important distinction is that a crawler tells you what can happen, while server logs + GSC tell you what Google is actually doing.

For a serious crawl-waste project, I'd build a URL classification such as:

URL
├── Indexable category
├── Indexable product
├── Indexable SEO facet
├── Non-indexable facet
├── Sort/order URL
├── Search URL
├── Pagination
└── Other parameter URL

Then compare URL count × internal links × Googlebot hits × organic traffic for each class. That usually exposes the biggest waste very quickly.

If you tell me roughly how many SKUs/URLs the store has and whether it's Shopify, Magento, WooCommerce, Salesforce Commerce Cloud, or custom, I can give you a concrete crawl setup and faceted-navigation audit workflow.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the most useful tools fall into a few buckets:

• Screaming Frog SEO Spider — crawl your site to identify orphan pages, excessive URL variants, deep click depth, internal-link patterns, canonicalization issues, and parameter-generated URLs.
• Google Search Console — monitor indexing, crawl-related signals, canonical selection, and which URL patterns Google is actually discovering/indexing.
• Ahrefs — useful for auditing internal links, finding pages with few/no internal links, analyzing competing sites, and identifying pages that deserve stronger internal-link equity.
• Semrush — its Site Audit can surface crawlability, internal-linking, canonical, redirect, and indexability problems at scale.
• Sitebulb — particularly good for visualizing crawl architecture, URL patterns, crawl depth, and internal-link relationships.
• Botify — better suited to very large ecommerce sites; it combines crawling, log-file analysis, and Googlebot behavior to identify where crawl budget is being spent.
• Oncrawl — strong for enterprise sites where you want to combine crawl data with server logs and analytics to distinguish valuable URLs from crawl waste.

For faceted navigation specifically

I'd combine a crawler with server-log analysis. The key is finding combinations such as:

/shoes?color=red&size=10&brand=nike

that generate huge numbers of crawlable URLs without providing enough unique search value.

A practical workflow is:

1. Crawl the site with Screaming Frog/Sitebulb.
2. Analyze Googlebot logs with Botify or Oncrawl.
3. Segment URLs by parameters, facets, status code, canonical, indexability, and organic traffic.
4. Identify facets that should be:
   • indexable and internally linked,
   • crawlable but noindex,
   • canonicalized,
   • or excluded from crawling altogether.
5. Strengthen internal links toward important category, subcategory, and product pages, rather than allowing faceted URLs to consume most of the site's internal-link graph.

One important distinction: noindex and canonicalization don't necessarily stop Googlebot from crawling URLs. If the goal is specifically to reduce crawl waste, you need to control URL discovery/crawl paths—not just whether those URLs appear in the index.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce SEO, the best approach is usually a combination of a crawler, log file analyzer, and visualization tool. Each solves a different part of the crawl waste problem.

1. Site crawlers (find crawl inefficiencies)

These simulate how search engines crawl your site.

Useful reports include:

• Pages deeper than 4–5 clicks
• Orphan pages
• Thin category pages
• Duplicate faceted URLs
• Internal PageRank distribution
• Crawl depth by directory

2. Log file analyzers (see what Google actually crawls)

This is the gold standard for identifying crawl waste.

Questions to answer:

• Is Googlebot spending 40% of its crawl on filtered URLs?
• Which faceted URLs get crawled repeatedly?
• Are product pages being crawled enough?
• Which parameter combinations waste crawl budget?

3. Internal linking optimization

These help identify pages that need more internal authority.

Look for:

• Important categories with few internal links
• Products more than 5 clicks from the homepage
• Broken link paths
• Weak hub pages

4. Faceted navigation analysis

Faceted navigation can generate millions of URL combinations.

Helpful tools include:

• Lumar
• Oncrawl
• JetOctopus
• Screaming Frog SEO Spider (using custom extraction and URL pattern filters)

Things to audit:

• Parameter combinations
• Infinite URL spaces
• Sort URLs
• Pagination
• Filter URLs
• Session IDs
• Duplicate canonicals

Typical issues include URLs like:

/shoes?size=9
/shoes?color=black
/shoes?size=9&color=black
/shoes?size=9&color=black&sort=price

where only a small subset may deserve indexing.

5. Google's own data

Don't overlook Google's first-party tools.

• Google Search Console: Crawl Stats, Indexing reports, URL Inspection, Page Indexing, and Internal Links reports help identify crawl bottlenecks and indexing patterns.
• Google Analytics (or another analytics platform): Prioritize crawl optimization around high-value landing pages.

6. Visualization tools

For very large ecommerce sites, visualizing the architecture helps uncover structural issues.

• Gephi — visualize internal link graphs.
• Sitebulb — built-in crawl maps.
• Oncrawl — internal PageRank and link flow visualizations.

Recommended stacks by site size

Best practices to reduce crawl waste

• Restrict indexation of low-value filter combinations using robots directives, noindex where appropriate, or URL handling strategies consistent with search engine guidance.
• Canonicalize duplicate parameter URLs to the preferred version.
• Ensure important category and product pages are reachable within a few clicks from the homepage.
• Add contextual internal links from buying guides, brand pages, and related categories to key commercial pages.
• Regularly compare crawl data with server logs to verify that search engine bots are spending their crawl budget on high-value URLs rather than faceted or duplicate pages.

For enterprise ecommerce sites (100k+ URLs), combining a crawler with server log analysis is the most effective way to identify and eliminate crawl waste caused by faceted navigation.

For ecommerce sites, the most effective approach is usually a combination of crawl analysis, log file analysis, internal linking visualization, and technical SEO auditing. No single tool covers all of these well.

Here's a breakdown by use case:

Best tools for faceted navigation

1. Screaming Frog SEO Spider

Excellent for discovering:

• Infinite crawl spaces
• Parameter combinations
• Duplicate category pages
• Canonicals
• Robots directives
• Noindex implementation
• Internal linking

Useful reports include:

• URL Parameters
• Duplicate Content
• Near Duplicates
• Crawl Depth
• Inlinks
• Response Codes

2. Sitebulb

Especially strong at visual diagnostics.

It highlights:

• Crawl traps
• Faceted navigation loops
• Excessive crawl depth
• Internal PageRank distribution
• Orphan pages
• Indexability conflicts

Its visual architecture reports are easier to interpret than raw crawl exports.

3. Botify

One of the strongest enterprise solutions because it combines:

• crawler data
• server logs
• Googlebot behavior
• indexation
• internal linking

This lets you answer questions like:

• Which filtered URLs consume most crawl budget?
• Which important categories Google never reaches?
• Which URLs receive internal links but no organic traffic?

Ideal for sites with millions of URLs.

4. OnCrawl

Strong for log-file analysis.

You can compare:

• discovered URLs
• crawled URLs
• indexed URLs
• URLs receiving Googlebot visits

Very useful when faceted navigation creates millions of low-value URLs.

5. JetOctopus

A more affordable alternative to Botify.

Useful features include:

• log analysis
• crawl visualization
• Googlebot behavior
• crawl budget reports
• JavaScript rendering

Often a good fit for mid-sized ecommerce businesses.

Tools specifically for internal linking

Screaming Frog

Exports:

• Inlinks
• Outlinks
• Crawl depth
• Link score
• Orphan URLs (with analytics/Search Console integration)

You can identify:

• pages with only one internal link
• deep category pages
• isolated product pages

Sitebulb

Provides:

• Link equity visualization
• Internal PageRank
• Hub analysis
• Crawl maps
• Click-depth charts

These reports make it easier to identify where important pages need stronger internal linking.

Gephi

Useful when exporting a crawl graph.

You can identify:

• disconnected clusters
• isolated categories
• over-linked navigation
• inefficient site architecture

This is especially helpful for marketplaces and very large catalogs.

Log analysis tools

Server logs reveal what search engines actually crawl, rather than what they could crawl.

Top options include:

• Splunk
• Elastic (ELK)
• JetOctopus
• OnCrawl
• Botify

Key metrics to monitor:

• Googlebot hits by directory
• Parameter URL frequency
• Crawl frequency by page type
• 404 crawl volume
• Redirect chains
• Low-value filter URLs receiving significant crawl activity

Google Search Console

Although limited compared with dedicated enterprise platforms, it remains valuable for identifying:

• indexed vs. non-indexed pages
• duplicate canonical issues
• crawl anomalies
• URL Inspection data
• crawl statistics
• sitemap coverage

Combining its Crawl Stats report with server logs helps validate where Google is spending its crawl resources.

Practical workflow

For most ecommerce sites (10,000–500,000 URLs), a strong workflow is:

1. Crawl the site with Screaming Frog or Sitebulb to map architecture, internal links, and faceted URL patterns.
2. Analyze server logs (or use JetOctopus, OnCrawl, or Botify) to see how Googlebot actually spends crawl budget.
3. Compare crawled, indexed, and linked URLs to identify low-value filter pages consuming crawl resources.
4. Review faceted navigation rules—such as URL parameters, canonical tags, robots directives, noindex usage where appropriate, and internal linking—to ensure only valuable combinations are discoverable.
5. Re-crawl after changes to confirm that crawl depth, orphan pages, and unnecessary parameterized URLs have been reduced.

If you're managing a large enterprise ecommerce site (500k+ URLs), platforms like Botify, Lumar, or OnCrawl paired with log analysis are generally the most comprehensive. For small to mid-sized stores, Screaming Frog combined with Sitebulb and Google Search Console provides excellent coverage at a much lower cost.

For ecommerce sites, the goal is to help search engines discover valuable pages efficiently while preventing them from spending crawl budget on low-value URL combinations (such as endless filter permutations). No single tool solves this—most SEO teams combine crawling, log analysis, and technical SEO platforms.

Here's a breakdown of the most useful categories.

Best tools for internal linking

Screaming Frog

• Finds orphan pages (when connected to analytics or Search Console)
• Measures crawl depth
• Identifies pages with few internal links
• Visualizes crawl architecture
• Detects broken pagination

Excellent for stores with a few thousand to hundreds of thousands of URLs.

Sitebulb

Especially good for visualizing:

• Internal PageRank flow
• Link equity distribution
• Weak category structures
• Pages buried too deep
• Navigation issues

Many ecommerce SEOs find its visual reports easier to interpret than raw crawl exports.

Botify / Oncrawl / Lumar

These shine on very large ecommerce sites (100k+ URLs).

They combine:

• Crawl data
• Log files
• Search Console
• Internal linking
• Indexability

This allows you to answer questions like:

• Which filtered URLs does Google crawl most?
• Which category pages never receive Googlebot visits?
• Are important products too deep in the architecture?

Best tools for faceted navigation

Google Search Console

Monitor:

• Unexpected indexed parameter URLs
• Crawled but not indexed pages
• Duplicate content patterns
• Soft 404s
• Parameter-generated URLs

The Crawl Stats report can also reveal spikes in crawling caused by faceted navigation.

Screaming Frog

Useful for identifying:

• Infinite crawl paths
• Parameter explosions
• Duplicate titles
• Duplicate canonicals
• URL parameters
• Filter combinations

You can configure it to crawl JavaScript so you can observe filter behavior as Google may see it.

Log file analyzers

This is often the most revealing approach.

Instead of asking "What URLs exist?" you ask:

"What URLs is Googlebot actually crawling?"

Examples:

• color=red&size=10
• sort=price_desc
• brand=nike&material=leather

You can quickly determine whether Googlebot is wasting thousands of requests on filter combinations.

Helpful visualization tools

Some enterprise crawlers include crawl visualizations, but for custom analysis you can export crawl data into:

• Gephi for graph/network visualization
• Tableau for dashboards
• Looker Studio for Search Console and crawl reporting

These help identify isolated sections, deep pages, and weak internal linking patterns.

Common recommendations for ecommerce sites

Regardless of the tool, you'll typically want to:

• Ensure important category and subcategory pages are reachable within a few clicks from the homepage.
• Reduce unnecessary crawl paths created by filters, sorts, session IDs, and search-result URLs.
• Use canonical tags where appropriate to consolidate duplicate or near-duplicate faceted URLs.
• Prevent indexing of low-value filter combinations while keeping useful landing pages indexable.
• Regularly review server logs to verify that Googlebot is spending time on high-value categories and products rather than parameter-generated URLs.

If you can only choose three tools

For most ecommerce teams, a strong combination is:

1. Google Search Console for index coverage, crawl statistics, and URL inspection.
2. Screaming Frog SEO Spider for technical crawling, internal link analysis, and faceted navigation audits.
3. Oncrawl or Botify if you have access to server logs and manage a large ecommerce catalog, since combining crawl data with log analysis provides the clearest view of crawl waste and optimization opportunities.