ChatGPT answers, unedited

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. TechRadar Search Engine Land

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. Search Engine Land

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. IndexProbe

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. Ahrefs

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. IndexProbe Search Engine Land

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

Yes. For product pages with lots of variants, parameters, and canonical tags, I’d use a combination rather than one tool:

For an ecommerce site, I'd structure the investigation like this

1. Crawl every variant URL

Use Screaming Frog or Sitebulb with your XML sitemaps plus discovered URLs. Segment URLs by things like:

• /product/red-shirt
• /product/red-shirt?size=M
• /product/red-shirt?color=red
• /product/red-shirt?variant=12345
• /product/red-shirt?utm_source=...

Then compare:

URL → status code → indexability → meta robots → canonical → canonical target → internal links → sitemap inclusion

Sitebulb is particularly useful here because it explicitly reports canonical targets that are themselves noindex, redirected, broken, disallowed, or canonicalized elsewhere. support.sitebulb.com

2. Compare your canonical strategy against Google's

This is the part crawlers can't fully replace.

Take representative URLs from each variant family and inspect them in Search Console. Compare:

rel=canonical declared by site
vs.
Google's selected canonical

Google's URL Inspection tool specifically exposes the Google-selected canonical for the indexed version. support.google.com

If you find lots of:

Declared canonical: /product/widget
Google canonical: /product/widget?color=blue

or the reverse, that's a strong signal to investigate duplicate/content and internal-linking signals.

3. Look at Googlebot's actual behavior

If the site is large enough for crawl-budget problems, add JetOctopus/log analysis. You want to answer questions such as:

• What percentage of Googlebot requests hit variant URLs?
• Are parameter URLs consuming disproportionate crawl activity?
• Are canonicalized URLs still being crawled heavily?
• Are important canonical product URLs being crawled?
• Are Googlebot requests hitting URLs that shouldn't be indexable?

That tells you whether the problem is merely indexation configuration or an actual crawl-budget/architecture problem. jetoctopus.com

My preferred stack

For a large ecommerce catalog, I'd use:

Screaming Frog → Search Console → server logs/JetOctopus

Screaming Frog gives you the technical URL universe, GSC tells you what Google actually decided, and logs tell you what Googlebot is actually spending time crawling.

If you're dealing with millions of product/variant URLs, I'd additionally build a database/BigQuery-style comparison that automatically flags:

variant URL → canonical → canonical status → GSC status → Google-selected canonical → crawl frequency

That is much more powerful than relying on a generic "SEO audit score."

For large ecommerce sites, I’d use a stack that combines crawl diagnostics + Google’s own indexing data + log analysis:

• Google Search Console — best for confirming whether variant URLs are indexed, canonicalized elsewhere, excluded, or affected by duplicate/alternate-page signals.
• Screaming Frog SEO Spider — excellent for crawling product/variant URL patterns at scale and auditing canonical tags, hreflang, status codes, noindex, pagination, internal links, and duplicate content.
• Sitebulb — particularly good for visualizing canonical/indexability relationships and finding patterns across large product catalogs.
• Semrush Site Audit — useful for automated recurring crawls and surfacing canonical, duplicate, redirect, and indexability problems.
• Ahrefs Site Audit — similar recurring-crawl capability, with strong internal-link and canonical diagnostics.
• JetOctopus / Oncrawl — worth considering for very large catalogs because they combine large-scale crawling with server-log analysis, helping distinguish what search engines can crawl from what Googlebot actually crawls.
• Google URL Inspection API — useful when you need programmatic inspection of selected URLs, though it isn't designed to inspect an entire catalog indiscriminately.

For product variants specifically

The important thing is to model the relationship between:

variant URL → canonical URL → indexability → internal links → sitemap inclusion → Google-selected canonical

For example, you want to catch situations such as:

• /product?color=red canonical → /product
• /product?color=blue canonical → /product
• but the blue variant is independently linked throughout the site
• variant URLs appear in XML sitemaps despite being canonicalized elsewhere
• canonical points to a URL that redirects, 404s, or is noindex
• conflicting canonicals between HTML, HTTP headers, and sitemap signals
• parameter combinations generating thousands of crawlable near-duplicates
• Google selecting a different canonical than the one declared
• faceted/variant URLs consuming crawl budget without providing unique search value

If you're dealing with hundreds of thousands or millions of URLs, I'd lean toward Screaming Frog/Sitebulb + Search Console + server logs, or JetOctopus/Oncrawl when log analysis is central.

For product pages with lots of variants—size/color parameters, variant URLs, faceted URLs, etc.—I’d use a combination of a crawler + Google’s own indexing data.

For your specific use case

I'd prioritize Screaming Frog + Search Console.

A useful crawl should let you segment URLs into something like:

/product/widget
/product/widget?color=red
/product/widget?color=blue
/product/widget?size=large
/product/widget?color=red&size=large

Then compare:

• HTTP status
• indexability
• robots meta
• canonical target
• canonical target's status/indexability
• hreflang, if applicable
• internal links
• XML sitemap inclusion
• rendered vs raw canonical
• duplicate-content similarity

The important distinction is that a crawler tells you what your site is declaring, while Search Console tells you what Google actually did with those declarations. Sitebulb, for example, explicitly exposes canonical targets that are malformed, noindex, 404, or disallowed, and can compare raw HTML against rendered canonical tags. support.sitebulb.comsupport.sitebulb.com

The particularly nasty variant problems to look for

For an ecommerce site, I'd build reports specifically for:

1. Variant URL is indexable but should canonicalize to parent
2. Variant URL canonicalizes to parent, but Google selects the variant anyway
3. Canonical points to a variant instead of the primary product
4. Canonical points through a redirect
5. Canonical points to a noindex/404 URL
6. Canonical chains — A → B → C
7. Different variants have conflicting canonicals
8. Variant URLs appear in XML sitemaps
9. Internal links predominantly point to non-canonical variants
10. JS changes the canonical between initial HTML and rendered HTML
11. Faceted/filter URLs accidentally become indexable
12. Products with genuinely unique variant content are being canonicalized away

If you're dealing with hundreds of thousands/millions of product-variant URLs, I'd go one step further: export the crawl data and build a URL-level ruleset that automatically flags these conditions. That's much more scalable than manually inspecting Search Console.

For variant-heavy ecommerce sites, the best tool stack is usually a crawler + Google’s own index data + log analysis:

• Screaming Frog — excellent for crawling product/variant URLs at scale. Check canonical targets, canonical chains, noindex, redirects, duplicate/near-duplicate pages, hreflang, and whether variant URLs are internally linked.
• Sitebulb — particularly useful for visualizing canonical/indexation relationships and finding patterns across large sets of product variants.
• Google Search Console — essential for comparing declared canonical vs. Google-selected canonical and investigating “Duplicate, Google chose a different canonical” and similar indexing states.
• Google Search Console URL Inspection API — useful if you need to automate URL-level inspection across a large sample of variants rather than checking URLs manually.
• Ahrefs or Semrush — useful for discovering indexed/linked variant URLs and monitoring organic visibility, though they shouldn't replace Search Console for Google's actual indexing decisions.
• Server-log analysis — tools such as Screaming Frog Log File Analyser can reveal whether Googlebot is spending crawl budget on parameterized variants, out-of-stock URLs, filtered URLs, or other pages you don't want indexed.

For your specific problem

I'd build a scalable audit around these checks:

The key distinction: a crawler tells you what your site declares, while Search Console tells you what Google actually chose. For canonical/indexation problems, you want both.

If you're dealing with tens or hundreds of thousands of variant URLs, I can also lay out a specific automated audit architecture (crawl → canonical clustering → GSC inspection → anomaly scoring → dashboard) and recommend which tools/APIs to use at each stage.

For large ecommerce catalogs with product variants, canonical tags, faceted URLs, and tens of thousands (or millions) of URLs, you'll typically need a combination of crawling, indexation, and log-analysis tools. No single tool catches everything.

Here's a practical stack.

1. Google Search Console (the ground truth)

Search Console is still the authoritative source for understanding how Google interprets your canonicalization.

Pay particular attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical tag
• Crawled – currently not indexed
• Discovered – currently not indexed

The URL Inspection tool also reports both:

• User-declared canonical
• Google-selected canonical

These are often different, which is the first sign your canonical strategy isn't working as intended. Google's documentation explains common reasons for this and how Google evaluates canonical signals. developers.google.com

2. Screaming Frog

For variant-heavy ecommerce sites, configure it to crawl:

• parameter URLs
• pagination
• faceted navigation
• XML sitemaps

Useful reports include:

• Canonical Errors
• Canonical Chains
• Multiple Canonicals
• Missing Canonicals
• Canonical to Redirect
• Non-indexable Canonical Targets

You can also compare:

• Canonical target
• Indexability
• Status code
• Robots directives

across every product variant.

3. Sitebulb

Sitebulb is particularly good for visualizing duplicate clusters.

It highlights situations like:

• 20 color variants all canonicalizing differently
• variant pages linking to each other incorrectly
• self-referencing canonicals mixed with primary-product canonicals
• parameter pages accidentally indexable

Many ecommerce SEO teams find these visual cluster reports easier to interpret than raw crawler exports.

4. Enterprise platforms (Botify, Oncrawl, Lumar)

Once you're managing hundreds of thousands or millions of URLs, crawl data alone isn't enough.

These tools combine:

• crawler data
• server log files
• Search Console
• sitemap analysis

This lets you answer questions like:

• Which variant URLs is Googlebot actually crawling?
• Which canonical URLs never get crawled?
• Are filter pages consuming crawl budget?
• Which products receive repeated crawl attempts but remain unindexed?

That's difficult to determine with standard crawlers alone.

5. Log file analysis

For very large catalogs, log analysis is invaluable.

Tools such as Botify and Oncrawl can reveal:

• excessive crawling of ?color=
• repeated crawling of discontinued variants
• ignored canonical targets
• crawl budget wasted on parameter URLs

This is especially helpful when a crawler says everything looks correct but Googlebot behaves differently.

6. Bulk URL inspection

If you need to verify Google's indexation status across thousands of URLs, bulk inspection tools built on the Search Console API can surface:

• indexed vs. non-indexed pages
• Google's exclusion reason
• last crawl date
• canonical-related exclusions

For example, urlIndexProbeturn0search3 is designed specifically for large-scale Search Console URL inspection. www.indexprobe.com

A scalable workflow for ecommerce

For catalogs with 100k+ products, a common workflow is:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export all product URLs, including variants and parameter URLs.
3. Compare declared canonical vs. actual canonical target.
4. Cross-reference with Search Console indexing reports.
5. Review Google-selected canonicals for mismatches.
6. Use log analysis to identify which variants Googlebot continues to crawl.
7. Set up recurring audits to detect new canonical conflicts after releases.

This combination catches issues such as:

• variant URLs being indexed instead of the primary product
• self-canonicals on duplicate variants
• canonicals pointing to redirects or 404s
• inconsistent internal linking that weakens canonical signals
• parameter URLs leaking into the index
• products omitted from XML sitemaps
• Google ignoring declared canonicals because pages differ too much—or not enough—to be clustered as intended. developers.google.com

For most organizations, Search Console + Screaming Frog is sufficient up to roughly 100,000 URLs. Above that, adding an enterprise platform such as Botify, Oncrawl, or Lumar provides much better visibility into crawl behavior and indexation at scale.

For product pages with variants, duplicate URLs, canonical tags, and large catalogs, you usually need a mix of technical SEO crawlers, Google tools, log analysis, and specialized ecommerce SEO platforms. The best tools depend on whether you need diagnosis, monitoring, or automated fixes.

1. Enterprise crawlers (best for finding canonical/indexation issues at scale)

Screaming Frog SEO Spider

Good for:

• Detecting:
  • Canonical tag mismatches
  • Canonical chains
  • Canonicalized pages still in XML sitemaps
  • Duplicate product URLs
  • Near-duplicate variant pages
  • Incorrect status codes
  • Pagination/faceted navigation issues
• Handles millions of URLs with database storage mode

Useful reports:

• Canonical Errors
• Duplicate Pages
• “Non-indexable URLs in Sitemap”
• “Canonicalised URLs”
• “Pages with missing canonicals”

Screaming Frog SEO Spider

Sitebulb

Good for:

• Visualization of indexation problems
• Internal linking analysis
• Crawl maps showing:
  • Variant pages competing with parent products
  • Canonical clusters
  • Orphaned products

Useful for communicating issues to developers/merchandising teams.

Sitebulb

DeepCrawl / Lumar

Better for enterprise ecommerce sites:

• Millions+ URLs
• Scheduled crawls
• Trend monitoring
• JavaScript rendering
• Custom extraction rules

Useful checks:

• “Is Google able to discover every variant?”
• “Are canonical tags stable over time?”
• “Did a template deployment change indexability?”

Lumar

2. Google-owned tools (must-have)

Google Search Console

Best source of truth for what Google actually indexed.

Check:

• Page indexing report
  • Crawled — currently not indexed
  • Duplicate without user-selected canonical
  • Google chose a different canonical
  • Excluded by noindex
• URL Inspection API
  • Test samples of variants
  • Compare declared vs Google-selected canonical

Google Search Console

Google Search Console API

Useful for scale:

• Pull indexing coverage data
• Monitor thousands of products
• Build dashboards in:
  • BigQuery
  • Looker Studio
  • Data warehouse systems

Google Search Console API documentation

3. Log analysis tools (best for knowing what Googlebot does)

For large ecommerce sites, crawl behavior often reveals problems before rankings drop.

Oncrawl

Good for:

• Googlebot crawl analysis
• Crawl budget issues
• Product URL discovery
• Comparing:
  • URLs crawled by Googlebot
  • URLs indexed
  • URLs generating traffic

Oncrawl

Botify

Enterprise-level:

• SEO log analysis
• Crawl budget optimization
• Indexation monitoring
• Large catalog diagnostics

Useful questions:

• Is Google wasting crawl budget on color/size variants?
• Are canonicalized variants consuming crawl resources?
• Are high-value products crawled often enough?

Botify

4. Ecommerce-specific SEO platforms

SearchPilot

Useful if you need:

• Testing canonical changes
• Measuring impact of template changes
• SEO experimentation

SearchPilot

Semrush / Ahrefs

Better for:

• Monitoring organic visibility
• Finding lost product rankings
• Spotting duplicate URL competition

Less powerful for deep canonical debugging than crawlers.

Semrush
Ahrefs

5. Custom checks you can build for variant-heavy catalogs

For serious ecommerce sites, I’d usually automate these checks:

A practical stack by site size

<100k product URLs

• Screaming Frog
• Google Search Console
• Sitebulb

100k–5M product URLs

• Lumar or Oncrawl
• Search Console API
• Log analysis

5M+ URLs / marketplaces

• Botify or Oncrawl
• BigQuery pipelines
• Custom canonical/indexation monitoring

For variant-heavy ecommerce, the biggest signal to watch is usually “Google chose a different canonical than the one you declared” plus “Crawled — currently not indexed” at scale. Those two reports often reveal whether your variant strategy is helping Google consolidate products or creating index bloat.

For large ecommerce sites with product variants (size, color, material, region) and canonical tags, no single tool catches every issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should always be your source of truth because it shows what Google actually indexed rather than what your site outputs.

Look closely at:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical tag
• Crawled – currently not indexed
• Discovered – currently not indexed

The URL Inspection tool also shows:

• User-declared canonical
• Google-selected canonical

If these differ, Google is ignoring your canonical for that URL. Google's documentation explains common reasons, including insufficient differentiation between variants and conflicting signals. developers.google.comwww.indexprobe.com

2. Screaming Frog

For catalogs under roughly 500k–1M URLs (or segmented crawls), Screaming Frog is still one of the best options.

Useful reports include:

• Canonical errors
• Canonical chains
• Canonical to redirects
• Multiple canonicals
• Missing canonicals
• Non-indexable canonicals
• Parameterized URLs
• Duplicate content clusters

For product variants you can quickly answer questions like:

• Are all color variants self-canonical?
• Are variants pointing to the parent?
• Are discontinued variants still indexable?
• Are faceted URLs accidentally canonicalized?

3. Sitebulb

Sitebulb excels at visualization.

It flags:

• canonical loops
• conflicting canonicals
• duplicate clusters
• orphan variants
• pagination issues
• parameter explosions

Its visual reports are often easier to interpret than raw crawler exports.

4. Lumar / Deepcrawl

If you're working with hundreds of thousands or millions of product pages, Lumar is designed for this scale.

It monitors:

• canonical consistency
• template regressions
• variant indexing
• JavaScript-rendered canonicals
• crawl budget waste

This is especially useful for Shopify Plus, Adobe Commerce, Salesforce Commerce Cloud, and other enterprise ecommerce platforms.

5. Botify

Botify combines three valuable data sources:

• crawl data
• Google Search Console data
• server log files

This lets you answer questions like:

• Google crawled 80,000 blue-shirt variants.
• Only 5,000 were indexed.
• 60,000 pointed to parent canonicals.
• 15,000 had conflicting internal links.

That level of correlation is difficult to achieve with standalone crawlers.

6. Log File Analysis

If crawl budget is a concern, server logs are invaluable.

They reveal whether Googlebot is spending time on:

• every color parameter
• session IDs
• filter URLs
• out-of-stock variants
• discontinued products

You can then compare crawl frequency against indexation to identify wasted crawl activity.

7. Bulk URL Inspection Tools

For validating large URL sets against Google's own data, tools built on the Google Search Console API can inspect thousands of URLs in bulk and report statuses such as "Alternate page with proper canonical tag" or "Google chose different canonical." www.indexprobe.com

Metrics worth tracking

For product variants, monitor:

• % indexed variants
• % alternate canonicals
• Google-selected vs declared canonical mismatch
• Canonicals pointing to redirects
• Canonicals pointing to non-200 pages
• Variant pages with self-canonicals
• Parent pages missing self-canonicals
• Parameter URLs in sitemaps
• Duplicate title clusters
• Duplicate content clusters
• Crawl frequency by variant type

A recommended stack

If I were auditing a large ecommerce site with many product variants, I'd use:

1. Google Search Console for actual indexation status.
2. Screaming Frog to crawl templates and canonical implementation.
3. Botify or Lumar for enterprise-scale monitoring and trend analysis.
4. Server log analysis to understand Googlebot's crawl behavior.
5. Periodic bulk URL inspections to verify Google's chosen canonical matches your intended canonical. developers.google.comwww.indexprobe.com

This combination covers both implementation issues (what your site outputs) and search engine behavior (what Google actually crawls, indexes, and treats as canonical).

If you're managing thousands to millions of product URLs with color, size, or configuration variants, no single tool will catch every indexation issue. The most effective approach combines Google's own data with enterprise crawlers and log analysis.

Here's how I'd categorize the best options:

What to look for

For product variants, the most valuable reports identify:

• Canonical points to a 404 or redirect
• Canonical chains (A → B → C)
• Canonical loops
• Self-canonical missing
• Multiple canonicals on one page
• Parameter URLs being indexed (?color=, ?size=, ?variant=)
• Variant URLs receiving internal links when they shouldn't
• Sitemap contains non-canonical URLs
• Google-selected canonical differs from declared canonical
• "Duplicate without user-selected canonical"
• "Alternate page with proper canonical"

Google recommends using the URL Inspection tool to compare your preferred canonical with Google's selected canonical because Google may choose a different URL if it believes another version is more appropriate. developers.google.com

At enterprise scale (100k+ products)

The workflow many technical SEO teams use looks like this:

1. Crawler (Screaming Frog, Sitebulb, Lumar, Botify)
   • Crawl every product page
   • Export canonicals
   • Export indexability
   • Export hreflang
   • Export internal links

2. Google Search Console
   • Pull Indexing reports
   • Use the URL Inspection API in bulk
   • Compare Google's canonical vs. declared canonical

3. Server logs
   • Verify Googlebot is spending crawl budget on canonical URLs rather than thousands of low-value variant URLs.

4. BI or SQL
   • Join crawl data with Search Console and product catalog data to identify recurring issues by template, category, or brand.

Useful automations

For ecommerce sites, it's worth creating rules that automatically flag:

• Canonical URL ≠ HTTP 200
• Canonical URL contains tracking parameters
• Variant canonical points to another variant instead of the primary product (if that's not your intended strategy)
• Canonical not included in sitemap
• Sitemap includes non-canonical URLs
• Indexable page with canonical pointing elsewhere (mixed signals)
• Duplicate title + duplicate description + different canonical

If you're using Shopify, Magento, Adobe Commerce, or BigCommerce

These platforms often generate variant URLs automatically. Common issues include:

• ?variant= URLs being crawled
• Color/size URLs entering the sitemap unintentionally
• Theme updates overwriting canonical logic
• Faceted navigation creating millions of crawlable URLs

A crawler combined with Search Console is usually enough to detect these at scale, and enterprise platforms like Botify or Oncrawl become particularly valuable once catalogs reach several hundred thousand URLs.

If you can share your platform (Shopify, Adobe Commerce/Magento, Salesforce Commerce Cloud, WooCommerce, BigCommerce, or a custom stack) and roughly how many product URLs you have, I can recommend a workflow tailored to that setup.

For large ecommerce catalogs with product variants, canonical tags, and faceted URLs, you'll typically need a combination of four tool types rather than a single solution.

My recommended stack

1. Screaming Frog SEO Spider (best overall)

Ideal for sites up to several hundred thousand URLs.

Useful reports:

• Canonical Errors
• Canonicalized URLs
• Missing Canonical
• Multiple Canonicals
• Pagination issues
• Parameter URLs
• Duplicate content

Custom extraction lets you compare:

• URL
• rel="canonical"
• meta robots
• hreflang
• structured data
• product variant parameters

You can also compare:

• /product
• /product?color=red
• /product?size=xl

to ensure they all canonicalize correctly.

2. Google Search Console

This tells you what Google actually decided—not just what your HTML says.

Watch these reports closely:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical tag
• Crawled – currently not indexed
• Discovered – currently not indexed
• Indexed pages

For individual URLs, use URL Inspection to compare:

• User-declared canonical
• Google-selected canonical

Google recommends this workflow when troubleshooting canonicalization. developers.google.comvisrank.org

3. Sitebulb

Excellent visual reports for ecommerce.

It highlights:

• canonical loops
• canonical chains
• multiple canonicals
• inconsistent canonicals
• duplicate product variants
• faceted navigation problems

Many SEO teams find it easier to interpret than raw crawl exports.

4. JetOctopus

A strong option for catalogs with millions of URLs.

It combines:

• crawler
• server logs
• Google Search Console

This helps answer questions like:

• Is Google wasting crawl budget on variant URLs?
• Are filtered URLs being crawled heavily?
• Are canonicalized URLs still consuming crawl resources?

5. Lumar (formerly Deepcrawl)

Built for enterprise ecommerce.

It can detect:

• template-wide canonical mistakes
• product variant duplication
• pagination issues
• JavaScript-rendered canonicals
• inconsistent internal linking

Very useful if thousands of pages are generated from the same templates.

6. Botify

One of the best tools for understanding:

• crawl → index → traffic

It shows:

• URLs crawled
• URLs indexed
• URLs receiving impressions
• canonical clusters
• crawl budget waste

This is particularly valuable for very large catalogs.

For Shopify stores

If you're working with Shopify variants, verify:

• Variant URLs (?variant=) point to the intended canonical.
• Collection URLs don't become accidental canonicals.
• Parameter URLs aren't ending up in XML sitemaps.
• Product schema aligns with your canonical strategy. developers.google.comvisrank.org

At very large scale (100k+ products)

Many SEO teams automate audits with exports from Screaming Frog or Sitebulb into a data warehouse or spreadsheet and flag patterns such as:

• Canonical ≠ self URL
• Canonical returns non-200 status
• Canonical points to a redirect
• Canonical chains
• Multiple canonicals
• Missing canonicals
• Canonical conflicts with noindex
• Canonical targets outside the sitemap
• Variant URLs receiving organic traffic despite canonicalization
• Google-selected canonical differing from the declared canonical

A practical "best-in-class" stack

• Free: Google Search Console + Screaming Frog
• Mid-market: Sitebulb + JetOctopus
• Enterprise: Botify or Lumar, supplemented with server log analysis

This combination gives you visibility into all three critical layers: what your site publishes (crawl), what Google chooses to index (Search Console), and how Googlebot actually behaves (logs).