ChatGPT answers, unedited

For large ecommerce catalogs with product variants, parameter URLs, and canonical tags, I’d use a combination rather than one tool:

• screamingfrog.co.uk — excellent for deep technical crawling. You can crawl variant/parameter URLs, inspect rel=canonical, robots directives, status codes, redirects, hreflang, and internal links. It’s particularly useful when you want to build custom extraction rules around your variant URL patterns.
• Sitebulb — especially good for diagnosing why URLs aren't indexable. Its indexability reporting identifies canonical conflicts, canonicals pointing to noindex/404/disallowed URLs, canonical chains/loops, parameterized URLs, and JS differences between raw and rendered HTML. Sitebulb Sitebulb Support
• ahrefs.com — useful for recurring/automated audits. It crawls URLs at scale and reports indexability, duplicates, redirects, JavaScript, robots, and sitemap problems. Ahrefs
• semrush.com — another good automated monitoring layer for canonicalization, duplicate content, redirects and indexability issues. Semrush
• Google Search Console — essential because crawlers tell you what your site says, whereas Search Console can show Google's view. URL Inspection exposes Google's selected canonical versus your declared canonical. Google explicitly notes that its chosen canonical can differ from the site's declared canonical. Google for Developers

For variants specifically

The key is to classify URLs into groups such as:

/product/shoe
/product/shoe?color=red
/product/shoe?color=blue
/product/shoe?size=10
/product/shoe?variant=12345

Then automatically flag patterns like:

Google's own documentation specifically treats sorting/filtering URLs and accidental URL variants as canonicalization cases, and recommends checking Google's selected canonical rather than assuming the declared one is being honored. Google for Developers

If you're talking hundreds of thousands/millions of URLs

I'd structure the stack as:

Crawler: Screaming Frog or Sitebulb
→ Search-engine reality: Google Search Console
→ Recurring monitoring: Ahrefs/Semrush
→ Very large enterprise catalog: add server-log analysis so you can see which variant URLs Googlebot is actually requesting.

The particularly useful output is a URL-level dataset, e.g.:

URL
HTTP status
Canonical declared
Canonical target status
Meta robots
Indexability
Google-selected canonical
URL parameters
Variant ID
Product ID
Internal inlinks
Sitemap presence
Crawl frequency

Then you can group by product ID / variant family and find systemic problems instead of manually inspecting individual PDPs.

One important distinction: a crawler can prove that your canonical implementation is wrong; Search Console is needed to determine whether Google actually selected a different canonical. Google's documentation explicitly says those can differ. Google for Developers

If you tell me whether this is Shopify, Magento, Salesforce Commerce Cloud, WooCommerce, or a custom ecommerce platform, I can give you a concrete at-scale audit workflow and the exact reports/filters to build.

For variant-heavy ecommerce sites, I’d use a stack rather than one tool. The key is separating:

1. What your site declares — canonicals, noindex, internal links, XML sitemaps.
2. What Google actually indexes — Google-selected canonical and index status.
3. Whether variant URLs are creating crawl/indexation bloat.

Tools I’d use

For product variants specifically

Suppose you have:

/product/shoe
/product/shoe?color=red
/product/shoe?color=blue
/product/shoe?size=10
/product/shoe?color=red&size=10

I'd configure Screaming Frog/JetOctopus to crawl the parameterized URLs and build a dataset containing:

URL
HTTP status
Indexability
noindex
User canonical
Google-selected canonical
Canonical status code
Canonical indexability
Internal links
Inlinks to canonical
Sitemap inclusion
URL parameters

Then look for patterns such as:

• Variant URL is indexable but canonicalizes to the parent product.
• Variant canonical points to another variant.
• Canonical points to a noindex URL.
• Canonical points through a redirect.
• Canonical URL returns 404/410.
• XML sitemap contains non-canonical variants.
• Internal links predominantly point to non-canonical variants.
• Google repeatedly selects a different canonical from the one you declared.
• Huge numbers of crawlable parameter combinations exist without search value.

The Google-selected canonical is especially important: a crawler can tell you that your <link rel="canonical"> is technically present, but only Google's indexed data tells you whether Google actually selected that URL. Search Console explicitly exposes both the user-declared and Google-selected canonical. support.google.com

My preferred setup

For a large ecommerce catalog, I'd use:

JetOctopus + Screaming Frog + Search Console + server logs

• JetOctopus: scale/crawl-budget/large URL universe
• Screaming Frog: detailed technical QA and repeatable crawls
• Search Console: Google's actual indexing/canonical decisions
• Server logs: what Googlebot is actually spending crawl resources on

And I'd keep Semrush or Ahrefs as the broader SEO monitoring layer rather than relying on them as the primary canonical diagnostic.

One important point: don't aim for 100% of variant URLs to be indexed. Google itself says the objective is generally to have the canonical versions of important pages indexed; duplicate/alternate URLs often being excluded is expected behavior. support.google.comsupport.google.com

If you tell me whether you're dealing with Shopify, Magento, BigCommerce, WooCommerce, or a custom platform, I can give you a specific variant/canonical audit workflow and the exact crawl filters to use.

For product pages with variants + canonical tags at scale, I’d use a combination rather than one tool:

• Google Search Console — the source of truth for what Google actually indexed. URL Inspection shows the indexed URL, indexability, and Google’s canonical/duplicate interpretation. The URL Inspection API lets you pull this programmatically for large URL sets, although Google imposes inspection quotas. Google Help Google for Developers
• Sitebulb — probably my top choice for diagnosing the site-wide pattern. It can bulk-audit self vs. non-self canonicals, canonical chains/loops, canonicals pointing to noindex/404/redirect URLs, parameterized URLs, robots directives, sitemaps, and duplicate content. It also compares response HTML vs. rendered HTML, which is particularly useful if your ecommerce platform changes canonicals with JavaScript. Sitebulb Support Sitebulb Support
• Screaming Frog SEO Spider — excellent for very granular crawl exports and custom extraction. I'd use it when you want to build a dataset such as variant URL → status → canonical → indexability → inlinks → sitemap inclusion and analyze it externally.
• Semrush Site Audit — useful for automated recurring monitoring. It checks crawlability, indexability, sitemap coverage and canonical tags, so it's good for catching regressions after ecommerce/template releases. Semrush
• A custom GSC + crawl-data pipeline — best when you have hundreds of thousands/millions of SKU/variant URLs. Combine a crawler's technical signals with GSC's actual indexing data rather than assuming that a 200 + self-canonical URL is necessarily indexed.

What I'd specifically look for with variants

Build a matrix something like:

The interesting problems are mismatches, not simply "pages with canonicals."

For example:

1. Variant has a canonical to the parent, but Google selects the variant URL anyway.
2. Canonical points to a URL that is itself canonicalized — a canonical chain.
3. Canonical points to a noindex, 404, redirect, or robots-blocked URL.
4. Variant URLs are accidentally in XML sitemaps despite being canonicalized.
5. Important variants are canonicalized away but are the only URLs receiving internal links.
6. JavaScript changes the canonical after the initial HTML response.
7. Different variants unexpectedly have different canonical targets.
8. Your faceted/filter parameters generate millions of crawlable URLs that weren't intended to be indexable.

Sitebulb is particularly strong for several of these canonical relationships and rendered-vs-response discrepancies. Sitebulb Support Sitebulb Support

If I were auditing a large ecommerce site, my stack would be: Screaming Frog/Sitebulb → GSC URL Inspection API → BigQuery/SQL for anomaly detection. The crawler tells you what your site says; GSC tells you what Google actually did with it. That's the combination that catches the subtle variant-indexation problems.

For product pages with lots of variants, I’d use a stack rather than one tool. The key is separating crawl/indexability problems from Google’s actual canonical/indexing decisions.

Best tools at scale

The important distinction: a crawler can tell you that /shirt?color=red has a canonical pointing to /shirt, but only Google's index data can tell you whether Google actually chose /shirt as canonical. Search Console exposes both the user-declared and Google-selected canonical. support.google.comdevelopers.google.com

For a large variant catalog, I'd build this workflow

1. Crawl the entire site with Screaming Frog/Sitebulb

Export at least:

• URL
• HTTP status
• indexability
• noindex
• canonical URL
• canonical status
• title/content hash
• internal inlinks
• sitemap inclusion
• hreflang, if applicable
• URL parameters
• redirect target

Then flag patterns such as:

Variant URL → canonical product URL       ✓ expected
Variant URL → different product          ✗
Variant URL → another variant             ✗
Self-canonical variant → intentional?     ?
200 indexable page → canonical elsewhere  ?
Canonical → 3xx/4xx/noindex URL            ✗
Sitemap URL ≠ canonical URL                ✗
Internal links → noncanonical variants     ⚠

Google specifically recommends consistency between internal links, sitemaps and canonical tags, and recommends self-referencing canonicals on indexable pages. For unique variant URLs, Google also documents using the canonical product URL on variant pages when appropriate. developers.google.com

2. Sample/segment URLs through Search Console

Don't try to inspect every URL manually. Group URLs by:

• product
• variant parameter
• template
• category
• inventory state
• canonical target
• crawl/index status

Then use the URL Inspection API for representative or high-value URLs in each group. Its response includes coverageState, indexingState, googleCanonical, userCanonical, robotsTxtState, lastCrawlTime, sitemap information and referring URLs. support.google.comdevelopers.google.com

The API currently has a quota of 2,000 inspection queries/day per property and 600/minute, so for enormous catalogs you need prioritization rather than blindly inspecting every SKU every day. developers.google.com

The highest-value automated check

I'd create a table like:

product_id
variant_url
variant_type
declared_canonical
google_canonical
index_status
robots_status
sitemap_status
internal_link_count
http_status
content_hash

Then generate alerts for:

Canonical disagreement

declared_canonical != google_canonical

Unexpected indexing

variant should be canonicalized
BUT
Google indexed variant

Missing indexing

variant/product should be indexable
BUT
Google says "Crawled - currently not indexed"

Canonical contamination

canonical → URL that is noindex / redirected / 404

Architecture problem

noncanonical variants receive substantial internal links

That combination is much more useful than simply running a "canonical errors" report.

One particularly important variant issue

If you're using query parameters such as:

/product/shirt?color=red

Google's current ecommerce guidance says that if the parameter is merely being used to identify a variant, you can use the parameter-free URL as the canonical, while giving each variant a separate URL when you want Google to understand the variants. developers.google.com

So I'd make the variant URL strategy explicit first, then configure the crawler to test deviations from that strategy.

If I were choosing only three tools: Screaming Frog + Google Search Console/API + Sitebulb. The crawler finds the problems, GSC tells you what Google actually did, and Sitebulb makes the patterns much easier to investigate.

For product pages with lots of variants—size, color, SKU, URL parameters, etc.—I’d use a combination of a crawler + Google’s own index data.

Best tool stack

• Screaming Frog SEO Spider — probably the best first-line tool for this. It can crawl canonical tags and HTTP-header canonicals at scale, flag missing/multiple/non-indexable canonicals, and compare raw HTML vs rendered HTML. That last part is especially useful if your ecommerce platform sets canonicals through JavaScript. Screaming Frog Screaming Frog Screaming Frog

screamingfrog.co.uk

• Google Search Console — essential for determining what Google actually did rather than merely what your HTML says. URL Inspection exposes both your user-declared canonical and Google's selected canonical. The Page Indexing report shows patterns such as duplicate URLs, "crawled—currently not indexed," and other exclusions. Google Help Google Help

search.google.com

• Sitebulb — useful when you want a more diagnostic crawl/audit interface rather than raw crawler exports. I'd particularly consider it for segmenting variant URL patterns and finding indexability/canonical problems across templates.
• Semrush Site Audit / **Ahrefs Site Audit — good for monitoring recurring technical problems and trends across a large ecommerce site. They're less useful than GSC for answering the crucial question, "Did Google choose the canonical I intended?"

For variants specifically

I'd build a crawl that groups URLs by the product/variant family, then checks:

The raw-vs-rendered canonical check is particularly valuable: Screaming Frog explicitly flags situations where JavaScript changes the canonical between the original and rendered HTML. Screaming Frog Screaming Frog

The important distinction

A crawler can tell you:

"This variant declares canonical /blue-shirt/."

GSC can tell you:

"Google actually selected /blue-shirt/?size=large instead."

That distinction is critical. Google says its selected canonical can differ from the user-declared canonical, and the live URL test cannot reliably predict canonical selection because that happens during indexing. Google Help

So for a large ecommerce catalog, my preferred workflow is:

Screaming Frog → identify every variant/canonical pattern → GSC Page Indexing → sample/segment with URL Inspection → fix template-level problems → monitor the pattern.

If you're dealing with hundreds of thousands or millions of product/variant URLs, I can also lay out a scalable audit architecture (crawl → URL normalization → variant clustering → canonical conflict detection → GSC validation) and the exact columns/rules I'd put in the dataset.

For large ecommerce catalogs, no single tool catches every variant/canonical issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should be your source of truth because it shows what Google actually decided.

Key reports:

• Alternate page with proper canonical tag
• Duplicate, Google chose different canonical than user
• Crawled – currently not indexed
• Discovered – currently not indexed
• Submitted URL not selected as canonical

For product variants, compare:

• canonical declared by your site
• canonical selected by Google
• sitemap inclusion
• impressions/clicks

If Google consistently ignores your canonical, that's often a sign that the variant pages differ more than expected—or that the canonical target is weak. Reddit IndexProbe

2. Screaming Frog SEO Spider

Still one of the best technical crawlers.

Useful features:

• Canonical extraction
• Canonical chains
• Canonical loops
• Self-referencing canonical validation
• Duplicate content detection
• Parameter URL crawling
• Custom extraction
• JavaScript rendering
• Search Console integration

For ecommerce, you can crawl:

/products/*

and compare

• canonical URL
• status code
• indexability
• robots directives
• selected variant URL

This quickly identifies templates producing incorrect canonicals. TechRadar

3. Sitebulb

Excellent visual reporting for large ecommerce sites.

It highlights:

• duplicate clusters
• canonical conflicts
• orphan pages
• parameter problems
• pagination
• faceted navigation

Many teams find it easier to interpret than raw crawl exports.

4. Ahrefs Site Audit

Very good when you also want ongoing monitoring.

Checks include:

• duplicate pages
• canonical inconsistencies
• orphan products
• internal linking
• indexability
• crawl depth

Useful if your catalog changes daily.

5. Semrush Site Audit

Strong for enterprise monitoring.

Especially useful for:

• scheduled crawls
• canonical errors
• duplicate titles
• duplicate descriptions
• thin product pages
• sitemap inconsistencies

6. JetOctopus

One of the strongest tools for very large ecommerce sites.

Strengths:

• crawl + log analysis
• Googlebot behavior
• crawl budget visualization
• canonical clusters
• indexability segmentation
• JavaScript rendering

If you have 500k+ products, this is a major step up from desktop crawlers.

7. Botify

Designed for enterprise ecommerce.

Combines:

• crawl data
• log files
• search performance
• indexing

Excellent for answering questions like:

• Which color variants receive Googlebot visits?
• Which variants never get crawled?
• Which canonical pages lose crawl budget?

8. Oncrawl

Another enterprise platform focused on technical SEO.

Useful reports:

• canonical mismatches
• duplicate clusters
• internal PageRank
• crawl depth
• orphan products
• log analysis

9. IndexProbe (or other bulk URL inspection tools)

These tools use the Google Search Console URL Inspection API to check indexing status at scale.

They help distinguish expected exclusions (for example, properly canonicalized variants) from genuine indexing problems by surfacing Google's official exclusion reasons. IndexProbe IndexProbe

Recommended workflow for variant pages

For catalogs with many product variants:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export: - URL
3. Canonical
4. Status code
5. Meta robots
6. Product template
7. Join this data with: - Google Search Console exports
8. XML sitemap
9. Product feed
10. Flag pages where: - Google chose a different canonical.
11. Canonical points to a non-200 page.
12. Variant canonicals point to another variant instead of the primary product.
13. Canonical URLs are missing from the sitemap.
14. Indexable pages contain noindex.
15. For very large catalogs (500k+ URLs), add log analysis to verify Googlebot isn't spending disproportionate crawl budget on parameterized or low-value variant URLs. Community experience with multi-million-product catalogs consistently emphasizes crawl efficiency, canonical rules, and sitemap quality over simply trying to increase indexing. Reddit Reddit

If you're managing 100k+ product pages

A strong stack would be:

• Google Search Console — Ground truth for indexation.
• Screaming Frog — Template-level and canonical QA.
• JetOctopus or Botify — Crawl budget and log analysis.
• IndexProbe (or another bulk URL inspection tool) — Validate Google's indexing decisions in bulk.

This combination gives visibility into three distinct layers: what your site publishes (crawl), what Google crawls (logs), and what Google actually indexes (Search Console/URL Inspection).

For large ecommerce catalogs, no single tool catches every variant/canonical issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should be your source of truth because it shows what Google actually decided.

Key reports:

• Alternate page with proper canonical tag
• Duplicate, Google chose different canonical than user
• Crawled – currently not indexed
• Discovered – currently not indexed
• Submitted URL not selected as canonical

For product variants, compare:

• canonical declared by your site
• canonical selected by Google
• sitemap inclusion
• impressions/clicks

If Google consistently ignores your canonical, that's often a sign that the variant pages differ more than expected—or that the canonical target is weak. Reddit IndexProbe

2. Screaming Frog SEO Spider

Still one of the best technical crawlers.

Useful features:

• Canonical extraction
• Canonical chains
• Canonical loops
• Self-referencing canonical validation
• Duplicate content detection
• Parameter URL crawling
• Custom extraction
• JavaScript rendering
• Search Console integration

For ecommerce, you can crawl:

/products/*

and compare

• canonical URL
• status code
• indexability
• robots directives
• selected variant URL

This quickly identifies templates producing incorrect canonicals. TechRadar

3. Sitebulb

Excellent visual reporting for large ecommerce sites.

It highlights:

• duplicate clusters
• canonical conflicts
• orphan pages
• parameter problems
• pagination
• faceted navigation

Many teams find it easier to interpret than raw crawl exports.

4. Ahrefs Site Audit

Very good when you also want ongoing monitoring.

Checks include:

• duplicate pages
• canonical inconsistencies
• orphan products
• internal linking
• indexability
• crawl depth

Useful if your catalog changes daily.

5. Semrush Site Audit

Strong for enterprise monitoring.

Especially useful for:

• scheduled crawls
• canonical errors
• duplicate titles
• duplicate descriptions
• thin product pages
• sitemap inconsistencies

6. JetOctopus

One of the strongest tools for very large ecommerce sites.

Strengths:

• crawl + log analysis
• Googlebot behavior
• crawl budget visualization
• canonical clusters
• indexability segmentation
• JavaScript rendering

If you have 500k+ products, this is a major step up from desktop crawlers.

7. Botify

Designed for enterprise ecommerce.

Combines:

• crawl data
• log files
• search performance
• indexing

Excellent for answering questions like:

• Which color variants receive Googlebot visits?
• Which variants never get crawled?
• Which canonical pages lose crawl budget?

8. Oncrawl

Another enterprise platform focused on technical SEO.

Useful reports:

• canonical mismatches
• duplicate clusters
• internal PageRank
• crawl depth
• orphan products
• log analysis

9. IndexProbe (or other bulk URL inspection tools)

These tools use the Google Search Console URL Inspection API to check indexing status at scale.

They help distinguish expected exclusions (for example, properly canonicalized variants) from genuine indexing problems by surfacing Google's official exclusion reasons. IndexProbe IndexProbe

Recommended workflow for variant pages

For catalogs with many product variants:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export: - URL
3. Canonical
4. Status code
5. Meta robots
6. Product template
7. Join this data with: - Google Search Console exports
8. XML sitemap
9. Product feed
10. Flag pages where: - Google chose a different canonical.
11. Canonical points to a non-200 page.
12. Variant canonicals point to another variant instead of the primary product.
13. Canonical URLs are missing from the sitemap.
14. Indexable pages contain noindex.
15. For very large catalogs (500k+ URLs), add log analysis to verify Googlebot isn't spending disproportionate crawl budget on parameterized or low-value variant URLs. Community experience with multi-million-product catalogs consistently emphasizes crawl efficiency, canonical rules, and sitemap quality over simply trying to increase indexing. Reddit Reddit

If you're managing 100k+ product pages

A strong stack would be:

• Google Search Console — Ground truth for indexation.
• Screaming Frog — Template-level and canonical QA.
• JetOctopus or Botify — Crawl budget and log analysis.
• IndexProbe (or another bulk URL inspection tool) — Validate Google's indexing decisions in bulk.

This combination gives visibility into three distinct layers: what your site publishes (crawl), what Google crawls (logs), and what Google actually indexes (Search Console/URL Inspection).

For large ecommerce catalogs, no single tool catches every variant/canonical issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should be your source of truth because it shows what Google actually decided.

Key reports:

• Alternate page with proper canonical tag
• Duplicate, Google chose different canonical than user
• Crawled – currently not indexed
• Discovered – currently not indexed
• Submitted URL not selected as canonical

For product variants, compare:

• canonical declared by your site
• canonical selected by Google
• sitemap inclusion
• impressions/clicks

If Google consistently ignores your canonical, that's often a sign that the variant pages differ more than expected—or that the canonical target is weak. Reddit IndexProbe

2. Screaming Frog SEO Spider

Still one of the best technical crawlers.

Useful features:

• Canonical extraction
• Canonical chains
• Canonical loops
• Self-referencing canonical validation
• Duplicate content detection
• Parameter URL crawling
• Custom extraction
• JavaScript rendering
• Search Console integration

For ecommerce, you can crawl:

/products/*

and compare

• canonical URL
• status code
• indexability
• robots directives
• selected variant URL

This quickly identifies templates producing incorrect canonicals. TechRadar

3. Sitebulb

Excellent visual reporting for large ecommerce sites.

It highlights:

• duplicate clusters
• canonical conflicts
• orphan pages
• parameter problems
• pagination
• faceted navigation

Many teams find it easier to interpret than raw crawl exports.

4. Ahrefs Site Audit

Very good when you also want ongoing monitoring.

Checks include:

• duplicate pages
• canonical inconsistencies
• orphan products
• internal linking
• indexability
• crawl depth

Useful if your catalog changes daily.

5. Semrush Site Audit

Strong for enterprise monitoring.

Especially useful for:

• scheduled crawls
• canonical errors
• duplicate titles
• duplicate descriptions
• thin product pages
• sitemap inconsistencies

6. JetOctopus

One of the strongest tools for very large ecommerce sites.

Strengths:

• crawl + log analysis
• Googlebot behavior
• crawl budget visualization
• canonical clusters
• indexability segmentation
• JavaScript rendering

If you have 500k+ products, this is a major step up from desktop crawlers.

7. Botify

Designed for enterprise ecommerce.

Combines:

• crawl data
• log files
• search performance
• indexing

Excellent for answering questions like:

• Which color variants receive Googlebot visits?
• Which variants never get crawled?
• Which canonical pages lose crawl budget?

8. Oncrawl

Another enterprise platform focused on technical SEO.

Useful reports:

• canonical mismatches
• duplicate clusters
• internal PageRank
• crawl depth
• orphan products
• log analysis

9. IndexProbe (or other bulk URL inspection tools)

These tools use the Google Search Console URL Inspection API to check indexing status at scale.

They help distinguish expected exclusions (for example, properly canonicalized variants) from genuine indexing problems by surfacing Google's official exclusion reasons. IndexProbe IndexProbe

Recommended workflow for variant pages

For catalogs with many product variants:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export: - URL
3. Canonical
4. Status code
5. Meta robots
6. Product template
7. Join this data with: - Google Search Console exports
8. XML sitemap
9. Product feed
10. Flag pages where: - Google chose a different canonical.
11. Canonical points to a non-200 page.
12. Variant canonicals point to another variant instead of the primary product.
13. Canonical URLs are missing from the sitemap.
14. Indexable pages contain noindex.
15. For very large catalogs (500k+ URLs), add log analysis to verify Googlebot isn't spending disproportionate crawl budget on parameterized or low-value variant URLs. Community experience with multi-million-product catalogs consistently emphasizes crawl efficiency, canonical rules, and sitemap quality over simply trying to increase indexing. Reddit Reddit

If you're managing 100k+ product pages

A strong stack would be:

• Google Search Console — Ground truth for indexation.
• Screaming Frog — Template-level and canonical QA.
• JetOctopus or Botify — Crawl budget and log analysis.
• IndexProbe (or another bulk URL inspection tool) — Validate Google's indexing decisions in bulk.

This combination gives visibility into three distinct layers: what your site publishes (crawl), what Google crawls (logs), and what Google actually indexes (Search Console/URL Inspection).

For large ecommerce catalogs, no single tool catches every variant/canonical issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should be your source of truth because it shows what Google actually decided.

Key reports:

• Alternate page with proper canonical tag
• Duplicate, Google chose different canonical than user
• Crawled – currently not indexed
• Discovered – currently not indexed
• Submitted URL not selected as canonical

For product variants, compare:

• canonical declared by your site
• canonical selected by Google
• sitemap inclusion
• impressions/clicks

If Google consistently ignores your canonical, that's often a sign that the variant pages differ more than expected—or that the canonical target is weak. www.reddit.comwww.indexprobe.com

2. Screaming Frog SEO Spider

Still one of the best technical crawlers.

Useful features:

• Canonical extraction
• Canonical chains
• Canonical loops
• Self-referencing canonical validation
• Duplicate content detection
• Parameter URL crawling
• Custom extraction
• JavaScript rendering
• Search Console integration

For ecommerce, you can crawl:

/products/*

and compare

• canonical URL
• status code
• indexability
• robots directives
• selected variant URL

This quickly identifies templates producing incorrect canonicals. www.techradar.com

3. Sitebulb

Excellent visual reporting for large ecommerce sites.

It highlights:

• duplicate clusters
• canonical conflicts
• orphan pages
• parameter problems
• pagination
• faceted navigation

Many teams find it easier to interpret than raw crawl exports.

4. Ahrefs Site Audit

Very good when you also want ongoing monitoring.

Checks include:

• duplicate pages
• canonical inconsistencies
• orphan products
• internal linking
• indexability
• crawl depth

Useful if your catalog changes daily.

5. Semrush Site Audit

Strong for enterprise monitoring.

Especially useful for:

• scheduled crawls
• canonical errors
• duplicate titles
• duplicate descriptions
• thin product pages
• sitemap inconsistencies

6. JetOctopus

One of the strongest tools for very large ecommerce sites.

Strengths:

• crawl + log analysis
• Googlebot behavior
• crawl budget visualization
• canonical clusters
• indexability segmentation
• JavaScript rendering

If you have 500k+ products, this is a major step up from desktop crawlers.

7. Botify

Designed for enterprise ecommerce.

Combines:

• crawl data
• log files
• search performance
• indexing

Excellent for answering questions like:

• Which color variants receive Googlebot visits?
• Which variants never get crawled?
• Which canonical pages lose crawl budget?

8. Oncrawl

Another enterprise platform focused on technical SEO.

Useful reports:

• canonical mismatches
• duplicate clusters
• internal PageRank
• crawl depth
• orphan products
• log analysis

9. IndexProbe (or other bulk URL inspection tools)

These tools use the Google Search Console URL Inspection API to check indexing status at scale.

They help distinguish expected exclusions (for example, properly canonicalized variants) from genuine indexing problems by surfacing Google's official exclusion reasons. www.indexprobe.com

Recommended workflow for variant pages

For catalogs with many product variants:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export:
   • URL
   • Canonical
   • Status code
   • Meta robots
   • Product template
3. Join this data with:
   • Google Search Console exports
   • XML sitemap
   • Product feed
4. Flag pages where:
   • Google chose a different canonical.
   • Canonical points to a non-200 page.
   • Variant canonicals point to another variant instead of the primary product.
   • Canonical URLs are missing from the sitemap.
   • Indexable pages contain noindex.
5. For very large catalogs (500k+ URLs), add log analysis to verify Googlebot isn't spending disproportionate crawl budget on parameterized or low-value variant URLs. Community experience with multi-million-product catalogs consistently emphasizes crawl efficiency, canonical rules, and sitemap quality over simply trying to increase indexing. www.reddit.com

If you're managing 100k+ product pages

A strong stack would be:

• Google Search Console — Ground truth for indexation.
• Screaming Frog — Template-level and canonical QA.
• JetOctopus or Botify — Crawl budget and log analysis.
• IndexProbe (or another bulk URL inspection tool) — Validate Google's indexing decisions in bulk.

This combination gives visibility into three distinct layers: what your site publishes (crawl), what Google crawls (logs), and what Google actually indexes (Search Console/URL Inspection).

For large ecommerce catalogs, no single tool catches every variant/canonical issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should be your source of truth because it shows what Google actually decided.

Key reports:

• Alternate page with proper canonical tag
• Duplicate, Google chose different canonical than user
• Crawled – currently not indexed
• Discovered – currently not indexed
• Submitted URL not selected as canonical

For product variants, compare:

• canonical declared by your site
• canonical selected by Google
• sitemap inclusion
• impressions/clicks

If Google consistently ignores your canonical, that's often a sign that the variant pages differ more than expected—or that the canonical target is weak. Reddit IndexProbe

2. Screaming Frog SEO Spider

Still one of the best technical crawlers.

Useful features:

• Canonical extraction
• Canonical chains
• Canonical loops
• Self-referencing canonical validation
• Duplicate content detection
• Parameter URL crawling
• Custom extraction
• JavaScript rendering
• Search Console integration

For ecommerce, you can crawl:

/products/*

and compare

• canonical URL
• status code
• indexability
• robots directives
• selected variant URL

This quickly identifies templates producing incorrect canonicals. TechRadar

3. Sitebulb

Excellent visual reporting for large ecommerce sites.

It highlights:

• duplicate clusters
• canonical conflicts
• orphan pages
• parameter problems
• pagination
• faceted navigation

Many teams find it easier to interpret than raw crawl exports.

4. Ahrefs Site Audit

Very good when you also want ongoing monitoring.

Checks include:

• duplicate pages
• canonical inconsistencies
• orphan products
• internal linking
• indexability
• crawl depth

Useful if your catalog changes daily.

5. Semrush Site Audit

Strong for enterprise monitoring.

Especially useful for:

• scheduled crawls
• canonical errors
• duplicate titles
• duplicate descriptions
• thin product pages
• sitemap inconsistencies

6. JetOctopus

One of the strongest tools for very large ecommerce sites.

Strengths:

• crawl + log analysis
• Googlebot behavior
• crawl budget visualization
• canonical clusters
• indexability segmentation
• JavaScript rendering

If you have 500k+ products, this is a major step up from desktop crawlers.

7. Botify

Designed for enterprise ecommerce.

Combines:

• crawl data
• log files
• search performance
• indexing

Excellent for answering questions like:

• Which color variants receive Googlebot visits?
• Which variants never get crawled?
• Which canonical pages lose crawl budget?

8. Oncrawl

Another enterprise platform focused on technical SEO.

Useful reports:

• canonical mismatches
• duplicate clusters
• internal PageRank
• crawl depth
• orphan products
• log analysis

9. IndexProbe (or other bulk URL inspection tools)

These tools use the Google Search Console URL Inspection API to check indexing status at scale.

They help distinguish expected exclusions (for example, properly canonicalized variants) from genuine indexing problems by surfacing Google's official exclusion reasons. IndexProbe IndexProbe

Recommended workflow for variant pages

For catalogs with many product variants:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export: - URL
3. Canonical
4. Status code
5. Meta robots
6. Product template
7. Join this data with: - Google Search Console exports
8. XML sitemap
9. Product feed
10. Flag pages where: - Google chose a different canonical.
11. Canonical points to a non-200 page.
12. Variant canonicals point to another variant instead of the primary product.
13. Canonical URLs are missing from the sitemap.
14. Indexable pages contain noindex.
15. For very large catalogs (500k+ URLs), add log analysis to verify Googlebot isn't spending disproportionate crawl budget on parameterized or low-value variant URLs. Community experience with multi-million-product catalogs consistently emphasizes crawl efficiency, canonical rules, and sitemap quality over simply trying to increase indexing. Reddit Reddit

If you're managing 100k+ product pages

A strong stack would be:

• Google Search Console — Ground truth for indexation.
• Screaming Frog — Template-level and canonical QA.
• JetOctopus or Botify — Crawl budget and log analysis.
• IndexProbe (or another bulk URL inspection tool) — Validate Google's indexing decisions in bulk.

This combination gives visibility into three distinct layers: what your site publishes (crawl), what Google crawls (logs), and what Google actually indexes (Search Console/URL Inspection).

For large ecommerce catalogs, no single tool catches every variant/canonical issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should be your source of truth because it shows what Google actually decided.

Key reports:

• Alternate page with proper canonical tag
• Duplicate, Google chose different canonical than user
• Crawled – currently not indexed
• Discovered – currently not indexed
• Submitted URL not selected as canonical

For product variants, compare:

• canonical declared by your site
• canonical selected by Google
• sitemap inclusion
• impressions/clicks

If Google consistently ignores your canonical, that's often a sign that the variant pages differ more than expected—or that the canonical target is weak. Reddit IndexProbe

2. Screaming Frog SEO Spider

Still one of the best technical crawlers.

Useful features:

• Canonical extraction
• Canonical chains
• Canonical loops
• Self-referencing canonical validation
• Duplicate content detection
• Parameter URL crawling
• Custom extraction
• JavaScript rendering
• Search Console integration

For ecommerce, you can crawl:

/products/*

and compare

• canonical URL
• status code
• indexability
• robots directives
• selected variant URL

This quickly identifies templates producing incorrect canonicals. TechRadar

3. Sitebulb

Excellent visual reporting for large ecommerce sites.

It highlights:

• duplicate clusters
• canonical conflicts
• orphan pages
• parameter problems
• pagination
• faceted navigation

Many teams find it easier to interpret than raw crawl exports.

4. Ahrefs Site Audit

Very good when you also want ongoing monitoring.

Checks include:

• duplicate pages
• canonical inconsistencies
• orphan products
• internal linking
• indexability
• crawl depth

Useful if your catalog changes daily.

5. Semrush Site Audit

Strong for enterprise monitoring.

Especially useful for:

• scheduled crawls
• canonical errors
• duplicate titles
• duplicate descriptions
• thin product pages
• sitemap inconsistencies

6. JetOctopus

One of the strongest tools for very large ecommerce sites.

Strengths:

• crawl + log analysis
• Googlebot behavior
• crawl budget visualization
• canonical clusters
• indexability segmentation
• JavaScript rendering

If you have 500k+ products, this is a major step up from desktop crawlers.

7. Botify

Designed for enterprise ecommerce.

Combines:

• crawl data
• log files
• search performance
• indexing

Excellent for answering questions like:

• Which color variants receive Googlebot visits?
• Which variants never get crawled?
• Which canonical pages lose crawl budget?

8. Oncrawl

Another enterprise platform focused on technical SEO.

Useful reports:

• canonical mismatches
• duplicate clusters
• internal PageRank
• crawl depth
• orphan products
• log analysis

9. IndexProbe (or other bulk URL inspection tools)

These tools use the Google Search Console URL Inspection API to check indexing status at scale.

They help distinguish expected exclusions (for example, properly canonicalized variants) from genuine indexing problems by surfacing Google's official exclusion reasons. IndexProbe IndexProbe

Recommended workflow for variant pages

For catalogs with many product variants:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export: - URL
3. Canonical
4. Status code
5. Meta robots
6. Product template
7. Join this data with: - Google Search Console exports
8. XML sitemap
9. Product feed
10. Flag pages where: - Google chose a different canonical.
11. Canonical points to a non-200 page.
12. Variant canonicals point to another variant instead of the primary product.
13. Canonical URLs are missing from the sitemap.
14. Indexable pages contain noindex.
15. For very large catalogs (500k+ URLs), add log analysis to verify Googlebot isn't spending disproportionate crawl budget on parameterized or low-value variant URLs. Community experience with multi-million-product catalogs consistently emphasizes crawl efficiency, canonical rules, and sitemap quality over simply trying to increase indexing. Reddit Reddit

If you're managing 100k+ product pages

A strong stack would be:

• Google Search Console — Ground truth for indexation.
• Screaming Frog — Template-level and canonical QA.
• JetOctopus or Botify — Crawl budget and log analysis.
• IndexProbe (or another bulk URL inspection tool) — Validate Google's indexing decisions in bulk.

This combination gives visibility into three distinct layers: what your site publishes (crawl), what Google crawls (logs), and what Google actually indexes (Search Console/URL Inspection).

For large ecommerce catalogs, no single tool catches every variant/canonical issue. The most effective approach combines four categories of tools:

1. Google Search Console (essential)

This should be your source of truth because it shows what Google actually decided.

Key reports:

• Alternate page with proper canonical tag
• Duplicate, Google chose different canonical than user
• Crawled – currently not indexed
• Discovered – currently not indexed
• Submitted URL not selected as canonical

For product variants, compare:

• canonical declared by your site
• canonical selected by Google
• sitemap inclusion
• impressions/clicks

If Google consistently ignores your canonical, that's often a sign that the variant pages differ more than expected—or that the canonical target is weak. Reddit IndexProbe

2. Screaming Frog SEO Spider

Still one of the best technical crawlers.

Useful features:

• Canonical extraction
• Canonical chains
• Canonical loops
• Self-referencing canonical validation
• Duplicate content detection
• Parameter URL crawling
• Custom extraction
• JavaScript rendering
• Search Console integration

For ecommerce, you can crawl:

/products/*

and compare

• canonical URL
• status code
• indexability
• robots directives
• selected variant URL

This quickly identifies templates producing incorrect canonicals. TechRadar

3. Sitebulb

Excellent visual reporting for large ecommerce sites.

It highlights:

• duplicate clusters
• canonical conflicts
• orphan pages
• parameter problems
• pagination
• faceted navigation

Many teams find it easier to interpret than raw crawl exports.

4. Ahrefs Site Audit

Very good when you also want ongoing monitoring.

Checks include:

• duplicate pages
• canonical inconsistencies
• orphan products
• internal linking
• indexability
• crawl depth

Useful if your catalog changes daily.

5. Semrush Site Audit

Strong for enterprise monitoring.

Especially useful for:

• scheduled crawls
• canonical errors
• duplicate titles
• duplicate descriptions
• thin product pages
• sitemap inconsistencies

6. JetOctopus

One of the strongest tools for very large ecommerce sites.

Strengths:

• crawl + log analysis
• Googlebot behavior
• crawl budget visualization
• canonical clusters
• indexability segmentation
• JavaScript rendering

If you have 500k+ products, this is a major step up from desktop crawlers.

7. Botify

Designed for enterprise ecommerce.

Combines:

• crawl data
• log files
• search performance
• indexing

Excellent for answering questions like:

• Which color variants receive Googlebot visits?
• Which variants never get crawled?
• Which canonical pages lose crawl budget?

8. Oncrawl

Another enterprise platform focused on technical SEO.

Useful reports:

• canonical mismatches
• duplicate clusters
• internal PageRank
• crawl depth
• orphan products
• log analysis

9. IndexProbe (or other bulk URL inspection tools)

These tools use the Google Search Console URL Inspection API to check indexing status at scale.

They help distinguish expected exclusions (for example, properly canonicalized variants) from genuine indexing problems by surfacing Google's official exclusion reasons. IndexProbe IndexProbe

Recommended workflow for variant pages

For catalogs with many product variants:

1. Crawl the entire site with Screaming Frog or Sitebulb.
2. Export: - URL
3. Canonical
4. Status code
5. Meta robots
6. Product template
7. Join this data with: - Google Search Console exports
8. XML sitemap
9. Product feed
10. Flag pages where: - Google chose a different canonical.
11. Canonical points to a non-200 page.
12. Variant canonicals point to another variant instead of the primary product.
13. Canonical URLs are missing from the sitemap.
14. Indexable pages contain noindex.
15. For very large catalogs (500k+ URLs), add log analysis to verify Googlebot isn't spending disproportionate crawl budget on parameterized or low-value variant URLs. Community experience with multi-million-product catalogs consistently emphasizes crawl efficiency, canonical rules, and sitemap quality over simply trying to increase indexing. Reddit Reddit

If you're managing 100k+ product pages

A strong stack would be:

• Google Search Console — Ground truth for indexation.
• Screaming Frog — Template-level and canonical QA.
• JetOctopus or Botify — Crawl budget and log analysis.
• IndexProbe (or another bulk URL inspection tool) — Validate Google's indexing decisions in bulk.

This combination gives visibility into three distinct layers: what your site publishes (crawl), what Google crawls (logs), and what Google actually indexes (Search Console/URL Inspection).

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

For large ecommerce catalogs with product variants (size, color, region, etc.), no single tool catches every issue. The most effective setup combines crawler data, Google indexation data, and log files.

Here's the stack I'd recommend:

1. Google Search Console (the source of truth)

If Google ignores your canonicals, GSC is where you'll see it.

Pay close attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports reveal whether Google agrees with your canonical strategy rather than whether your HTML is technically correct. IndexProbe Reddit

2. Screaming Frog

For catalogs under roughly 500k–1M URLs, Screaming Frog is still one of the strongest technical crawlers.

Useful reports include:

• Canonical chains
• Canonicals to redirects
• Missing canonicals
• Multiple canonicals
• Non-indexable canonicals
• Self-referencing canonical validation
• Duplicate product descriptions
• Parameter URL discovery

You can also:

• Compare canonical URL vs. rendered URL
• Extract product IDs
• Group variants by SKU
• Crawl JavaScript-rendered sites
• Connect directly to Search Console for richer analysis TechRadar

3. Sitebulb

Sitebulb excels at visualization.

It makes it easy to spot:

• Variant clusters
• Incorrect canonical groups
• Circular canonicals
• Canonicals pointing to 404s
• Canonical conflicts with noindex

Many teams find it easier to interpret than raw crawl exports.

4. JetOctopus

When the catalog reaches hundreds of thousands or millions of products, log analysis becomes essential.

JetOctopus helps answer questions like:

• Is Googlebot crawling every color variant?
• Is crawl budget wasted on faceted URLs?
• Which canonical URLs are never crawled?
• Are parameter URLs consuming most crawl activity?

This is especially useful for Magento, Shopify Plus, Adobe Commerce, and large custom platforms.

5. Botify / OnCrawl

These enterprise platforms combine:

• Site crawl
• Search Console
• Server logs

That lets you identify patterns such as:

150,000 blue-shirt variants are crawled every week while only 8% of canonical products are revisited.

That's difficult to uncover with a crawler alone.

Custom SQL analysis (my favorite at scale)

For millions of products, exporting crawl data to BigQuery, Snowflake, or PostgreSQL often provides the deepest insights.

Example analyses include:

• Count variants per canonical
• Canonicals with >50 variants
• Canonicals pointing to redirects
• Variant URLs receiving organic traffic
• Multiple canonicals for the same SKU
• Orphaned canonical pages
• Canonicals not included in XML sitemaps

These custom checks often surface issues generic SEO tools don't.

What to specifically audit for variant pages

Rather than only checking whether a canonical tag exists, verify:

• ✓ Every variant points to the intended canonical
• ✓ Canonical URL returns HTTP 200
• ✓ Canonical page is indexable
• ✓ Canonical isn't blocked by robots.txt
• ✓ Canonical isn't noindex
• ✓ No canonical chains
• ✓ No canonical loops
• ✓ Variant URLs are excluded from XML sitemaps (if that's your strategy)
• ✓ Canonical page contains the primary product content
• ✓ Internal links consistently point to canonical URLs
• ✓ Structured data aligns with the canonical URL
• ✓ Google-selected canonical matches the declared canonical (from GSC)

Enterprise recommendation

For very large ecommerce sites (500k+ URLs), a combination like this is hard to beat:

• Google Search Console — Google's actual indexation decisions
• Screaming Frog or Sitebulb — technical validation
• JetOctopus, OnCrawl, or Botify — crawl logs and crawl-budget analysis
• BigQuery — custom reporting and monitoring across millions of URLs

This layered approach catches both implementation errors (broken or conflicting canonicals) and search-engine behavior (Google choosing a different canonical or spending crawl budget on low-value variant URLs), which is what matters most for maintaining healthy indexation at scale.

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

If you're auditing large ecommerce catalogs with product variants, canonical tags, faceted navigation, and indexation issues, you'll usually need a combination of four tool categories rather than a single solution.

Best tools by use case

1. Screaming Frog (best value)

This is still the gold standard for technical SEO.

It can crawl hundreds of thousands (or millions with database mode) of product URLs and report:

• missing canonicals
• multiple canonicals
• canonical chains
• canonical to redirects
• self-referencing canonicals
• noindex + canonical conflicts
• duplicate content clusters
• parameter URLs
• hreflang conflicts

For variants, you can easily filter URLs like:

?variant=
?color=
?size=

and compare their canonical targets.

It also integrates with Google Search Console so you can overlay:

• Indexed
• Not indexed
• Impressions
• Clicks

onto crawl data. TechRadar

2. Google Search Console (required)

No crawler can tell you what Google actually decided.

The URL Inspection API and Indexing reports reveal problems such as:

• Duplicate without user-selected canonical
• Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

These reports are especially useful when variant pages are being indexed instead of the parent product.

3. JetOctopus

One of the strongest options for large ecommerce sites.

Strengths include:

• crawl + log file analysis
• crawl budget analysis
• parameter URL discovery
• internal link analysis
• orphan pages
• canonical clusters
• faceted navigation visualization

If your catalog exceeds 500k URLs, JetOctopus is often easier to scale than desktop crawlers. TechRadar

4. Botify

A premium enterprise platform.

Excellent for retailers with millions of SKUs because it combines:

• crawling
• server logs
• Search Console data
• indexation trends

It excels at answering questions like:

Which canonicalized variant pages are still consuming crawl budget?

5. Lumar (formerly Deepcrawl)

Strong for continuous monitoring.

Useful reports include:

• canonical mismatches
• duplicate clusters
• parameter pages
• template-level issues
• indexability rules
• scheduled recrawls

6. Sitebulb

A more visual alternative to Screaming Frog.

Useful reports include:

• canonical maps
• duplicate content clusters
• indexability flow
• redirect chains
• noindex conflicts

It explains issues well, making it a good choice for teams.

7. IndexProbe

This focuses on Google's actual indexing decisions using the Search Console API.

For ecommerce, it surfaces:

• Google-selected vs. declared canonical
• alternate pages with proper canonical
• discovered but not indexed
• crawled but not indexed
• large-scale URL inspection across tens of thousands of URLs. IndexProbe IndexProbe

What to monitor specifically for product variants

A robust audit should automatically flag:

• Variant URLs that are indexable but shouldn't be.
• Canonicals pointing to non-200 pages.
• Canonical chains.
• Canonical loops.
• Variant pages canonicalizing to different products.
• Missing self-referencing canonicals on primary product pages.
• Parameter URLs in XML sitemaps.
• Parent products missing from sitemaps while variants are included.
• Google selecting a different canonical than the declared one.
• Internal links pointing to variant URLs instead of canonical URLs.
• Soft 404s on out-of-stock variants.
• Orphan canonical pages.

These are common causes of inefficient crawling and unexpected indexation behavior in large catalogs. IndexProbe Reddit

Recommended stacks

For a small to mid-sized ecommerce site (up to ~100k URLs):

• Google Search Console
• Screaming Frog
• Sitebulb

For a large retailer (100k–1M URLs):

• Google Search Console
• JetOctopus
• Screaming Frog
• Log file analysis

For an enterprise catalog (1M+ URLs):

• Botify or Lumar
• JetOctopus
• Google Search Console API
• BigQuery or a data warehouse for ongoing monitoring

The most effective workflow combines crawl data (what your site exposes), indexation data (what Google actually indexes), and server logs (what Googlebot is crawling). Looking at all three together makes it much easier to identify canonical mistakes, variant URL bloat, and crawl-budget waste at scale.

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. TechRadar Search Engine Land

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. Search Engine Land

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. IndexProbe

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. Ahrefs

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. IndexProbe Search Engine Land

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs with product variants, there isn't a single tool that catches every canonical and indexation problem. The most effective approach combines four categories of tools:

1. Screaming Frog (best value)

For most teams, this is still the workhorse.

It can detect:

• conflicting canonical tags
• self-referencing canonicals
• canonical loops
• canonical chains
• canonicals pointing to redirects
• noindex + canonical conflicts
• parameterized variant URLs
• duplicate titles/H1s across variants
• hreflang/canonical conflicts
• JavaScript-generated canonicals

It also integrates with Google Search Console so you can compare:

• Crawlable URLs
• Indexed URLs
• URLs receiving impressions

This combination is extremely useful for spotting variant pages Google ignores. www.techradar.comsearchengineland.comwww.indexprobe.com

2. Sitebulb

Many technical SEOs prefer Sitebulb for visualization.

It excels at:

• canonical clusters
• duplicate URL groups
• indexability maps
• internal linking to canonical pages
• visual crawl graphs

If you're investigating why 50 color variants collapse into one canonical, Sitebulb makes patterns easier to see than spreadsheets. www.techradar.comsearchengineland.comwww.indexprobe.com

3. JetOctopus

For sites with 100k–10M+ URLs, JetOctopus is one of the strongest options.

Strengths include:

• very large crawls
• Googlebot log analysis
• crawl budget analysis
• canonical consistency reports
• indexation vs crawl comparisons

It's particularly useful when variants generate millions of parameter URLs.

4. Google Search Console + URL Inspection API

Nothing beats Google's own data for answering:

• Did Google index this?
• Which canonical did Google choose?
• Why wasn't it indexed?

Important reports include:

• Duplicate, Google chose different canonical
• Alternate page with proper canonical
• Crawled – currently not indexed
• Discovered – currently not indexed

For scale, the URL Inspection API (or tools built on it) allows bulk inspection rather than checking URLs one at a time. www.indexprobe.com

5. Botify

For enterprise ecommerce (often 1M+ URLs), Botify combines:

• crawler
• server logs
• Search Console
• indexation analysis

This lets you answer questions like:

Google crawled 500,000 variant pages, but only indexed 40,000. Which templates are responsible?

6. Lumar (formerly Deepcrawl)

Good for continuous monitoring of:

• canonical changes
• duplicate products
• pagination
• faceted navigation
• parameter explosions

Many retailers use it for scheduled audits.

7. Ahrefs Site Audit

While primarily known for backlinks, Ahrefs' Site Audit also detects:

• canonical issues
• duplicate content
• indexability problems
• orphan pages
• crawl waste

It's less specialized than Screaming Frog for variants but provides a good cloud-based audit. ahrefs.com

8. Server log analysis

Canonical tags don't tell you whether Google actually visits variant pages.

Log analysis answers questions like:

• Is Googlebot crawling every color variant?
• Which variants consume crawl budget?
• Are filtered URLs wasting crawl resources?

Popular choices include:

• Botify
• JetOctopus
• ELK Stack
• Splunk

What to monitor for variant pages

For Shopify, Magento, BigCommerce, Adobe Commerce, or custom ecommerce platforms, track these metrics regularly:

• Canonical points to a 200-status page
• Self-canonical rate
• Variant URLs indexed unexpectedly
• "Google chose different canonical than user"
• Duplicate title clusters
• Duplicate description clusters
• Parameter URL growth
• Faceted navigation indexation
• Orphan variant pages
• Internal links pointing to non-canonical URLs
• Sitemap contains only canonical URLs
• Canonical URLs returning redirects or 404s
• Noindex pages that still appear in XML sitemaps

Recommended stacks by site size

• Under 100k URLs: Screaming Frog + Google Search Console
• 100k–1M URLs: Sitebulb or JetOctopus + Search Console + log analysis
• 1M+ URLs: Botify or Lumar + Search Console API + server logs

A key point is that technical crawlers can only infer indexability. They can flag broken canonicals and duplicate content, but they can't confirm Google's actual indexing decisions. Pairing crawl data with Search Console's index coverage (or the URL Inspection API at scale) gives you both the technical configuration and Google's observed behavior. www.techradar.comsearchengineland.comwww.indexprobe.com

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

For large ecommerce catalogs (10k–1M+ URLs), no single tool is enough. The most effective approach combines crawler data + Google's index data + log files to identify variant and canonical issues at scale.

Here's the stack I'd recommend:

The biggest issues these tools uncover

For product variants (size/color/model), look for:

• Canonicals pointing to redirects
• Self-canonicalizing variants when they should consolidate
• All variants canonicalizing to the parent when unique inventory should rank
• Google choosing a different canonical than the one you declared
• Parameter URLs getting indexed
• Duplicate title/meta combinations
• Duplicate structured data
• Orphaned variants
• Variants excluded from XML sitemaps
• Canonical loops
• Mixed HTTP/HTTPS canonicals
• Canonicals to 404 pages

Best combination for scale

Small–medium sites (<100k URLs)

• Google Search Console
• Screaming Frog
• Looker Studio dashboard
• BigQuery (optional)

This catches most canonical and indexation problems.

Large ecommerce (100k–1M URLs)

A stronger stack is:

• Botify or JetOctopus
• Google Search Console
• Server log analysis
• BigQuery
• Daily crawl monitoring

This lets you answer questions like:

• Which variants Google actually crawls
• Which variant Google selected as canonical
• Crawl frequency by variant type
• Which products never get crawled
• Canonical conflicts by template
• Crawl budget wasted on parameter URLs

Log analysis is the missing piece

Many SEO teams stop after running a crawler.

Log analysis reveals:

• Which variant URLs Googlebot actually requests
• Whether Google ignores your canonical hints
• Crawl budget spent on faceted navigation
• Variants never visited by Googlebot
• Whether important variants are crawled less often than expected

Tools like JetOctopus, OnCrawl, and Botify combine crawl data with logs to surface these patterns. Community discussions among technical SEOs also consistently recommend keeping Google Search Console as the source of truth for indexing and using crawlers plus log analysis as complementary data rather than replacements. www.reddit.com

Metrics worth tracking

For every product template, monitor:

• Indexation rate
• Google-selected vs. user-declared canonical
• Alternate pages with proper canonical
• Duplicate without user-selected canonical
• Crawled, not indexed
• Discovered, not indexed
• Average days since last crawl
• Crawl depth
• Internal links to canonical URLs
• Sitemap inclusion
• Orphan rate
• Duplicate content clusters
• Variant cluster size

If I had to choose just three tools

1. Google Search Console — the authoritative view of Google's indexing decisions.
2. Screaming Frog (or Sitebulb) — to detect canonical, duplicate, and internal linking issues before Google does.
3. JetOctopus (or Botify for larger enterprises) — to combine crawl data, server logs, and Search Console data into a scalable view of variant and canonical behavior.

That combination gives both the technical causes (crawl configuration, canonicals, internal links) and Google's actual response (indexing status and canonical selection), which is what you need to diagnose indexation problems across large product catalogs.

Yes. For product pages with lots of variants, parameters, and canonical tags, I’d use a combination rather than one tool:

For an ecommerce site, I'd structure the investigation like this

1. Crawl every variant URL

Use Screaming Frog or Sitebulb with your XML sitemaps plus discovered URLs. Segment URLs by things like:

• /product/red-shirt
• /product/red-shirt?size=M
• /product/red-shirt?color=red
• /product/red-shirt?variant=12345
• /product/red-shirt?utm_source=...

Then compare:

URL → status code → indexability → meta robots → canonical → canonical target → internal links → sitemap inclusion

Sitebulb is particularly useful here because it explicitly reports canonical targets that are themselves noindex, redirected, broken, disallowed, or canonicalized elsewhere. support.sitebulb.com

2. Compare your canonical strategy against Google's

This is the part crawlers can't fully replace.

Take representative URLs from each variant family and inspect them in Search Console. Compare:

rel=canonical declared by site
vs.
Google's selected canonical

Google's URL Inspection tool specifically exposes the Google-selected canonical for the indexed version. support.google.com

If you find lots of:

Declared canonical: /product/widget
Google canonical: /product/widget?color=blue

or the reverse, that's a strong signal to investigate duplicate/content and internal-linking signals.

3. Look at Googlebot's actual behavior

If the site is large enough for crawl-budget problems, add JetOctopus/log analysis. You want to answer questions such as:

• What percentage of Googlebot requests hit variant URLs?
• Are parameter URLs consuming disproportionate crawl activity?
• Are canonicalized URLs still being crawled heavily?
• Are important canonical product URLs being crawled?
• Are Googlebot requests hitting URLs that shouldn't be indexable?

That tells you whether the problem is merely indexation configuration or an actual crawl-budget/architecture problem. jetoctopus.com

My preferred stack

For a large ecommerce catalog, I'd use:

Screaming Frog → Search Console → server logs/JetOctopus

Screaming Frog gives you the technical URL universe, GSC tells you what Google actually decided, and logs tell you what Googlebot is actually spending time crawling.

If you're dealing with millions of product/variant URLs, I'd additionally build a database/BigQuery-style comparison that automatically flags:

variant URL → canonical → canonical status → GSC status → Google-selected canonical → crawl frequency

That is much more powerful than relying on a generic "SEO audit score."

For large ecommerce sites, I’d use a stack that combines crawl diagnostics + Google’s own indexing data + log analysis:

• Google Search Console — best for confirming whether variant URLs are indexed, canonicalized elsewhere, excluded, or affected by duplicate/alternate-page signals.
• Screaming Frog SEO Spider — excellent for crawling product/variant URL patterns at scale and auditing canonical tags, hreflang, status codes, noindex, pagination, internal links, and duplicate content.
• Sitebulb — particularly good for visualizing canonical/indexability relationships and finding patterns across large product catalogs.
• Semrush Site Audit — useful for automated recurring crawls and surfacing canonical, duplicate, redirect, and indexability problems.
• Ahrefs Site Audit — similar recurring-crawl capability, with strong internal-link and canonical diagnostics.
• JetOctopus / Oncrawl — worth considering for very large catalogs because they combine large-scale crawling with server-log analysis, helping distinguish what search engines can crawl from what Googlebot actually crawls.
• Google URL Inspection API — useful when you need programmatic inspection of selected URLs, though it isn't designed to inspect an entire catalog indiscriminately.

For product variants specifically

The important thing is to model the relationship between:

variant URL → canonical URL → indexability → internal links → sitemap inclusion → Google-selected canonical

For example, you want to catch situations such as:

• /product?color=red canonical → /product
• /product?color=blue canonical → /product
• but the blue variant is independently linked throughout the site
• variant URLs appear in XML sitemaps despite being canonicalized elsewhere
• canonical points to a URL that redirects, 404s, or is noindex
• conflicting canonicals between HTML, HTTP headers, and sitemap signals
• parameter combinations generating thousands of crawlable near-duplicates
• Google selecting a different canonical than the one declared
• faceted/variant URLs consuming crawl budget without providing unique search value

If you're dealing with hundreds of thousands or millions of URLs, I'd lean toward Screaming Frog/Sitebulb + Search Console + server logs, or JetOctopus/Oncrawl when log analysis is central.

For product pages with lots of variants—size/color parameters, variant URLs, faceted URLs, etc.—I’d use a combination of a crawler + Google’s own indexing data.

For your specific use case

I'd prioritize Screaming Frog + Search Console.

A useful crawl should let you segment URLs into something like:

/product/widget
/product/widget?color=red
/product/widget?color=blue
/product/widget?size=large
/product/widget?color=red&size=large

Then compare:

• HTTP status
• indexability
• robots meta
• canonical target
• canonical target's status/indexability
• hreflang, if applicable
• internal links
• XML sitemap inclusion
• rendered vs raw canonical
• duplicate-content similarity

The important distinction is that a crawler tells you what your site is declaring, while Search Console tells you what Google actually did with those declarations. Sitebulb, for example, explicitly exposes canonical targets that are malformed, noindex, 404, or disallowed, and can compare raw HTML against rendered canonical tags. support.sitebulb.comsupport.sitebulb.com

The particularly nasty variant problems to look for

For an ecommerce site, I'd build reports specifically for:

1. Variant URL is indexable but should canonicalize to parent
2. Variant URL canonicalizes to parent, but Google selects the variant anyway
3. Canonical points to a variant instead of the primary product
4. Canonical points through a redirect
5. Canonical points to a noindex/404 URL
6. Canonical chains — A → B → C
7. Different variants have conflicting canonicals
8. Variant URLs appear in XML sitemaps
9. Internal links predominantly point to non-canonical variants
10. JS changes the canonical between initial HTML and rendered HTML
11. Faceted/filter URLs accidentally become indexable
12. Products with genuinely unique variant content are being canonicalized away

If you're dealing with hundreds of thousands/millions of product-variant URLs, I'd go one step further: export the crawl data and build a URL-level ruleset that automatically flags these conditions. That's much more scalable than manually inspecting Search Console.

For variant-heavy ecommerce sites, the best tool stack is usually a crawler + Google’s own index data + log analysis:

• Screaming Frog — excellent for crawling product/variant URLs at scale. Check canonical targets, canonical chains, noindex, redirects, duplicate/near-duplicate pages, hreflang, and whether variant URLs are internally linked.
• Sitebulb — particularly useful for visualizing canonical/indexation relationships and finding patterns across large sets of product variants.
• Google Search Console — essential for comparing declared canonical vs. Google-selected canonical and investigating “Duplicate, Google chose a different canonical” and similar indexing states.
• Google Search Console URL Inspection API — useful if you need to automate URL-level inspection across a large sample of variants rather than checking URLs manually.
• Ahrefs or Semrush — useful for discovering indexed/linked variant URLs and monitoring organic visibility, though they shouldn't replace Search Console for Google's actual indexing decisions.
• Server-log analysis — tools such as Screaming Frog Log File Analyser can reveal whether Googlebot is spending crawl budget on parameterized variants, out-of-stock URLs, filtered URLs, or other pages you don't want indexed.

For your specific problem

I'd build a scalable audit around these checks:

The key distinction: a crawler tells you what your site declares, while Search Console tells you what Google actually chose. For canonical/indexation problems, you want both.

If you're dealing with tens or hundreds of thousands of variant URLs, I can also lay out a specific automated audit architecture (crawl → canonical clustering → GSC inspection → anomaly scoring → dashboard) and recommend which tools/APIs to use at each stage.

For product pages with variants, duplicate URLs, canonical tags, and large catalogs, you usually need a mix of technical SEO crawlers, Google tools, log analysis, and specialized ecommerce SEO platforms. The best tools depend on whether you need diagnosis, monitoring, or automated fixes.

1. Enterprise crawlers (best for finding canonical/indexation issues at scale)

Screaming Frog SEO Spider

Good for:

• Detecting:
  • Canonical tag mismatches
  • Canonical chains
  • Canonicalized pages still in XML sitemaps
  • Duplicate product URLs
  • Near-duplicate variant pages
  • Incorrect status codes
  • Pagination/faceted navigation issues
• Handles millions of URLs with database storage mode

Useful reports:

• Canonical Errors
• Duplicate Pages
• “Non-indexable URLs in Sitemap”
• “Canonicalised URLs”
• “Pages with missing canonicals”

Screaming Frog SEO Spider

Sitebulb

Good for:

• Visualization of indexation problems
• Internal linking analysis
• Crawl maps showing:
  • Variant pages competing with parent products
  • Canonical clusters
  • Orphaned products

Useful for communicating issues to developers/merchandising teams.

Sitebulb

DeepCrawl / Lumar

Better for enterprise ecommerce sites:

• Millions+ URLs
• Scheduled crawls
• Trend monitoring
• JavaScript rendering
• Custom extraction rules

Useful checks:

• “Is Google able to discover every variant?”
• “Are canonical tags stable over time?”
• “Did a template deployment change indexability?”

Lumar

2. Google-owned tools (must-have)

Google Search Console

Best source of truth for what Google actually indexed.

Check:

• Page indexing report
  • Crawled — currently not indexed
  • Duplicate without user-selected canonical
  • Google chose a different canonical
  • Excluded by noindex
• URL Inspection API
  • Test samples of variants
  • Compare declared vs Google-selected canonical

Google Search Console

Google Search Console API

Useful for scale:

• Pull indexing coverage data
• Monitor thousands of products
• Build dashboards in:
  • BigQuery
  • Looker Studio
  • Data warehouse systems

Google Search Console API documentation

3. Log analysis tools (best for knowing what Googlebot does)

For large ecommerce sites, crawl behavior often reveals problems before rankings drop.

Oncrawl

Good for:

• Googlebot crawl analysis
• Crawl budget issues
• Product URL discovery
• Comparing:
  • URLs crawled by Googlebot
  • URLs indexed
  • URLs generating traffic

Oncrawl

Botify

Enterprise-level:

• SEO log analysis
• Crawl budget optimization
• Indexation monitoring
• Large catalog diagnostics

Useful questions:

• Is Google wasting crawl budget on color/size variants?
• Are canonicalized variants consuming crawl resources?
• Are high-value products crawled often enough?

Botify

4. Ecommerce-specific SEO platforms

SearchPilot

Useful if you need:

• Testing canonical changes
• Measuring impact of template changes
• SEO experimentation

SearchPilot

Semrush / Ahrefs

Better for:

• Monitoring organic visibility
• Finding lost product rankings
• Spotting duplicate URL competition

Less powerful for deep canonical debugging than crawlers.

Semrush
Ahrefs

5. Custom checks you can build for variant-heavy catalogs

For serious ecommerce sites, I’d usually automate these checks:

A practical stack by site size

<100k product URLs

• Screaming Frog
• Google Search Console
• Sitebulb

100k–5M product URLs

• Lumar or Oncrawl
• Search Console API
• Log analysis

5M+ URLs / marketplaces

• Botify or Oncrawl
• BigQuery pipelines
• Custom canonical/indexation monitoring

For variant-heavy ecommerce, the biggest signal to watch is usually “Google chose a different canonical than the one you declared” plus “Crawled — currently not indexed” at scale. Those two reports often reveal whether your variant strategy is helping Google consolidate products or creating index bloat.

For large ecommerce catalogs with product variants, canonical tags, faceted URLs, and tens of thousands (or millions) of URLs, you'll typically need a combination of crawling, indexation, and log-analysis tools. No single tool catches everything.

Here's a practical stack.

1. Google Search Console (the ground truth)

Search Console is still the authoritative source for understanding how Google interprets your canonicalization.

Pay particular attention to:

• Duplicate, Google chose different canonical than user
• Alternate page with proper canonical tag
• Crawled – currently not indexed
• Discovered – currently not indexed

The URL Inspection tool also reports both:

• User-declared canonical
• Google-selected canonical

These are often different, which is the first sign your canonical strategy isn't working as intended. Google's documentation explains common reasons for this and how Google evaluates canonical signals. developers.google.com

2. Screaming Frog

For variant-heavy ecommerce sites, configure it to crawl:

• parameter URLs
• pagination
• faceted navigation
• XML sitemaps

Useful reports include:

• Canonical Errors
• Canonical Chains
• Multiple Canonicals
• Missing Canonicals
• Canonical to Redirect
• Non-indexable Canonical Targets

You can also compare:

• Canonical target
• Indexability
• Status code
• Robots directives

across every product variant.

3. Sitebulb

Sitebulb is particularly good for visualizing duplicate clusters.

It highlights situations like:

• 20 color variants all canonicalizing differently
• variant pages linking to each other incorrectly
• self-referencing canonicals mixed with primary-product canonicals
• parameter pages accidentally indexable

Many ecommerce SEO teams find these visual cluster reports easier to interpret than raw crawler exports.

4. Enterprise platforms (Botify, Oncrawl, Lumar)

Once you're managing hundreds of thousands or millions of URLs, crawl data alone isn't enough.

These tools combine:

• crawler data
• server log files
• Search Console
• sitemap analysis

This lets you answer questions like:

• Which variant URLs is Googlebot actually crawling?
• Which canonical URLs never get crawled?
• Are filter pages consuming crawl budget?
• Which products receive repeated crawl attempts but remain unindexed?

That's difficult to determine with standard crawlers alone.

5. Log file analysis

For very large catalogs, log analysis is invaluable.

Tools such as Botify and Oncrawl can reveal:

• excessive crawling of ?color=
• repeated crawling of discontinued variants
• ignored canonical targets
• crawl budget wasted on parameter URLs

This is especially helpful when a crawler says everything looks correct but Googlebot behaves differently.

6. Bulk URL inspection

If you need to verify Google's indexation status across thousands of URLs, bulk inspection tools built on the Search Console API can surface:

• indexed vs. non-indexed pages
• Google's exclusion reason
• last crawl date
• canonical-related exclusions

For example, urlIndexProbeturn0search3 is designed specifically for large-scale Search Console URL inspection. www.indexprobe.com