ChatGPT answers, unedited

For a large catalog, the scalable approach is to treat product content quality like a data quality pipeline, not a copywriting project. The goal is to continuously detect problems, prioritize them, generate fixes, and measure improvement.

A practical architecture looks like this:

1. Build a catalog content health audit

Create automated checks for every SKU:

Duplicate content checks

Detect:

• Exact duplicate descriptions
• Near-duplicate descriptions (same text with small changes)
• Manufacturer/supplier copy reused across competitors
• Variant pages where only color/size changes but descriptions are identical
• Duplicate title + meta combinations

Useful methods:

• Hashing for exact matches
• TF-IDF / cosine similarity for older SEO systems
• Embeddings + clustering for semantic duplicates (better for large catalogs)

Tools that can help:

• screamingfrog.co.uk — crawl-based audits for duplicate titles, descriptions, and indexation issues
• semrush.com — large-scale technical SEO monitoring
• ahrefs.com — crawl and SEO issue detection
• Ecommerce-focused auditors can also flag duplicate product descriptions and thin pages at catalog scale. WebAudit

2. Define “thin content” rules

Don't rely on word count alone. A 150-word description can still be weak.

Score products on signals like:

A useful product content score:

Content Score =
40% uniqueness
25% attribute completeness
20% buyer usefulness
15% SEO metadata quality

3. Store content rules in your PIM or catalog system

For thousands of SKUs, don't manage this in spreadsheets.

Common systems:

• akeneo.com
• salsify.com
• plytix.com
• inriver.com

Use them to enforce:

• Required attributes by category
• Minimum description standards
• Approval workflows
• Version history
• Supplier content cleanup

4. Use AI for rewriting — but with guardrails

For a huge catalog, AI works best when it rewrites from structured product facts:

Good input:

Product:
10-inch cast iron skillet

Facts:
- Pre-seasoned
- Oven safe to 500°F
- Weight: 5.5 lbs
- Compatible with induction
- Lifetime warranty

Audience:
Home cooks and campers

Bad input:

Rewrite this description:
[paste supplier paragraph]

The first approach creates differentiated descriptions. The second often creates “spun” duplicates.

Some catalog enrichment tools specifically focus on bulk rewriting and approval workflows. Catalog Cleanup AI Ecomptimize

5. Create a prioritization model

Do not rewrite 100,000 SKUs in random order.

Rank products by:

Priority =
Organic traffic
× Revenue potential
× Duplicate severity
× Conversion opportunity

Example queue:

Tier 1

• Top 5,000 revenue products
• Indexed but declining pages
• Products with duplicate manufacturer copy

Tier 2

• Category leaders
• High-margin products
• Products with impressions but low clicks

Tier 3

• Long-tail SKUs
• Discontinued inventory
• Low-demand variants

6. Monitor continuously

Set up dashboards showing:

Content KPIs

• % products with unique descriptions
• Average description quality score
• Missing attribute rate
• Duplicate cluster count

SEO KPIs

• Indexed product pages
• Organic impressions
• Click-through rate
• Ranking changes
• Cannibalization issues

Commerce KPIs

• Conversion rate changes after rewrites
• Add-to-cart rate
• Revenue per product page

Connect:

• search.google.com
• analytics.google.com
• Your ecommerce platform data warehouse

7. A scalable workflow

A mature process usually looks like:

Nightly catalog sync
        ↓
Duplicate + thin-content detection
        ↓
Product quality scoring
        ↓
Priority queue
        ↓
AI draft generation
        ↓
Human approval sampling
        ↓
Publish changes
        ↓
Measure SEO + conversion impact

Tool stack examples

Small/medium catalog (1k–20k SKUs)

• Screaming Frog / Semrush
• Shopify or WooCommerce
• AI rewriting workflow
• Spreadsheet approval queue

Enterprise catalog (50k–millions of SKUs)

• PIM (Akeneo/Salsify/etc.)
• Data warehouse
• Embedding-based duplicate detection
• AI enrichment service
• Automated QA rules
• Merchandising approval workflow

The biggest mistake at scale is treating duplicate and thin content as a one-time cleanup. Large catalogs constantly create new issues through supplier imports, new variants, marketplace feeds, and seasonal launches, so the system needs ongoing detection and remediation. nexcommerce.ai

For a huge ecommerce catalog, I’d treat this as a continuous data-quality pipeline, not a one-off SEO audit.

A scalable setup

1. Detect duplicates at several levels

Use different tests because “duplicate” can mean very different things:

• Exact duplicates: identical descriptions/product copy.
• Near duplicates: same template with a few changed words, dimensions, colors, etc.
• Template-heavy pages: the actual product copy is tiny relative to navigation/footer/boilerplate.
• Duplicate metadata: same title/H1/meta description across many SKUs.
• URL duplicates: parameters, faceted navigation, HTTP/HTTPS, trailing slashes, variant URLs, etc.

For crawling and near-duplicate detection, Screaming Frog is particularly useful: its SEO Spider uses MinHash for near-duplicate detection and lets you define which page content should be compared. Screaming Frog Screaming Frog

Sitebulb is another good option for large-scale content auditing; it checks duplicate/near-duplicate content, word counts, titles, H1s and meta descriptions. Sitebulb Sitebulb

For an enterprise team, Semrush Site Audit can automate recurring crawls and identify duplicate-content and other technical issues; its current enterprise offering is designed for very large sites. Semrush Semrush

2. Don't use word count as your definition of "thin"

A 150-word description can be excellent for one product and inadequate for another.

Instead, create a content-quality score from multiple signals, for example:

thin_score =
    low_unique_word_count
  + low_unique_character_count
  + high_template_ratio
  + duplicate_similarity
  + missing_product_attributes
  + missing_use_cases
  + low organic engagement

For products, I'd extract structured fields such as:

• brand
• product type
• material
• dimensions
• compatibility
• use cases
• differentiating features
• specifications
• what's included
• warranty/returns information
• relevant FAQs

Then flag products where the description doesn't actually communicate the information available in your catalog/PIM.

3. Build a duplicate-content cluster instead of a giant URL list

This is one of the biggest scalability improvements.

Instead of:

URL A → duplicate
URL B → duplicate
URL C → duplicate

create clusters:

Cluster 1842
├── /blue-widget
├── /blue-widget-large
├── /blue-widget-v2
├── /blue-widget-model-x
└── /blue-widget-compatible

Similarity: 91–97%

Then assign the cluster a canonical/product-family decision:

• genuinely different products → improve differentiation
• variants that shouldn't have separate search pages → consolidate/canonicalize
• accidental duplicate URLs → redirect/canonicalize
• legitimate regional/language versions → retain with appropriate international signals
• genuinely unique products → rewrite descriptions

This prevents your content team from manually evaluating thousands of individual URLs.

4. Automate the actual rewriting

Don't have an LLM blindly rewrite every short description.

A much safer pipeline is:

PIM/catalog → validation → duplicate clustering → content generation → QA → human approval → publish

Give the generation system the structured product facts, not merely the old description.

For example:

{
  "product": "ABC-123",
  "brand": "Acme",
  "category": "Outdoor jacket",
  "material": "3-layer nylon",
  "waterproof_rating": "20,000 mm",
  "features": [
    "pit zips",
    "adjustable hood",
    "YKK zippers"
  ],
  "sizes": ["S", "M", "L", "XL"],
  "use_cases": ["hiking", "skiing"]
}

Then impose rules such as:

• never invent specifications
• don't repeat the product title 10 times
• don't copy manufacturer text verbatim
• describe meaningful differentiators
• preserve regulated/safety claims exactly
• produce different copy for genuinely different products
• reject output when required attributes are missing

5. Put the monitoring directly into your catalog pipeline

This is much better than discovering problems months later with an SEO crawler.

Every time a SKU is created or modified:

Product changed
      ↓
Normalize text
      ↓
Exact duplicate check
      ↓
Similarity search
      ↓
Thin-content checks
      ↓
Required-attribute checks
      ↓
Quality threshold?
   ↙          ↘
 YES          NO
publish      content queue

For millions of products, you can store embeddings or MinHash signatures and use approximate-nearest-neighbor search to identify similar descriptions without comparing every product against every other product.

6. Use crawlers as your independent QA layer

Your internal catalog system tells you what should exist. A crawler tells you what actually reached the website.

I'd run something like:

• Screaming Frog → deep technical/content crawling and near-duplicate investigation. Screaming Frog
• Sitebulb → content/duplicate auditing and visualization. Sitebulb Sitebulb Support
• Semrush Site Audit → recurring automated monitoring and issue tracking. Semrush Semrush
• Your PIM/database + custom scripts → the scalable source-of-truth layer.

The dashboard I'd build

Track these weekly:

The particularly useful metric is new problems introduced since the previous catalog release. That turns SEO from periodic cleanup into regression testing.

Tool choice

If you're dealing with 100K+ SKUs, I'd generally use a combination rather than expect one SEO platform to do everything:

PIM/database + custom similarity/quality pipeline + Screaming Frog/Sitebulb + Semrush monitoring.

The crawler catches what customers and search engines actually see; the internal pipeline lets you prevent the next 100,000 bad descriptions from being generated in the first place.

One important caveat: duplicate content isn't automatically an SEO penalty. The appropriate action depends on why the pages are duplicated and whether they represent genuinely distinct products/variants. Even Semrush recommends contextual review rather than mechanically treating every similarity match as something to delete. Semrush Semrush

5. Put the monitoring directly into your catalog pipeline

This is much better than discovering problems months later with an SEO crawler.

Every time a SKU is created or modified:

Product changed
      ↓
Normalize text
      ↓
Exact duplicate check
      ↓
Similarity search
      ↓
Thin-content checks
      ↓
Required-attribute checks
      ↓
Quality threshold?
   ↙          ↘
 YES          NO
publish      content queue

For millions of products, you can store embeddings or MinHash signatures and use approximate-nearest-neighbor search to identify similar descriptions without comparing every product against every other product.

6. Use crawlers as your independent QA layer

Your internal catalog system tells you what should exist. A crawler tells you what actually reached the website.

I'd run something like:

For a huge catalog, the key is to stop treating duplicate/thin content as a copywriting project and treat it as a data quality pipeline:

Detect → prioritize → generate fixes → approve → publish → monitor

A scalable setup usually has four layers.

1. Build a catalog content health dashboard

Create a score for every SKU:

Useful fields to export:

• SKU
• URL
• category
• brand
• description length
• word count
• duplicate cluster ID
• organic traffic
• revenue
• conversion rate
• inventory status

Then prioritize by business impact, not just SEO severity:

1. High revenue + duplicate copy
2. High impressions + thin content
3. Products with backlinks
4. New products launching soon
5. Long-tail inventory

2. Use crawlers for detection

For very large catalogs, you typically need a crawler rather than manually checking pages.

Common options:

• Screaming Frog — good for technical crawling, duplicate metadata, similarity checks, and custom extraction.
• Lumar — enterprise crawling and scheduled monitoring for large sites.
• Semrush — broader SEO monitoring, keyword tracking, and site audits. Semrush

For ecommerce-specific audits, some tools focus directly on product-page duplication, thin descriptions, and catalog issues. WebAudit

At enterprise scale, many teams combine:

• crawler → finds problems
• data warehouse → stores catalog quality metrics
• workflow tool → assigns fixes
• CMS/PIM → publishes approved changes

3. Detect duplicates with similarity, not only exact matches

Exact matching catches only the easy cases.

A better pipeline:

A. Normalize text

Before comparison:

• lowercase
• remove HTML
• remove boilerplate ("free shipping", warranty text)
• remove brand/legal blocks
• stem words

B. Generate similarity scores

Methods:

• TF-IDF cosine similarity (simple)
• MinHash / locality-sensitive hashing (fast for millions of pages)
• embeddings + vector search (best for semantic similarity)

Example:

Product A:
"Premium leather hiking boots designed for outdoor trails..."

Product B:
"Durable leather boots made for hiking and outdoor adventures..."

Similarity: 92%
Action: review as duplicate

Store results as clusters:

Cluster 1842
├── Boot SKU 10021
├── Boot SKU 10045
├── Boot SKU 10391
└── Boot SKU 10922

Then fix the cluster with one content strategy instead of four separate rewrites.

4. Create scalable rewriting rules

Avoid "rewrite every description with AI."

That often creates thousands of mediocre variations.

Instead create templates by category.

Example:

Running shoes template

Opening:
What makes this model different?

Performance:
- cushioning
- weight
- terrain
- durability

Fit:
- sizing
- width
- comfort

Best for:
- beginner runners
- marathon training
- trail use

Specifications:
- materials
- dimensions
- warranty

Then generate from structured product data:

Product facts
      ↓
Category template
      ↓
AI draft
      ↓
Human/brand QA
      ↓
Publish

The AI should expand known facts, not invent specifications.

5. Automate monitoring

Set recurring checks:

Daily

• new SKU scan
• missing descriptions
• duplicate titles
• supplier feed changes

Weekly

• duplicate clusters
• pages losing rankings
• pages with declining CTR

Monthly

• full catalog quality score
• category-level gaps
• content refresh campaigns

Tools that can help automate parts of this include ecommerce SEO platforms and catalog optimization tools; some specifically offer bulk catalog analysis and rewriting workflows. Katapic Catalog Cleanup AI

6. Connect fixes to your PIM/CMS

The workflow should look like:

Supplier feed
     ↓
PIM / Product database
     ↓
Quality scanner
     ↓
Issue queue
     ↓
AI enrichment
     ↓
Approval workflow
     ↓
Shopify / Magento / custom storefront

Avoid editing directly in the storefront. Large catalogs need versioning and rollback.

Example enterprise stack

10k–100k SKUs

• Screaming Frog or Lumar → crawl
• Semrush/Ahrefs → rankings + visibility
• BigQuery/Snowflake → content database
• Python scripts → similarity detection
• LLM API → rewrite drafts
• PIM → approval + publishing

Shopify-heavy catalog

• Shopify export/API
• catalog audit tool
• AI rewrite workflow
• bulk import with approval

Metrics to track

Don't just measure "number of descriptions fixed."

Track:

• % duplicate descriptions remaining
• % products with unique value propositions
• organic clicks per product page
• impressions per SKU
• conversion rate after rewrite
• revenue per optimized SKU
• indexed pages with useful content

A mature catalog operation treats product descriptions like a continuously monitored asset, similar to inventory or pricing data.

Yes. For a huge catalog, I’d treat this as a content-quality pipeline, not a one-off SEO audit. The scalable setup is:

crawl → detect → score → generate/fix → validate → monitor

1. Detect duplicates at the URL level

Use a crawler that can process your entire catalog and export page-level data.

• Sitebulb is particularly good for this because it detects exact duplicates and near-duplicates, including matching chunks of text. Sitebulb
• Screaming Frog is excellent when you want maximum control over custom extraction, APIs, scheduled crawls, and very large technical audits.
• Ahrefs Site Audit is useful as a second layer for duplicate/near-duplicate clusters and canonical issues. Ahrefs
• Google Search Console should be your validation layer: watch for things like "Duplicate without user-selected canonical" and Google selecting a different canonical. Ahrefs

Don't just flag identical descriptions. Calculate a similarity score between product pages. For example:

The exact word threshold shouldn't be treated as an SEO rule. Thin means insufficiently useful, not simply short.

2. Build a product-content score

This is where you make the process scalable.

Create a row for every SKU in a database/data warehouse:

SKU
URL
brand
category
product_name
description
description_length
description_similarity_cluster
title_similarity_cluster
unique_word_ratio
indexable
canonical
organic_clicks
organic_impressions
conversions
revenue
reviews_count
images
specifications
content_score
priority_score

Then calculate something like:

Priority = traffic opportunity × content deficiency × business value

So a 40-word description on a $500 product getting 2,000 impressions/month gets fixed before a 40-word description on a discontinued $8 product.

3. Cluster before rewriting

This is probably the biggest scalability win.

Suppose you have:

• 18,000 black shoes
• 7,000 red shoes
• 4,000 blue shoes

Don't ask an AI to independently rewrite 29,000 descriptions. You'll end up with 29,000 variations of the same generic copy.

Instead:

Cluster → establish content template → generate product-specific fields → QA

For example:

Cluster: running shoes

Required unique information:
- intended runner
- terrain
- cushioning
- heel-to-toe drop
- upper material
- fit
- stability
- weight
- use cases
- meaningful differentiator

Then have the generation system pull these facts from your PIM/product database, rather than inventing them.

4. Use AI for enrichment, not invention

A good scalable workflow looks like:

PIM
 ↓
Product attributes
 ↓
Existing manufacturer copy
 ↓
Reviews / FAQs
 ↓
Search/query data
 ↓
AI enrichment
 ↓
SEO/content rules
 ↓
Human QA for high-value products
 ↓
CMS

Give the model structured facts and require it to return structured fields:

{
  "short_description": "...",
  "key_benefits": [],
  "use_cases": [],
  "technical_summary": "...",
  "buyer_questions": [],
  "content_confidence": 0.0
}

That makes it much easier to automatically reject bad outputs.

Never let the model fill missing product specifications by guessing.

5. Create automated quality gates

Before generated content reaches production, automatically check:

• duplicate/similarity score
• minimum useful-content threshold
• prohibited claims
• unsupported specifications
• keyword stuffing
• repeated sentences
• manufacturer-copy overlap
• brand/product-name consistency
• readability
• HTML validity
• schema consistency

You can even have a second model act as a content QA classifier rather than another writer.

Example:

IF similarity > 85%
    → rewrite

IF unsupported_claim = true
    → reject

IF unique_content_score < threshold
    → rewrite

IF product_attributes_used < 70%
    → reject

IF content_quality_score >= 85
    → publish

6. Don't forget duplicate URLs

This is separate from duplicate copy.

Ecommerce catalogs commonly generate duplicates through:

• color/size parameters
• filters
• sorting
• pagination
• tracking parameters
• variant URLs
• internal search
• faceted navigation

Google specifically warns that faceted navigation can create enormous URL spaces and waste crawling resources. Google for Developers Google for Developers

For products with variant URLs, Google recommends appropriate canonicalization, and its ecommerce guidance recommends self-referencing canonicals for indexable pages. Google for Developers

So your monitoring system should distinguish:

Content duplicate

Two different URLs contain essentially the same copy.

from

URL duplicate

Multiple URLs represent essentially the same product/page.

Those require different fixes.

7. Tools I'd put in the stack

Core technical layer

• screamingfrog.co.uk — crawling, extraction, duplicate detection, technical auditing.
• sitebulb.com — particularly strong for duplicate/near-duplicate analysis and prioritization. Sitebulb
• search.google.com — Google's actual indexing/canonical/search-performance feedback.
• ahrefs.com or semrush.com — competitive/search-demand data plus another site-audit layer.

Content/data layer

• Your PIM as the source of truth.
• A warehouse such as BigQuery/Snowflake/Postgres for SKU-level content scoring.
• An LLM/API for controlled enrichment and rewriting.
• Your CMS/API for automated publishing.

Optional external-duplication layer

For catalogs where manufacturers supply identical copy to many retailers, something like copyscape.com can identify copies appearing elsewhere on the web. Ahrefs

8. The monitoring dashboard I'd actually build

Have one dashboard with:

CATALOG
──────────────
Total SKUs             184,392
Indexable products     161,804
Products needing copy   27,431

DUPLICATION
──────────────
Exact duplicates        8,219
Near-duplicate clusters 4,087
Manufacturer-copy       19,332

THIN CONTENT
──────────────
Critical                 4,821
Needs improvement       22,610

TECHNICAL
──────────────
Wrong canonicals           913
Duplicate URLs           7,201
Unwanted facet URLs     184,000+

QUALITY
──────────────
Generated this month    12,400
Passed QA               11,730
Rejected                   670

Then trend those numbers weekly.

The important part is that you don't want humans looking at 184,000 product pages. Humans should review the exceptions and the highest-value products; everything else should be handled by rules, clustering, and automated QA.

A practical architecture

If you're dealing with 100k+ SKUs, I'd build it roughly like this:

Screaming Frog/Sitebulb → warehouse → similarity clustering → content scoring → AI enrichment → automated QA → CMS/PIM → crawl → Search Console → warehouse

That gives you a closed feedback loop rather than a giant spreadsheet of SEO problems.

Also, don't make "unique text" the sole objective. Google's ecommerce guidance emphasizes useful, interesting content and understanding product/category relationships; merely making every description different isn't necessarily enough. Google for Developers

If you tell me roughly how many SKUs you have (10k / 100k / 1M+) and whether you're on Shopify, Magento, BigCommerce, or a custom stack, I can map this into a concrete tool stack + database schema + automated workflow.

Content/data layer

• Your PIM as the source of truth.
• A warehouse such as BigQuery/Snowflake/Postgres for SKU-level content scoring.
• An LLM/API for controlled enrichment and rewriting.
• Your CMS/API for automated publishing.

Optional external-duplication layer

For a huge ecommerce catalog, I’d treat this as a content-quality pipeline, not a one-time SEO audit. The scalable setup is: detect → classify → prioritize → generate/rewrite → QA → publish → re-crawl.

1. Detect duplicates at multiple levels

Don't rely on one "duplicate content" percentage.

• Exact duplicates: hash the normalized product description/body. Great for catching syndicated or accidentally copied descriptions.
• Near duplicates: use MinHash/SimHash or embeddings to find descriptions that differ only by product name, color, size, etc.
• Template overlap: strip navigation, boilerplate, shipping text, manufacturer disclaimers, etc. before comparing the actual product content.
• Cross-product similarity: embeddings are particularly useful for finding 500 products that are essentially saying the same thing in different words.

screamingfrog.co.uk is particularly useful here: its crawler can identify exact duplicates and configurable near-duplicates, with near-duplicate detection based on MinHash. Screaming Frog Screaming Frog

For a very large site, semrush.com is another good monitoring layer; it can schedule recurring crawls and identify duplicate-content problems and low-content pages. Semrush Semrush

2. Don't define "thin" as simply "under 300 words"

For products, word count alone is a weak signal.

Build a product-content score from fields such as:

This lets you distinguish:

Good short page:
A $9 replacement screw with complete specifications and little search demand.

Bad thin page:
A high-value product with 37 words copied across 200 nearly identical SKUs.

Tools such as sitebulb.com can help with this because they combine word-count/thin-content analysis with duplicate-content detection. Sitebulb

3. Build a prioritization score

This is where scalability really comes from.

Instead of asking "Which 100,000 descriptions are bad?", calculate something like:

Priority = SEO opportunity × content problem × business value

For example:

priority =
  organic_opportunity
  × product_revenue
  × duplicate/thin severity
  × indexability

Then your queue might look like:

1. 2,400 high-revenue products with >90% duplicate descriptions
2. 8,700 indexed products with extremely thin content
3. 15,000 products with moderate duplication but significant search demand
4. 80,000 low-value SKUs where rewriting probably isn't worth the cost

That prevents your content team from spending six months rewriting pages nobody searches for.

4. Separate "fix" into four actions

Don't automatically rewrite everything.

A. Consolidate

If 15 URLs represent essentially the same product, determine whether they should really be separate pages.

Potential fixes include:

• 301 redirect
• canonicalization
• removing unnecessary URL variants
• consolidating product pages

Semrush likewise recommends canonicalization, redirects, or adding genuinely unique content depending on the situation. Semrush

B. Keep separate + differentiate

For legitimate product variants, retain separate pages when they have meaningful user/search value—but make their content materially useful and distinct.

For example, don't produce:

"The Nike Air Zoom X is a comfortable running shoe available in blue."
"The Nike Air Zoom X is a comfortable running shoe available in red."

Instead differentiate based on actual attributes, use cases, sizing, materials, compatibility, specifications, etc.

C. Expand

For genuinely valuable thin pages, generate content from structured product data, not from the existing description alone.

Your PIM/catalog should ideally provide:

brand
model
product_type
materials
dimensions
compatibility
features[]
use_cases[]
specifications{}
certifications[]
warranty
manufacturer_data

Then use those fields to produce consistent but product-specific copy.

D. Leave alone

This is important.

A similarity flag isn't automatically an SEO problem. Legitimate variants can naturally have highly similar descriptions. Screaming Frog explicitly recommends manually reviewing near duplicates and deciding whether pages should be improved, consolidated, removed, or left alone. Screaming Frog

5. For AI rewriting, use a structured generation pipeline

I would not give an LLM:

"Rewrite these 500,000 product descriptions to be unique."

Instead:

PIM data
   ↓
Content-quality rules
   ↓
Duplicate/semantic clustering
   ↓
Priority scoring
   ↓
LLM generation
   ↓
Automated factual QA
   ↓
Human review for high-value products
   ↓
PIM/CMS
   ↓
Site
   ↓
Crawler + Search Console
   ↓
Monitoring

The model should receive the authoritative product attributes and be explicitly prohibited from inventing missing specifications.

Then automatically check:

• Does every specification match the PIM?
• Did the model invent a feature?
• Is the description sufficiently different from its cluster?
• Does it contain required product attributes?
• Is it within the desired length?
• Is it readable?
• Does it accidentally copy another product?
• Did it preserve brand/legal terminology?

6. The scalable tool stack

I'd choose tools by layer rather than trying to find one tool that does everything.

Crawl + duplicate detection

Screaming Frog — excellent for technical crawling, exact duplicates and configurable near-duplicate analysis. Screaming Frog Screaming Frog

Sitebulb — strong visual auditing plus word counts, duplicate detection and content analysis. Sitebulb

Semrush Site Audit — useful when you want recurring monitoring, issue tracking and broader SEO reporting. Semrush Semrush

At millions of products

For truly enormous catalogs, I'd move the core detection logic into your data warehouse/data pipeline rather than making a crawler do all the work.

For example:

Product DB / PIM
       ↓
Warehouse
       ↓
SQL rules ───────→ missing/thin fields
       ↓
Embeddings ──────→ semantic duplicates
       ↓
Similarity clusters
       ↓
Priority queue
       ↓
Content generation
       ↓
QA
       ↓
PIM

The crawler then becomes the external validation layer, rather than the primary database.

7. Monitor it continuously

Create a dashboard with metrics such as:

• % products with unique descriptions
• % products below your content-quality threshold
• number of duplicate clusters
• average similarity within product clusters
• indexed thin pages
• high-revenue thin pages
• high-revenue duplicate pages
• pages rewritten this month
• pages successfully re-crawled
• organic clicks/impressions before vs. after
• conversion rate before vs. after

Semrush's Site Audit supports recurring audits and trend monitoring, which is useful for catching regressions after catalog changes. Semrush Semrush

The key architectural decision

If you're talking about 100K+ or millions of SKUs, I'd use:

PIM/database → warehouse + embeddings → automated scoring → LLM content pipeline → CMS/PIM → Screaming Frog/Semrush monitoring

rather than trying to manage the catalog manually inside an SEO tool.

The SEO tools are excellent detectors and validators; your catalog data + warehouse should be the system of record.

If you tell me roughly whether you're dealing with 50K, 500K, or 5M+ products and what platform you're on (Shopify, Magento/Adobe Commerce, Salesforce Commerce, custom, etc.), I can lay out a concrete scalable architecture and tool stack.

• % products with unique descriptions
• % products below your content-quality threshold
• number of duplicate clusters
• average similarity within product clusters
• indexed thin pages
• high-revenue thin pages
• high-revenue duplicate pages
• pages rewritten this month
• pages successfully re-crawled
• organic clicks/impressions before vs. after
• conversion rate before vs. after

Semrush's Site Audit supports recurring audits and trend monitoring, which is useful for catching regressions after catalog changes.

The key architectural decision

If you're talking about 100K+ or millions of SKUs, I'd use:

"Rewrite these 500,000 product descriptions to be unique."

Instead:

PIM data
   ↓
Content-quality rules
   ↓
Duplicate/semantic clustering
   ↓
Priority scoring
   ↓
LLM generation
   ↓
Automated factual QA
   ↓
Human review for high-value products
   ↓
PIM/CMS
   ↓
Site
   ↓
Crawler + Search Console
   ↓
Monitoring

The model should receive the authoritative product attributes and be explicitly prohibited from inventing missing specifications.

Then automatically check:

• Does every specification match the PIM?
• Did the model invent a feature?
• Is the description sufficiently different from its cluster?
• Does it contain required product attributes?
• Is it within the desired length?
• Is it readable?
• Does it accidentally copy another product?
• Did it preserve brand/legal terminology?

6. The scalable tool stack

I'd choose tools by layer rather than trying to find one tool that does everything.

Crawl + duplicate detection

Screaming Frog — excellent for technical crawling, exact duplicates and configurable near-duplicate analysis.

Sitebulb — strong visual auditing plus word counts, duplicate detection and content analysis.

Semrush Site Audit — useful when you want recurring monitoring, issue tracking and broader SEO reporting.

At millions of products

Yes. For a huge catalog, I’d treat this as a content-quality pipeline, not a one-off SEO audit.

The key is to separate true duplicates, near-duplicates, and thin-but-unique pages—because the fixes are different. Google explicitly recommends canonicalization for duplicate/very similar URLs, while useful ecommerce pages should provide substantive value rather than just low word counts. Google for Developers Semrush

A scalable setup

1. Crawl the entire catalog continuously

Use Screaming Frog SEO Spider, Sitebulb, or Ahrefs Site Audit.

For each product URL, export at least:

• URL
• canonical URL
• HTTP status
• indexability
• title/meta description
• H1
• product description
• word/character count
• internal links
• structured-data status
• product ID/SKU
• category/brand
• organic traffic
• conversions/revenue

Sitebulb, for example, can identify URLs containing large identical chunks of body content and group similar pages. Sitebulb Ahrefs also has duplicate-content reporting in Site Audit. Ahrefs

2. Detect duplicates algorithmically

Don't rely on an SEO crawler's simple "duplicate title" report.

Build a similarity layer over your product descriptions:

SKU → description → normalized text → similarity fingerprint

Use several levels:

• Exact duplicate: normalized text hash is identical.
• Near duplicate: cosine similarity / TF-IDF / MinHash / SimHash exceeds a threshold.
• Template-heavy: e.g. 85% manufacturer boilerplate + 15% unique attributes.
• Variant duplicate: different color/size URLs contain essentially the same content.
• Cross-category duplicate: same product description reused in multiple taxonomy locations.

For very large catalogs, MinHash/LSH is particularly useful because you don't want to compare every product against every other product.

A useful output looks like:

3. Score "thin" based on usefulness, not word count

This is important. 500 words of generic fluff isn't necessarily better than 150 useful words.

Create a product-content score from fields such as:

Content score =
  description completeness
+ unique attributes
+ specifications
+ use cases
+ compatibility information
+ FAQs
+ reviews/UGC
+ imagery/video
+ buying guidance

Then prioritize pages using something like:

Priority =
  organic traffic potential
× revenue/value
× content deficiency
× duplicate severity

That prevents your team from spending three hours rewriting a $9 product with zero demand while ignoring a high-margin product generating thousands of searches.

Google's ecommerce guidance emphasizes useful, interesting content and making important products discoverable through site structure and links. Google for Developers

4. Fix each problem differently

Exact/near duplicates

Use:

• rel="canonical" when multiple URLs genuinely represent the same product/content.
• 301 redirects when an old/duplicate URL should disappear.
• Better URL/variant architecture where appropriate.
• Consistent internal links to the canonical URL.
• XML sitemaps containing your preferred URLs.

Google says redirects and canonical tags are strong canonicalization signals, while sitemap inclusion is weaker. Google for Developers

Don't use robots.txt as your primary duplicate-content solution; Google specifically says it isn't a canonicalization mechanism. Google for Developers

Manufacturer-copy products

Don't just run the manufacturer description through an LLM and call it fixed.

Create a structured enrichment template:

Short summary
Key benefits
Who it's for
Important specifications
Compatibility
Dimensions/materials
Use cases
What's included
Installation/setup
Common questions
Customer-generated insights

Then generate/enrich from structured product facts in your PIM, with human review for important products.

Screaming Frog specifically calls out manufacturer-supplied product descriptions as a common ecommerce duplicate-content problem and recommends adding genuinely original material such as reviews/UGC. Screaming Frog

Thin but strategically important products

Prioritize enrichment rather than deleting them.

For example:

"Nike running shoe, black, men's."

is technically unique but provides little value.

Instead:

intended runner + cushioning characteristics + terrain + fit + materials + sizing guidance + notable features + compatibility/use case + genuine FAQs.

The goal is more information density, not simply more words.

5. Put the monitoring into your data pipeline

For a 100k+ SKU catalog, I'd avoid making SEO the system of record.

A good architecture is:

PIM / Product DB
       ↓
Content-quality pipeline
       ↓
Duplicate detection ───┐
Thin-content scoring ──┤
Schema validation ─────┤
Canonical validation ──┤
Indexability checks ───┤
       ↓                │
SEO issue database ←────┘
       ↓
Dashboard
       ↓
CMS/PIM remediation
       ↓
Re-crawl
       ↓
Google Search Console

That lets you monitor new problems automatically instead of discovering them during quarterly audits.

The tool stack I'd consider

Google recommends monitoring product structured data in Search Console after deployment and after template/code changes. Google for Developers

If I were managing 500k SKUs

I'd make the system automatically produce four queues every week:

1. 🔴 Duplicate URLs — canonical/redirect/architecture fix.
2. 🟠 Duplicate descriptions — rewrite/enrich.
3. 🟡 Thin high-value products — prioritize for editorial/AI enrichment.
4. 🟢 Healthy products — monitor only.

And I'd add a fifth queue:

"Newly degraded" — products that were previously unique/useful but became duplicated or thin after a supplier feed, catalog migration, or template change.

That last one is particularly valuable at scale.

The big principle is: don't try to manually audit 500,000 pages. Build a scoring and clustering system that tells humans which 500 pages are worth touching this week.

Screaming Frog specifically calls out manufacturer-supplied product descriptions as a common ecommerce duplicate-content problem and recommends adding genuinely original material such as reviews/UGC. Screaming Frog

Thin but strategically important products

Prioritize enrichment rather than deleting them.

For example:

"Nike running shoe, black, men's."

is technically unique but provides little value.

Instead:

intended runner + cushioning characteristics + terrain + fit + materials + sizing guidance + notable features + compatibility/use case + genuine FAQs.

The goal is more information density, not simply more words.

5. Put the monitoring into your data pipeline

For a 100k+ SKU catalog, I'd avoid making SEO the system of record.

A good architecture is:

PIM / Product DB
       ↓
Content-quality pipeline
       ↓
Duplicate detection ───┐
Thin-content scoring ──┤
Schema validation ─────┤
Canonical validation ──┤
Indexability checks ───┤
       ↓                │
SEO issue database ←────┘
       ↓
Dashboard
       ↓
CMS/PIM remediation
       ↓
Re-crawl
       ↓
Google Search Console

That lets you monitor new problems automatically instead of discovering them during quarterly audits.

The tool stack I'd consider

Google recommends monitoring product structured data in Search Console after deployment and after template/code changes.

For a very large product catalog, you need to treat duplicate/thin content as a data quality monitoring problem, not a one-time SEO audit. The scalable approach is a pipeline:

1. Detect problems automatically
2. Prioritize by business impact
3. Generate or assign fixes
4. Push approved changes back into your PIM/CMS
5. Continuously monitor regressions

Here is a practical stack and workflow.

1. Build a catalog content health layer

Create a product-level scoring system (SKU-level, not URL-level):

For huge catalogs, avoid manually exporting spreadsheets. Store these checks in a database or data warehouse so you can query:

• "Show all duplicate descriptions affecting indexed products"
• "Find the worst 1,000 product pages by traffic opportunity"
• "Show categories where 80% of products have thin copy"

2. Crawl your storefront continuously

Use enterprise crawlers to catch issues Google actually sees:

• Duplicate URLs from variants
• Duplicate titles/meta descriptions
• Canonical problems
• Indexation problems
• Missing structured data

Tools commonly used for this include:

• Screaming Frog — strong for custom crawls and large exports.
• Ahrefs — useful for site auditing, search visibility, and prioritization.
• Semrush — combines technical audits with keyword and competitive analysis. Semrush

For ecommerce-specific auditing, some newer tools focus directly on product descriptions, catalog fields, and duplicate listings. WebAudit StoreCheck

3. Detect duplicate descriptions at scale

Use multiple levels of matching:

Level 1: Exact duplicate detection

Fast and cheap:

description_hash = SHA256(description_text)

Anything with the same hash is identical.

Good for:

• Manufacturer copy reused everywhere
• Accidental duplication

Level 2: Near duplicate detection

Use:

• TF-IDF cosine similarity
• MinHash/LSH
• Embeddings + vector search

Example:

"Premium leather hiking boot with waterproof lining..."

and

"Waterproof leather boot designed for hiking..."

should be grouped even though they are not identical.

A scalable architecture:

Product DB
    |
    v
Text cleaning
    |
    v
Embedding generation
    |
    v
Vector database
    |
    v
Similarity clusters
    |
    v
Content rewrite queue

4. Define what "thin" means by category

A common mistake is using one global word count.

A phone case and a refrigerator need different content depth.

Create category rules:

Example:

Running shoes

• Materials present
• Cushioning explained
• Fit guidance
• Use cases
• Size information

Laptop

• Processor
• RAM/storage
• Ports
• Battery
• Display details
• Compatibility

A product description score might be:

Content Score =
40% attribute completeness
25% uniqueness
20% search intent coverage
15% readability

5. Use AI for rewriting — but with guardrails

Do not blindly regenerate 500,000 descriptions.

Better workflow:

1. Identify worst performers
2. Provide structured product facts
3. Generate improved copy
4. Validate against attributes
5. Human approve high-value items

Good AI inputs:

Product:
Brand:
Category:
Specifications:
Customer questions:
Competitor gaps:
Tone:
Forbidden claims:

Avoid:

"Write a better description for this SKU"

because it often invents features.

6. Prioritize fixes using business impact

A duplicate description on a product receiving 50,000 monthly visits matters more than one on an obscure SKU.

Create an opportunity score:

Priority =
Organic traffic potential
× Conversion value
× Content weakness
× Search demand

Typical order:

1. High traffic + duplicate descriptions
2. Best-selling products with thin copy
3. Products ranking positions 5–20
4. Long-tail products
5. Low inventory/discontinued items last

7. Connect monitoring to your catalog system

The ideal setup:

PIM / ERP
    |
    v
Content Quality Engine
    |
    +--> Duplicate detector
    |
    +--> Thin content scorer
    |
    +--> AI improvement queue
    |
    v
Approval workflow
    |
    v
Shopify / Magento / Custom CMS

Useful enterprise components:

• PIM: Akeneo, Salsify, Pimcore
• Data processing: SQL + Python pipelines
• Search/indexing: Elasticsearch or vector databases
• Monitoring: SEO crawler + Search Console data

8. Set up recurring alerts

Examples:

Daily:

• New products missing descriptions
• New duplicate clusters
• Products published with copied manufacturer text

Weekly:

• Biggest content quality drops
• Categories with declining uniqueness
• Pages losing impressions

Monthly:

• Full catalog quality score

A realistic enterprise stack

For a catalog with 100k–1M SKUs:

• Crawler: Screaming Frog / enterprise SEO crawler
• SEO data: Search Console + Ahrefs/Semrush
• Catalog source: PIM/ERP export
• Duplicate engine: embeddings + vector database
• Workflow: Jira/Asana/custom dashboard
• Generation: LLM-assisted rewrite pipeline
• Approval: merchandising team review

The biggest scaling win usually comes from not rewriting everything. First cluster and score the catalog, then spend human and AI effort on the few thousand products where better content can actually change revenue.

because it often invents features.

6. Prioritize fixes using business impact

A duplicate description on a product receiving 50,000 monthly visits matters more than one on an obscure SKU.

Create an opportunity score:

Priority =
Organic traffic potential
× Conversion value
× Content weakness
× Search demand

Typical order:

1. High traffic + duplicate descriptions
2. Best-selling products with thin copy
3. Products ranking positions 5–20
4. Long-tail products
5. Low inventory/discontinued items last

7. Connect monitoring to your catalog system

The ideal setup:

PIM / ERP
    |
    v
Content Quality Engine
    |
    +--> Duplicate detector
    |
    +--> Thin content scorer
    |
    +--> AI improvement queue
    |
    v
Approval workflow
    |
    v
Shopify / Magento / Custom CMS

Useful enterprise components:

For a huge catalog, treat duplicate/thin product copy as a data-quality pipeline, not a one-off SEO project. The scalable approach is:

1. Detect → prioritize → rewrite → QA → monitor
2. Automate the first pass, but keep approval controls before publishing.

1. Build a catalog content health score

Create a score per SKU using signals like:

Your output should be a queue like:

• 500 products: exact duplicate descriptions
• 2,000 products: thin descriptions
• 10,000 products: missing attributes
• 50,000 products: acceptable

Don't rewrite everything.

2. Use crawling + similarity detection

For large catalogs, combine:

Technical SEO crawlers

Good for finding pages, metadata issues, indexing problems, and duplicate page patterns:

• Screaming Frog SEO Spider — crawl millions of URLs with the right configuration.
• Semrush — useful for audits, rankings, and content issues.
• Ahrefs — useful for organic visibility and competitor/content research.

Similarity engines

For product-description duplication, use text fingerprinting:

• Exact hash matching (fastest)
• TF-IDF cosine similarity
• MinHash/LSH clustering
• Embedding similarity (best for meaning-level duplicates)

Example:

Products A and B:

"Premium stainless steel bottle keeps beverages cold for 24 hours..."
"Double-wall steel bottle maintains cold drinks for a full day..."

A basic duplicate checker may miss this; embeddings will catch it.

3. Segment before rewriting

A common mistake is generating thousands of generic descriptions.

Create rewrite rules by product family:

Example:

Running shoes

• cushioning
• terrain
• weight
• drop
• fit
• runner type

Laptops

• processor
• RAM
• display
• battery
• ports
• use case

Furniture

• dimensions
• materials
• assembly
• room fit
• maintenance

The content template should pull from structured product data, not invent details.

4. Tools that work at catalog scale

Catalog auditing / cleanup tools

• screamingfrog.co.uk — technical crawling and large-scale URL/content audits.
• semrush.com — SEO auditing, keyword tracking, and competitive research.
• ahrefs.com — backlink, ranking, and content analysis.

For ecommerce-specific bulk content workflows, tools in this category include platforms that audit product descriptions and generate prioritized fixes. Ryze AI StoreCheck

AI rewriting / enrichment layer

Look for tools that support:

• CSV/API ingestion
• product attribute grounding
• category-specific prompts
• approval workflows
• rollback/version history
• bulk publishing

Examples:

• ecommercelayer.com — focuses on bulk catalog rewriting workflows and review queues. EcommerceLayer
• getperennial.com — focuses on ecommerce product/category content optimization workflows. Perennial

5. Create an automated workflow

A mature setup looks like:

PIM / Shopify / Magento / ERP
              |
              v
       Catalog export/API
              |
              v
     Content quality scanner
              |
     +--------+---------+
     |                  |
 Duplicate clusters   Thin pages
     |                  |
     +--------+---------+
              |
              v
       Rewrite engine
              |
              v
     Human approval queue
              |
              v
        Publish updates
              |
              v
     Monitor rankings/conversions

6. Measure whether fixes worked

Track:

SEO metrics

• Indexed product pages
• Organic impressions
• Product-page rankings
• Rich-result eligibility

Engagement metrics

• CTR from search
• Add-to-cart rate
• Conversion rate
• Revenue per landing page

Content metrics

• % unique descriptions
• Average description depth
• Attribute completeness
• Duplicate cluster count

7. Prioritize fixes using an impact model

A simple formula:

Priority score =
(Search impressions × conversion value × ranking opportunity)
÷ rewrite effort

Start with:

1. High-impression products with bad copy
2. Best sellers with supplier descriptions
3. Category pages driving product discovery
4. Long-tail products with zero visibility

Avoid spending weeks rewriting products nobody searches for.

For a catalog of 10k+ SKUs, I would typically combine:

• a crawler (technical detection),
• a database layer for similarity scoring,
• an AI enrichment pipeline grounded in product attributes,
• and a review/publish workflow.

The key is not “generate more descriptions”; it is building a system that continuously identifies which descriptions are worth improving and why.

For a huge ecommerce catalog, I’d treat this as a content-quality pipeline, not a one-off SEO audit. The scalable setup is:

crawl → classify → prioritize → generate/fix → validate → monitor

1. Detect duplicates at multiple levels

You want to distinguish true duplicates, near-duplicates, and legitimate variants.

• screamingfrog.co.uk — excellent for large technical crawls and near-duplicate detection. Its near-duplicate system uses MinHash and lets you adjust the similarity threshold; 90% is the default. Screaming Frog Screaming Frog
• sitebulb.com — particularly nice for this use case because it reports exact duplicates and similar content, plus duplicate titles, H1s and meta descriptions. Sitebulb Support Sitebulb
• semrush.com — useful if you also want broader technical SEO monitoring and historical issue tracking. Semrush Semrush

For a catalog with hundreds of thousands or millions of SKUs, I'd generally use the crawler to identify clusters, rather than manually inspecting URLs.

For example:

Don't automatically treat every duplicate as an SEO problem. Some duplication is intentional; the important question is which URL should exist/index and whether separate pages provide enough unique value. Sitebulb and Screaming Frog both explicitly recommend contextual review rather than blindly fixing every duplicate. Sitebulb Support Screaming Frog

2. Build a thin-content score instead of using word count alone

Word count is a useful signal, but "500 words = good" is a bad rule.

For every product, I'd calculate something like:

Content Quality Score =

• description length
• uniqueness vs. catalog
• product-specific facts covered
• search demand/value
• organic traffic
• conversions/revenue
• backlinks
• indexability
• similarity to competing products
• presence of useful attributes/specifications

A 150-word description for a highly specialized product might be perfectly adequate. Conversely, 800 words of manufacturer boilerplate can still be terrible.

Sitebulb, for example, provides word-count analysis specifically to surface potentially thin pages while allowing you to determine the appropriate threshold for your site. Sitebulb Support Sitebulb

3. Create a catalog-level "content inventory"

This is where scalability really comes from.

Export your catalog into a warehouse/dataframe with roughly:

product_id
url
category
brand
sku
title
description
word_count
description_hash
similarity_cluster
canonical_url
indexability
organic_clicks
organic_impressions
revenue
conversions
backlinks
search_demand
last_updated
content_quality_score

Then cluster products based on description similarity.

For example:

Cluster 1842
├── SKU-1001  98% similar
├── SKU-1002  97% similar
├── SKU-1003  96% similar
├── SKU-1004  96% similar
└── SKU-1005  95% similar

Now you don't have 5,000 problems.

You have one content template problem affecting 5,000 SKUs.

That's the key insight for huge catalogs.

4. Fix the source of duplication first

If descriptions are coming from a PIM, manufacturer feed, ERP, marketplace feed, or supplier API, don't just rewrite the pages.

Fix the upstream process.

A good architecture is:

Manufacturer data

→ structured product attributes
→ internal product-content model
→ unique editorial/AI enrichment
→ CMS/PIM
→ ecommerce site
→ crawler monitoring

Otherwise you'll spend months rewriting 200,000 descriptions and have the same problem when the next catalog import overwrites them.

5. Use AI for enrichment—but constrain it heavily

For thousands of products, AI can make the actual rewriting scalable, but don't ask:

"Write an SEO product description for this product."

Instead give the model structured facts:

{
  "brand": "...",
  "product": "...",
  "category": "...",
  "materials": ["..."],
  "dimensions": "...",
  "features": ["...", "..."],
  "use_cases": ["...", "..."],
  "compatibility": ["..."],
  "existing_description": "..."
}

Then enforce rules such as:

• Never invent specifications.
• Don't claim certifications unless supplied.
• Don't change numerical values.
• Don't repeat the manufacturer description.
• Include genuinely product-specific information.
• Don't keyword-stuff.
• Preserve important technical terminology.
• Flag insufficient source data instead of hallucinating.

I'd also have the system produce a confidence/coverage score rather than automatically publishing everything.

6. Prioritize instead of rewriting everything

This is where your business data becomes extremely valuable.

A useful priority formula is:

Priority = SEO opportunity × business value × content deficiency

So:

P1: high-revenue + high-search-demand + highly duplicated/thin
P2: high-search-demand + mediocre content
P3: low traffic but strategic category/product
P4: low-value long-tail products
P5: discontinued/low-value products → consolidate or remove

You might discover that rewriting 8,000 products produces 80% of the potential gain, while the remaining 400,000 aren't worth touching manually.

7. Monitor it continuously

Don't make this a quarterly crawl.

I'd set up:

Daily/weekly

• New products with missing descriptions
• Description changes
• Duplicate-description clusters
• Very short descriptions
• Products accidentally inheriting manufacturer copy

Monthly

• Full crawl
• Similarity analysis
• Indexability/canonical audit
• Organic performance comparison
• Content-quality trend

After every catalog import

• Run a delta check against the previous product dataset.

That last one is especially important. You want the system to catch:

"Supplier feed just created 17,432 near-duplicate descriptions."

before Google finds them.

My recommended stack

For a serious catalog, I'd use something like:

Crawler: Screaming Frog or Sitebulb
SEO monitoring: Semrush
Search data: Google Search Console + your rank/keyword platform
Product data: PIM/ERP/catalog database
Processing: SQL + Python/data warehouse
Similarity: MinHash/embeddings
Generation: LLM API with structured product attributes
QA: automated validation + human review for high-value products
Deployment: API/batch job back into the PIM/CMS

Sitebulb is particularly attractive if duplicate/thin-content auditing is the primary problem because it can extract custom content during crawling as well as analyze duplicates and on-page data. Sitebulb Support

The scalable workflow I'd actually implement

                 PRODUCT CATALOG
                       │
                       ▼
                ┌──────────────┐
                │ Crawl / Data │
                │    Import    │
                └──────┬───────┘
                       ▼
             ┌───────────────────┐
             │ Content Analysis  │
             │                   │
             │ duplicate         │
             │ near-duplicate    │
             │ thin              │
             │ missing           │
             │ stale             │
             └─────────┬─────────┘
                       ▼
               PRIORITY SCORING
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
      Consolidate   Rewrite      Leave
      /canonical    /enrich      alone
          │            │
          └──────┬─────┘
                 ▼
             QA / Rules
                 │
                 ▼
             PIM / CMS
                 │
                 ▼
             Live Site
                 │
                 ▼
          Crawl + GSC Data
                 │
                 └──────────► repeat

The biggest mistake to avoid: building an AI system that blindly rewrites every thin product page. The scalable solution is to first determine which pages deserve unique content, which should be consolidated, and which are legitimate variants, then automate only the appropriate action.

If you tell me roughly how many SKUs you have (50k / 500k / 5M+) and what platform you're on (Shopify, Magento/Adobe Commerce, Salesforce Commerce, custom, etc.), I can lay out a concrete architecture and tool stack—including how I'd implement the duplicate clustering and automated rewrite/QA pipeline.

For thousands of products, AI can make the actual rewriting scalable, but don't ask:

"Write an SEO product description for this product."

Instead give the model structured facts:

{
  "brand": "...",
  "product": "...",
  "category": "...",
  "materials": ["..."],
  "dimensions": "...",
  "features": ["...", "..."],
  "use_cases": ["...", "..."],
  "compatibility": ["..."],
  "existing_description": "..."
}

Then enforce rules such as:

• Never invent specifications.
• Don't claim certifications unless supplied.
• Don't change numerical values.
• Don't repeat the manufacturer description.
• Include genuinely product-specific information.
• Don't keyword-stuff.
• Preserve important technical terminology.
• Flag insufficient source data instead of hallucinating.

I'd also have the system produce a confidence/coverage score rather than automatically publishing everything.

6. Prioritize instead of rewriting everything

This is where your business data becomes extremely valuable.

A useful priority formula is:

Priority = SEO opportunity × business value × content deficiency

So:

P1: high-revenue + high-search-demand + highly duplicated/thin
P2: high-search-demand + mediocre content
P3: low traffic but strategic category/product
P4: low-value long-tail products
P5: discontinued/low-value products → consolidate or remove

You might discover that rewriting 8,000 products produces 80% of the potential gain, while the remaining 400,000 aren't worth touching manually.

7. Monitor it continuously

Don't make this a quarterly crawl.

I'd set up:

Daily/weekly

• New products with missing descriptions
• Description changes
• Duplicate-description clusters
• Very short descriptions
• Products accidentally inheriting manufacturer copy

Monthly

• Full crawl
• Similarity analysis
• Indexability/canonical audit
• Organic performance comparison
• Content-quality trend

After every catalog import

• Run a delta check against the previous product dataset.

That last one is especially important. You want the system to catch:

"Supplier feed just created 17,432 near-duplicate descriptions."

before Google finds them.

My recommended stack

For a serious catalog, I'd use something like:

Crawler: Screaming Frog or Sitebulb
SEO monitoring: Semrush
Search data: Google Search Console + your rank/keyword platform
Product data: PIM/ERP/catalog database
Processing: SQL + Python/data warehouse
Similarity: MinHash/embeddings
Generation: LLM API with structured product attributes
QA: automated validation + human review for high-value products
Deployment: API/batch job back into the PIM/CMS

Sitebulb is particularly attractive if duplicate/thin-content auditing is the primary problem because it can extract custom content during crawling as well as analyze duplicates and on-page data.

The scalable workflow I'd actually implement

For a huge catalog, the key is to stop treating duplicate/thin content as a “content writing” problem and build a catalog quality pipeline:

Detect → Prioritize → Generate fixes → Human approve → Publish → Monitor

At 10k–1M SKUs, manual review does not scale. You need automated scoring, clustering, and workflows.

1. Build a catalog content health score

Create a score per product page using signals like:

Store this in a dashboard so your team works on the highest-value pages first.

2. Use crawlers to find problems automatically

For large catalogs, start with a crawler that can identify:

• Duplicate titles
• Duplicate meta descriptions
• Near-identical product descriptions
• Missing content fields
• Canonical problems
• Index bloat from variants/facets

Common choices:

• Screaming Frog SEO Spider — good for technical crawling, exports, and custom extraction rules.
• Sitebulb — strong visual audits and prioritization.
• semrush.com — combines technical SEO auditing with ecommerce-focused reporting.
• Ahrefs — useful for organic performance, competing pages, and keyword gaps.

For very large sites, combine crawler output with your product database rather than crawling blindly.

3. Detect near-duplicates with similarity clustering

Exact duplicate checks are not enough. The biggest issue is usually:

“Same supplier description with a few words changed.”

Use semantic similarity:

Approaches

• TF-IDF / cosine similarity (cheap, traditional)
• Sentence embeddings + vector database (better)
• LLM classification for edge cases

Example workflow:

Product descriptions
        ↓
Generate embeddings
        ↓
Cluster similar descriptions
        ↓
Flag clusters >80% similarity
        ↓
Rewrite only highest-value SKUs

Tools that can help:

• OpenAI embeddings/API for similarity pipelines
• Pinecone or similar vector databases for large-scale matching

4. Prioritize rewrites instead of rewriting everything

A common mistake is rewriting 100,000 descriptions equally.

Use a priority formula:

Priority =
(Search impressions)
× (Revenue potential)
× (Duplicate/thin score)
× (Conversion opportunity)

Typical order:

1. Products already getting impressions but not clicks
2. Best-selling products with weak copy
3. Category leaders
4. Long-tail products
5. Low-demand SKUs

5. Generate descriptions with guardrails

Avoid “AI rewrite everything” because it creates thousands of similar pages.

A scalable setup:

Input data

• Product specs
• Brand rules
• Category template
• Customer questions
• Competitor SERP patterns

Output structure

Example:

Short summary:
Who it's for:
Key benefits:
Technical specifications:
Use cases:
Comparison points:
FAQs:

Rules:

• Never invent specs
• Preserve manufacturer facts
• Add category-specific differentiation
• Include buyer intent language

Platforms built for catalog-scale rewriting include tools such as ecommercelayer.com and getperennial.com, which focus on bulk product/category optimization workflows.

6. Connect the workflow to your PIM or CMS

The ideal architecture:

ERP / PIM
   |
   ↓
Content quality engine
   |
   ├── Duplicate detector
   ├── Thin-content scorer
   ├── AI rewrite queue
   └── Approval workflow
          |
          ↓
 Shopify / Magento / WooCommerce / custom CMS

Useful catalog systems:

• Akeneo
• Salsify
• Pimcore

They help separate product data management from storefront publishing.

7. Monitor continuously

Set up recurring checks:

Weekly

• New duplicate descriptions
• New missing attributes
• Products added without SEO fields

Monthly

• Pages losing impressions
• Products with declining CTR
• Cannibalized keywords

Quarterly

• Rewrite underperforming product groups
• Update templates
• Remove obsolete variants

Google Search Console data is especially useful for finding pages that Google sees but users ignore.

Example stack for a 100k SKU store

Budget-conscious

• Screaming Frog crawl
• Google Search Console
• BigQuery export
• Custom similarity script
• LLM rewrite queue

Mid-market

• Semrush/Sitebulb
• PIM system
• Vector similarity database
• AI content workflow

Enterprise

• Enterprise crawler
• PIM + DAM integration
• Automated scoring service
• Human approval queues
• Continuous monitoring

The scalable mindset is: don’t ask “how do we write 100,000 descriptions?” Ask “which 5,000 pages are worth improving, what pattern caused the weakness, and how do we prevent it from returning?”

• Semrush/Sitebulb
• PIM system
• Vector similarity database
• AI content workflow

Enterprise

• Enterprise crawler
• PIM + DAM integration
• Automated scoring service
• Human approval queues
• Continuous monitoring

The scalable mindset is: don’t ask “how do we write 100,000 descriptions?” Ask “which 5,000 pages are worth improving, what pattern caused the weakness, and how do we prevent it from returning?”

For a huge product catalog, the key is to stop treating duplicate/thin content as a one-time SEO audit and build a continuous catalog quality system:

1. Detect problems automatically
2. Prioritize which SKUs matter
3. Generate/fix content at scale
4. Prevent new bad listings from entering the catalog

A scalable setup usually looks like this:

1. Build a catalog content health score

Create a score per SKU (0–100) using signals like:

The output should be something like:

SKU        Issue                         Priority
12345      Description copied 92%        High
67890      45-word description           Medium
54321      Missing specs                 High

Do not start by rewriting everything. Start with the products that have the biggest revenue/search impact.

2. Use crawling + SEO tools for detection

For very large catalogs, combine several types of tools:

Technical crawlers

Good for finding:

• duplicate titles
• duplicate meta descriptions
• indexation problems
• canonical issues
• thin pages

Examples:

• Screaming Frog
• DeepCrawl (now part of Lumar)
• Semrush
• Ahrefs

Ecommerce-focused catalog auditors

These are designed around SKU-level problems rather than just URLs. Some tools specifically audit product descriptions, duplicate copy, missing fields, and catalog consistency. StoreCheck StoreCheck

Examples:

• Content Status — product content monitoring and scheduled audits are aimed at large ecommerce catalogs. HubSpot Blog
• WebAudit — ecommerce audits that include duplicate content and thin product page checks. WebAudit

3. Detect duplicate descriptions with similarity matching

Simple exact-match checks are not enough. Suppliers often create:

Product A

Premium stainless steel bottle keeps drinks cold for 24 hours.

Product B

Durable stainless steel bottle keeps beverages cold all day.

Search engines may see these as nearly identical.

Use:

Text similarity methods

• TF-IDF cosine similarity
• MinHash / locality-sensitive hashing
• Sentence embeddings
• Vector databases

Example workflow:

Product descriptions
        ↓
Generate embeddings
        ↓
Compare similarity
        ↓
Flag pairs >85% similar
        ↓
Send to content team

At enterprise scale, teams often store description embeddings in:

• Pinecone
• Amazon Web Services vector databases
• Google Cloud vector search systems

4. Create content templates instead of free-form rewriting

AI rewriting thousands of descriptions without structure usually creates inconsistent output.

Use category-specific templates:

Running shoes

Opening:
[Product] is designed for [runner type] who needs [benefit].

Key benefits:
- Cushioning:
- Fit:
- Materials:
- Best for:
- Care instructions:

Furniture

Overview
Materials
Dimensions
Room placement
Design style
Care instructions
Frequently asked questions

Templates make AI output consistent and easier to QA.

5. Use AI for controlled bulk remediation

A scalable workflow:

Audit finds 100,000 weak SKUs

↓

Filter:
- High traffic
- High margin
- High search potential

↓

AI drafts improvements

↓

Human approval for top categories

↓

Publish through PIM/API

↓

Re-crawl and measure

Important safeguards:

• Generate only from verified product attributes
• Do not invent specifications
• Keep brand tone rules
• Require structured outputs

6. Connect it to your PIM or product database

The best long-term solution is a quality gate before products go live.

Example:

New SKU arrives:

Supplier feed
      ↓
PIM
      ↓
Content quality checks
      ↓
Pass → publish
Fail → content queue

Block:

• empty descriptions
• supplier-copy descriptions
• missing attributes
• duplicate titles

Common enterprise PIM platforms:

• Akeneo
• Salsify
• Pimcore

7. Monitor continuously

Set up recurring checks:

Daily

• New SKUs missing descriptions
• Supplier overwrites
• Duplicate imports

Weekly

• New duplicate clusters
• Products losing content quality score
• Indexation changes

Monthly

• Rewrite campaigns
• Category benchmarks
• Search performance impact

Some catalog monitoring systems are built specifically to catch listing drift and changes across thousands of SKUs. Nextract

A practical enterprise stack

For a 50k–1M SKU catalog:

Data layer

• PIM + product database
• Data warehouse

Detection

• Screaming Frog/Lumar/Semrush
• Custom similarity pipeline

Prioritization

• Search Console data
• Revenue data
• Conversion rate
• Margin

Generation

• LLM + category templates
• Human approval workflow

Publishing

• PIM APIs
• Ecommerce platform APIs

Monitoring

• Scheduled crawls
• Content health dashboard

The biggest mistake at scale is asking “how do we rewrite 500,000 descriptions?” The better question is “which 20,000 product pages are hurting us most, and how do we create a system that prevents the next 20,000?”

Yes. For a huge catalog, I’d treat this as a data-quality pipeline, not a one-time SEO audit. The scalable setup is:

1. Build a catalog-level content inventory

Export every product into a table/data warehouse with at least:

• Product ID / SKU
• URL
• Product title
• Description
• Brand
• Category
• Attributes/variants
• Canonical URL
• Indexability
• Organic clicks/impressions
• Revenue/conversions
• Backlinks
• Inventory status

Then calculate automated fields such as:

• Description word count
• Exact-description duplicate hash
• Near-duplicate similarity score
• % of description shared with other products
• Missing description
• Reused manufacturer copy
• Duplicate title/H1/meta description
• Organic traffic/revenue

This gives you a product-content health score rather than asking someone to manually inspect 100,000 pages.

2. Use a crawler for the technical layer

My shortlist:

• screamingfrog.co.uk — excellent for crawling, extracting product-page fields, finding exact/near duplicates, canonicals, indexability, etc. Its near-duplicate system can identify pages around a configurable similarity threshold. Screaming Frog
• sitebulb.com — particularly nice for large-scale duplicate/thin-content analysis and prioritization. It explicitly reports word counts, duplicate content and near-duplicates; its Cloud product supports audits up to 10m URLs. Sitebulb Sitebulb
• ahrefs.com or semrush.com — use these primarily for the SEO impact layer: which products actually have impressions, rankings, links, traffic and keyword opportunities. Ahrefs also has duplicate-content checks in Site Audit. Ahrefs
• At truly enterprise scale, look at Botify, Lumar or JetOctopus rather than trying to make a desktop crawler your entire monitoring system. The Sharp Digital

3. Don't treat every duplicate as a problem

This is crucial.

Google explicitly says some duplicate content is normal. The correct treatment depends on why the duplicate exists. Google Developers

I'd classify every duplicate cluster automatically:

Google's canonicalization documentation specifically calls out things like filtering/sorting URLs and accidental URL variants as common sources of duplication. Google Developers

4. Detect thin descriptions with more than word count

Don't use "under 200 words = bad" as your rule.

A 75-word description can be excellent for a simple commodity product, while 400 words of generic manufacturer fluff can be useless.

I'd create a score roughly like:

Content quality score =

uniqueness + product-specific information + search demand + conversion value + completeness

For example, automatically flag a product when:

description_words < category_threshold
OR
unique_text_ratio < 40%
OR
description_similarity_to_another_product > 85%
OR
manufacturer_copy = true
OR
missing_key_attributes > 2

Then prioritize based on:

SEO opportunity × business value × content problem severity

That prevents your team from spending three hours rewriting a $12 product with zero impressions while ignoring a $900 product generating 20,000 monthly searches.

5. Use AI for rewriting—but put it behind a structured system

I would not dump 500,000 descriptions into an LLM and ask it to "make them unique."

Instead, create a product-content template from your structured product data:

PRODUCT
├── Brand
├── Product type
├── Primary differentiators
├── Materials
├── Dimensions
├── Compatibility
├── Use cases
├── Key specifications
├── Warranty
└── Important limitations

Then have the generation system produce copy from those facts.

For example:

Input:
Brand = Acme
Product = 24" commercial shelving unit
Material = powder-coated steel
Capacity = 800 lb
Depth = 24"
Use = restaurant / warehouse
Warranty = 5 years

The resulting description should emphasize facts that distinguish this SKU, rather than merely swapping adjectives in a manufacturer's paragraph.

That also makes regeneration much easier when product attributes change.

6. Create duplicate clusters—not just duplicate URLs

This is where a custom pipeline becomes extremely powerful.

Instead of:

Product A is 92% similar to Product B.

Build:

Cluster #1847
├── 247 products
├── 91% average similarity
├── Same manufacturer
├── Same base product
├── Difference: size/color
└── $428k annual revenue

Now you can make one decision for the whole cluster.

For variants, Google's current structured-data guidance specifically supports ProductGroup, variesBy, hasVariant, and productGroupID to communicate relationships between product variants and reduce repetitive information. Google for Developers

7. Put the monitoring on a recurring schedule

I'd build a pipeline like:

PIM / Product Database
        ↓
Nightly/weekly content export
        ↓
Duplicate + similarity detection
        ↓
Thin-content scoring
        ↓
SEO/traffic/revenue enrichment
        ↓
Priority queue
        ↓
AI/human content generation
        ↓
CMS/PIM
        ↓
Crawler
        ↓
Google Search Console
        ↓
Performance monitoring

Then your dashboard can say:

1.8M product URLs

42,381 severe duplicate clusters
18,920 thin products
7,412 missing descriptions
3,104 high-value products needing improvement
842 products improved this month
Organic clicks: +11.4%

That's much more scalable than having an SEO manager manually inspect pages.

My preferred stack

For a 10k–100k product catalog:

Sitebulb or Screaming Frog + Google Search Console + Ahrefs/Semrush + your PIM/database + an LLM/content-generation layer.

For 100k–millions of URLs:

Enterprise crawler (Botify/Lumar/JetOctopus) + data warehouse + custom similarity/content-quality pipeline + GSC + SEO platform + automated PIM/CMS workflow.

The important architectural decision is that the crawler finds problems, but your product database becomes the source of truth for fixing them.

And I'd keep structured product data separate from prose. Google recommends relevant ecommerce structured data, and product/merchant markup can communicate things such as price, availability, shipping and returns independently of the descriptive copy. Google for Developers Google for Developers

If you tell me roughly how many SKUs/URLs you have (e.g. 50k, 500k, 5M) and what platform you're on (Shopify, Magento, BigCommerce, custom, etc.), I can lay out a concrete architecture—including the exact tools, database fields, duplicate-detection algorithm, and an automated remediation workflow.

For a huge catalog, the key is to stop treating duplicate/thin content as a writing problem and treat it as a catalog quality pipeline:

Detect → prioritize → generate fixes → review → publish → monitor

A scalable setup usually combines a crawler, a catalog database/PIM, content scoring, and (optionally) AI-assisted rewriting.

1. Build a catalog content health score

Create automated rules that flag every SKU:

Duplicate content signals

• Exact duplicate descriptions
• Near-duplicate descriptions (same text with product name/color swapped)
• Manufacturer copy reused across competitors
• Duplicate title/meta descriptions
• Multiple variants competing for the same query

Thin content signals

Examples:

• Description under a minimum word count
• No benefits/use cases
• No unique attributes
• Missing FAQs, compatibility, sizing, materials, care info
• Missing structured attributes

A simple scoring model:

This prevents your team from wasting time rewriting thousands of low-value products first.

2. Use crawlers to find problems automatically

For large catalogs, manual audits do not scale. Tools commonly used for this include:

• screamingfrog.co.uk — useful for crawling large sites and finding duplicate titles, metadata issues, and content problems.
• semrush.com — provides large-scale technical SEO auditing.
• ahrefs.com — useful for content and technical SEO monitoring.
• jetoctopus.com — built around large-scale crawling and log analysis, including ecommerce SEO issues. Tech SEO Hub

For ecommerce-specific auditing, some platforms also focus directly on product-page problems such as duplicate descriptions and thin pages. WebAudit

3. Detect near-duplicates with embeddings, not just exact matching

Exact matching catches only the easy cases.

For thousands or millions of SKUs, use semantic similarity:

Example:

Product A:

Lightweight waterproof hiking jacket with breathable membrane...

Product B:

Breathable rain shell jacket designed for wet-weather hiking...

Different wording, same content.

A scalable workflow:

1. Export all descriptions.
2. Convert descriptions into embeddings.
3. Cluster products by similarity.
4. Review clusters above a threshold (for example, 85–95% similarity).

Common stack:

• Vector database: - pinecone.io
• weaviate.io
• opensearch.org
• Processing: - Python pipelines
• LLM APIs for classification/rewrite suggestions

4. Connect your product data system

A PIM (Product Information Management system) helps prevent the problem from returning.

Useful PIM capabilities:

• Required description fields
• Category-specific templates
• Approval workflows
• Attribute completeness checks
• Version history

Examples:

• akeneo.com
• salsify.com
• pimcore.com

5. Use AI carefully for rewriting

AI works best when it has structured product facts:

Good input:

Product:
10-inch stainless steel skillet

Facts:
- Oven safe to 500°F
- Pre-seasoned surface
- Cast iron construction
- Weight: 5.5 lb

Audience:
Home cooks and campers

Tone:
Helpful, premium

Bad input:

Rewrite this description:
"Great skillet for cooking."

The goal is not "spin the text." It is to create genuinely differentiated pages.

Tools focused on ecommerce content enrichment and bulk rewriting exist, including catalog-focused platforms that detect duplicate/thin descriptions and generate revisions with review workflows. Catalog Cleanup AI Ecomptimize

6. Create reusable product description frameworks

Instead of writing every SKU from scratch, create templates by category.

Example:

Running shoes template

• Best for
• Key performance benefits
• Cushioning
• Fit guidance
• Materials
• Care
• Comparison vs similar models

Furniture template

• Design style
• Materials
• Dimensions
• Room placement
• Assembly
• Maintenance

This gives consistency without creating thousands of identical pages.

7. Automate monitoring

Run checks weekly or daily:

Daily

• New products missing descriptions
• Supplier-imported copy
• Duplicate SKUs

Weekly

• New duplicate clusters
• Pages losing rankings
• Products with declining conversion

Monthly

• Refresh old winners
• Rewrite high-value pages
• Remove obsolete catalog pages

Connect:

• search.google.com
• Analytics
• Revenue data
• Inventory lifecycle

A practical enterprise workflow

Step 1: Crawl catalog
↓
Step 2: Score every product page
↓
Step 3: Rank fixes by revenue/search impact
↓
Step 4: Generate improved copy from structured attributes
↓
Step 5: Human approval for top-value products
↓
Step 6: Bulk publish
↓
Step 7: Monitor continuously

For a catalog with 50k–1M SKUs, I would typically avoid trying to rewrite everything. A better approach is:

1. Fix indexed products driving revenue.
2. Fix products with impressions but poor rankings.
3. Fix category leaders.
4. Then handle the long tail with automated rules.

That gives measurable SEO and conversion gains without creating a massive content-maintenance project.

Yes. For a huge ecommerce catalog, I’d treat duplicate/thin content as a data pipeline problem, not a one-off SEO audit.

A scalable setup

1. Build a catalog-level content dataset

Export your product catalog into a table with at least:

• SKU / product ID
• URL
• brand
• product type/category
• title
• description
• attributes/specifications
• canonical URL
• indexability
• organic clicks/impressions
• conversions/revenue
• inventory status

Then calculate automatically:

• description word/character count
• exact-content hash
• normalized-content hash
• similarity score against other products
• percentage of description shared with other products
• missing attributes
• duplicate titles/H1s
• AI/content-quality score, if useful

The key is not to use word count alone as your definition of "thin." A 150-word description can be excellent for one product and inadequate for another.

2. Use a crawler for the web-facing reality

Two particularly useful options are:

• Screaming Frog SEO Spider — excellent for extracting product-page data, finding exact/near duplicates, low-content pages, canonical issues, duplicate titles/H1s, etc. Its near-duplicate system can be configured around similarity thresholds. Screaming Frog Screaming Frog
• Sitebulb Cloud — particularly attractive when you're talking about hundreds of thousands or millions of URLs. Its cloud crawler supports audits up to 10m URLs and recurring crawls, and it explicitly analyzes duplicate and similar content plus word counts. Sitebulb Sitebulb

screamingfrog.co.uk
sitebulb.com

For a very large catalog, I'd lean toward Sitebulb Cloud or an enterprise crawler, while using your own catalog database for the deeper product-to-product analysis.

3. Cluster duplicates instead of generating a report of 500,000 URLs

This is where scalability really matters.

Suppose you have:

Red Nike Running Shoe — Size 10
Blue Nike Running Shoe — Size 10
Black Nike Running Shoe — Size 10

and 95% of the description is identical.

Don't create 3,000 individual "fix this URL" tickets.

Instead, algorithmically create a duplicate cluster:

Cluster 8472
├── 1,842 product URLs
├── shared description: 91%
├── primary product: SKU 18492
├── revenue at risk: $X
└── recommended action: rewrite template + differentiate attributes

Then prioritize clusters by organic traffic × revenue × indexability × similarity.

That turns an impossible SEO project into a manageable queue.

4. Separate the fixes into four buckets

A. True duplicates

If they're genuinely the same product/page, consolidate them or establish the appropriate canonical/indexing architecture.

B. Variant pages that should exist separately

Keep them, but make their content meaningfully differentiated: dimensions, material, compatibility, use cases, fit, color-specific information, etc.

C. Supplier/manufacturer copy

Don't simply rewrite every description with superficial synonym swaps. Add information that actually helps the shopper: specifications, comparisons, compatibility, use cases, original testing, FAQs, imagery, etc. Google's current guidance specifically emphasizes added value rather than copied/cookie-cutter product content. Google for Developers

D. Truly low-value products

For products with little search demand, no inventory, no differentiation, or negligible business value, consider whether they should be indexable at all rather than spending resources writing elaborate copy.

5. Automate the rewrite pipeline

For thousands of products, I'd create a workflow like:

Catalog/PIM
     ↓
Content extraction
     ↓
Duplicate + similarity detection
     ↓
Quality scoring
     ↓
Priority queue
     ↓
Content generation
     ↓
Human/brand QA
     ↓
PIM/CMS
     ↓
Deploy
     ↓
Crawl + Search Console monitoring

The important bit: don't let an LLM autonomously rewrite your entire catalog.

Give it structured product facts and a content specification, then validate that its output:

• doesn't invent specifications
• doesn't change dimensions/numbers
• doesn't make unsupported claims
• uses the actual product attributes
• isn't just a paraphrase of another SKU
• meets your category-specific requirements

For example, your content generator could receive:

Product:
Brand: Acme
Category: Hiking backpack
Capacity: 35L
Weight: 1.2kg
Material: 210D nylon
Water resistance: IPX4

Required:
- 120–180 word description
- 3 concrete differentiators
- intended use
- important limitations
- no unsupported claims
- don't repeat manufacturer copy

That's considerably safer than "write an SEO description for this product."

6. Monitor continuously, not quarterly

Set up automated thresholds such as:

Screaming Frog and Sitebulb can both support the crawling/monitoring side; Sitebulb also provides recurring crawls and audit comparisons for tracking changes over time. Sitebulb Sitebulb

The stack I'd use

For a 100k–1m+ SKU catalog, roughly:

Crawler: Sitebulb Cloud or Screaming Frog
SEO/query data: Google Search Console + your analytics
Keyword/competitive data: Ahrefs or Semrush
Catalog source: PIM/product database
Processing: SQL/Python/data warehouse
Similarity: embeddings + exact hashes + lexical similarity
Generation: LLM constrained by structured product data
QA: automated validation + human review for high-value products
Dashboard: Looker Studio/BI tool

The crucial distinction is that your crawler finds problems; your catalog database decides what the problem actually is; your prioritization system decides what is worth fixing.

Also, don't obsess over eliminating every instance of duplicate text. Search engines can canonicalize/filter naturally occurring duplicates. At large scale, the bigger objective is making sure you control which URL is indexed and that genuinely distinct products have sufficiently differentiated content. Google's current canonicalization guidance explicitly emphasizes making clustered pages sufficiently different. Google for Developers

If you tell me roughly how many SKUs you have (100k, 1m, 10m+) and what platform/PIM you're using (Shopify, Magento, Salesforce Commerce, custom, etc.), I can lay out a concrete architecture—including the database schema, similarity algorithm, scoring formula, and automation workflow.

Red Nike Running Shoe — Size 10
Blue Nike Running Shoe — Size 10
Black Nike Running Shoe — Size 10

and 95% of the description is identical.

Don't create 3,000 individual "fix this URL" tickets.

Instead, algorithmically create a duplicate cluster:

Cluster 8472
├── 1,842 product URLs
├── shared description: 91%
├── primary product: SKU 18492
├── revenue at risk: $X
└── recommended action: rewrite template + differentiate attributes

Then prioritize clusters by organic traffic × revenue × indexability × similarity.

That turns an impossible SEO project into a manageable queue.

4. Separate the fixes into four buckets

A. True duplicates

If they're genuinely the same product/page, consolidate them or establish the appropriate canonical/indexing architecture.

B. Variant pages that should exist separately

Keep them, but make their content meaningfully differentiated: dimensions, material, compatibility, use cases, fit, color-specific information, etc.

C. Supplier/manufacturer copy

Don't simply rewrite every description with superficial synonym swaps. Add information that actually helps the shopper: specifications, comparisons, compatibility, use cases, original testing, FAQs, imagery, etc. Google's current guidance specifically emphasizes added value rather than copied/cookie-cutter product content. Google for Developers

D. Truly low-value products

For products with little search demand, no inventory, no differentiation, or negligible business value, consider whether they should be indexable at all rather than spending resources writing elaborate copy.

5. Automate the rewrite pipeline

For thousands of products, I'd create a workflow like:

Catalog/PIM
     ↓
Content extraction
     ↓
Duplicate + similarity detection
     ↓
Quality scoring
     ↓
Priority queue
     ↓
Content generation
     ↓
Human/brand QA
     ↓
PIM/CMS
     ↓
Deploy
     ↓
Crawl + Search Console monitoring

The important bit: don't let an LLM autonomously rewrite your entire catalog.

Give it structured product facts and a content specification, then validate that its output:

• doesn't invent specifications
• doesn't change dimensions/numbers
• doesn't make unsupported claims
• uses the actual product attributes
• isn't just a paraphrase of another SKU
• meets your category-specific requirements

For example, your content generator could receive:

Yes. For a large catalog, I’d treat duplicate/thin content as a data pipeline rather than a page-by-page SEO project.

The scalable setup

1. Build a catalog-level content dataset

Export every product into a table/database with at least:

• Product ID / SKU
• URL
• Brand
• Product type/category
• Product title
• Description
• Key attributes/specs
• Manufacturer description
• Canonical URL
• Indexability
• Organic clicks/impressions
• Revenue/conversions
• Backlinks
• Inventory status
• Last content update

Then calculate:

• Description word/character count
• % of description shared with another product
• Exact duplicate hash
• Near-duplicate similarity score
• Missing attributes
• Duplicate title/H1/meta description
• Organic traffic/revenue per product

This lets you prioritize commercially important thin pages, rather than blindly rewriting 100,000 SKUs.

2. Crawl the site continuously

For the crawling/diagnostic layer, sitebulb.com is particularly well suited to this problem. It can detect exact and near-duplicate HTML, duplicate titles/H1s/meta descriptions, and use configurable word-count thresholds to flag thin pages. Its Cloud product is designed for very large crawls, reportedly up to 10 million URLs per audit. Sitebulb Support Sitebulb Sitebulb

I'd also consider Screaming Frog for engineering-heavy audits, particularly if your team wants extensive custom extraction.

3. Don't use "word count < X" as your definition of thin

This is important.

A 150-word product page can be excellent if it completely answers the buying question. A 500-word page can be useless if it's 90% boilerplate.

Instead, create a content completeness score, e.g.:

Content Score =
  20% unique description
+ 15% product-specific attributes
+ 15% use cases
+ 10% compatibility information
+ 10% dimensions/specifications
+ 10% FAQs
+ 10% original imagery/media
+ 10% customer/review content

Adjust the weights by category.

For example, a laptop should probably have detailed specifications and compatibility information; a shirt needs material, fit, sizing, care, and style information.

4. Detect duplicates at two levels

Exact duplicates

Hash normalized descriptions:

normalize(description)
        ↓
remove HTML
lowercase
remove whitespace
        ↓
SHA-256
        ↓
group by hash

Anything with the same hash is an exact duplicate.

Near duplicates

For products that differ only by:

"Blue men's Nike running shoe, size 10"

vs.

"Black men's Nike running shoe, size 10"

use similarity detection rather than exact matching.

At scale, I'd use embeddings/vector similarity or MinHash/SimHash rather than comparing every product against every other product. That turns an otherwise enormous pairwise-comparison problem into something much more manageable.

A crawler such as Sitebulb already provides near-duplicate detection based on similar content, which is useful for the initial audit. Sitebulb Support

5. Create automated remediation rules

Don't send every flagged page to an editor.

Build a decision tree:

Google specifically recommends canonicalization to consolidate duplicate/similar URLs and redirects when you actually want duplicate URLs removed. Google for Developers

Don't automatically canonicalize every thin product to its category. A product page and category page serve different search intents; the better solution is often to improve the product, consolidate genuinely redundant products, or keep the page for users without treating it as a priority SEO landing page.

6. Use AI for the first draft, not blind publishing

This is where you can get enormous leverage.

Give your generation system structured product data:

{
  "brand": "...",
  "product": "...",
  "category": "...",
  "materials": [],
  "dimensions": [],
  "features": [],
  "compatibility": [],
  "use_cases": [],
  "warranty": "...",
  "existing_description": "..."
}

Have it produce content from facts in the catalog, rather than asking:

"Write an SEO description for this product."

The latter tends to produce thousands of interchangeable descriptions.

Instead, enforce templates such as:

Product-specific summary
↓
Key differentiators
↓
Who it's for / use cases
↓
Specifications
↓
Compatibility
↓
Care / installation / warranty
↓
FAQs

Then run automated validation:

• Did it invent a specification?
• Does it mention attributes that aren't in the source data?
• Is it substantially different from sibling products?
• Does it contain forbidden claims?
• Is it below the minimum useful-content threshold?
• Does it accidentally copy the manufacturer?
• Does it contradict structured product data?

7. Prioritize with an opportunity score

This is probably the biggest scalability win.

Don't optimize products in alphabetical order. Score them:

Priority =
  Search opportunity
× Revenue potential
× Current ranking potential
× Content deficiency
× Product importance

For example:

Tier 1

• High impressions
• Page 2–3 rankings
• High-margin products
• Thin/duplicate descriptions

→ Human-reviewed rewrite.

Tier 2

• Moderate demand
• Good products
• Clear content gaps

→ AI-assisted enrichment + sampling/QC.

Tier 3

• Little/no demand
• No backlinks
• Poor inventory
• Highly redundant

→ Consolidate, remove, or leave alone depending on business requirements.

That can turn a 500,000-SKU problem into, say, 5,000 high-value pages requiring serious attention.

A practical tool stack

Sitebulb also supports custom content extraction/search, which is useful for pulling product-specific fields and identifying patterns across a catalog. Sitebulb

The architecture I'd actually build

             PIM / ERP
                 │
                 ▼
          Product data warehouse
                 │
       ┌─────────┴─────────┐
       ▼                   ▼
 Duplicate engine      Content scorer
       │                   │
       └─────────┬─────────┘
                 ▼
          Priority queue
                 │
        ┌────────┴────────┐
        ▼                 ▼
   Consolidate       Enrich/rewrite
        │                 │
        └────────┬────────┘
                 ▼
              CMS/PIM
                 │
                 ▼
              Website
                 │
          ┌──────┴──────┐
          ▼             ▼
       Crawler       GSC/GA4
          │             │
          └──────┬──────┘
                 ▼
          Monitoring loop

The crucial idea is that the crawler detects problems, but your product database should be the system of record for fixing them.

And don't measure success simply by "number of duplicate pages fixed." Track organic clicks, indexed pages, rankings, revenue, conversion rate, and crawl/indexation changes by product cohort.

If you tell me roughly whether you're dealing with 10k, 100k, or 1M+ products and what platform you're on (Shopify, Magento, Salesforce Commerce Cloud, custom, etc.), I can lay out a concrete stack and workflow—including the database schema, duplicate-detection logic, AI generation pipeline, and monitoring dashboard.

"Black men's Nike running shoe, size 10"

use similarity detection rather than exact matching.

At scale, I'd use embeddings/vector similarity or MinHash/SimHash rather than comparing every product against every other product. That turns an otherwise enormous pairwise-comparison problem into something much more manageable.

A crawler such as Sitebulb already provides near-duplicate detection based on similar content, which is useful for the initial audit. Sitebulb Support

5. Create automated remediation rules

Don't send every flagged page to an editor.

Build a decision tree:

Google specifically recommends canonicalization to consolidate duplicate/similar URLs and redirects when you actually want duplicate URLs removed. Google for Developers

Don't automatically canonicalize every thin product to its category. A product page and category page serve different search intents; the better solution is often to improve the product, consolidate genuinely redundant products, or keep the page for users without treating it as a priority SEO landing page.

6. Use AI for the first draft, not blind publishing

This is where you can get enormous leverage.

Give your generation system structured product data:

{
  "brand": "...",
  "product": "...",
  "category": "...",
  "materials": [],
  "dimensions": [],
  "features": [],
  "compatibility": [],
  "use_cases": [],
  "warranty": "...",
  "existing_description": "..."
}

Have it produce content from facts in the catalog, rather than asking:

"Write an SEO description for this product."

The latter tends to produce thousands of interchangeable descriptions.

Instead, enforce templates such as:

Product-specific summary
↓
Key differentiators
↓
Who it's for / use cases
↓
Specifications
↓
Compatibility
↓
Care / installation / warranty
↓
FAQs

Then run automated validation:

• Did it invent a specification?
• Does it mention attributes that aren't in the source data?
• Is it substantially different from sibling products?
• Does it contain forbidden claims?
• Is it below the minimum useful-content threshold?
• Does it accidentally copy the manufacturer?
• Does it contradict structured product data?

7. Prioritize with an opportunity score

This is probably the biggest scalability win.

Don't optimize products in alphabetical order. Score them:

Priority =
  Search opportunity
× Revenue potential
× Current ranking potential
× Content deficiency
× Product importance

For example:

Tier 1

• High impressions
• Page 2–3 rankings
• High-margin products
• Thin/duplicate descriptions

→ Human-reviewed rewrite.

Tier 2

• Moderate demand
• Good products
• Clear content gaps

→ AI-assisted enrichment + sampling/QC.

Tier 3

• Little/no demand
• No backlinks
• Poor inventory
• Highly redundant

→ Consolidate, remove, or leave alone depending on business requirements.

That can turn a 500,000-SKU problem into, say, 5,000 high-value pages requiring serious attention.

A practical tool stack

Sitebulb also supports custom content extraction/search, which is useful for pulling product-specific fields and identifying patterns across a catalog.

The architecture I'd actually build

For a huge ecommerce catalog, I’d treat this as a data-quality pipeline, not a one-off SEO audit. The scalable setup is:

1. Detect duplicates in layers

You want to distinguish true duplicates, near-duplicates, and legitimate variants.

• Exact duplicates: identical descriptions/titles/H1s.
• Near-duplicates: descriptions that differ only by color, size, SKU, or a few words.
• Template duplication: thousands of pages where only one or two product attributes change.
• Cross-product duplication: different SKUs using the same manufacturer/vendor copy.
• Variant duplication: legitimate variants that shouldn't each require radically different copy.

For crawling and detection, sitebulb.com is particularly useful: its duplicate-content system checks HTML, titles, meta descriptions and H1s, and can also identify similar/near-duplicate pages. Its cloud crawler is designed for very large sites. Sitebulb Support Sitebulb

For a very large enterprise site, I'd also evaluate Botify or Screaming Frog alongside Sitebulb, depending on whether you need enterprise crawling, APIs/data pipelines, or a more hands-on crawler.

2. Don't define "thin" as simply "under X words"

That's one of the biggest mistakes.

A 120-word product description can be perfectly useful if the product itself is simple. A 500-word description can still be terrible if it's generic manufacturer copy.

Build a content-quality score per product, something like:

Content Score =
  20% description completeness
  15% uniqueness
  15% product-specific information
  10% attribute coverage
  10% search demand/relevance
  10% organic traffic
  10% conversions/revenue
  10% indexing/search visibility

Then add hard flags:

description missing
description < minimum threshold
description identical to another SKU
description >90% similar to another SKU
manufacturer copy detected
only differs from sibling SKU by color/size
no unique product attributes

Sitebulb can already help with word counts and thin-content identification, but I'd augment it with your own catalog-level scoring. Sitebulb

3. Use embeddings for near-duplicate detection

This is where you get much more scalable than traditional SEO crawlers.

Export something like:

SKU
URL
brand
category
title
description
attributes
canonical
traffic
revenue
conversion_rate
index_status

Generate an embedding for each product description, then perform similarity searches.

For example:

SKU A → embedding
SKU B → embedding
SKU C → embedding
...

Then flag:

similarity > 0.95 → probable duplicate
0.85–0.95       → investigate
0.70–0.85       → potentially related
< 0.70          → probably distinct

Don't blindly use those thresholds as universal rules—calibrate them against manually reviewed examples from your catalog.

This lets you discover clusters such as:

8,400 products → 1,200 description clusters → 340 clusters responsible for most duplication.

That's far more actionable than a spreadsheet containing 8,400 URLs.

4. Fix the catalog according to priority

Don't rewrite everything.

I'd prioritize:

Tier 1 — highest value

• High-revenue products
• Products already getting organic impressions
• Products with strong search demand
• Products ranking positions 4–20
• Products with substantial duplicate/near-duplicate content

Tier 2

• Products with impressions but almost no clicks
• Important category/subcategory products
• Products with strong backlinks
• Products whose competitors have substantially better content

Tier 3

• Low-demand, low-margin, rarely searched products
• Discontinued products
• Near-identical variants
• Products with essentially no organic opportunity

For Tier 3, the right answer may be consolidation, canonicalization, noindexing, or leaving the page alone, rather than spending money generating unique copy.

5. Generate descriptions from structured product data

For thousands of products, don't ask an LLM:

"Write a description for this product."

Instead give it a controlled schema:

{
  "product": "...",
  "brand": "...",
  "category": "...",
  "materials": [],
  "dimensions": {},
  "features": [],
  "use_cases": [],
  "compatibility": [],
  "target_customer": "...",
  "differentiators": [],
  "limitations": []
}

Then impose rules such as:

• Never invent specifications.
• Don't repeat the product title verbatim.
• Don't use generic filler.
• Mention important differentiating attributes.
• Explain why an attribute matters where appropriate.
• Don't claim superiority without evidence.
• Don't create artificial keyword variations.
• Maintain consistent terminology by category.

The result should be programmatic content generation with human QA, rather than mass AI rewriting.

6. Build a "content factory"

A good architecture looks like this:

PIM / ERP
   ↓
Product data warehouse
   ↓
Quality checks
   ├── missing attributes
   ├── duplicate SKU data
   ├── bad manufacturer copy
   └── incomplete descriptions
   ↓
Embedding / similarity engine
   ↓
Duplicate + thin-content scoring
   ↓
Prioritization
   ↓
LLM content generation
   ↓
Automated validation
   ├── factuality
   ├── required attributes
   ├── similarity
   ├── prohibited claims
   └── formatting
   ↓
Human QA sampling
   ↓
CMS/PIM
   ↓
Site
   ↓
Crawler + Search Console monitoring

The important part is that the PIM/database becomes the source of truth, not the SEO tool.

7. Monitor continuously

I'd run:

• Nightly/weekly: product-data validation
• Weekly: duplicate/near-duplicate detection on changed/new products
• Weekly: content-quality score changes
• Monthly: full crawl
• Monthly: Google Search Console performance/indexation analysis
• Quarterly: larger content-quality audit

Google recommends using Search Console to monitor ecommerce search performance and Product/Merchant Listings structured-data issues; its documentation also recommends combining product structured data with Merchant Center data where appropriate. Google Developers Google Developers

8. Don't accidentally "fix" legitimate variants

This is especially important for ecommerce.

If you have:

Nike Shoe – Black
Nike Shoe – White
Nike Shoe – Red

you don't necessarily need three completely unrelated essays.

Instead, establish:

Shared product facts
        +
variant-specific information
        +
unique purchasing/use-case information

Google explicitly supports ProductGroup/Product structured data for variants and notes that grouping variants can reduce duplicated information. Google for Developers

My recommended stack

For a catalog of 100K+ products, I'd consider:

The key insight is: don't make your SEO crawler responsible for fixing the catalog. Use it to discover problems, combine that data with product/revenue/search data, then have a data pipeline decide which products deserve intervention and what kind of intervention they need.

If you tell me roughly whether you're dealing with 50K, 500K, or 5M+ SKUs and what platform you're on (Shopify, Magento/Adobe Commerce, BigCommerce, custom, etc.), I can lay out a concrete architecture—including the database schema, duplicate-scoring algorithm, and automated AI rewriting workflow.

I'd prioritize:

Tier 1 — highest value

• High-revenue products
• Products already getting organic impressions
• Products with strong search demand
• Products ranking positions 4–20
• Products with substantial duplicate/near-duplicate content

Tier 2

• Products with impressions but almost no clicks
• Important category/subcategory products
• Products with strong backlinks
• Products whose competitors have substantially better content

Tier 3

• Low-demand, low-margin, rarely searched products
• Discontinued products
• Near-identical variants
• Products with essentially no organic opportunity

For Tier 3, the right answer may be consolidation, canonicalization, noindexing, or leaving the page alone, rather than spending money generating unique copy.

5. Generate descriptions from structured product data

For thousands of products, don't ask an LLM:

"Write a description for this product."

Instead give it a controlled schema:

{
  "product": "...",
  "brand": "...",
  "category": "...",
  "materials": [],
  "dimensions": {},
  "features": [],
  "use_cases": [],
  "compatibility": [],
  "target_customer": "...",
  "differentiators": [],
  "limitations": []
}

Then impose rules such as:

• Never invent specifications.
• Don't repeat the product title verbatim.
• Don't use generic filler.
• Mention important differentiating attributes.
• Explain why an attribute matters where appropriate.
• Don't claim superiority without evidence.
• Don't create artificial keyword variations.
• Maintain consistent terminology by category.

The result should be programmatic content generation with human QA, rather than mass AI rewriting.

6. Build a "content factory"

A good architecture looks like this:

PIM / ERP
   ↓
Product data warehouse
   ↓
Quality checks
   ├── missing attributes
   ├── duplicate SKU data
   ├── bad manufacturer copy
   └── incomplete descriptions
   ↓
Embedding / similarity engine
   ↓
Duplicate + thin-content scoring
   ↓
Prioritization
   ↓
LLM content generation
   ↓
Automated validation
   ├── factuality
   ├── required attributes
   ├── similarity
   ├── prohibited claims
   └── formatting
   ↓
Human QA sampling
   ↓
CMS/PIM
   ↓
Site
   ↓
Crawler + Search Console monitoring

The important part is that the PIM/database becomes the source of truth, not the SEO tool.

7. Monitor continuously

I'd run:

• Nightly/weekly: product-data validation
• Weekly: duplicate/near-duplicate detection on changed/new products
• Weekly: content-quality score changes
• Monthly: full crawl
• Monthly: Google Search Console performance/indexation analysis
• Quarterly: larger content-quality audit

Google recommends using Search Console to monitor ecommerce search performance and Product/Merchant Listings structured-data issues; its documentation also recommends combining product structured data with Merchant Center data where appropriate.

8. Don't accidentally "fix" legitimate variants

This is especially important for ecommerce.

If you have:

Nike Shoe – Black
Nike Shoe – White
Nike Shoe – Red

you don't necessarily need three completely unrelated essays.

Instead, establish:

Yes. For a huge ecommerce catalog, I’d treat duplicate/thin content as a data-quality pipeline rather than a one-off SEO audit.

The scalable setup

1. Crawl the catalog continuously

Use a crawler to collect, per product URL:

• URL / SKU / product ID
• canonical URL
• indexability
• title, H1, meta description
• main description text
• word count
• structured data
• category/brand/product type
• organic traffic and conversions
• internal links
• last content update

Good tools:

• screamingfrog.co.uk — particularly useful for large-scale duplicate and near-duplicate detection. It supports exact duplicates and configurable near-duplicate analysis; its default near-duplicate threshold is 90%. Screaming Frog Screaming Frog
• semrush.com — better as an ongoing monitoring/dashboard layer, with scheduled crawls and duplicate-content checks. Semrush
• siteliner.com — useful as a lightweight secondary duplicate-content check, though its standard offering is much smaller than what you'd want for a massive catalog. Siteliner Siteliner

For tens/hundreds of thousands or millions of products, I'd eventually move the content analysis itself into your data warehouse, rather than depending entirely on a GUI crawler.

2. Separate "duplicate" into different problems

This is important because the fixes are different.

Don't make word count = quality. A 250-word product description can be useless, while a concise 120-word description can completely answer the buyer's question.

3. Build a product-content scoring system

For every SKU, calculate something like:

content_score =
    25% uniqueness
  + 20% product-specific detail
  + 20% search-intent coverage
  + 15% completeness
  + 10% readability
  + 10% business importance

Then classify:

A = healthy
B = minor improvement
C = thin / highly similar
D = critical

The key is not to rewrite everything.

Imagine 500,000 products:

• 40,000 exact/near duplicates
• 90,000 thin
• 30,000 high-revenue products
• 15,000 products receiving meaningful organic traffic

Your priority should be the intersection of:

content problem × search opportunity × commercial value

rather than simply "fix every page under 200 words."

Screaming Frog itself exposes a low-content-pages filter based on word count, while emphasizing that near-duplicates need contextual review because similar product configurations can legitimately have similar content. Screaming Frog Screaming Frog

4. Use embeddings for massive catalogs

This is where I'd go beyond traditional SEO tools.

Put every product description through an embedding model and store the vectors in something like:

• PostgreSQL + pgvector
• Pinecone
• Weaviate
• Elasticsearch/OpenSearch
• BigQuery vector search

Then compare products within sensible groups, such as:

category + brand
category + product_type
manufacturer + model family

Don't compare every product against every other product.

For example:

Nike / running shoes
        ↓
embedding similarity
        ↓
SKU 18392 ↔ SKU 18397 = 97%
SKU 18392 ↔ SKU 28114 = 42%

Now you can automatically surface clusters like:

"These 63 product pages are essentially the same description with color/size/model substitutions."

That's much more scalable than manually reviewing pages.

5. Detect thinness semantically, not just by length

I'd create a product-type content schema.

For example, a laptop might be expected to cover:

brand
model
processor
RAM
storage
display
GPU
battery
ports
dimensions
weight
warranty
compatibility
ideal use case
key differentiators

A running shoe might instead need:

upper
midsole
outsole
cushioning
drop
weight
terrain
fit
water resistance
recommended use

Then your system can say:

Product 18392

87 words
91% similarity to 14 other pages
Missing: cushioning, drop, terrain, fit
Manufacturer description detected
Organic clicks: 320/month
Revenue: $18,400/month

Priority: P0

That's dramatically more actionable than "thin content detected."

6. Automate the remediation

For a large catalog, don't have an editor manually write every description from scratch.

Create a pipeline:

PIM / Product database
        ↓
content quality rules
        ↓
duplicate detection
        ↓
thin-content detection
        ↓
product-type schema
        ↓
LLM enrichment
        ↓
automated QA
        ↓
human approval
        ↓
PIM/CMS
        ↓
website
        ↓
SEO monitoring

The LLM should be given structured product facts, not asked to invent marketing copy.

For example:

{
  "brand": "...",
  "model": "...",
  "material": "...",
  "dimensions": "...",
  "features": ["...", "..."],
  "compatibility": ["..."],
  "use_cases": ["..."]
}

Then impose hard rules:

• Never invent specifications.
• Never change measurements.
• Never claim certifications unless supplied.
• Don't copy manufacturer sentences verbatim.
• Don't produce near-identical descriptions for variants.
• Don't stuff keywords.
• Preserve important technical terminology.

7. Add a duplicate-content "gate" before publishing

This is one of the highest-value changes.

When a new/updated product description is generated:

new description
      ↓
exact duplicate check
      ↓
similarity check
      ↓
required-field check
      ↓
product-type coverage check
      ↓
fact validation
      ↓
PASS → publish
FAIL → regenerate/review

For example:

Similarity to existing catalog:
96% → FAIL

Unique information:
31% → FAIL

Required attributes:
8/10 → FAIL

Factual validation:
PASS

→ send back for regeneration

That prevents your catalog from creating the same problem faster than your SEO team can fix it.

8. Monitor it with a warehouse dashboard

I'd have a dashboard with roughly these KPIs:

Catalog health

• % products with unique descriptions
• % near-duplicate products
• % thin products
• median description length
• % missing required attributes
• % manufacturer-copy content
• % products failing content QA

SEO impact

• organic clicks by content-quality bucket
• impressions by bucket
• indexed URLs by bucket
• CTR
• rankings
• pages with declining organic traffic

Business priority

• revenue by content-quality bucket
• conversions by bucket
• organic revenue by bucket
• inventory status
• product margin

Then you can answer:

"We fixed 12,000 descriptions, which eliminated 78% of our high-similarity clusters and improved organic traffic to those products by 14%."

rather than:

"We fixed some duplicate content."

My preferred tool stack

For a serious catalog, I'd use:

Crawling: screamingfrog.co.uk
SEO monitoring: semrush.com
Data warehouse: BigQuery/Snowflake/Redshift
Similarity: embeddings + pgvector/OpenSearch/Pinecone
Generation: LLM API
Source of truth: PIM
Workflow: CMS/PIM + approval queue
Reporting: Looker/Tableau/Power BI

Screaming Frog is particularly useful for establishing the initial baseline because it can export exact and near-duplicate groups in bulk. Screaming Frog Semrush is useful for recurring monitoring and scheduled audits. Semrush

The big architectural idea

Don't build "an SEO tool that finds thin pages." Build a catalog content-quality system.

The ideal loop is:

Detect → cluster → prioritize → generate → validate → approve → publish → recrawl → measure.

That scales much better than having SEO specialists periodically export a spreadsheet of "pages under 200 words."

For example:

{
  "brand": "...",
  "model": "...",
  "material": "...",
  "dimensions": "...",
  "features": ["...", "..."],
  "compatibility": ["..."],
  "use_cases": ["..."]
}

Then impose hard rules:

• Never invent specifications.
• Never change measurements.
• Never claim certifications unless supplied.
• Don't copy manufacturer sentences verbatim.
• Don't produce near-identical descriptions for variants.
• Don't stuff keywords.
• Preserve important technical terminology.

7. Add a duplicate-content "gate" before publishing

This is one of the highest-value changes.

When a new/updated product description is generated:

new description
      ↓
exact duplicate check
      ↓
similarity check
      ↓
required-field check
      ↓
product-type coverage check
      ↓
fact validation
      ↓
PASS → publish
FAIL → regenerate/review

For example:

Similarity to existing catalog:
96% → FAIL

Unique information:
31% → FAIL

Required attributes:
8/10 → FAIL

Factual validation:
PASS

→ send back for regeneration

That prevents your catalog from creating the same problem faster than your SEO team can fix it.

8. Monitor it with a warehouse dashboard

I'd have a dashboard with roughly these KPIs:

Catalog health

• % products with unique descriptions
• % near-duplicate products
• % thin products
• median description length
• % missing required attributes
• % manufacturer-copy content
• % products failing content QA

SEO impact

• organic clicks by content-quality bucket
• impressions by bucket
• indexed URLs by bucket
• CTR
• rankings
• pages with declining organic traffic

Business priority

• revenue by content-quality bucket
• conversions by bucket
• organic revenue by bucket
• inventory status
• product margin

Then you can answer:

"We fixed 12,000 descriptions, which eliminated 78% of our high-similarity clusters and improved organic traffic to those products by 14%."

rather than:

"We fixed some duplicate content."

My preferred tool stack

For a serious catalog, I'd use:

For a huge ecommerce catalog, I’d treat this as a data-quality pipeline, not a one-off SEO audit. The scalable setup is:

1. Build a catalog-level content dataset

Export one row per product/SKU/URL with fields such as:

• URL
• SKU/product ID
• title
• brand
• category
• description
• specs/attributes
• canonical URL
• indexability
• organic clicks/impressions
• conversions/revenue
• inventory status
• word/character count
• last content update

Then calculate automated flags:

Duplicate score

• Exact description match → 1.0
• Near-duplicate similarity → e.g. >0.85
• Same title + same attributes + very similar description → high-risk cluster

Thinness score
Don't use a crude "<300 words = thin" rule. Product descriptions naturally vary in length. Compare each product against its category/template peers.

For example:

content_score = description_length + attribute_coverage + unique_information + search_performance

A $2 screw doesn't need a 1,000-word description; a complex appliance probably needs substantially more information.

2. Use a crawler to continuously detect problems

My shortlist:

• sitebulb.com — particularly strong for this use case. It detects exact and near-duplicate content, and its cloud crawler can handle very large sites. Its duplicate report can compare HTML, titles, meta descriptions and H1s. Sitebulb Support Sitebulb
• screamingfrog.co.uk — excellent for detailed crawling, custom extraction and finding duplicate/near-duplicate and low-content pages. It can extract custom product fields using XPath/CSS/regex. Screaming Frog Screaming Frog
• ahrefs.com — useful as the performance layer: organic traffic, rankings, cannibalization, GSC data and content opportunities. Its enterprise tooling also exposes APIs for automation. Ahrefs
• semrush.com — another strong all-in-one option if you want scheduled crawling and duplicate/on-page issue monitoring. Its Site Audit currently has 140+ checks and supports scheduled recrawls. Semrush

If you're talking millions of URLs, I'd lean toward Sitebulb Cloud or an API/data pipeline rather than repeatedly running desktop crawls. Sitebulb currently advertises audits up to 10 million URLs in Cloud. Sitebulb

3. Cluster duplicates instead of fixing URLs individually

This is the big scalability trick.

Suppose you have:

/blue-widget-small
/blue-widget-medium
/blue-widget-large
/blue-widget-xl

and 95% of each description is identical.

Don't create four manual tickets.

Create a duplicate cluster:

Cluster 1842
 ├─ canonical: /blue-widget
 ├─ /blue-widget-small
 ├─ /blue-widget-medium
 ├─ /blue-widget-large
 └─ /blue-widget-xl

Then classify the cluster:

• True duplicate → canonicalize/redirect/consolidate where appropriate
• Legitimate variants → retain separate pages but make their content meaningfully differentiated
• Faceted/search URLs → usually control indexation rather than rewriting thousands of descriptions
• Manufacturer copy → rewrite/enrich the commercially important products
• Out-of-stock/dead products → handle according to your catalog strategy

Sitebulb specifically supports finding both exact and similar content and provides URL/path information that helps identify these underlying patterns. Sitebulb Support

4. Generate descriptions from structured product data

For a massive catalog, don't have writers start with blank pages.

Create a product-description template + structured inputs:

Brand
Product type
Primary use
3–8 differentiating attributes
Materials
Dimensions
Compatibility
Warranty
What's included
Important limitations

Then generate a draft from those fields.

The important part is that the system should not merely paraphrase the existing manufacturer description. Give it product-specific facts that make each page materially different.

For example:

BAD:
"Experience superior performance with our high-quality
stainless steel water bottle..."

GOOD:
"Built for 32 oz capacity, this double-wall stainless bottle
fits standard car cup holders and keeps drinks cold for up
to X hours. The 3.1-inch opening accommodates ice cubes..."

The second version contains information tied to that specific product.

5. Put humans only where they add value

Don't send 500,000 products to editors.

Create a priority score such as:

priority =
  organic_traffic
× conversion_value
× duplicate_risk
× thinness_risk
× business_value

Then divide the catalog:

This prevents your team from spending six months rewriting pages nobody visits.

6. Create a continuous monitoring loop

I'd build this:

             PRODUCT DATABASE
                    ↓
             Content pipeline
                    ↓
       ┌────────────┴────────────┐
       ↓                         ↓
Duplicate detection       Thinness scoring
       ↓                         ↓
       └────────────┬────────────┘
                    ↓
             Priority queue
                    ↓
          AI/template rewrite
                    ↓
             Human QA sample
                    ↓
               Publish
                    ↓
          Crawl + GSC/SEO data
                    ↓
             Re-score pages

The really important part is the re-score loop. Otherwise you'll fix 100,000 descriptions and discover six months later that your CMS introduced another 300,000 duplicates.

7. Track the right KPIs

Don't make "number of words" your primary KPI.

Track:

• % of indexable product URLs with unique descriptions
• % of pages in duplicate clusters
• % of products below your category-specific content threshold
• organic clicks per product
• impressions/product
• indexed pages/product
• organic conversion rate
• revenue from optimized products
• number of pages consolidated
• number of new duplicate/thin pages created each week

Google's ecommerce documentation also emphasizes making sure search engines can discover the full set of ecommerce content, so I'd monitor indexability and crawl architecture alongside the actual descriptions. Google for Developers

My recommended stack

For 100K–1M+ products, I'd use:

CMS/PIM → warehouse → Python/SQL similarity pipeline → Sitebulb Cloud → Google Search Console → Ahrefs → content generation → CMS API

For a smaller catalog:

Screaming Frog + Semrush/Ahrefs + spreadsheet/database + controlled AI rewriting

And one important caveat: duplicate content isn't automatically an SEO penalty. The objective isn't "make every URL contain completely different words." The objective is to make indexable pages useful and sufficiently differentiated where there is a genuine reason for them to rank, while consolidating URLs that don't deserve separate search results.

If you tell me roughly how many products/URLs you have (100K, 1M, 10M+) and what platform you're on (Shopify, Magento/Adobe Commerce, BigCommerce, custom, etc.), I can lay out a concrete architecture—including the duplicate algorithm, database schema, scoring formula, and an automated rewrite/QA workflow.

My recommended stack

For 100K–1M+ products, I'd use:

CMS/PIM → warehouse → Python/SQL similarity pipeline → Sitebulb Cloud → Google Search Console → Ahrefs → content generation → CMS API

For a smaller catalog:

Screaming Frog + Semrush/Ahrefs + spreadsheet/database + controlled AI rewriting

For a huge ecommerce catalog, I’d treat duplicate/thin content as a data-quality pipeline, not a one-off SEO audit.

A scalable setup

1. Build a product-content inventory

Pull your catalog from the PIM/ERP/database into a table or warehouse with at least:

• SKU / product ID
• canonical URL
• product title
• description
• category / brand
• product type
• variant/parent ID
• indexability
• organic traffic/revenue
• conversion rate
• inventory status
• last content update

Then calculate basic signals such as:

• description word count
• character count
• percentage of description shared with other products
• duplicate title/H1 flags
• similarity to sibling products
• traffic/revenue opportunity

Don't use word count alone to define "thin." A 150-word description can be perfectly useful for one product and inadequate for another. Use it as a screening signal, then combine it with uniqueness, search demand, product complexity and business value.

2. Crawl the live site for the technical layer

For large catalogs, sitebulb.com and screamingfrog.co.uk are particularly useful.

Sitebulb can detect exact duplicates, near-duplicates, duplicate titles/descriptions/H1s and thin pages, and its cloud crawler is designed for very large sites. Sitebulb Sitebulb

Screaming Frog can identify exact and near-duplicate pages, low-content pages and duplicated page elements; its near-duplicate analysis uses configurable similarity thresholds. Screaming Frog Screaming Frog

I'd use the crawler to answer "what does Google actually see?", while your catalog database answers "what products do we have?"

3. Cluster duplicates instead of fixing URLs individually

This is where the process becomes scalable.

For example, instead of producing 14,000 individual tickets:

SKU A → duplicate
SKU B → duplicate
SKU C → duplicate

create clusters:

This distinction matters because duplicate detection is not automatically a "rewrite everything" problem. Some duplicates should be canonicalized, consolidated, redirected, or left alone depending on the architecture. Sitebulb Support Screaming Frog

4. Score pages for remediation priority

I'd create something like:

Content Priority Score =

SEO opportunity × business value × content deficiency × duplication severity

For example, a product with:

• $50k/month organic revenue
• 5,000 monthly impressions
• 92% similarity to 20 sibling products
• 45-word description

should jump ahead of a $30 product that gets virtually no traffic.

That prevents the classic enterprise SEO failure of spending six months fixing thousands of technically interesting but commercially irrelevant URLs.

Automate the actual rewriting

For thousands or millions of SKUs, don't have writers manually open product pages one at a time.

Instead, create a structured content-generation pipeline:

PIM data → quality checks → duplicate detection → content generation → human/automated QA → PIM → website → crawl validation

Give the generator structured product attributes rather than simply saying "write an SEO description."

For example:

Product:
Brand:
Model:
Product type:
Materials:
Dimensions:
Features:
Use cases:
Compatibility:
Warranty:
Target customer:

Existing description:
...

Required output:
- Unique description
- No unsupported claims
- No invented specifications
- Include important differentiators
- Avoid manufacturer boilerplate
- Match brand voice

The important part is grounding the copy in your actual product attributes. Otherwise, generating 100,000 descriptions can simply turn one content-quality problem into 100,000 hallucinated-content problems.

Add an automated QA gate

Before new descriptions reach production, run checks for:

• minimum useful-information threshold
• duplicate/near-duplicate similarity
• unsupported claims
• missing important attributes
• repeated phrases
• title/description duplication
• prohibited claims
• readability
• brand/style compliance
• required product terminology

Then send only exceptions to humans.

For example:

100,000 products
      ↓
95,000 pass automatically
      ↓
3,500 need enrichment
      ↓
1,200 have duplicate-content conflicts
      ↓
300 require human review

That's much more scalable than having humans review 100,000 pages.

Tools I'd put in the stack

Crawler / duplicate detection

• sitebulb.com — excellent for duplicate/near-duplicate analysis, thin-content analysis and recurring large-site crawls. Sitebulb Sitebulb
• screamingfrog.co.uk — excellent for deep technical crawling, custom extraction and duplicate/near-duplicate analysis. Screaming Frog Screaming Frog
• semrush.com — useful as a broader SEO layer for site auditing and competitive/search analysis; its Site Audit also flags highly similar pages. Semrush

Search/indexation monitoring

• search.google.com — use this for indexing/search-performance feedback rather than relying exclusively on crawler metrics.
• Google's Merchant listings/Product reports are particularly useful for checking product structured-data issues. Google for Developers

Data layer

• Your PIM/commerce database → warehouse (BigQuery, Snowflake, etc.)
• SQL/dbt for deterministic quality rules
• Python for large-scale similarity clustering if your catalog is enormous
• Embeddings/vector search if you want semantic similarity rather than just matching text

Content layer

• Your CMS/PIM as the source of truth
• An LLM/content-generation service for first drafts
• Human review only for high-value or low-confidence exceptions

The key architectural decision

I would not make your crawler the system of record.

Use:

PIM/database = truth about products
Crawler = truth about what is actually published
Search Console/analytics = truth about performance
Similarity engine = truth about content overlap
LLM = content production assistant
QA rules = production gate

Then run the pipeline continuously—e.g. nightly data-quality checks and weekly/monthly crawls—rather than doing a massive "duplicate content cleanup" once a year.

That gives you a system that can handle 100k+ products and continuously catch new duplicates/thin pages as suppliers, variants and merchandising teams add inventory.

For a huge ecommerce catalog, I’d treat this as a continuous content-quality pipeline, not a one-time SEO audit. The scalable setup is: crawl → cluster → score → prioritize → generate/rewrite → QA → monitor.

1. Detect duplicate product pages at scale

Use a crawler that can compare both exact and near duplicates.

• screamingfrog.co.uk — particularly good for this. Its near-duplicate analysis uses similarity matching, lets you adjust the threshold, and can show the differences between similar pages. Screaming Frog Screaming Frog
• semrush.com — useful when you want scheduled crawling, issue prioritization, and reporting for an SEO/content team. Semrush flags pages with very high similarity and also identifies low-content pages. Semrush Semrush
• siteliner.com — useful as a secondary duplicate-content check, though its free version is limited to 250 pages and Premium supports up to 30,000 pages. Siteliner Siteliner

For a catalog with hundreds of thousands or millions of URLs, I'd make your own product-level similarity pipeline the primary system and use crawlers for validation.

The important distinction

Don't just flag:

Product A = 92% similar to Product B

Instead, cluster products by meaningful product identity:

brand + model + product type + important attributes

Then ask whether each URL deserves to exist independently.

For example:

• Red Nike running shoe, size 10
• Blue Nike running shoe, size 10
• Nike running shoe, men's
• Nike running shoe, clearance
• Nike running shoe?sort=price

These shouldn't automatically become five independently indexable pages.

Google can group near-identical pages and select a canonical URL, particularly where differences are only things such as filtering, sorting, or product attributes. Google Help

2. Detect thin descriptions separately

Don't use a simple rule like "<300 words = bad."

Instead create a product-content score using fields such as:

Screaming Frog, for example, has a configurable low-content report and defaults to identifying pages below 200 words. Screaming Frog

But word count should be a diagnostic, not your definition of quality. A 120-word product page can be excellent if it contains the information a buyer needs; a 500-word manufacturer's boilerplate description can still be useless.

3. Build a prioritization score

This is where the process becomes scalable.

I'd create something like:

Content opportunity score =

traffic potential × business value × content deficiency × duplication severity

Then bucket products:

• P0: High revenue + high impressions + thin/duplicate → fix immediately
• P1: High search opportunity + weak content → rewrite
• P2: Duplicate variants with little independent value → consolidate/canonicalize
• P3: Low-value/no-demand products → don't spend editorial resources
• P4: Good pages → leave alone and monitor

This prevents the classic ecommerce mistake of spending six months rewriting 80,000 products nobody searches for.

4. Generate descriptions from structured product data

For a massive catalog, don't have AI "rewrite the existing description."

Give the model structured source data:

Product:
Brand: Acme
Model: X200
Category: Trail running shoe

Attributes:
Weight: 285g
Drop: 8mm
Waterproof: Yes
Upper: Gore-Tex
Outsole: Vibram
Use: Trail running

Audience:
Intermediate/advanced trail runners

Differentiators:
- Waterproof construction
- Aggressive outsole
- Lightweight for its category

Then generate from those facts.

That gives you much more control over:

• factual accuracy
• differentiation
• terminology
• brand voice
• required attributes
• prohibited claims

And importantly, don't deliberately manufacture uniqueness by changing synonyms. If 5,000 products are essentially the same product, the solution isn't 5,000 AI paraphrases.

5. Create reusable content templates

This is probably the biggest scalability win.

Instead of asking AI to invent every description, establish a schema:

Product description
├── 1-sentence value proposition
├── What it is
├── Key benefits
├── Important specifications
├── Best for
├── Differentiating characteristics
└── Compatibility / sizing / care information

Then use category-specific schemas.

For example, electronics need things like:

• compatibility
• ports
• dimensions
• power
• included accessories

Apparel needs:

• material
• fit
• sizing
• construction
• care

Furniture needs:

• dimensions
• materials
• assembly
• capacity
• room/use case

That produces genuine information density rather than generic SEO prose.

6. Automate QA before publishing

I'd put every generated description through automated checks:

IF missing_required_attribute
    → reject

IF unsupported_claim_detected
    → reject

IF similarity_to_existing_description > 0.85
    → regenerate

IF description_length < category_minimum
    → regenerate

IF prohibited_phrase_detected
    → reject

IF product_data_changed_since_generation
    → queue for refresh

You can also run semantic similarity using embeddings rather than relying solely on literal word matching. This is particularly useful for detecting descriptions that are technically different but communicate essentially the same thing.

7. Monitor Google separately

Your internal crawler tells you what you think is wrong. search.google.com tells you what Google is actually doing with your pages.

Pay particular attention to:

• indexed vs non-indexed products
• Google-selected canonical
• duplicate/alternate pages
• impressions with very low CTR
• pages losing impressions after changes
• products that never receive search visibility

Google recommends fixing pages that have been clustered together by making their content sufficiently different, and notes that reassessment can take time after changes. Google for Developers

8. My recommended stack

For a serious ecommerce operation, I'd use:

Discovery/crawling
→ Screaming Frog + scheduled Semrush crawls

Catalog data
→ PIM/database/export from your ecommerce platform

Similarity
→ exact hashes + MinHash/SimHash + embeddings

Scoring
→ SQL/dbt/Python pipeline

Generation
→ LLM API + category-specific templates + structured product attributes

QA
→ deterministic rules + LLM evaluation + sampling by humans

SEO validation
→ Google Search Console + crawler

Workflow
→ Jira/Linear/your CMS/PIM so content issues become actual production tasks

Dashboard
→ Looker Studio/Tableau/Power BI or your warehouse BI layer

The architecture I'd actually build

                 PRODUCT CATALOG / PIM
                         │
                         ▼
                 ┌─────────────────┐
                 │ Content pipeline│
                 └────────┬────────┘
                          │
          ┌───────────────┼────────────────┐
          ▼               ▼                ▼
     Exact matching   Near matching   Thin-content
       (hashes)       (MinHash)        scoring
          │               │                │
          └───────────────┼────────────────┘
                          ▼
                  PRIORITY SCORE
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
       Consolidate/canonical      Rewrite
              │                       │
              │                Structured data
              │                       +
              │                    AI draft
              │                       │
              │                       ▼
              │                     QA
              │                       │
              └───────────┬───────────┘
                          ▼
                       PUBLISH
                          │
                          ▼
                 SEARCH CONSOLE DATA
                          │
                          ▼
                  MONITOR / RE-SCORE

The key is that AI should be the remediation layer, not the detection layer. Detection and prioritization should be deterministic and data-driven; AI then fills the content gaps using your authoritative product attributes.

If you tell me roughly whether your catalog is 10k, 100k, 1M+ products and what platform you're on (Shopify, Magento/Adobe Commerce, BigCommerce, custom, etc.), I can lay out a concrete stack and workflow—including how I'd automate the duplicate clustering and AI rewriting.

• dimensions
• materials
• assembly
• capacity
• room/use case

That produces genuine information density rather than generic SEO prose.

6. Automate QA before publishing

I'd put every generated description through automated checks:

For a huge ecommerce catalog, I’d treat duplicate/thin content as a data-quality pipeline, not as a page-by-page SEO project.

Google’s own guidance emphasizes informative product descriptions, and duplicate content generally isn’t a “penalty” by itself—the bigger issue is that when pages are substantially similar, Google may choose one URL over another. support.google.com

A scalable setup

1. Crawl the whole catalog

• Screaming Frog SEO Spider — excellent for exact/near duplicates, low-content pages, custom extraction, and scheduled crawls. It can identify near-duplicates using similarity thresholds and extract things like SKU, price, headings, and description fields. www.screamingfrog.co.ukwww.screamingfrog.co.uk
• Sitebulb — particularly good if you need cloud crawling, prioritization, dashboards, and recurring audits. Its duplicate-content analysis covers HTML, titles, meta descriptions, H1s and similar content; Cloud supports very large crawls. www.screamingfrog.co.uksitebulb.comwww.screamingfrog.co.ukdevelopers.google.comsitebulb.com
• For millions of URLs, consider an enterprise crawler rather than relying exclusively on desktop crawling.

2. Build a product-content dataset

Export each PDP into a table/data warehouse with fields like:

This is where scalability really comes from: you analyze products as records rather than manually auditing URLs.

3. Use multiple levels of duplicate detection

Don't just search for identical descriptions.

I'd run:

Level A — exact duplicates

Hash normalized descriptions:

lowercase
→ strip HTML
→ remove whitespace
→ normalize punctuation
→ hash

Anything with the same hash goes into a duplicate cluster.

Level B — near duplicates

Use cosine similarity, MinHash/LSH, or embeddings to catch things like:

"Men's waterproof hiking jacket made from..."

vs.

"Waterproof hiking jacket for men made from..."

A crawler such as Screaming Frog can already do near-duplicate detection, but a warehouse-based similarity pipeline gives you much more control at very large scale. www.screamingfrog.co.ukwww.screamingfrog.co.uk

Level C — templated thin content

This is often the biggest ecommerce problem.

For example:

Nike Air Zoom Pegasus 41
Nike Air Zoom Pegasus 41 Women's
Nike Air Zoom Pegasus 41 Wide
Nike Air Zoom Pegasus 41 Blue

If 90% of the description is identical and only color/size changes, don't treat every page as an independent copywriting project.

Instead, identify the shared template + differentiating attributes.

4. Don't use "word count" as your definition of thin

A 150-word description can be much more useful than a 500-word generic one.

I'd create a content-quality score based on:

Content score =
  uniqueness
+ product-specific information
+ attribute coverage
+ search demand
+ organic traffic
+ conversion value
+ category importance

For example:

That prevents the common mistake of spending six months rewriting thousands of pages that don't matter.

5. Cluster before generating anything

This is the big scalability trick.

Suppose you have 500,000 products but only 3,000 product families.

Instead of generating 500,000 descriptions independently:

500,000 products
       ↓
3,000 product families
       ↓
shared content framework
       ↓
product-specific attributes
       ↓
human/automated QA
       ↓
500,000 validated descriptions

For each family, define required facts:

[Brand]
[Product type]
[Primary material]
[Dimensions]
[Compatibility]
[Key features]
[Use case]
[Warranty]
[Shipping/availability]

Then generate or assemble descriptions from structured product data, rather than asking an LLM to invent prose from scratch.

Google explicitly says automated/AI-generated content should prioritize accuracy, quality and relevance; for ecommerce, product data needs to remain accurate. developers.google.com

6. Use AI as the production layer, not the detection layer

I'd use AI for:

• rewriting supplier descriptions
• expanding genuinely useful product information
• turning structured attributes into readable prose
• creating differentiated descriptions within a product family
• identifying missing product facts
• classifying whether two descriptions are substantively redundant

But don't let AI decide what facts are true.

Your PIM/ERP/product feed should be the source of truth.

A good architecture is:

PIM / ERP
    ↓
Product data warehouse
    ↓
Duplicate + thin-content detection
    ↓
Priority scoring
    ↓
AI content generation
    ↓
Automated factual/SEO QA
    ↓
Human review for high-value products
    ↓
CMS
    ↓
Crawl + Search Console monitoring
    ↺

7. Prioritize using SEO + money

I'd create a simple priority score such as:

Priority =
  organic impressions
× revenue/conversion value
× duplication/thinness severity
× indexability importance

Then divide pages into:

Tier 1 — high revenue + high search demand
→ human-reviewed rewrite

Tier 2 — meaningful traffic + moderate revenue
→ AI-assisted rewrite + automated QA

Tier 3 — long tail
→ structured/template-generated content

Tier 4 — obsolete/low-value variants
→ investigate consolidation, canonicalization, redirects, or whether they should be indexable at all.

Don't automatically noindex every thin page. Some product variants are legitimate landing pages, while others may be better consolidated. The right answer depends on whether the URL provides unique value and deserves search visibility.

The tool stack I'd use

Lean stack

• Screaming Frog
• Google Search Console
• Google Analytics
• SQL/BigQuery
• Google Sheets/Looker Studio
• LLM/API for controlled rewriting

Larger catalog

• Sitebulb Cloud or enterprise crawler
• Semrush/Ahrefs for search/competitive data
• BigQuery/Snowflake
• PIM
• LLM API
• automated QA pipeline
• Search Console API
• scheduled crawls

Sitebulb is particularly attractive when you want recurring cloud crawls and team-wide access to the same audit data. www.screamingfrog.co.uksitebulb.comwww.screamingfrog.co.ukdevelopers.google.comsitebulb.com

One important distinction

I'd actually run two separate monitoring systems:

SEO crawler
→ "Are the pages technically/indexably healthy?"

Content intelligence pipeline
→ "Are our 500,000 product descriptions actually differentiated and useful?"

The crawler is great for finding the symptoms. Your warehouse/content pipeline is what makes fixing hundreds of thousands of products economically feasible.

If you tell me roughly how many SKUs you have (e.g. 50k, 500k, 5M), your ecommerce platform (Shopify, Magento, BigCommerce, custom, etc.), and whether you have a PIM, I can map this into a concrete architecture and tool stack.

For a huge product catalog, the key is to stop treating duplicate/thin content as a manual copywriting problem and build a content quality monitoring pipeline:

Detect → prioritize → enrich → approve → publish → monitor again

A scalable setup usually combines a crawler, a content database/warehouse, scoring rules, and (optionally) AI-assisted rewriting.

1. Build a catalog content health score

Create a score per SKU/product page using signals like:

Store the score in a dashboard so your team works worst-first, not category-by-category.

2. Use enterprise crawlers for detection

For thousands or millions of URLs, use crawlers that can process the whole catalog:

• Screaming Frog SEO Spider — useful for crawling product URLs, extracting page elements, finding duplicate titles/descriptions, canonicals, and technical issues. Paul Vengeons
• Semrush — useful for ecommerce audits, keyword gaps, rankings, and prioritizing pages that have search opportunity. Semrush
• JetOctopus — built around large-site crawling and can surface duplicate/thin product pages and crawl issues. TechSEO AI

For very large catalogs, export crawl data into a warehouse (BigQuery, Snowflake, etc.) and analyze it at SKU level.

3. Detect duplicate descriptions programmatically

Don't only look for exact duplicates. Use similarity detection.

A practical stack:

• Exact matching

• Hash normalized descriptions.
• Finds copy/paste duplicates instantly.
• Near-duplicate matching

• Convert descriptions into embeddings.
• Cluster products with high cosine similarity.
• Example: - Product A: "Premium waterproof hiking boot with durable leather..."
• Product B: "Durable leather hiking boot designed to withstand rain..."
• Different words, same underlying copy.

Tools commonly used:

• Vector databases: Pinecone, Weaviate, Elasticsearch
• Data processing: Python + NLP libraries
• Warehouse scoring pipelines

4. Prioritize which products to rewrite

Do not rewrite every SKU first.

Rank opportunities by:

1. Revenue potential
2. Organic impressions but low CTR
3. Ranking positions 5–30 (quick wins)
4. Duplicate severity
5. Inventory importance
6. Conversion rate

Example priority formula:

Priority Score =
(Search Impressions × Revenue Potential × Duplicate Severity)
÷ Rewrite Cost

A 100,000-SKU catalog might only need the top 5,000 pages fixed initially.

5. Use AI rewriting carefully

AI works well when it is fed structured product facts:

Good input:

• Brand
• Product type
• Materials
• Dimensions
• Features
• Target customer
• Use cases
• Warranty info
• Approved claims

Avoid:

• "Rewrite this product description" with no product data.

The workflow should be:

1. Generate draft
2. Check against product facts
3. Compare against original
4. Human approval for high-value products
5. Bulk publish lower-risk changes

Some catalog-focused tools are built specifically around this workflow, including bulk rewriting and review queues. Catalog Cleanup AI EcommerceLayer

6. Add ongoing monitoring

A mature system runs weekly/monthly:

Automated alerts

• New duplicate descriptions
• New supplier-imported copy
• Products falling below content score
• New thin pages
• Template changes creating thousands of duplicates

Dashboard metrics

Track:

• % products with unique descriptions
• Average content score
• Organic traffic per optimized SKU
• Conversion rate change
• Index coverage
• Number of duplicate clusters

Example scalable architecture

PIM / Ecommerce Platform
          |
          v
Catalog Export (daily)
          |
          v
Content Quality Pipeline
          |
  -----------------------
  |          |          |
Duplicate   Thin     SEO Gap
Detection   Score    Detection
  |          |          |
  -----------------------
          |
          v
Prioritization Dashboard
          |
          v
AI Rewrite + Human Review
          |
          v
CMS / PIM Publish
          |
          v
Crawler Monitoring

Tool combinations by catalog size

1,000–20,000 products

• Screaming Frog
• Semrush
• Spreadsheet/database scoring
• AI-assisted rewrites

20,000–500,000 products

• Enterprise crawler (JetOctopus, Screaming Frog at scale, similar)
• Data warehouse
• Embedding-based duplicate detection
• PIM integration
• Automated rewrite workflows

500,000+ products

• Treat it like a data engineering problem: - Product information management system
• Automated content QA
• Vector similarity search
• Human review only for priority tiers

The biggest mistake with large catalogs is creating a giant rewrite project. The scalable approach is to create a continuous content quality system that catches bad product pages as they appear.

For a huge ecommerce catalog, I’d treat this as an automated content-quality pipeline, not a one-time SEO audit. The key is separating true duplicate URLs from legitimate product variants, then scoring thinness based on useful product information, not simply word count.

A scalable setup

1. Crawl continuously for technical duplicates

Use screamingfrog.co.uk or semrush.com as your crawler layer.

Screaming Frog is particularly useful for this because it can identify:

• Exact duplicate pages
• Near-duplicate pages using configurable similarity thresholds
• Low-content pages
• Duplicate titles/meta descriptions
• Canonical/indexability problems

Its near-duplicate system uses MinHash and defaults to a 90% similarity threshold, which you can tune. Screaming Frog Screaming Frog

For an enterprise catalog, I'd run this on a schedule rather than manually.

Output: a table like:

2. Don't use word count as your definition of "thin"

This is a major trap.

A 150-word product page can be perfectly useful for a simple product. A 500-word page can still be thin if 450 words are generic boilerplate.

Instead, create a Product Content Score based on fields that actually differentiate the product:

• Unique product description
• Material/specifications
• Dimensions
• Compatibility
• Use cases
• What's included
• Care/installation information
• Warranty/returns information
• Original imagery
• FAQs
• Product-specific benefits
• Structured product data

Then flag products based on missing information + similarity + business value, rather than "under 200 words."

Screaming Frog itself cautions that there isn't a universal minimum word count; its default low-content filter is 200 words, but that threshold is configurable. Screaming Frog Screaming Frog

3. Build duplicate detection into your data pipeline

For tens or hundreds of thousands of products, don't make your crawler do everything.

At the catalog/database level, calculate:

Exact duplicate

hash(normalized_description)

Near duplicate

similarity(description_A, description_B)

Semantic duplicate

embedding(description_A) ↔ embedding(description_B)

The last one is particularly valuable because two descriptions can use different wording while communicating essentially the same information.

I'd cluster products into groups such as:

• 0–60% similarity: probably unique
• 60–85%: review
• 85–95%: likely duplicate template
• 95%+: very likely duplicate

Those aren't Google thresholds—they're operational thresholds I'd use to prioritize human review.

Screaming Frog's semantic-similarity functionality is also moving beyond simple text matching toward embeddings, which is useful for this type of problem. Screaming Frog

4. Treat product families differently

This is crucial for ecommerce.

Suppose you sell:

• Nike Air Max, men's
• Nike Air Max, women's
• Nike Air Max, size 10
• Nike Air Max, red
• Nike Air Max, blue

You don't necessarily want five radically different descriptions.

Instead, establish a canonical product/family structure:

Product family
│
├── Shared factual information
├── Variant-specific information
├── Color-specific information
├── Size/spec information
└── Unique merchandising copy

Then decide whether each variation deserves its own indexable URL based on:

Does this URL have distinct search intent and enough unique value?

If not, consolidate/canonicalize rather than forcing an AI to invent 200 words for every SKU.

Near-duplicate pages can be legitimate—for example, different product configurations—so tools such as Screaming Frog explicitly recommend reviewing them in context rather than automatically deleting or rewriting everything. Screaming Frog Screaming Frog

5. Use AI for the rewrite queue, not the detection system

I'd avoid:

"Give every product a new 300-word SEO description."

That creates a gigantic pile of mediocre, interchangeable copy.

Instead, feed the AI structured product data:

PRODUCT
Brand:
Model:
Category:
Materials:
Dimensions:
Features:
Compatibility:
Use cases:
Customer questions:
Existing description:
Competitor/product-family descriptions:

Then ask it to identify:

1. What information is missing?
2. What information is duplicated?
3. What makes this product different?
4. What customer questions aren't answered?
5. What factual claims can safely be made from the catalog data?
6. What should be rewritten?

The AI should never invent specifications, certifications, materials, compatibility, performance claims, etc.

6. Prioritize with an impact score

This is where scalability really comes from.

Don't rewrite products in SKU order.

Give every URL a score roughly like:

Priority =
  organic traffic potential
× revenue importance
× duplicate/thin severity
× search opportunity
× indexation importance

So:

10,000 products × low traffic × obsolete inventory

shouldn't beat:

500 products × high search demand × high revenue × poor descriptions.

I'd create four queues:

• P0 — Technical: duplicate URLs, canonical/indexation problems
• P1 — High-value thin: important products with inadequate content
• P2 — Near-duplicate clusters: rewrite/consolidate families
• P3 — Long tail: improve opportunistically

7. The tool stack I'd use

Semrush's current Site Audit supports 140+ technical/on-page checks, scheduled crawls, and large crawl limits depending on tier, making it useful as the recurring monitoring layer. Semrush

The architecture I'd aim for

                 PRODUCT DATABASE / PIM
                         │
                         ▼
                ┌─────────────────┐
                │ Content Profiler│
                └────────┬────────┘
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
     Exact hashes   Similarity       Embeddings
          │              │              │
          └──────────────┼──────────────┘
                         ▼
                  CONTENT CLUSTERS
                         │
                         ▼
               PRIORITY / IMPACT SCORE
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
       Fix URL       Rewrite        Leave alone
      structure       with AI
          │              │
          └──────────────┼──────────────┘
                         ▼
                    QA / REVIEW
                         │
                         ▼
                     PIM / CMS
                         │
                         ▼
                  RECRAWL + MEASURE

The important part is the closed loop: detect → prioritize → fix → recrawl → measure.

And don't automatically treat duplicate content as a Google penalty. The practical problem is usually indexation/canonicalization, wasted crawling, diluted signals, and pages that aren't differentiated enough to deserve separate visibility. Screaming Frog makes the same distinction when explaining its duplicate detection. Screaming Frog Screaming Frog

If you're dealing with 100k+ SKUs, I'd lean toward a custom pipeline around your PIM/database + embeddings, with Screaming Frog/Semrush used as the external SEO validation layer. That will scale substantially better than trying to manage the catalog inside an SEO tool alone.

Then ask it to identify:

1. What information is missing?
2. What information is duplicated?
3. What makes this product different?
4. What customer questions aren't answered?
5. What factual claims can safely be made from the catalog data?
6. What should be rewritten?

The AI should never invent specifications, certifications, materials, compatibility, performance claims, etc.

6. Prioritize with an impact score

This is where scalability really comes from.

Don't rewrite products in SKU order.

Give every URL a score roughly like:

Priority =
  organic traffic potential
× revenue importance
× duplicate/thin severity
× search opportunity
× indexation importance

So:

10,000 products × low traffic × obsolete inventory

shouldn't beat:

500 products × high search demand × high revenue × poor descriptions.

I'd create four queues:

• P0 — Technical: duplicate URLs, canonical/indexation problems
• P1 — High-value thin: important products with inadequate content
• P2 — Near-duplicate clusters: rewrite/consolidate families
• P3 — Long tail: improve opportunistically

7. The tool stack I'd use

Semrush's current Site Audit supports 140+ technical/on-page checks, scheduled crawls, and large crawl limits depending on tier, making it useful as the recurring monitoring layer.

The architecture I'd aim for

For a huge ecommerce catalog, I’d treat this as a continuous data-quality pipeline, not a one-off SEO audit.

A scalable setup

1. Crawl the catalog to find the problems

Use a crawler to collect, per product URL:

• Product description word count
• Exact duplicate descriptions
• Near-duplicate descriptions
• Duplicate titles/H1s/meta descriptions
• Canonical URL
• Indexability
• HTTP status
• Internal links
• Organic traffic/impressions
• Conversion/revenue

sitebulb.com is particularly useful here: its duplicate-content analysis handles both exact duplicates and similar/near-duplicate pages, while its on-page reports can flag thin pages based on configurable word-count thresholds. Sitebulb Support Sitebulb

For very large catalogs, Sitebulb Cloud advertises crawling up to 10 million URLs per audit and recurring crawls, which makes it a reasonable monitoring layer rather than something you run once a year. Sitebulb

2. Don't define "thin" as simply "under X words"

This is a common mistake.

A 120-word product description can be perfectly useful for a simple commodity product, while a 500-word description can still be useless if it's mostly boilerplate.

I'd score products using something like:

Content quality score =

• Description uniqueness
• Description length
• Number of unique product attributes
• Search impressions/clicks
• Conversion/revenue
• Product/category importance
• Competitive content gap

Then create buckets:

• P0: duplicate + high-traffic/high-revenue product
• P1: duplicate/near-duplicate + indexable
• P2: thin + meaningful search demand
• P3: thin + little/no demand
• P4: discontinued/variant/filter URLs that shouldn't be indexed

That prevents your team from wasting weeks rewriting products nobody searches for.

3. Detect duplicate families, not just duplicate URLs

This is where automation gets powerful.

Imagine 20,000 products where manufacturers supplied essentially the same description:

"The XYZ Widget features premium construction..."

Don't create 20,000 independent tickets.

Cluster them:

Supplier/manufacturer template
        ↓
8,431 products
        ↓
127 description clusters
        ↓
12 high-value clusters
        ↓
rewrite templates + product-specific attributes

You can use text similarity/embeddings to cluster descriptions and then have an LLM classify why they're similar.

For example:

• Exact duplicate
• Manufacturer boilerplate
• Same product family
• Variant legitimately sharing copy
• Accidentally copied
• Thin but unique
• Thin because product data is incomplete

That classification is much more valuable than a raw duplicate percentage.

4. Fix the catalog upstream

This is arguably the most important part.

If your product descriptions originate in a PIM, ERP, supplier feed, or spreadsheet, don't continually fix them on the website.

Build fields such as:

product_id
manufacturer_description
seo_description
key_features
materials
dimensions
compatibility
use_cases
unique_selling_points
content_status
content_quality_score
last_reviewed

Then generate the SEO description from structured product attributes + editorial rules.

For example:

Manufacturer copy
        +
Product attributes
        +
Category-specific requirements
        +
Brand voice
        ↓
SEO description

That makes it much harder for 5,000 variants to accidentally inherit exactly the same paragraph.

5. Use AI for the rewriting, but rules for the decisions

I wouldn't have an LLM blindly rewrite your entire catalog.

Instead:

Programmatic layer decides:

• Which products need work
• What information is missing
• What competitor/product attributes matter
• Whether the page should be indexed
• Which duplicate cluster it belongs to
• Minimum/maximum content requirements

AI layer does:

• Drafting
• Rephrasing
• Combining structured attributes naturally
• Creating differentiated descriptions
• Identifying missing information
• Quality scoring

Human/editorial layer does:

• Approves templates
• Reviews high-value products
• Handles unusual products
• Audits AI output

That can take you from manually rewriting 50 products/day to processing thousands while keeping humans focused on exceptions.

The tools I'd combine

I wouldn't buy every SEO platform. A crawler + Search Console + one serious SEO research platform + your catalog database is generally a much cleaner architecture.

One important distinction: duplicate content vs duplicate URLs

A huge ecommerce catalog can appear to have a "duplicate content problem" when the real problem is URL proliferation.

Filters can generate thousands or millions of combinations. Google specifically warns that faceted navigation can cause overcrawling and recommends controlling which facet URLs are crawlable/indexable; canonicalization can be part of the solution where appropriate. Google for Developers

So your monitoring should separately report:

CONTENT DUPLICATION
├── exact product copy
├── near-duplicate product copy
├── duplicate titles
└── duplicate meta descriptions

URL DUPLICATION
├── tracking parameters
├── filters
├── sorting
├── session URLs
└── alternate product URLs

Don't solve both with "rewrite the content."

The monitoring dashboard I'd build

Every week, track:

• % products with unique descriptions
• % products below your category-specific content threshold
• # exact duplicate clusters
• # near-duplicate clusters
• # indexable products without useful copy
• # products with zero organic impressions
• # products receiving organic traffic
• Organic clicks/impressions for remediated products
• Revenue from remediated products
• New duplicate content introduced since last crawl
• New thin pages introduced since last crawl

And most importantly:

New problems → automatically enter the remediation queue.

Google also recommends ensuring important ecommerce products are reachable through normal site navigation, with sitemaps/merchant feeds as additional discovery mechanisms. Google for Developers

If I were implementing this from scratch

I'd build:

Crawler → warehouse → duplicate/similarity engine → quality scoring → prioritized queue → AI/content generation → human QA → PIM/CMS → recurring crawl → performance measurement

The key insight is that the crawler shouldn't be the system of record. Your product database should be. The crawler detects what Google sees; your catalog/PIM should control what gets fixed.

If you tell me roughly whether you're dealing with 10k, 100k, 1M+ products and what platform you're on (Shopify, Magento/Adobe Commerce, Salesforce Commerce, custom, etc.), I can suggest a concrete architecture and tool stack for that scale.

That makes it much harder for 5,000 variants to accidentally inherit exactly the same paragraph.

5. Use AI for the rewriting, but rules for the decisions

I wouldn't have an LLM blindly rewrite your entire catalog.

Instead:

Programmatic layer decides:

• Which products need work
• What information is missing
• What competitor/product attributes matter
• Whether the page should be indexed
• Which duplicate cluster it belongs to
• Minimum/maximum content requirements

AI layer does:

• Drafting
• Rephrasing
• Combining structured attributes naturally
• Creating differentiated descriptions
• Identifying missing information
• Quality scoring

Human/editorial layer does:

• Approves templates
• Reviews high-value products
• Handles unusual products
• Audits AI output

That can take you from manually rewriting 50 products/day to processing thousands while keeping humans focused on exceptions.

The tools I'd combine

For a huge catalog, I’d treat this as a data-quality + SEO monitoring pipeline, not a one-off content audit. The key is to identify which products deserve unique copy rather than trying to rewrite every SKU.

1. Build a catalog-level content score

Export your product feed/database with at least:

• SKU / product ID
• URL
• Product name
• Brand
• Category
• Product description
• Specs/attributes
• Canonical URL
• Indexability
• Organic clicks/impressions
• Revenue/conversions
• Backlinks
• Inventory/status

Then calculate per product:

Content quality score =

• description length
• uniqueness vs other products
• uniqueness vs manufacturer feed
• presence of useful attributes/specs
• search demand
• organic traffic
• conversion/revenue
• indexability

Don't use word count alone as your definition of "thin." A 100-word description containing genuinely useful product information can be better than 500 words of boilerplate.

2. Use a crawler to detect duplicates automatically

For this specific problem, Sitebulb is particularly interesting because it detects both exact duplicates and near-duplicates, and can flag thin content based on configurable word-count thresholds. Its cloud version is designed for very large ecommerce sites, with audits advertised up to 10 million URLs. sitebulb.com

Screaming Frog SEO Spider is another excellent option. Its near-duplicate analysis can identify pages with roughly 90% similarity by default, with the threshold adjustable. www.screamingfrog.co.uk

For an enterprise catalog, I'd use one of those as the crawling/diagnostic layer, rather than trying to detect duplication manually in spreadsheets.

3. Separate duplicates into different buckets

This is where the system becomes much more useful.

This distinction matters because duplicate content isn't automatically something you should "fix" by rewriting everything. Ecommerce sites naturally have repeated elements, and Google's systems can choose between substantially similar pages rather than treating every duplicate as a manual penalty. support.google.com

4. Create a "content opportunity" queue

Instead of:

"We have 300,000 thin products. Rewrite 300,000 descriptions."

Do:

"Which 20,000 products could generate the most incremental value?"

For example:

Priority = search opportunity × commercial value × content deficiency × indexability

That could give you a queue like:

1. 2,400 products with high impressions + thin descriptions
2. 5,100 products ranking positions 5–20 + near-duplicate copy
3. 8,000 products with strong sales but manufacturer descriptions
4. 50,000 low-demand products → leave alone or handle programmatically

This is dramatically more scalable.

5. Automate the actual rewriting carefully

For thousands of products, I'd make your PIM/product database the source of truth and generate copy from structured attributes rather than asking an AI model to invent descriptions.

For example:

INPUT
Brand
Product type
Material
Dimensions
Compatibility
Features
Use cases
Warranty
Existing description

        ↓

QUALITY RULES
Required attributes
Forbidden claims
Brand terminology
Minimum useful information
Duplicate phrase limits

        ↓

GENERATED DESCRIPTION

        ↓

VALIDATION
Missing attributes?
Unsupported claims?
Too similar to another SKU?
Too short?
Too generic?

        ↓

HUMAN REVIEW
Only high-value / low-confidence products

The important part is the validation layer. Don't publish AI-generated copy simply because it is grammatically different from the original.

6. Monitor duplication continuously

Set up recurring crawls and compare each crawl against the previous one. Sitebulb, for example, supports scheduled audits and historical comparison, which is useful when products are constantly being added, changed, or discontinued. sitebulb.comsitebulb.com

I'd have alerts for:

• New duplicate-description clusters
• >X% of new products below your content threshold
• Manufacturer descriptions appearing unchanged
• Large increases in near-duplicate pages
• Newly indexable thin pages
• Canonical changes
• Product pages disappearing from the index
• Templates accidentally replacing unique descriptions with boilerplate

That turns the problem from "SEO team audits catalog every quarter" into "bad content gets caught within days of entering the catalog."

A scalable stack

I'd roughly structure it like this:

Crawler:
Sitebulb or Screaming Frog

Search/performance data:
Google Search Console

Keyword/competitor data:
Ahrefs or Semrush

Catalog source:
PIM / Shopify / Magento / Salesforce Commerce / your internal product database

Processing:
SQL + Python/ETL + embeddings or similarity detection

Generation:
LLM API behind strict templates, product attributes, and validation rules

Reporting:
Looker Studio / Power BI / your warehouse BI

The big architectural idea is: crawl → warehouse → score → prioritize → generate → validate → publish → recrawl.

For a catalog of 100k+ SKUs, I'd strongly favor building the duplicate/thin-content detection in your data warehouse and using the crawler primarily to validate what is actually rendered/indexable. That prevents you from repeatedly doing expensive full-catalog analysis when most of the underlying product data hasn't changed.

For a huge catalog, you need to treat duplicate/thin product content as a data quality pipeline, not a one-time SEO audit. The scalable approach is:

Detect → prioritize → generate improvements → review → publish → monitor

1. Build a catalog content health layer

Create a database/report that scores every SKU on:

A simple scoring model:

Content Opportunity Score =
(duplicate risk × 40%)
+ (thin content risk × 30%)
+ (organic traffic opportunity × 20%)
+ (conversion opportunity × 10%)

This prevents wasting time rewriting low-value SKUs first.

2. Use crawlers for detection

For enterprise catalogs, common tools include:

• Screaming Frog SEO Spider — good for crawling large sites, finding duplicate titles/descriptions, word-count issues, and exportable URL-level data.
• Semrush — useful for site audits, keyword overlap, and content monitoring.
• Ahrefs — useful for organic performance signals and competing pages.
• JetOctopus — built around large-scale crawling and ecommerce SEO analysis, including thin and duplicate product page detection. jetoctopus.com

For very large catalogs (100k+ URLs), look for:

• API access
• scheduled crawls
• exports into BigQuery/Snowflake
• URL segmentation by category, brand, inventory status, margin, etc.

3. Detect duplicates beyond exact matching

Exact matching is easy:

hash(description_text)

But ecommerce problems are usually near duplicates:

Example:

Product A:

"Premium leather wallet with RFID protection and six card slots."

Product B:

"High-quality leather wallet featuring RFID blocking technology and six card pockets."

Different words, same content.

Use:

• embeddings/vector similarity
• cosine similarity
• clustering

A common workflow:

1. Generate embeddings for every product description.
2. Compare products within the same category.
3. Flag anything above a similarity threshold.

Example thresholds:

• 95%+ similarity → likely duplicate
• 80–95% → human review
• <80% → probably unique

4. Fix thin descriptions with structured generation

Avoid "rewrite everything with AI." At catalog scale, use templates plus product attributes.

Bad:

"This blue shirt is comfortable and stylish."

Better:

[Product purpose]
+ [material]
+ [unique features]
+ [buyer problem solved]
+ [specifications]
+ [care/use information]
+ [comparison guidance]

Example inputs:

Category: hiking backpack
Material: recycled nylon
Capacity: 35L
Features:
- waterproof zippers
- laptop sleeve
- hip belt

Generate:

• short description
• long description
• bullets
• FAQs
• comparison points
• metadata

Then run quality checks before publishing.

5. Add human review where it matters

Do not manually review every SKU.

Create tiers:

Tier 1 — automatic

Low-risk:

• missing bullets
• missing specs
• obvious duplicate supplier copy

Tier 2 — editorial review

High-value:

• top sellers
• high-margin products
• category leaders
• products ranking positions 5–20

Tier 3 — deprioritize

Low impact:

• discontinued items
• products with no impressions
• duplicate variants that should be consolidated

6. Automate monitoring

Set recurring checks:

Weekly

• new duplicate descriptions
• supplier feed changes
• missing descriptions
• products falling below word/spec thresholds

Monthly

• organic traffic changes
• index coverage
• rankings
• conversion impact

Useful integrations:

• Google Search Console API
• ecommerce platform API
• SEO crawler API
• data warehouse

7. Tools specifically aimed at catalog-scale content cleanup

Depending on your stack:

Shopify-heavy catalogs

Tools exist that combine catalog auditing and bulk rewriting workflows, such as Catalog Cleanup AI, which focuses on detecting duplicate descriptions and rewriting product content with review workflows. catalogcleanup.app

Enterprise ecommerce teams

Look at:

• JetOctopus for crawling/diagnostics jetoctopus.com
• Screaming Frog + custom data pipeline
• Semrush/Ahrefs for SEO impact measurement
• Content operations platforms connected to your PIM/CMS

8. A scalable architecture

A mature setup often looks like:

Product Database / PIM
          |
          v
Content Audit Pipeline
          |
          +--> Duplicate Detector
          |
          +--> Thin Content Scorer
          |
          +--> SEO Performance Data
          |
          v
Priority Queue
          |
          v
AI + Editorial Workflow
          |
          v
CMS / Ecommerce Platform
          |
          v
Monitoring + Regression Alerts

For a catalog with 10k, 100k, or 1M+ SKUs, the biggest mistake is trying to "write better descriptions." The scalable win is building a system that continuously identifies the highest-value content gaps and fixes those first.

For a huge catalog, you need to treat product-content quality like a data pipeline, not a one-off SEO cleanup. The scalable approach is:

Detect → prioritize → generate fixes → review → publish → monitor

1. Build a catalog content health score

Create a score for every SKU/product URL based on signals like:

• Duplicate description percentage - Exact matches
• Near-duplicates (same wording with product names swapped)
• Supplier/manufacturer copy reused elsewhere
• Description length and completeness - Missing descriptions
• Very short copy
• Missing benefits/use cases
• SEO fields - Duplicate titles
• Duplicate meta descriptions
• Missing structured data
• Performance signals - Organic traffic
• Revenue
• Conversion rate
• Backlinks
• Index status

Then prioritize:

1. High-revenue products with poor content
2. Products ranking positions 5–30 (often easiest wins)
3. Category leaders
4. Long-tail SKUs with zero differentiation

2. Use crawlers to find problems automatically

For large catalogs, use enterprise crawlers that can compare thousands or millions of pages.

Common options:

• Screaming Frog — good for crawling, duplicate titles/descriptions, content analysis, and exports into data pipelines.
• Semrush — combines technical auditing, rankings, and ecommerce SEO monitoring; its ecommerce tooling includes audits for duplicate content and technical issues. Semrush
• JetOctopus — designed for large sites and specifically highlights thin/duplicate product pages, crawl issues, and ecommerce-scale analysis. Tech SEO Platform

For very large catalogs, avoid relying only on a monthly crawl. Run continuous monitoring on:

• New products
• Updated products
• Imported supplier feeds
• Variant creation

3. Detect near-duplicates, not just exact duplicates

Exact matching misses most ecommerce duplication.

Use text similarity methods:

• Shingling / n-grams
• Cosine similarity on embeddings
• MinHash / locality-sensitive hashing
• Vector databases for clustering

Example:

These should cluster together:

"Premium leather wallet with RFID protection and six card slots."
"A premium RFID leather wallet featuring six card pockets."

They are not identical strings, but they are effectively the same page content.

A practical stack:

• Export product descriptions daily
• Create embeddings
• Store vectors
• Cluster similar products
• Flag clusters above a similarity threshold (for example, 85–90%)

4. Fix content with templates + controlled generation

Do not simply ask AI to "rewrite everything." That creates inconsistent copy and possible factual errors.

Create category-specific content frameworks.

Example:

Running shoes template

• Product positioning
• Best use case
• Cushioning/fit
• Materials
• Durability
• Who it's for
• Comparison against similar models

Coffee maker template

• Brewing method
• Capacity
• Counter space
• Cleaning
• Ideal user
• Key differentiator

Then generate only from verified product attributes.

Tools that focus on bulk catalog enrichment include platforms such as catalogcleanup.app and ecommercelayer.com, which are built around large-scale product copy improvements and review workflows. Catalog Cleanup AI EcommerceLayer

5. Keep humans in the approval loop

For thousands of SKUs, use:

• Batch approval
• Before/after diffs
• Category editors
• Brand voice rules
• Forbidden claims lists
• Product fact validation

A good workflow:

1. AI proposes changes
2. Merchandising team reviews samples
3. Approve a category batch
4. Publish
5. Measure organic traffic and conversion impact

6. Monitor after fixes

Set up recurring checks:

Weekly

• New duplicate descriptions
• Newly imported supplier copy
• Missing descriptions
• Pages losing impressions

Monthly

• Organic clicks by product group
• Rankings
• Conversion rate changes
• Index coverage

Quarterly

• Refresh stale products
• Rewrite low-performing pages
• Merge overlapping products

7. Suggested architecture for a very large catalog

A scalable setup might look like:

Data sources

• Shopify / Magento / Salesforce Commerce Cloud / PIM
• Product database
• Search Console
• Analytics

↓

Content quality engine

• Crawl data
• Similarity detection
• Rules engine
• AI enrichment

↓

Workflow layer

• Jira / Asana / CMS queue
• Approval dashboards

↓

Publishing

• PIM → ecommerce platform

↓

Monitoring

• SEO crawler
• Rankings
• Revenue attribution

Tool choices by catalog size

The biggest mistake at scale is rewriting everything. The better approach is to build a product content quality system that continuously finds the worst pages, fixes the highest-value ones, and prevents the catalog from degrading again.

For a huge catalog, the scalable approach is to treat product content like a data quality pipeline, not a copywriting project. The goal is to continuously detect problems, prioritize them, generate fixes, and measure impact.

A practical architecture:

1. Build a catalog content audit layer

Run automated checks across every SKU:

Duplicate content signals

• Exact duplicate descriptions
• Near-duplicate descriptions (same template, swapped product name/color)
• Manufacturer/supplier copy reused across the web
• Duplicate title + description combinations
• Variant pages with almost identical text

Thin content signals

Flag products with:

• Description length below a threshold (e.g., <100–150 words depending on category)
• Missing benefits/use cases
• Missing structured attributes
• Only a spec list with no shopper guidance
• Missing FAQs, compatibility notes, sizing info, etc.

Tools commonly used:

• Screaming Frog — crawl product pages and identify duplicate titles, descriptions, missing elements.
• Semrush Site Audit — monitor technical/content issues.
• Ahrefs Site Audit — useful for large-scale crawling and content analysis.
• Product-content audit tools that score catalogs and flag duplicate/thin descriptions are also emerging; some ecommerce-focused platforms analyze descriptions and metadata in bulk. app.getcassian.comcatalogcleanup.app

2. Create a product content scoring model

Don't just output "bad/good." Give every SKU a score.

Example:

Then prioritize:

Tier 1

• High traffic + thin/duplicate content
• Best ROI

Tier 2

• High-margin products
• Products ranking on page 2–3
• Products with declining organic traffic

Tier 3

• Long-tail SKUs with little demand

3. Store product facts separately from marketing copy

This is where many catalogs fail.

Keep structured data:

Product facts:
- Material
- Dimensions
- Weight
- Compatibility
- Warranty
- Features
- Use cases

Generated content:
- Short description
- Long description
- Buying guide
- FAQs
- Comparison copy

Your AI or copy system should rewrite from facts, not from the old description. Otherwise you just create a prettier duplicate.

4. Use AI for bulk rewriting — with guardrails

For thousands of SKUs, use generation workflows:

Input:

Product name
Category
Attributes
Specifications
Customer questions
Brand voice rules
Competitor positioning

Output:

SEO title
Meta description
Short description
Long description
FAQ
Bullet benefits

Important controls:

• Require factual grounding
• Reject missing-spec hallucinations
• Run similarity checks after generation
• Human approval before publishing

Some ecommerce content platforms specifically support bulk rewriting, duplicate detection, and approval workflows rather than silently changing catalog text. catalogcleanup.app

5. Add similarity monitoring after cleanup

The job never ends because catalogs change.

Run recurring checks:

Daily

• New products missing descriptions
• Supplier-imported copy
• Duplicate variants

Weekly

• New duplicate clusters
• Content regressions
• Pages losing rankings

Monthly

• Refresh underperforming products

6. Connect content quality to SEO and revenue

Don't optimize every SKU equally. Tie fixes to:

• Organic impressions
• CTR
• Conversion rate
• Add-to-cart rate
• Revenue per session
• Search query coverage

Example prioritization:

Product A
20,000 impressions/month
Position 12
Thin description

→ Rewrite first

Product B
20 impressions/month
Duplicate description

→ Lower priority

7. Enterprise-scale stack example

A common setup:

Catalog source

• Shopify / Magento / BigCommerce / PIM

↓

Crawler + SEO monitoring

• Screaming Frog
• Semrush
• Ahrefs

↓

Data warehouse

• BigQuery / Snowflake

↓

Content quality engine

• Custom rules + embeddings similarity search

↓

AI enrichment

• LLM workflow with product attributes

↓

Approval workflow

• Merchandising team review

↓

Publishing

• API back into ecommerce platform

Metrics worth tracking

After implementation, monitor:

• % of SKUs with unique descriptions
• Average description quality score
• Duplicate-description count
• Thin-content count
• Organic clicks per optimized SKU
• Conversion lift on rewritten products
• Time from product creation → SEO-ready

For a catalog with tens of thousands of SKUs, the biggest win usually comes from automated detection + prioritization + controlled bulk rewriting, not trying to manually improve everything. www.under.io

For a large ecommerce catalog (10,000+ SKUs), the goal isn't to manually "find duplicate pages"—it's to build an automated content quality pipeline that continuously scores, prioritizes, and fixes problems.

A scalable workflow typically looks like this:

1. Use a crawler as your "content inventory"

Enterprise crawlers can scan hundreds of thousands (or even millions) of URLs and export:

• duplicate page titles
• duplicate meta descriptions
• duplicate H1s
• exact duplicate body text
• canonical issues
• word counts
• near-empty pages

These tools are the industry standard:

• Screaming Frog SEO Spider (excellent for exports and automation)
• Sitebulb (better visual reporting)
• Botify or Deepcrawl/Lumar (enterprise scale)

Rather than opening pages individually, export everything into Excel or a data warehouse and sort by issue count.

2. Detect near duplicates, not just exact copies

Exact duplicates are only half the problem.

Many catalogs contain pages like:

Blue Running Shoe

and

Red Running Shoe

where 98% of the description is identical.

Traditional SEO crawlers won't always flag these.

Instead use similarity scoring:

• cosine similarity on embeddings
• MinHash
• SimHash
• sentence embeddings
• vector search

This lets you identify pages that are, say:

• 95% identical
• 90% identical
• 80% identical

instead of only exact copies.

Modern embedding-based approaches are much better at finding semantically similar content at scale. arxiv.org

3. Score thin content automatically

Instead of asking:

Is this page thin?

build a quality score.

For example:

Description length
+ unique features
+ specifications
+ FAQs
+ reviews
+ images
+ alt text
+ internal links
= Product Quality Score

Example thresholds:

• <75 words → critical
• 75–150 → needs improvement
• 150–300 → acceptable
• 300+ → comprehensive

Many ecommerce audit platforms automate checks for thin descriptions, duplicate copy, missing metadata, and related content quality signals. blog.hubspot.com

4. Prioritize by business impact

Don't rewrite 50,000 products alphabetically.

Instead calculate:

Priority =
Organic traffic
× Revenue
× Conversion rate
× Content quality score

This usually identifies the top few hundred products where better content will have the biggest payoff.

5. Connect your product database

If you have a Product Information Management (PIM) system, treat it as the source of truth.

Useful fields include:

• SKU
• manufacturer
• category
• description
• specifications
• attributes
• GTIN
• brand

Run automated rules such as:

Description < 100 words

OR

Description equals manufacturer description

OR

Description duplicated >5 times

OR

Missing benefits section

OR

Missing FAQs

Schedule these checks nightly or weekly.

6. Use AI for rewriting—but keep humans in the loop

For large catalogs, AI is the only practical way to generate unique descriptions quickly.

A good workflow is:

1. Detect weak pages
2. Generate improved copy from structured attributes
3. Preserve technical specs
4. Add benefits, use cases, and FAQs
5. Human review before publishing

Research and production systems have shown this approach can scale effectively while improving engagement, especially when paired with quality control. arxiv.org

7. Build dashboards instead of spreadsheets

Track metrics like:

• Duplicate descriptions
• Near-duplicate descriptions
• Average description length
• Pages under 100 words
• Pages with missing metadata
• Pages missing alt text
• Pages not indexed
• Average content quality score
• Products updated this week

This turns content maintenance into an ongoing operational process rather than a one-time cleanup.

A scalable tech stack

For most mid-size to enterprise ecommerce teams, a practical stack looks like:

• Crawling: Screaming Frog or Sitebulb
• Search performance: Google Search Console
• Storage: BigQuery, Snowflake, or PostgreSQL
• Similarity detection: Sentence Transformers/OpenAI embeddings + a vector database (e.g., FAISS or Milvus)
• Content generation: LLMs integrated into your CMS or PIM
• Reporting: Looker Studio, Power BI, or Tableau

This combination allows you to monitor hundreds of thousands of product pages, automatically surface the highest-impact duplicate and thin-content issues, and prioritize fixes based on SEO and business value rather than manual review.

For a large ecommerce catalog (10,000+ SKUs), the goal isn't to manually "find duplicate pages"—it's to build an automated content quality pipeline that continuously scores, prioritizes, and fixes problems.

A scalable workflow typically looks like this:

1. Use a crawler as your "content inventory"

Enterprise crawlers can scan hundreds of thousands (or even millions) of URLs and export:

• duplicate page titles
• duplicate meta descriptions
• duplicate H1s
• exact duplicate body text
• canonical issues
• word counts
• near-empty pages

These tools are the industry standard:

• Screaming Frog SEO Spider (excellent for exports and automation)
• Sitebulb (better visual reporting)
• Botify or Deepcrawl/Lumar (enterprise scale)

Rather than opening pages individually, export everything into Excel or a data warehouse and sort by issue count.

2. Detect near duplicates, not just exact copies

Exact duplicates are only half the problem.

Many catalogs contain pages like:

Blue Running Shoe

and

Red Running Shoe

where 98% of the description is identical.

Traditional SEO crawlers won't always flag these.

Instead use similarity scoring:

• cosine similarity on embeddings
• MinHash
• SimHash
• sentence embeddings
• vector search

This lets you identify pages that are, say:

• 95% identical
• 90% identical
• 80% identical

instead of only exact copies.

Modern embedding-based approaches are much better at finding semantically similar content at scale. arxiv.org

3. Score thin content automatically

Instead of asking:

Is this page thin?

build a quality score.

For example:

Description length
+ unique features
+ specifications
+ FAQs
+ reviews
+ images
+ alt text
+ internal links
= Product Quality Score

Example thresholds:

• <75 words → critical
• 75–150 → needs improvement
• 150–300 → acceptable
• 300+ → comprehensive

Many ecommerce audit platforms automate checks for thin descriptions, duplicate copy, missing metadata, and related content quality signals. blog.hubspot.com

4. Prioritize by business impact

Don't rewrite 50,000 products alphabetically.

Instead calculate:

Priority =
Organic traffic
× Revenue
× Conversion rate
× Content quality score

This usually identifies the top few hundred products where better content will have the biggest payoff.

5. Connect your product database

If you have a Product Information Management (PIM) system, treat it as the source of truth.

Useful fields include:

• SKU
• manufacturer
• category
• description
• specifications
• attributes
• GTIN
• brand

Run automated rules such as:

Description < 100 words

OR

Description equals manufacturer description

OR

Description duplicated >5 times

OR

Missing benefits section

OR

Missing FAQs

Schedule these checks nightly or weekly.

6. Use AI for rewriting—but keep humans in the loop

For large catalogs, AI is the only practical way to generate unique descriptions quickly.

A good workflow is:

1. Detect weak pages
2. Generate improved copy from structured attributes
3. Preserve technical specs
4. Add benefits, use cases, and FAQs
5. Human review before publishing

Research and production systems have shown this approach can scale effectively while improving engagement, especially when paired with quality control. arxiv.org

7. Build dashboards instead of spreadsheets

Track metrics like:

• Duplicate descriptions
• Near-duplicate descriptions
• Average description length
• Pages under 100 words
• Pages with missing metadata
• Pages missing alt text
• Pages not indexed
• Average content quality score
• Products updated this week

This turns content maintenance into an ongoing operational process rather than a one-time cleanup.

A scalable tech stack

For most mid-size to enterprise ecommerce teams, a practical stack looks like:

• Crawling: Screaming Frog or Sitebulb
• Search performance: Google Search Console
• Storage: BigQuery, Snowflake, or PostgreSQL
• Similarity detection: Sentence Transformers/OpenAI embeddings + a vector database (e.g., FAISS or Milvus)
• Content generation: LLMs integrated into your CMS or PIM
• Reporting: Looker Studio, Power BI, or Tableau

This combination allows you to monitor hundreds of thousands of product pages, automatically surface the highest-impact duplicate and thin-content issues, and prioritize fixes based on SEO and business value rather than manual review.

For a large ecommerce catalog (10,000+ SKUs), the goal isn't to manually "find duplicate pages"—it's to build an automated content quality pipeline that continuously scores, prioritizes, and fixes problems.

A scalable workflow typically looks like this:

1. Use a crawler as your "content inventory"

Enterprise crawlers can scan hundreds of thousands (or even millions) of URLs and export:

• duplicate page titles
• duplicate meta descriptions
• duplicate H1s
• exact duplicate body text
• canonical issues
• word counts
• near-empty pages

These tools are the industry standard:

• Screaming Frog SEO Spider (excellent for exports and automation)
• Sitebulb (better visual reporting)
• Botify or Deepcrawl/Lumar (enterprise scale)

Rather than opening pages individually, export everything into Excel or a data warehouse and sort by issue count.

2. Detect near duplicates, not just exact copies

Exact duplicates are only half the problem.

Many catalogs contain pages like:

Blue Running Shoe

and

Red Running Shoe

where 98% of the description is identical.

Traditional SEO crawlers won't always flag these.

Instead use similarity scoring:

• cosine similarity on embeddings
• MinHash
• SimHash
• sentence embeddings
• vector search

This lets you identify pages that are, say:

• 95% identical
• 90% identical
• 80% identical

instead of only exact copies.

Modern embedding-based approaches are much better at finding semantically similar content at scale. arxiv.org

3. Score thin content automatically

Instead of asking:

Is this page thin?

build a quality score.

For example:

Description length
+ unique features
+ specifications
+ FAQs
+ reviews
+ images
+ alt text
+ internal links
= Product Quality Score

Example thresholds:

• <75 words → critical
• 75–150 → needs improvement
• 150–300 → acceptable
• 300+ → comprehensive

Many ecommerce audit platforms automate checks for thin descriptions, duplicate copy, missing metadata, and related content quality signals. blog.hubspot.com

4. Prioritize by business impact

Don't rewrite 50,000 products alphabetically.

Instead calculate:

Priority =
Organic traffic
× Revenue
× Conversion rate
× Content quality score

This usually identifies the top few hundred products where better content will have the biggest payoff.

5. Connect your product database

If you have a Product Information Management (PIM) system, treat it as the source of truth.

Useful fields include:

• SKU
• manufacturer
• category
• description
• specifications
• attributes
• GTIN
• brand

Run automated rules such as:

Description < 100 words

OR

Description equals manufacturer description

OR

Description duplicated >5 times

OR

Missing benefits section

OR

Missing FAQs

Schedule these checks nightly or weekly.

6. Use AI for rewriting—but keep humans in the loop

For large catalogs, AI is the only practical way to generate unique descriptions quickly.

A good workflow is:

1. Detect weak pages
2. Generate improved copy from structured attributes
3. Preserve technical specs
4. Add benefits, use cases, and FAQs
5. Human review before publishing

Research and production systems have shown this approach can scale effectively while improving engagement, especially when paired with quality control. arxiv.org

7. Build dashboards instead of spreadsheets

Track metrics like:

• Duplicate descriptions
• Near-duplicate descriptions
• Average description length
• Pages under 100 words
• Pages with missing metadata
• Pages missing alt text
• Pages not indexed
• Average content quality score
• Products updated this week

This turns content maintenance into an ongoing operational process rather than a one-time cleanup.

A scalable tech stack

For most mid-size to enterprise ecommerce teams, a practical stack looks like:

• Crawling: Screaming Frog or Sitebulb
• Search performance: Google Search Console
• Storage: BigQuery, Snowflake, or PostgreSQL
• Similarity detection: Sentence Transformers/OpenAI embeddings + a vector database (e.g., FAISS or Milvus)
• Content generation: LLMs integrated into your CMS or PIM
• Reporting: Looker Studio, Power BI, or Tableau

This combination allows you to monitor hundreds of thousands of product pages, automatically surface the highest-impact duplicate and thin-content issues, and prioritize fixes based on SEO and business value rather than manual review.

For a large ecommerce catalog (10,000+ SKUs), the goal isn't to manually "find duplicate pages"—it's to build an automated content quality pipeline that continuously scores, prioritizes, and fixes problems.

A scalable workflow typically looks like this:

1. Use a crawler as your "content inventory"

Enterprise crawlers can scan hundreds of thousands (or even millions) of URLs and export:

• duplicate page titles
• duplicate meta descriptions
• duplicate H1s
• exact duplicate body text
• canonical issues
• word counts
• near-empty pages

These tools are the industry standard:

• Screaming Frog SEO Spider (excellent for exports and automation)
• Sitebulb (better visual reporting)
• Botify or Deepcrawl/Lumar (enterprise scale)

Rather than opening pages individually, export everything into Excel or a data warehouse and sort by issue count.

2. Detect near duplicates, not just exact copies

Exact duplicates are only half the problem.

Many catalogs contain pages like:

Blue Running Shoe

and

Red Running Shoe

where 98% of the description is identical.

Traditional SEO crawlers won't always flag these.

Instead use similarity scoring:

• cosine similarity on embeddings
• MinHash
• SimHash
• sentence embeddings
• vector search

This lets you identify pages that are, say:

• 95% identical
• 90% identical
• 80% identical

instead of only exact copies.

Modern embedding-based approaches are much better at finding semantically similar content at scale. arxiv.org

3. Score thin content automatically

Instead of asking:

Is this page thin?

build a quality score.

For example:

Description length
+ unique features
+ specifications
+ FAQs
+ reviews
+ images
+ alt text
+ internal links
= Product Quality Score

Example thresholds:

• <75 words → critical
• 75–150 → needs improvement
• 150–300 → acceptable
• 300+ → comprehensive

Many ecommerce audit platforms automate checks for thin descriptions, duplicate copy, missing metadata, and related content quality signals. blog.hubspot.com

4. Prioritize by business impact

Don't rewrite 50,000 products alphabetically.

Instead calculate:

Priority =
Organic traffic
× Revenue
× Conversion rate
× Content quality score

This usually identifies the top few hundred products where better content will have the biggest payoff.

5. Connect your product database

If you have a Product Information Management (PIM) system, treat it as the source of truth.

Useful fields include:

• SKU
• manufacturer
• category
• description
• specifications
• attributes
• GTIN
• brand

Run automated rules such as:

Description < 100 words

OR

Description equals manufacturer description

OR

Description duplicated >5 times

OR

Missing benefits section

OR

Missing FAQs

Schedule these checks nightly or weekly.

6. Use AI for rewriting—but keep humans in the loop

For large catalogs, AI is the only practical way to generate unique descriptions quickly.

A good workflow is:

1. Detect weak pages
2. Generate improved copy from structured attributes
3. Preserve technical specs
4. Add benefits, use cases, and FAQs
5. Human review before publishing

Research and production systems have shown this approach can scale effectively while improving engagement, especially when paired with quality control. arxiv.org

7. Build dashboards instead of spreadsheets

Track metrics like:

• Duplicate descriptions
• Near-duplicate descriptions
• Average description length
• Pages under 100 words
• Pages with missing metadata
• Pages missing alt text
• Pages not indexed
• Average content quality score
• Products updated this week

This turns content maintenance into an ongoing operational process rather than a one-time cleanup.

A scalable tech stack

For most mid-size to enterprise ecommerce teams, a practical stack looks like:

• Crawling: Screaming Frog or Sitebulb
• Search performance: Google Search Console
• Storage: BigQuery, Snowflake, or PostgreSQL
• Similarity detection: Sentence Transformers/OpenAI embeddings + a vector database (e.g., FAISS or Milvus)
• Content generation: LLMs integrated into your CMS or PIM
• Reporting: Looker Studio, Power BI, or Tableau

This combination allows you to monitor hundreds of thousands of product pages, automatically surface the highest-impact duplicate and thin-content issues, and prioritize fixes based on SEO and business value rather than manual review.

For a large ecommerce catalog (10,000+ SKUs), the goal isn't to manually "find duplicate pages"—it's to build an automated content quality pipeline that continuously scores, prioritizes, and fixes problems.

A scalable workflow typically looks like this:

1. Use a crawler as your "content inventory"

Enterprise crawlers can scan hundreds of thousands (or even millions) of URLs and export:

• duplicate page titles
• duplicate meta descriptions
• duplicate H1s
• exact duplicate body text
• canonical issues
• word counts
• near-empty pages

These tools are the industry standard:

• Screaming Frog SEO Spider (excellent for exports and automation)
• Sitebulb (better visual reporting)
• Botify or Deepcrawl/Lumar (enterprise scale)

Rather than opening pages individually, export everything into Excel or a data warehouse and sort by issue count.

2. Detect near duplicates, not just exact copies

Exact duplicates are only half the problem.

Many catalogs contain pages like:

Blue Running Shoe

and

Red Running Shoe

where 98% of the description is identical.

Traditional SEO crawlers won't always flag these.

Instead use similarity scoring:

• cosine similarity on embeddings
• MinHash
• SimHash
• sentence embeddings
• vector search

This lets you identify pages that are, say:

• 95% identical
• 90% identical
• 80% identical

instead of only exact copies.

Modern embedding-based approaches are much better at finding semantically similar content at scale. arxiv.org

3. Score thin content automatically

Instead of asking:

Is this page thin?

build a quality score.

For example:

Description length
+ unique features
+ specifications
+ FAQs
+ reviews
+ images
+ alt text
+ internal links
= Product Quality Score

Example thresholds:

• <75 words → critical
• 75–150 → needs improvement
• 150–300 → acceptable
• 300+ → comprehensive

Many ecommerce audit platforms automate checks for thin descriptions, duplicate copy, missing metadata, and related content quality signals. blog.hubspot.com

4. Prioritize by business impact

Don't rewrite 50,000 products alphabetically.

Instead calculate:

Priority =
Organic traffic
× Revenue
× Conversion rate
× Content quality score

This usually identifies the top few hundred products where better content will have the biggest payoff.

5. Connect your product database

If you have a Product Information Management (PIM) system, treat it as the source of truth.

Useful fields include:

• SKU
• manufacturer
• category
• description
• specifications
• attributes
• GTIN
• brand

Run automated rules such as:

Description < 100 words

OR

Description equals manufacturer description

OR

Description duplicated >5 times

OR

Missing benefits section

OR

Missing FAQs

Schedule these checks nightly or weekly.

6. Use AI for rewriting—but keep humans in the loop

For large catalogs, AI is the only practical way to generate unique descriptions quickly.

A good workflow is:

1. Detect weak pages
2. Generate improved copy from structured attributes
3. Preserve technical specs
4. Add benefits, use cases, and FAQs
5. Human review before publishing

Research and production systems have shown this approach can scale effectively while improving engagement, especially when paired with quality control. arxiv.org

7. Build dashboards instead of spreadsheets

Track metrics like:

• Duplicate descriptions
• Near-duplicate descriptions
• Average description length
• Pages under 100 words
• Pages with missing metadata
• Pages missing alt text
• Pages not indexed
• Average content quality score
• Products updated this week

This turns content maintenance into an ongoing operational process rather than a one-time cleanup.

A scalable tech stack

For most mid-size to enterprise ecommerce teams, a practical stack looks like:

• Crawling: Screaming Frog or Sitebulb
• Search performance: Google Search Console
• Storage: BigQuery, Snowflake, or PostgreSQL
• Similarity detection: Sentence Transformers/OpenAI embeddings + a vector database (e.g., FAISS or Milvus)
• Content generation: LLMs integrated into your CMS or PIM
• Reporting: Looker Studio, Power BI, or Tableau

This combination allows you to monitor hundreds of thousands of product pages, automatically surface the highest-impact duplicate and thin-content issues, and prioritize fixes based on SEO and business value rather than manual review.