ChatGPT answers, unedited

For very large websites (hundreds of thousands to millions of URLs), the “best” tool depends on whether you prioritize SEO auditing, raw crawling speed, API automation, or custom pipelines. The strongest options are:

Best overall choices

1. Best SEO crawler for most large sites: Screaming Frog

For finding:

• 404/410 errors
• 3xx redirects
• redirect chains
• redirect loops
• broken internal links
• external broken links
• bad canonicals
• hreflang issues

…it remains one of the most capable tools.

For very large sites, configure:

• Database storage mode instead of RAM mode
• Crawl limits and URL exclusions
• Custom user agents if needed
• Crawl from XML sitemaps when possible instead of discovering everything

2. Best for millions of URLs: JetOctopus / Botify

For sites like:

• ecommerce catalogs
• marketplaces
• news sites
• documentation platforms

cloud crawlers are usually better because they provide:

• distributed crawling
• faster throughput
• scheduled crawls
• historical comparisons
• API access

3. Best technical approach at extreme scale: build a crawler pipeline

For millions to tens of millions of pages, a common architecture is:

URL frontier
     |
     v
Distributed workers
     |
     +--> HTTP fetch
     |
     +--> Link extraction
     |
     +--> Redirect tracking
     |
     +--> Status storage
     |
     v
Database / analytics layer

Typical stack:

• Go/Rust workers for high concurrency
• Redis/Kafka for URL queues
• PostgreSQL/ClickHouse for results
• Headless browsers only for JavaScript-heavy pages
• Bloom filters for duplicate URL detection

A simple crawler should record:

source_url
target_url
status_code
redirect_count
redirect_path
canonical_url
crawl_timestamp
response_time

This makes redirect-chain analysis easy.

Tools specifically good at redirect-chain detection

• Screaming Frog — easiest and most SEO-friendly
• curl + custom scripts — good for targeted validation
• Sitebulb — excellent reporting
• JetOctopus/Botify — best for enterprise monitoring

For maximum efficiency, avoid these common mistakes

• Crawling the entire site from links alone when XML sitemaps exist
• Rendering JavaScript for every URL unless necessary
• Following infinite URL patterns (?sort=, filters, calendars)
• Ignoring URL normalization (/page vs /page/, uppercase, encoded URLs)
• Treating every external timeout as a broken link
• Not storing redirect paths (only final status)

My typical recommendation by site size

• <100k URLs: Screaming Frog
• 100k–5M URLs: Screaming Frog + database mode, or JetOctopus
• 5M–50M URLs: JetOctopus/Botify/Oncrawl
• 50M+ URLs: custom distributed crawler

For most SEO teams auditing a large production website, a combination of Screaming Frog for deep technical audits + a cloud crawler for ongoing monitoring is usually the strongest setup.

If you're talking about very large sites—hundreds of thousands to millions of URLs—I'd separate the tools into two categories: dedicated crawlers and cloud/enterprise crawlers.

My shortlist

1. Screaming Frog — best if you want maximum control

For broken links and redirect chains specifically, this is probably where I'd start.

It can identify:

• 4xx/5xx broken URLs and their source pages
• redirecting internal links
• multi-hop redirect chains
• redirect loops
• redirects that ultimately land on errors
• canonical/redirect chains
• the original source URL behind a chain

Its “All Redirects” report can map the entire chain, and its “Always follow redirects” option follows redirects until the final non-3xx destination. Screaming Frog Screaming Frog

For large crawls, it now has database storage rather than requiring the entire crawl to live in RAM. Screaming Frog says a suitable SSD/16 GB machine can handle roughly 10 million URLs, depending heavily on the site and configuration. Screaming Frog Screaming Frog

Particularly good workflow: crawl only HTML → export every 3xx/4xx/5xx → inspect source → destination → final destination.

2. JetOctopus — my pick when scale/speed matters most

If you're dealing with millions or tens of millions of URLs, I'd seriously consider a cloud crawler rather than running everything locally.

JetOctopus is particularly attractive because you can combine crawling with log-file data, which lets you distinguish:

“This URL is technically broken”

from

“This URL is actually being requested by Google/users and is therefore important.”

That's a much more useful prioritization for a huge site.

3. Lumar — best enterprise option

For large organizations where crawling is something you want to run continuously rather than as an occasional desktop audit, Lumar is compelling.

I'd look at it when you need:

• scheduled crawling
• automated technical SEO monitoring
• very large sites
• reporting across teams
• API/integration workflows
• enterprise-scale infrastructure

It's generally more of a platform than simply “a crawler.”

4. Oncrawl — best when logs are central to the problem

Oncrawl becomes particularly useful if your goal isn't merely:

Find every broken link.

but rather:

Find broken/redirected URLs that matter to search engines and understand how crawling is being wasted.

Combining crawl data with server logs can expose things like:

• Googlebot repeatedly hitting redirects
• old URLs consuming crawl activity
• orphaned URLs
• redirect-heavy sections
• crawl-budget problems

That's extremely valuable on large ecommerce, publishing, marketplace, and faceted-navigation sites.

What I'd actually use

For a 1M–10M URL website, my preference would be:

Screaming Frog + log analysis

if you need maximum technical control.

For 10M+ URLs or recurring crawls, I'd lean toward:

JetOctopus / Lumar / Oncrawl

because cloud infrastructure becomes increasingly attractive.

And for a normal 100K–1M URL site, I'd probably just use Screaming Frog. Its database mode, SSD storage, configurable crawl limits, and redirect-chain reports make it unusually capable for this use case. Screaming Frog Screaming Frog

One important optimization

Don't crawl a giant site indiscriminately.

For broken-link/redirect auditing, configure the crawler to focus on HTML and link relationships, rather than downloading every image, CSS file, JavaScript asset, etc. Screaming Frog specifically recommends disabling unnecessary resource crawling to reduce memory and processing requirements on large sites. Screaming Frog

If you tell me roughly how many URLs you're dealing with (1M, 10M, 100M+) and whether you need to crawl the whole site or just analyze a URL list, I can recommend the fastest architecture and settings.

Find broken/redirected URLs that matter to search engines and understand how crawling is being wasted.

Combining crawl data with server logs can expose things like:

• Googlebot repeatedly hitting redirects
• old URLs consuming crawl activity
• orphaned URLs
• redirect-heavy sections
• crawl-budget problems

That's extremely valuable on large ecommerce, publishing, marketplace, and faceted-navigation sites.

What I'd actually use

For a 1M–10M URL website, my preference would be:

Screaming Frog + log analysis

if you need maximum technical control.

For 10M+ URLs or recurring crawls, I'd lean toward:

JetOctopus / Lumar / Oncrawl

because cloud infrastructure becomes increasingly attractive.

And for a normal 100K–1M URL site, I'd probably just use Screaming Frog. Its database mode, SSD storage, configurable crawl limits, and redirect-chain reports make it unusually capable for this use case.

One important optimization

Don't crawl a giant site indiscriminately.

For crawling very large websites (millions of URLs) to find broken links, redirect chains, redirect loops, bad canonicalization, and crawl issues, the best tool depends on whether you need a desktop crawler, cloud-scale crawling, API access, or custom infrastructure.

Best overall enterprise crawlers

Screaming Frog SEO Spider

screamingfrog.co.uk

Best for: Deep technical SEO audits, large but manageable sites.

Strengths:

• Finds: - 404/410 errors
• 3xx redirect chains
• redirect loops
• broken internal links
• orphan pages (with integrations)
• bad canonicals
• hreflang issues
• Very mature rules engine
• Supports JavaScript rendering
• Can connect to databases and APIs
• Custom extraction via XPath/CSS selectors

Scale:

• Millions of URLs possible with enough RAM and configuration
• Database storage mode helps with very large crawls

Limitations:

• Primarily a desktop application
• Large crawls require tuning

Sitebulb

sitebulb.com

Best for: Teams that want strong visualization and diagnostics.

Strengths:

• Excellent issue prioritization
• Crawl maps
• Link analysis
• Redirect chain reporting
• Easier interpretation than raw crawl exports

Limitations:

• Less suited than custom/cloud crawlers for tens of millions of URLs

Best for very large-scale crawling

Botify

botify.com

Best for: Enterprise sites with millions to billions of pages.

Strengths:

• Cloud-based crawling
• Designed for huge websites
• Log file analysis
• Crawl budget analysis
• Large-scale internal linking analysis
• Historical comparisons

Common users:

• Large publishers
• Marketplaces
• Travel sites
• Ecommerce platforms

Deepcrawl (now part of Lumar)

lumar.io

Best for: Enterprise technical SEO monitoring.

Strengths:

• Handles very large crawls
• Scheduled monitoring
• API access
• Detailed issue detection
• Good for regression detection after releases

Best developer-friendly crawlers

Apache Nutch

Best for: Building your own crawler infrastructure.

Strengths:

• Open source
• Distributed crawling
• Hadoop integration
• Can crawl very large URL sets

Weaknesses:

• Requires engineering effort
• You build much of the analysis pipeline yourself

Scrapy

scrapy.org

Best for: Custom link auditing pipelines.

Strengths:

• Very fast asynchronous crawling
• Easy to extend
• Good for: - custom broken-link detection
• API-driven audits
• continuous monitoring

Typical stack:

• Scrapy → crawl
• Redis/Kafka → queue
• PostgreSQL/ClickHouse → storage
• Grafana → monitoring

Best command-line tools

wget

Good for:

• Quick link checks
• Small sites

Example:

wget --spider -r -l 10 -o crawl.log https://example.com

Not recommended for:

• Millions of URLs
• Complex redirect analysis

curl

Useful for spot checks:

curl -I https://example.com/page

Good for debugging individual redirect chains, not crawling.

Best tools specifically for redirect-chain analysis

For extremely large sites (10M+ URLs), I would typically use:

Option 1: Enterprise SaaS

• Botify or Lumar
• Connect: - XML sitemaps
• server logs
• Google Search Console
• analytics

Option 2: Build your own pipeline

A common architecture:

URL source
   |
   v
Queue (Kafka / Redis)
   |
   v
Async crawler (Scrapy / custom Go crawler)
   |
   v
HTTP analyzer
   |
   +--> status code
   +--> redirect chain
   +--> canonical target
   +--> response time
   |
   v
Storage (ClickHouse / BigQuery)
   |
   v
Dashboards + alerts

For very large sites, avoid crawling purely from discovered links. Seed from:

• XML sitemaps
• internal link graph
• database exports
• URL inventories
• server logs

My shortlist by scenario

• Up to ~5 million URLs: Screaming Frog with database storage
• 5–100M URLs: Botify or Lumar
• Need custom rules/API: Scrapy or a custom Go/Rust crawler
• Need SEO team usability: Screaming Frog or Sitebulb
• Need continuous monitoring after deployments: Lumar/Botify
• Need raw crawl data for analysis: custom crawler + ClickHouse/BigQuery

For a very large website where broken links and redirect chains are the primary goal, I would prioritize a crawler that stores every HTTP hop, not just the final response. Many basic crawlers report “301 → 200” but fail to expose chains like:

/old-page
  301 → /temporary-page
  302 → /new-page
  301 → /final-page
  200

Those hidden chains are where large sites usually accumulate performance and SEO problems.

• XML sitemaps
• internal link graph
• database exports
• URL inventories
• server logs

My shortlist by scenario

• Up to ~5 million URLs: Screaming Frog with database storage
• 5–100M URLs: Botify or Lumar
• Need custom rules/API: Scrapy or a custom Go/Rust crawler
• Need SEO team usability: Screaming Frog or Sitebulb
• Need continuous monitoring after deployments: Lumar/Botify
• Need raw crawl data for analysis: custom crawler + ClickHouse/BigQuery

For a very large website where broken links and redirect chains are the primary goal, I would prioritize a crawler that stores every HTTP hop, not just the final response. Many basic crawlers report “301 → 200” but fail to expose chains like:

/old-page
  301 → /temporary-page
  302 → /new-page
  301 → /final-page
  200

If you're crawling very large websites (100k–100M+ URLs), the "best" tool depends on whether your priority is speed, SEO diagnostics, custom analysis, or enterprise-scale crawling.

Here's how I'd rank the major options.

1. Screaming Frog (best overall)

Pros

• Finds all internal broken links
• Finds external broken links
• Detects redirect chains
• Detects redirect loops
• Reports orphan URLs (with log files/sitemaps)
• Highly configurable
• JavaScript rendering
• Can crawl millions of URLs if enough RAM/disk is available

Typical workflow:

Crawl
↓
Response Codes
↓
Filter 4xx

Crawl
↓
Reports
↓
Redirect Chains

For sites under ~500k URLs, it's hard to beat.

2. JetOctopus (best cloud crawler)

If your site is

• ecommerce
• marketplace
• publisher
• millions of pages

JetOctopus is excellent.

Advantages:

• doesn't require local resources
• scheduled crawls
• redirect chain visualization
• crawl budget analysis
• integrates server logs
• very fast cloud infrastructure

3. Lumar (formerly Deepcrawl)

Enterprise standard.

Very strong at:

• redirect chains
• canonical issues
• broken links
• crawl path visualization
• massive websites

Usually chosen by Fortune 500 SEO teams.

4. Scrapy (best if you're technical)

If you're comfortable with Python:

Scrapy
↓
Request URL
↓
status
↓
Location header
↓
continue following redirects
↓
record redirect depth
↓
report chain

Advantages:

• unlimited customization
• can run distributed
• parallel requests
• API integrations
• export to databases
• incremental crawling

For engineering teams this is often the best long-term solution.

5. StormCrawler

Designed specifically for very large distributed crawls.

Can crawl

• millions
• tens of millions
• hundreds of millions

URLs using

• Apache Storm
• Elasticsearch
• Kafka

It's much more infrastructure-heavy than SEO tools but scales exceptionally well.

6. Sitebulb

Excellent if you want explanations rather than raw data.

It automatically surfaces:

• redirect chains
• redirect loops
• broken links
• internal 404s
• external 404s

Very user-friendly.

Fastest open-source stack

For engineers, I'd recommend:

Scrapy
+
aiohttp
+
Redis
+
PostgreSQL

or

StormCrawler
+
Kafka
+
Elasticsearch

These can comfortably crawl millions of URLs per day on suitable infrastructure.

Detecting redirect chains efficiently

The key is to avoid repeatedly requesting the same redirects.

A good crawler should cache:

A
→ B
→ C
→ D (200)

Then whenever another page links to A, it already knows the full chain without re-fetching each hop.

Store something like:

URL
Final URL
Hop count
Status
Chain

This dramatically reduces duplicate network requests.

Detecting broken links efficiently

Rather than only reporting 404s, capture:

• 404 Not Found
• 410 Gone
• 451 Unavailable
• DNS failures
• TLS failures
• timeout
• connection refused
• redirect loops
• redirect chains longer than 3–5 hops
• redirects ending in 404s

The most actionable report links each broken destination back to every referring page and anchor text, making fixes straightforward.

If I were building an enterprise crawler

I'd use:

• Scrapy for crawling and URL discovery
• aiohttp (or another async HTTP client) for high-concurrency requests
• Redis for the URL frontier and deduplication
• PostgreSQL (or ClickHouse for very large datasets) to store crawl results
• Elasticsearch/OpenSearch for fast querying and dashboards

This combination scales well into the millions of URLs while giving you complete control over redirect handling, broken-link detection, retry logic, and reporting.

For most SEO professionals, Screaming Frog remains the best desktop choice, while JetOctopus or Lumar are stronger options once you're regularly auditing multi-million-page sites. For engineering teams that need maximum flexibility and scale, a custom crawler built on Scrapy is usually the strongest long-term solution.

If you're crawling very large websites (100k–100M+ URLs), the "best" tool depends on whether your priority is speed, SEO diagnostics, custom analysis, or enterprise-scale crawling.

Here's how I'd rank the major options.

1. Screaming Frog (best overall)

Pros

• Finds all internal broken links
• Finds external broken links
• Detects redirect chains
• Detects redirect loops
• Reports orphan URLs (with log files/sitemaps)
• Highly configurable
• JavaScript rendering
• Can crawl millions of URLs if enough RAM/disk is available

Typical workflow:

Crawl
↓
Response Codes
↓
Filter 4xx

Crawl
↓
Reports
↓
Redirect Chains

For sites under ~500k URLs, it's hard to beat.

2. JetOctopus (best cloud crawler)

If your site is

• ecommerce
• marketplace
• publisher
• millions of pages

JetOctopus is excellent.

Advantages:

• doesn't require local resources
• scheduled crawls
• redirect chain visualization
• crawl budget analysis
• integrates server logs
• very fast cloud infrastructure

3. Lumar (formerly Deepcrawl)

Enterprise standard.

Very strong at:

• redirect chains
• canonical issues
• broken links
• crawl path visualization
• massive websites

Usually chosen by Fortune 500 SEO teams.

4. Scrapy (best if you're technical)

If you're comfortable with Python:

Scrapy
↓
Request URL
↓
status
↓
Location header
↓
continue following redirects
↓
record redirect depth
↓
report chain

Advantages:

• unlimited customization
• can run distributed
• parallel requests
• API integrations
• export to databases
• incremental crawling

For engineering teams this is often the best long-term solution.

5. StormCrawler

Designed specifically for very large distributed crawls.

Can crawl

• millions
• tens of millions
• hundreds of millions

URLs using

• Apache Storm
• Elasticsearch
• Kafka

It's much more infrastructure-heavy than SEO tools but scales exceptionally well.

6. Sitebulb

Excellent if you want explanations rather than raw data.

It automatically surfaces:

• redirect chains
• redirect loops
• broken links
• internal 404s
• external 404s

Very user-friendly.

Fastest open-source stack

For engineers, I'd recommend:

Scrapy
+
aiohttp
+
Redis
+
PostgreSQL

or

StormCrawler
+
Kafka
+
Elasticsearch

These can comfortably crawl millions of URLs per day on suitable infrastructure.

Detecting redirect chains efficiently

The key is to avoid repeatedly requesting the same redirects.

A good crawler should cache:

A
→ B
→ C
→ D (200)

Then whenever another page links to A, it already knows the full chain without re-fetching each hop.

Store something like:

URL
Final URL
Hop count
Status
Chain

This dramatically reduces duplicate network requests.

Detecting broken links efficiently

Rather than only reporting 404s, capture:

• 404 Not Found
• 410 Gone
• 451 Unavailable
• DNS failures
• TLS failures
• timeout
• connection refused
• redirect loops
• redirect chains longer than 3–5 hops
• redirects ending in 404s

The most actionable report links each broken destination back to every referring page and anchor text, making fixes straightforward.

If I were building an enterprise crawler

I'd use:

• Scrapy for crawling and URL discovery
• aiohttp (or another async HTTP client) for high-concurrency requests
• Redis for the URL frontier and deduplication
• PostgreSQL (or ClickHouse for very large datasets) to store crawl results
• Elasticsearch/OpenSearch for fast querying and dashboards

This combination scales well into the millions of URLs while giving you complete control over redirect handling, broken-link detection, retry logic, and reporting.

For most SEO professionals, Screaming Frog remains the best desktop choice, while JetOctopus or Lumar are stronger options once you're regularly auditing multi-million-page sites. For engineering teams that need maximum flexibility and scale, a custom crawler built on Scrapy is usually the strongest long-term solution.

If you're crawling very large websites (100k–100M+ URLs), the "best" tool depends on whether your priority is speed, SEO diagnostics, custom analysis, or enterprise-scale crawling.

Here's how I'd rank the major options.

1. Screaming Frog (best overall)

Pros

• Finds all internal broken links
• Finds external broken links
• Detects redirect chains
• Detects redirect loops
• Reports orphan URLs (with log files/sitemaps)
• Highly configurable
• JavaScript rendering
• Can crawl millions of URLs if enough RAM/disk is available

Typical workflow:

Crawl
↓
Response Codes
↓
Filter 4xx

Crawl
↓
Reports
↓
Redirect Chains

For sites under ~500k URLs, it's hard to beat.

2. JetOctopus (best cloud crawler)

If your site is

• ecommerce
• marketplace
• publisher
• millions of pages

JetOctopus is excellent.

Advantages:

• doesn't require local resources
• scheduled crawls
• redirect chain visualization
• crawl budget analysis
• integrates server logs
• very fast cloud infrastructure

3. Lumar (formerly Deepcrawl)

Enterprise standard.

Very strong at:

• redirect chains
• canonical issues
• broken links
• crawl path visualization
• massive websites

Usually chosen by Fortune 500 SEO teams.

4. Scrapy (best if you're technical)

If you're comfortable with Python:

Scrapy
↓
Request URL
↓
status
↓
Location header
↓
continue following redirects
↓
record redirect depth
↓
report chain

Advantages:

• unlimited customization
• can run distributed
• parallel requests
• API integrations
• export to databases
• incremental crawling

For engineering teams this is often the best long-term solution.

5. StormCrawler

Designed specifically for very large distributed crawls.

Can crawl

• millions
• tens of millions
• hundreds of millions

URLs using

• Apache Storm
• Elasticsearch
• Kafka

It's much more infrastructure-heavy than SEO tools but scales exceptionally well.

6. Sitebulb

Excellent if you want explanations rather than raw data.

It automatically surfaces:

• redirect chains
• redirect loops
• broken links
• internal 404s
• external 404s

Very user-friendly.

Fastest open-source stack

For engineers, I'd recommend:

Scrapy
+
aiohttp
+
Redis
+
PostgreSQL

or

StormCrawler
+
Kafka
+
Elasticsearch

These can comfortably crawl millions of URLs per day on suitable infrastructure.

Detecting redirect chains efficiently

The key is to avoid repeatedly requesting the same redirects.

A good crawler should cache:

A
→ B
→ C
→ D (200)

Then whenever another page links to A, it already knows the full chain without re-fetching each hop.

Store something like:

URL
Final URL
Hop count
Status
Chain

This dramatically reduces duplicate network requests.

Detecting broken links efficiently

Rather than only reporting 404s, capture:

• 404 Not Found
• 410 Gone
• 451 Unavailable
• DNS failures
• TLS failures
• timeout
• connection refused
• redirect loops
• redirect chains longer than 3–5 hops
• redirects ending in 404s

The most actionable report links each broken destination back to every referring page and anchor text, making fixes straightforward.

If I were building an enterprise crawler

I'd use:

• Scrapy for crawling and URL discovery
• aiohttp (or another async HTTP client) for high-concurrency requests
• Redis for the URL frontier and deduplication
• PostgreSQL (or ClickHouse for very large datasets) to store crawl results
• Elasticsearch/OpenSearch for fast querying and dashboards

This combination scales well into the millions of URLs while giving you complete control over redirect handling, broken-link detection, retry logic, and reporting.

For most SEO professionals, Screaming Frog remains the best desktop choice, while JetOctopus or Lumar are stronger options once you're regularly auditing multi-million-page sites. For engineering teams that need maximum flexibility and scale, a custom crawler built on Scrapy is usually the strongest long-term solution.

If you're crawling very large websites (100k–100M+ URLs), the "best" tool depends on whether your priority is speed, SEO diagnostics, custom analysis, or enterprise-scale crawling.

Here's how I'd rank the major options.

1. Screaming Frog (best overall)

Pros

• Finds all internal broken links
• Finds external broken links
• Detects redirect chains
• Detects redirect loops
• Reports orphan URLs (with log files/sitemaps)
• Highly configurable
• JavaScript rendering
• Can crawl millions of URLs if enough RAM/disk is available

Typical workflow:

Crawl
↓
Response Codes
↓
Filter 4xx

Crawl
↓
Reports
↓
Redirect Chains

For sites under ~500k URLs, it's hard to beat.

2. JetOctopus (best cloud crawler)

If your site is

• ecommerce
• marketplace
• publisher
• millions of pages

JetOctopus is excellent.

Advantages:

• doesn't require local resources
• scheduled crawls
• redirect chain visualization
• crawl budget analysis
• integrates server logs
• very fast cloud infrastructure

3. Lumar (formerly Deepcrawl)

Enterprise standard.

Very strong at:

• redirect chains
• canonical issues
• broken links
• crawl path visualization
• massive websites

Usually chosen by Fortune 500 SEO teams.

4. Scrapy (best if you're technical)

If you're comfortable with Python:

Scrapy
↓
Request URL
↓
status
↓
Location header
↓
continue following redirects
↓
record redirect depth
↓
report chain

Advantages:

• unlimited customization
• can run distributed
• parallel requests
• API integrations
• export to databases
• incremental crawling

For engineering teams this is often the best long-term solution.

5. StormCrawler

Designed specifically for very large distributed crawls.

Can crawl

• millions
• tens of millions
• hundreds of millions

URLs using

• Apache Storm
• Elasticsearch
• Kafka

It's much more infrastructure-heavy than SEO tools but scales exceptionally well.

6. Sitebulb

Excellent if you want explanations rather than raw data.

It automatically surfaces:

• redirect chains
• redirect loops
• broken links
• internal 404s
• external 404s

Very user-friendly.

Fastest open-source stack

For engineers, I'd recommend:

Scrapy
+
aiohttp
+
Redis
+
PostgreSQL

or

StormCrawler
+
Kafka
+
Elasticsearch

These can comfortably crawl millions of URLs per day on suitable infrastructure.

Detecting redirect chains efficiently

The key is to avoid repeatedly requesting the same redirects.

A good crawler should cache:

A
→ B
→ C
→ D (200)

Then whenever another page links to A, it already knows the full chain without re-fetching each hop.

Store something like:

URL
Final URL
Hop count
Status
Chain

This dramatically reduces duplicate network requests.

Detecting broken links efficiently

Rather than only reporting 404s, capture:

• 404 Not Found
• 410 Gone
• 451 Unavailable
• DNS failures
• TLS failures
• timeout
• connection refused
• redirect loops
• redirect chains longer than 3–5 hops
• redirects ending in 404s

The most actionable report links each broken destination back to every referring page and anchor text, making fixes straightforward.

If I were building an enterprise crawler

I'd use:

• Scrapy for crawling and URL discovery
• aiohttp (or another async HTTP client) for high-concurrency requests
• Redis for the URL frontier and deduplication
• PostgreSQL (or ClickHouse for very large datasets) to store crawl results
• Elasticsearch/OpenSearch for fast querying and dashboards

This combination scales well into the millions of URLs while giving you complete control over redirect handling, broken-link detection, retry logic, and reporting.

For most SEO professionals, Screaming Frog remains the best desktop choice, while JetOctopus or Lumar are stronger options once you're regularly auditing multi-million-page sites. For engineering teams that need maximum flexibility and scale, a custom crawler built on Scrapy is usually the strongest long-term solution.

If you're crawling very large websites (100k–100M+ URLs), the "best" tool depends on whether your priority is speed, SEO diagnostics, custom analysis, or enterprise-scale crawling.

Here's how I'd rank the major options.

1. Screaming Frog (best overall)

Pros

• Finds all internal broken links
• Finds external broken links
• Detects redirect chains
• Detects redirect loops
• Reports orphan URLs (with log files/sitemaps)
• Highly configurable
• JavaScript rendering
• Can crawl millions of URLs if enough RAM/disk is available

Typical workflow:

Crawl
↓
Response Codes
↓
Filter 4xx

Crawl
↓
Reports
↓
Redirect Chains

For sites under ~500k URLs, it's hard to beat.

2. JetOctopus (best cloud crawler)

If your site is

• ecommerce
• marketplace
• publisher
• millions of pages

JetOctopus is excellent.

Advantages:

• doesn't require local resources
• scheduled crawls
• redirect chain visualization
• crawl budget analysis
• integrates server logs
• very fast cloud infrastructure

3. Lumar (formerly Deepcrawl)

Enterprise standard.

Very strong at:

• redirect chains
• canonical issues
• broken links
• crawl path visualization
• massive websites

Usually chosen by Fortune 500 SEO teams.

4. Scrapy (best if you're technical)

If you're comfortable with Python:

Scrapy
↓
Request URL
↓
status
↓
Location header
↓
continue following redirects
↓
record redirect depth
↓
report chain

Advantages:

• unlimited customization
• can run distributed
• parallel requests
• API integrations
• export to databases
• incremental crawling

For engineering teams this is often the best long-term solution.

5. StormCrawler

Designed specifically for very large distributed crawls.

Can crawl

• millions
• tens of millions
• hundreds of millions

URLs using

• Apache Storm
• Elasticsearch
• Kafka

It's much more infrastructure-heavy than SEO tools but scales exceptionally well.

6. Sitebulb

Excellent if you want explanations rather than raw data.

It automatically surfaces:

• redirect chains
• redirect loops
• broken links
• internal 404s
• external 404s

Very user-friendly.

Fastest open-source stack

For engineers, I'd recommend:

Scrapy
+
aiohttp
+
Redis
+
PostgreSQL

or

StormCrawler
+
Kafka
+
Elasticsearch

These can comfortably crawl millions of URLs per day on suitable infrastructure.

Detecting redirect chains efficiently

The key is to avoid repeatedly requesting the same redirects.

A good crawler should cache:

A
→ B
→ C
→ D (200)

Then whenever another page links to A, it already knows the full chain without re-fetching each hop.

Store something like:

URL
Final URL
Hop count
Status
Chain

This dramatically reduces duplicate network requests.

Detecting broken links efficiently

Rather than only reporting 404s, capture:

• 404 Not Found
• 410 Gone
• 451 Unavailable
• DNS failures
• TLS failures
• timeout
• connection refused
• redirect loops
• redirect chains longer than 3–5 hops
• redirects ending in 404s

The most actionable report links each broken destination back to every referring page and anchor text, making fixes straightforward.

If I were building an enterprise crawler

I'd use:

• Scrapy for crawling and URL discovery
• aiohttp (or another async HTTP client) for high-concurrency requests
• Redis for the URL frontier and deduplication
• PostgreSQL (or ClickHouse for very large datasets) to store crawl results
• Elasticsearch/OpenSearch for fast querying and dashboards

This combination scales well into the millions of URLs while giving you complete control over redirect handling, broken-link detection, retry logic, and reporting.

For most SEO professionals, Screaming Frog remains the best desktop choice, while JetOctopus or Lumar are stronger options once you're regularly auditing multi-million-page sites. For engineering teams that need maximum flexibility and scale, a custom crawler built on Scrapy is usually the strongest long-term solution.

If the goal is very large sites—hundreds of thousands to millions of URLs—and specifically finding 404s, broken links, redirect chains/loops, and links pointing to redirects, I’d shortlist these:

1. Lumar — my pick for enormous sites

Lumar is particularly compelling when crawl speed itself is the bottleneck. Its current crawler advertises up to 450 URLs/sec non-rendered and 350 URLs/sec rendered, and is explicitly designed for sites with millions of URLs. It also reports HTTP status codes, redirect chains, broken resources, link structure, etc. www.lumar.iosupport.sitebulb.com

For a 5–20M URL site, this is the kind of architecture I'd look at first.

Best when: crawl completion time and enterprise-scale infrastructure matter more than having a desktop application.

2. Oncrawl — best for crawl + logs

Oncrawl is particularly strong if you don't just want to find broken URLs but want to answer "which broken/redirected URLs actually matter?"

It can crawl millions of URLs, render JavaScript, and combine crawl data with log-file and SEO data. Its crawler specifically handles redirect loops/chains and can follow up to 30 redirects for analysis. www.oncrawl.com

That's valuable on huge sites because you can prioritize things like:

• redirect chains receiving lots of Googlebot hits
• 404s receiving significant traffic
• sections generating excessive crawl waste
• redirects that are disproportionately expensive to crawl

Best when: you're doing serious enterprise technical SEO rather than simply running a broken-link report.

3. Sitebulb Cloud — best combination of usability and scale

Sitebulb has become a strong option for large sites. Its Cloud crawler is designed for millions of URLs, including JavaScript rendering at scale. sitebulb.com

For your specific problem, it has particularly nice reporting: it identifies broken internal links, internal links to redirects, and redirect chains, and you can export redirect-chain data for further analysis. support.sitebulb.com

I'd choose it over a more enterprise-oriented platform if the people actually operating the crawler need to explore and understand the results easily.

4. Screaming Frog SEO Spider — best power-user option

Don't underestimate Screaming Frog for large sites.

Its database storage mode is designed specifically for large crawls. The documented default is 5M URLs, but it can go beyond that with appropriate hardware/configuration; Screaming Frog gives an example of ~10M URLs with a 500GB SSD and 16GB RAM. www.screamingfrog.co.uk

For serious crawling I'd use:

• database storage
• SSD/NVMe
• plenty of RAM
• carefully configured URL/parameter exclusions
• crawl only the resource types you actually need
• separate crawls for different subdomains/sections where appropriate

Screaming Frog is particularly good when you want precise control over exactly what gets crawled rather than handing everything to a cloud crawler.

Best when: you have an SEO engineer/technical person who wants maximum control.

5. JetOctopus — very good for massive sites

JetOctopus is worth serious consideration for database-driven sites with enormous URL counts. Its positioning is specifically around very large crawling, with no simultaneous-crawl or project limits advertised, and it combines crawling with log analysis. jetoctopus.com

I'd put it particularly high on the list for large ecommerce/faceted sites, where URL explosions and crawl-budget issues are common.

6. Ahrefs Site Audit — excellent if you already use Ahrefs

Ahrefs catches both broken pages and redirect chains, and its Site Audit can check external links as well. help.ahrefs.com

Its verified-site crawler can be configured up to 30,000 URLs/minute, although actual speed should be constrained by what your server can handle. help.ahrefs.com

The catch for truly gigantic sites is the crawl-credit model: depending on the subscription, Site Audit analyzes 100K–5M pages/month. help.ahrefs.com

So I wouldn't make Ahrefs my primary crawler for a 20M+ URL site, but it's excellent as a complementary SEO data source.

What I'd actually use

For a 1M+ URL website, my choices would be:

If you want one tool:
→ Lumar or Sitebulb Cloud

If you're an enterprise SEO team:
→ Oncrawl + Lumar

If you're technical and want maximum control:
→ Screaming Frog

If it's a gigantic ecommerce/database site:
→ JetOctopus or Lumar

If you're already paying for Ahrefs:
→ Use Ahrefs Site Audit alongside your primary crawler rather than necessarily replacing it.

One important distinction

For broken-link/redirect work, don't just look for URLs returning 404. The most useful dataset is really:

source URL → linked URL → HTTP status → redirect hop 1 → hop 2 → ... → final URL/status

That lets you distinguish:

• direct 404 — A → 404
• link to redirect — A → B (301) → C (200)
• redirect chain — A → B (301) → C (301) → D (200)
• broken redirect — A → B (301) → C (404)
• redirect loop — A → B → A
• redirect to another domain
• internal links that should be updated to the final destination

For your particular use case, Sitebulb and Lumar are especially attractive because they expose this link/redirect relationship rather than merely giving you a list of bad destination URLs. www.lumar.iosupport.sitebulb.com

If you're talking about something truly huge (10M–100M+ URLs), I'd narrow the field considerably and look at Lumar vs. Oncrawl vs. a custom crawler architecture, because at that scale crawl scheduling, deduplication, URL-parameter control, concurrency, storage, and incremental crawling become more important than the UI.

For very large sites, I’d narrow the field to a few serious crawlers. The best choice depends on whether you want a powerful desktop crawler, cloud-scale crawling, or an API/automation-heavy setup.

My shortlist

1. Screaming Frog — my default recommendation

Screaming Frog is particularly strong if your primary task is:

crawl everything → identify 4xx/5xx → trace redirects → find pages linking to the bad/redirected URLs → export the remediation list.

Its Redirect Chains report maps the entire chain, counts hops, identifies the source, and flags redirect loops. It also has a Redirects to Errors report, which is particularly useful for finding cases where a URL redirects to a 404/5xx. www.screamingfrog.co.uk

For very large crawls, its database storage mode is important. Screaming Frog says its default limit is 5 million URLs, but it can go beyond that with appropriate hardware; its documentation gives roughly 2 million URLs with 4 GB allocated RAM, 5 million with ~10 GB, and 10 million with ~16 GB as rough examples. www.screamingfrog.co.uk

For your use case, I'd choose this if you have the infrastructure to run it.

2. Lumar — if "very large" means genuinely enormous

Lumar is more compelling once you're talking about enterprise sites with millions of URLs, especially if you don't want to manage a powerful crawling workstation/VM yourself.

Lumar claims crawling speeds up to 450 pages/sec and specifically supports millions of URLs, with redirect chains, HTTP status codes, broken resources and link structure among its technical SEO checks. www.lumar.io

The big advantage is that the crawling infrastructure is cloud-based, so you can throw substantial workloads at it without turning your local machine into a database server.

I'd pick Lumar over Screaming Frog for a 10M–100M+ enterprise property where crawl throughput and centralized workflows matter more than having the most granular desktop controls.

3. Sitebulb — excellent alternative

Sitebulb has both desktop and cloud versions and explicitly supports sites ranging from thousands to 5 million URLs. sitebulb.com

Its biggest strength is arguably analysis and presentation rather than raw crawling. If you're auditing a giant site and need to quickly understand why thousands of broken links or redirects exist, its visualization and issue prioritization are excellent.

I'd put it behind Screaming Frog for a pure "find every bad URL as efficiently as possible" workflow, but ahead of it if communicating findings to stakeholders is important.

4. Oncrawl — best for crawl-data analysis

Oncrawl is worth considering when crawling isn't an isolated task but part of a broader enterprise SEO data pipeline.

Its redirect tooling identifies chains and loops and can follow chains to a substantial depth; it also supports using crawl data to analyze migrations and structural problems. help.oncrawl.com

I'd favor it when you're combining crawl data with things like log files, organic traffic, indexation and site architecture rather than simply wanting a list of broken links.

The important part: crawl architecture

For a really large website, the crawler itself isn't the whole solution. I'd structure the process roughly like this:

1. Discover URLs from multiple sources

• XML sitemaps
• Internal links
• Canonicals
• hreflang
• structured data
• redirects
• server logs
• existing URL inventories

2. Do a lightweight HTTP crawl first

Check:

URL → status → Location → final status

This lets you identify enormous numbers of redirect/broken URL problems without rendering pages unnecessarily.

3. Separately crawl HTML for link relationships

You want to know not merely:

/old-page = 301

but:

/article-123 → /old-page → /new-page

That makes the remediation actionable.

4. Classify redirect problems

I'd distinguish:

• 301 → 200 — generally okay, but potentially unnecessary
• 302 → 200 — investigate
• 301 → 301 → 200 — redirect chain
• 301 → 301 → 301 → 200 — high-priority chain
• 301 → 404 — broken redirect
• 301 → 5xx — server failure
• URL → itself — redirect loop
• URL → A → B → A — loop
• internal link → redirect — update the source link

The last category is particularly valuable: don't just report redirects; identify the internal links causing them.

What I'd use

If you told me the site was:

• <1M URLs: Screaming Frog
• 1–5M: Screaming Frog in database mode, or Sitebulb Cloud
• 5–20M: Lumar or a carefully configured Screaming Frog deployment
• 20M+ / enterprise: Lumar/Oncrawl, potentially supplemented by a custom HTTP crawler
• Continuous monitoring: Lumar/Oncrawl rather than repeatedly running a desktop crawl

For Screaming Frog specifically, its database storage mode is designed for this kind of workload, and the vendor recommends an SSD; its stated recommended hardware for million-URL crawling is a 64-bit machine with an SSD and around 16 GB RAM. www.screamingfrog.co.uk

If your main objective is specifically "find broken links and redirect chains on a 10M+ URL site as fast as possible," I'd start with Lumar for cloud crawling and Screaming Frog as the technical-audit benchmark.

If the site is very large—hundreds of thousands to millions of URLs—the best choice depends on whether you prioritize raw crawl power, cloud scalability, or analysis.

1. Screaming Frog — best all-around

Screaming Frog SEO Spider

For most technical SEO teams, this would be my first choice. It has extremely granular crawl data, flexible filtering/export, custom extraction, JavaScript rendering, and excellent handling of redirects and broken URLs.

For large crawls, Screaming Frog recommends database storage on an SSD. Their documentation says roughly 4 GB allocated RAM can handle up to ~2M URLs, 8 GB around ~5M, and 16 GB around ~10M depending on the site and configuration. www.screamingfrog.co.uk

Particularly good for:

• 404/410 discovery
• Finding every internal link pointing to a broken URL
• 301/302 chains
• Redirect loops
• Redirect → 404 situations
• Canonical/redirect inconsistencies
• Exporting raw crawl data for your own analysis

If you have a 1–10M URL site and can run a well-configured machine/VM, I'd seriously consider this before moving to a cloud platform.

2. JetOctopus — best for truly enormous sites

JetOctopus

This is the one I'd investigate first if "very large" means 10M, 50M, or 100M+ URLs.

JetOctopus says it is designed for large websites and supports 100M+ URLs, with crawler speeds advertised up to 250 pages/sec. It is cloud-based, so you don't have to dedicate a monster workstation to the crawl. jetoctopus.com

It's especially interesting if you want to combine:

crawl data + Google Search Console + server logs

rather than just asking "what links are broken?"

For example, you can distinguish:

50,000 broken URLs

from:

50,000 broken URLs, of which 3,200 are still being requested by Googlebot and 740 receive significant organic traffic.

That's a much more useful prioritization.

3. Sitebulb — best analysis experience

Sitebulb

Sitebulb is particularly good when you don't just want a giant CSV of problems—you want the crawler to help you understand and prioritize them.

Its link auditing specifically identifies broken internal links and internal links to redirects. Its redirect checks also cover redirect chains, loops, redirect-to-error situations, and related problems. sitebulb.com

Sitebulb has both desktop and cloud versions and explicitly supports sites ranging from thousands to millions of URLs. sitebulb.com

One important consideration: on extremely large sites, link-analysis data can become enormous. Sitebulb gives an example of a 1.6M-URL crawl producing 142.6M links and taking 36 GB with link analysis enabled. support.sitebulb.com

So I'd use its configuration controls aggressively on giant sites.

4. Oncrawl — best for crawl + server-log analysis

Oncrawl

Oncrawl is particularly compelling for enterprise sites where you're trying to understand how search engines actually crawl the site, rather than just discovering links.

It supports millions of URLs, JavaScript rendering, technical auditing, and combining crawl data with log data. www.oncrawl.com

It also explicitly analyzes redirect chains and loops and can export the affected URLs. help.oncrawl.com

I'd choose it when the question is something like:

"Which redirects are wasting Googlebot crawl activity?"

rather than simply:

"Which URLs redirect?"

What I'd use for your specific problem

If your primary objective is finding broken links + redirect chains efficiently, I'd narrow it down to:

Up to ~2M URLs:
→ Screaming Frog

2–10M URLs:
→ Screaming Frog with database storage, or Sitebulb Cloud

10M–100M+ URLs:
→ JetOctopus

Huge enterprise site + server logs:
→ JetOctopus or Oncrawl

You want the easiest interface for diagnosing problems:
→ Sitebulb

There's also a useful distinction between finding a redirect and finding the links that cause it. For example:

/old-page → 301 → /old-page-2 → 301 → /new-page

A good crawler should let you identify both the chain and the pages internally linking to /old-page, so you can change those internal links directly to /new-page.

That's why I'd favor a real crawler over simple HTTP-status checkers for this job. Sitebulb, for example, explicitly exposes the referring URL → target URL relationship for broken links. support.sitebulb.com

My overall ranking for this particular use case:
Screaming Frog → JetOctopus → Sitebulb → Oncrawl, with JetOctopus moving to #1 if you're talking about tens of millions of URLs.

If the goal is very large sites—hundreds of thousands to millions+ of URLs—and specifically broken links, redirect chains, loops, and redirected internal links, I’d shortlist these:

1. Screaming Frog — my default recommendation

Screaming Frog SEO Spider

For actually diagnosing why links are broken, I'd start here. It gives you granular URL-to-URL data, inlinks/outlinks, status codes, redirect paths, and extensive filtering/export capabilities.

The important thing for huge sites is Database Storage mode. Screaming Frog says its default 5-million-URL limit isn't a hard limit; with appropriate hardware, it gives an example of roughly 10 million URLs on a 500 GB SSD + 16 GB RAM. www.screamingfrog.co.uk

It's particularly good if you need reports like:

/old-page → 301 → /new-page → 301 → /final-page → 200

and then want to know every page internally linking to /old-page.

I'd choose it when: you have an SEO/engineering person who wants raw crawl data and maximum control.

2. Botify — best for true enterprise scale

Botify

Botify's SiteCrawler is designed as a cloud crawler for large sites and can crawl/render JavaScript while collecting a very large technical dataset. It explicitly reports broken links and other crawlability problems and supports ongoing monitoring/prioritization. support.botify.com

I'd choose it when: you're dealing with millions/tens of millions of URLs, multiple properties, complicated JS sites, or need a shared enterprise platform rather than a desktop crawler.

3. Ahrefs Site Audit — easiest cloud solution

Ahrefs Site Audit

Ahrefs is surprisingly good for this particular job. Its current Site Audit can crawl up to 170,000 URLs/minute under the appropriate conditions and explicitly detects redirect chains, redirect loops, broken redirects, 4xx/5xx pages, and pages linking to broken URLs. ahrefs.com

Its current plans can analyze 100K–5M pages/month, so it's worth checking the crawl-credit model against your site's size. help.ahrefs.com

I'd choose it when: you want cloud infrastructure and don't want to maintain a crawler workstation/VM.

4. Sitebulb — excellent for diagnosing the results

Sitebulb

Sitebulb is especially nice when the problem isn't merely "which URLs are broken?" but:

"Show me the structure of the problem and help me decide what to fix first."

It explicitly supports broken internal links, internal links to redirects, and dedicated redirect-chain exports. sitebulb.comhelp.ahrefs.com

It can also work at surprisingly large scale: Sitebulb documents a crawl of 1.6 million internal URLs, noting that link-analysis data can become enormous. support.sitebulb.com

I'd choose it when: humans need to analyze and communicate the crawl findings, rather than just dump millions of rows into a database.

5. Oncrawl — particularly interesting if you have server logs

Oncrawl

This becomes more interesting for a giant site because you can combine crawler data + server-log data. That's useful for answering questions such as:

• Which broken URLs are actually being requested?
• Which redirect chains are Googlebot encountering?
• Which 404s have significant crawl activity?
• Which pages are linking to redirects?
• Are there URLs being crawled that aren't discoverable through internal links?

For a huge ecommerce/content site, that distinction is valuable: a crawler tells you what exists in the site's link graph; logs tell you what real crawlers/users are actually requesting.

What I'd actually use

For a 500K–5M URL site, my preference would be:

Screaming Frog + server logs

Screaming Frog gives you the detailed URL/link graph, while logs give you the real-world crawl/request picture.

For 5M+ URLs, I'd lean toward:

Botify / Oncrawl / JetOctopus

because cloud/distributed crawling becomes much more attractive than trying to push everything through one machine.

For a smaller team that wants minimal infrastructure:

Ahrefs Site Audit

is probably the easiest answer.

For SEO consultants who need to investigate and present the problems:

Sitebulb is exceptionally pleasant.

One important optimization

Don't simply crawl every URL and then look for 404s. For a massive site, configure the crawler to prioritize:

1. Internal HTML
2. Sitemap URLs
3. URLs discovered from internal links
4. Redirect destinations
5. 4xx/5xx URLs
6. Redirect chains/loops
7. External links only if they're part of your objective

Screaming Frog specifically recommends narrowing large crawls by subfolder/subdomain, URL parameters, depth, and resource types rather than blindly crawling everything. www.screamingfrog.co.uk

And for redirect chains, the most actionable output isn't just:

A → 301 → B → 301 → C → 200

It's:

A → B → C, plus every internal page linking to A.

That lets engineering replace the internal link with C and eliminate the chain at the source. Ahrefs and Sitebulb both support this kind of inlink analysis. sitebulb.comhelp.ahrefs.com

If you tell me roughly whether the site is 100K, 1M, 10M, or 50M+ URLs, I can narrow this down to the best 2–3 tools and the most efficient crawl architecture.

If your priority is very large sites + finding broken links + fully resolving redirect chains efficiently, I’d shortlist these:

1. Screaming Frog — best if you want control

For pure technical link crawling, Screaming Frog SEO Spider is probably my first choice.

It can crawl up to 5 million URLs per crawl on a licensed installation, with controls for crawl limits, URL parameters, subdomains, redirects, response retries, etc. It also has a dedicated Redirect Chains report and can follow redirects to the final destination. www.screamingfrog.co.uk

That makes it particularly good for something like:

/page-a
   ↓ 301
/page-b
   ↓ 302
/page-c
   ↓ 301
/page-d
   ↓ 200

You can identify the complete chain rather than merely knowing /page-a is a redirect.

Best when: you have an SEO/engineering person who wants granular crawl configuration and exports.

2. Sitebulb — best overall for huge-site diagnostics

I'd pick Sitebulb if you care not just about finding the bad URLs, but understanding where they're linked from and how serious they are.

Its link analysis distinguishes the referring URL from the target URL, so you can get something like:

500 pages → /old-product
/old-product → 301 → /new-product

rather than simply seeing that /old-product is a redirect. It also has dedicated reporting for redirect chains and can export the chains for further analysis. support.sitebulb.com

For genuinely enormous sites, Sitebulb Cloud is particularly interesting: it advertises crawling millions of URLs, including JavaScript rendering at scale, without using your local machine's resources. sitebulb.com

One important consideration: Sitebulb itself notes that large crawls can explode in size because link-analysis data grows much faster than URL counts. It gives an example of a 1.6-million-URL crawl generating 142.6 million links and consuming substantially more disk space when link analysis was enabled. support.sitebulb.com

Best when: you're dealing with hundreds of thousands to millions of URLs and need useful analysis rather than just raw crawl output.

3. JetOctopus — particularly attractive for multi-million-URL sites

For cloud-first crawling at very large scale, I'd investigate JetOctopus.

It's a better fit than a desktop crawler when you don't want your laptop/workstation tied up for hours or days. It's particularly useful for sites where you want to repeatedly crawl large URL inventories and compare results.

Best when: your site is millions of URLs and cloud crawling speed/scale is more important than having every possible desktop configuration option.

4. Oncrawl — best if this becomes an enterprise data problem

Oncrawl is worth considering if the crawler is going to feed a larger SEO/data-analysis operation.

For example, you might want to combine:

• crawl data
• internal links
• log files
• Google Search Console data
• analytics
• indexation information

That's more powerful than simply asking "which URLs return 404?"

Best when: you're an enterprise SEO team and want to correlate crawl problems with search-engine behavior and traffic.

5. Ahrefs — excellent if you already have it

Ahrefs Site Audit specifically detects:

• 404 pages
• broken redirects
• redirect chains
• redirect loops
• pages linking to redirecting URLs

It can also identify the pages containing links to the problematic URLs, which is important because the real fix is often changing the source link, not adding yet another redirect. help.ahrefs.com

So if you already pay for Ahrefs, I wouldn't necessarily buy another crawler just for this.

What I'd actually use

For a very large website, I'd use a two-stage approach:

Stage 1 — exhaustive crawl

Use Screaming Frog, Sitebulb Cloud, or JetOctopus to crawl the entire URL universe.

Prioritize:

HTML pages
internal links
redirects
4xx/5xx responses
canonical URLs
XML sitemaps

Don't blindly crawl every external resource, image, JS file, query parameter, etc. On huge sites those can multiply the crawl enormously. Sitebulb explicitly recommends restricting unnecessary external URLs, resources, subdomains and parameters on large crawls. support.sitebulb.com

Stage 2 — targeted HTTP verification

For every URL that is:

3xx
4xx
5xx

run a lightweight HTTP checker that records the entire redirect path, e.g.:

requested URL
→ status
→ Location
→ status
→ Location
→ final status

That gives you a clean dataset like:

This separation is important because on a multi-million-URL site, you don't want expensive processing on every URL when only a fraction need deep redirect analysis.

My ranking

If I were choosing today:

1. Sitebulb Cloud — best combination of scale + link analysis
2. Screaming Frog — best control and technical depth
3. JetOctopus — excellent for massive cloud crawls
4. Oncrawl — best enterprise/data-heavy workflow
5. Ahrefs — excellent if SEO/backlink data is already part of your stack
6. Semrush — good all-purpose SEO crawler

For a site in the 10M–100M+ URL range, though, I would stop thinking of this as a normal "SEO crawler" problem and build a distributed HTTP/link-crawling pipeline around sitemap inventories, internal-link extraction, and targeted redirect verification. That's substantially more efficient than repeatedly throwing a conventional GUI crawler at the whole site.

If your main goal is finding broken links and redirect chains on very large sites, I’d narrow the field considerably. The best choice depends mostly on whether you mean 1–10M URLs or tens/hundreds of millions.

My shortlist

1. Screaming Frog — my default recommendation

For technical SEO work, this is probably the first tool I'd reach for.

Its current crawler can use database storage and crawl millions of URLs. Screaming Frog says its default 5M URL limit isn't a hard limit, and gives an example of approximately 10M URLs with a 500GB SSD and 16GB RAM. www.screamingfrog.co.uk

More importantly for your particular use case, it has dedicated reports for:

• 404/410 and other 4xx errors
• redirects
• redirect-to-error
• redirect chains
• redirect loops
• redirect + canonical chains
• number of redirect hops
• source URLs and final destinations

It can also follow every redirect in List Mode, which is particularly useful if you have a known list of millions of URLs rather than wanting to discover everything through crawling. www.screamingfrog.co.uk

For a 1–10M URL site, I'd probably choose Screaming Frog first.

2. Sitebulb — excellent alternative

Sitebulb is particularly good if you want the crawler to help interpret the data, rather than giving you a giant spreadsheet to analyze.

It explicitly supports sites ranging from thousands to 5M URLs, and its cloud version is designed for large uninterrupted audits. sitebulb.comwww.screamingfrog.co.uk

For your specific problem, Sitebulb has:

• broken internal links
• links pointing to redirects
• redirect chains
• redirect-chain exports
• incoming-link analysis showing exactly which pages contain the problematic links

Its distinction between a broken URL and a broken link pointing to that URL is also useful when prioritizing fixes. support.sitebulb.com

I'd pick Sitebulb over Screaming Frog if non-technical stakeholders need to understand the results.

3. Oncrawl — when you really mean large

If you're talking about 10M+ URLs, multiple sites, scheduled crawls, or enterprise infrastructure, I'd start looking at cloud-native crawlers.

Oncrawl explicitly positions its crawler for millions of URLs without artificial limits on speed, depth, or site size and supports JavaScript rendering. www.oncrawl.com

That's a significant distinction from a desktop crawler: you're not relying on someone's workstation sitting there for 18 hours.

Oncrawl also has dedicated analysis around redirect chains/loops and can be used for recurring technical monitoring. www.oncrawl.com

4. Botify — enterprise option

For a very large enterprise site, Botify SiteCrawler is another serious option.

It provides HTTP-code analysis for 2xx/3xx/4xx/5xx URLs and specifically identifies redirect chains and loops, including the URLs involved. support.botify.com

The interesting part is the surrounding dataset: Botify can combine crawl data with search-engine crawl information, organic visits, orphan URLs, etc. support.botify.com

That's useful if the question isn't merely:

"Where are my 404s?"

but instead:

"Which broken/redirected URLs are actually important to Google and users?"

It also supports exporting millions of rows through its API/data infrastructure, which becomes important at enterprise scale. developers.botify.com

5. Semrush — good, but not my first choice for this job

Semrush Site Audit absolutely handles broken links and redirect chains. www.semrush.com

But its current published crawl limits are substantially more constrained than dedicated enterprise crawlers: for example, its Business tier lists 1M pages/month and 100K pages per campaign. www.semrush.com

So I'd use it when you already want the broader Semrush ecosystem—not because it's the best tool for crawling a 10M+ URL site.

What I'd actually use

Under ~2M URLs:
Screaming Frog → easiest recommendation.

2–10M URLs:
Screaming Frog with database storage + SSD, or Sitebulb Cloud. Screaming Frog specifically recommends database storage for large crawls and gives hardware guidance up through roughly 10M URLs. www.screamingfrog.co.uk

10–50M+ URLs:
I'd move toward Oncrawl or Botify rather than trying to brute-force everything through a desktop crawler.

50M–100M+ URLs:
I'd strongly favor an enterprise/cloud crawler and probably combine it with server-log analysis and sitemap/URL-list crawling rather than attempting a naïve "crawl every link" crawl.

One important optimization

For your use case, don't configure the crawler to collect everything.

If the primary objective is:

broken links + redirect chains

you can dramatically reduce crawl cost by focusing on:

1. HTML pages
2. internal links
3. HTTP status codes
4. redirect destinations/hops
5. referring URLs
6. XML sitemaps
7. optionally external links

and initially exclude images, CSS, JS, fonts, etc.

Screaming Frog specifically recommends narrowing large crawls by URL path, query parameters, resource types and external links; Sitebulb similarly recommends excluding unnecessary resources/external URLs when crawling very large sites. sitebulb.comwww.screamingfrog.co.uk

If you tell me roughly how large the site is (e.g. 5M, 20M, 100M URLs) and whether you need to crawl from links, sitemaps, or a supplied URL list, I can give you a much more specific setup—including the fastest/cheapest architecture.

For crawling very large websites (hundreds of thousands to millions of URLs) to find broken links, redirect chains, redirect loops, bad canonicals, orphan pages, and crawl inefficiencies, the best tools depend on whether you want a desktop crawler, cloud-scale crawler, API pipeline, or custom infrastructure.

Best large-scale website crawlers

Best choices by use case

1. Maximum crawl depth and technical control

Best pick: Screaming Frog SEO Spider

Great for:

• finding 404/410 errors
• identifying 301/302 chains
• detecting redirect loops
• checking internal links pointing to redirects
• auditing hreflang, canonicals, pagination
• exporting URL-level data for processing

For very large sites:

• run headless/cloud instances
• increase RAM allocation
• crawl from XML sitemaps instead of discovering everything
• export in batches

2. Millions of URLs with scheduled monitoring

Best picks: Lumar or Oncrawl

Better when you need:

• weekly/monthly automated crawls
• dashboards
• historical comparisons
• multiple websites
• API access
• team workflows

Typical enterprise workflow:

1. Crawl production site weekly
2. Compare broken URLs against previous crawl
3. Alert when redirect chains increase
4. Combine with server logs

3. Developer-friendly / custom pipelines

For engineering teams, consider:

• Apache Nutch — scalable open-source crawler framework
• Scrapy — Python crawling framework
• Playwright — browser-based crawling for JavaScript-heavy sites

A custom crawler stack often looks like:

URL frontier
    ↓
Distributed crawler workers
    ↓
HTTP fetchers
    ↓
Redirect resolver
    ↓
URL database
    ↓
Broken-link reports

Common infrastructure:

• Redis/Kafka for URL queues
• PostgreSQL/ClickHouse for crawl data
• S3/object storage for raw responses
• Kubernetes for worker scaling

Tools specifically good at redirect-chain detection

Redirect checking is slightly different from general crawling. These are strong:

Screaming Frog SEO Spider

Excellent for:

• URL → 301 → 302 → 200 chains
• internal links hitting redirects
• redirect loops

Sitebulb

Good reporting around:

• "redirect chains"
• "redirect hops"
• SEO impact

Dedicated HTTP tools:

• httpstatus.io — quick redirect-chain checks
• Redirect Checker by WhereGoes — simple chain tracing

For extremely large sites (10M+ URLs)

A practical architecture:

Discovery

• XML sitemaps
• internal link extraction
• URL databases
• server logs

Crawler

• Scrapy cluster or custom Go/Rust crawler

Storage

• ClickHouse for billions of crawl records
• PostgreSQL for metadata

Checks

• HTTP status
• redirect depth
• canonical consistency
• robots rules
• response time
• content duplication

Example crawl record:

source_url
destination_url
status_code
redirect_count
canonical_url
crawl_timestamp
response_time_ms

Then queries become easy:

"Show all internal URLs with more than 2 redirects"

SELECT *
FROM crawl_links
WHERE redirect_count > 2;

My recommended stack by site size

<100k URLs

• Screaming Frog

100k–5M URLs

• Screaming Frog + Lumar/Oncrawl for monitoring

5M–50M URLs

• Lumar/Oncrawl or custom crawler

50M+ URLs

• Custom distributed crawler + ClickHouse

For most SEO teams, Screaming Frog is still the best crawler for deep technical audits, while Lumar/Oncrawl are better when crawling becomes an ongoing infrastructure problem rather than a one-off audit.

If your priority is very large sites + fast detection of broken links and redirect chains, I’d shortlist these:

1. Lumar — best when "very large" really means huge

Lumar's crawler is probably where I'd start for millions to tens of millions of URLs. Lumar currently advertises crawl speeds of up to 450 URLs/second without rendering and 350 URLs/second rendered. www.lumar.io

It's particularly attractive if you're trying to minimize the time between:

crawl → identify bad URL → identify every page linking to it → fix

rather than just running a traditional SEO audit.

Use it when: you're dealing with enterprise sites, huge ecommerce catalogs, multiple domains/subdomains, or need repeatable high-speed crawling.

2. Sitebulb — my favorite for diagnosing the problems

Sitebulb is particularly strong at turning crawl data into actionable link problems.

It explicitly distinguishes a broken link from merely a broken URL: for example, Page A linking to a 404 Page B is the broken link that needs fixing. support.sitebulb.com

Its Cloud product is designed for large-scale crawling and can handle sites ranging from thousands to 5 million URLs, with no crawl-credit model on its enterprise offering. sitebulb.com

One important optimization: on enormous sites, don't blindly crawl every resource, parameterized URL, external URL, and subdomain. Sitebulb's own guidance notes that limiting unnecessary data can dramatically reduce crawl time and storage. support.sitebulb.com

Use it when: you want excellent visualization, link-level diagnostics, and a crawler that an SEO team can actually work with.

3. Ahrefs — excellent if you want SEO intelligence alongside crawling

Ahrefs Site Audit is surprisingly capable at high crawl rates. Verified sites can be configured for up to 30,000 URLs/minute, subject to what your infrastructure can handle. www.lumar.iositebulb.comhelp.ahrefs.comahrefs.com

It detects things like:

• 404/410 broken pages
• pages linking to broken pages
• broken redirects
• redirect loops
• redirect chains
• external broken links

Ahrefs specifically recommends replacing chains with a direct 301 → 200 where possible. help.ahrefs.com

Its current Site Audit plans go as high as 5 million pages per project, although crawl-credit limits vary by plan. ahrefs.com

Use it when: you're already using Ahrefs for backlinks/keywords and don't want a separate crawling stack.

4. Screaming Frog — still fantastic for technical investigations

Screaming Frog SEO Spider is the tool I'd give a technical SEO person who wants maximum control.

Its big advantage isn't necessarily raw cloud-scale throughput. It's the sheer number of crawl controls, custom extractions, JavaScript rendering options, filters, exports, integrations, and ways to investigate individual URLs.

For a massive site, though, I'd favor a cloud crawler if your goal is continuous large-scale monitoring rather than occasional forensic crawling.

5. Oncrawl — strongest when crawl data needs to feed analysis

Oncrawl is worth considering if your crawler is part of a larger enterprise SEO/data workflow.

I'd put it behind Lumar/Sitebulb specifically for your stated goal, but ahead of them if you need to combine crawl data with things like log files, analytics, backlinks, and large-scale SEO segmentation.

What I'd choose

For 1–5M URLs:
Sitebulb Cloud or Ahrefs

For 5M+ URLs / extremely large enterprise sites:
Lumar

For technical SEO investigations where flexibility matters more than raw throughput:
Screaming Frog

For SEO + crawl + backlink/data analysis in one platform:
Ahrefs or Oncrawl

One important point about redirect chains

Don't just look for:

301 → 301 → 200

You ideally want the crawler to give you the source URL, every hop, final URL, status code of every hop, and the pages containing the source link.

For example:

/page-old
  ↓ 301
/page-old-2
  ↓ 302
/page-new
  ↓ 301
/page-final
  ↓ 200

The most useful report is therefore something like:

Source page
    ↓
/page-old
    ↓ 301
/page-old-2
    ↓ 302
/page-new
    ↓ 301
/page-final (200)

That lets you prioritize internal links that should be changed directly, rather than merely identifying redirects that exist.

If you're crawling 10M–100M+ URLs, I'd actually consider a different architecture: a fast crawler + URL/status database + a second-stage validator, rather than relying entirely on a traditional SEO crawler. That can be substantially faster and cheaper for finding only broken links and redirect chains.

For very large websites (100k to millions of URLs), the best crawler depends on whether you prioritize speed, technical SEO, customization, or enterprise-scale distributed crawling.

Here's how the major options compare.

Best choices by website size

Under 100,000 URLs

Screaming Frog is still difficult to beat.

Advantages:

• Fast multithreaded crawler
• Excellent redirect chain reports
• Finds:
  • 404s
  • 410s
  • redirect loops
  • redirect chains
  • orphan pages
  • broken internal links
  • broken external links
• Exports directly to CSV/Excel
• JavaScript rendering if needed

100k–2 million URLs

Cloud crawlers become more attractive.

Top choices:

• JetOctopus
• Oncrawl
• Ahrefs Site Audit

These avoid desktop RAM limitations and can crawl continuously.

Millions of URLs

Enterprise options include:

• JetOctopus
• Oncrawl
• Custom crawlers built with frameworks such as Apache Nutch or Apache StormCrawler
• Custom crawlers using Scrapy

Distributed crawlers are usually preferable once a crawl can no longer fit comfortably on one machine.

If you only care about broken links

The fastest tools are generally:

1. Screaming Frog
2. SiteOne Crawler
3. JetOctopus

These can quickly produce reports such as:

• Internal 404s
• External 404s
• Broken images
• Broken CSS
• Broken JavaScript
• Broken PDFs

If you only care about redirect chains

Screaming Frog is especially strong.

Typical reports include:

• 301 → 301 → 301
• 302 → 301
• Infinite loops
• Redirects longer than one hop
• Redirects to 404
• Mixed HTTP/HTTPS chains

It also highlights opportunities to update internal links so they point directly to the final destination instead of relying on redirects.

For developers who want automation

If you're comfortable writing code, these are excellent choices:

• Scrapy — highly customizable and scalable
• Crawlee — modern JavaScript/TypeScript crawler with support for static and dynamic sites
• Playwright — useful for JavaScript-heavy sites where rendering is required
• Heritrix — designed for very large archival crawls

With these frameworks, you can parallelize requests, follow redirects, record redirect chains, validate status codes, and export results to databases or data warehouses.

My recommendation

For different use cases, I'd recommend:

• Technical SEO audits (best overall): Screaming Frog
• Cloud crawling of millions of URLs: JetOctopus
• Enterprise websites with log analysis: Oncrawl
• Open-source automation: Scrapy or Crawlee
• Large JavaScript applications: Crawlee with Playwright or Playwright directly

For organizations managing websites with millions of pages, a common pattern is to use a cloud crawler (such as JetOctopus or Oncrawl) for routine monitoring and a custom crawler built with Scrapy or Crawlee for specialized checks, integrations, or workflows that commercial tools don't support.

For very large websites (100k to 100M+ URLs), the best tool depends on whether your priority is crawl speed, SEO auditing, customization, or continuous monitoring.

Here's how I'd categorize them.

If your only goal is broken links + redirect chains

These are the fastest choices:

1. Screaming Frog ⭐⭐⭐⭐⭐

Still the gold standard.

Pros

• Extremely configurable
• Detects
  • 404s
  • 410s
  • 5xx
  • redirect chains
  • redirect loops
  • mixed content
  • canonicals
  • orphan URLs
• JavaScript rendering
• XML sitemap comparison
• Can import millions of URLs
• API integrations

For huge sites:

• Increase RAM allocation
• Use Database Storage mode instead of RAM mode
• Crawl with 50–100+ threads depending on the server
• Disable unnecessary extraction (images, JS, CSS) when only checking links

2. JetOctopus ⭐⭐⭐⭐⭐

If your site has millions of pages.

Advantages:

• Cloud crawler
• Extremely fast
• Handles ecommerce sites with tens of millions of URLs
• Excellent redirect-chain reports
• Internal PageRank visualization
• Incremental crawls

Much easier than trying to make Screaming Frog crawl 20M pages.

3. Lumar (Deepcrawl)

Excellent for enterprise websites.

Strengths:

• Scheduled crawling
• Trend analysis
• Crawl budget optimization
• Redirect reports
• Broken internal/external links

Ideal if you're monitoring large sites continuously.

If you want maximum crawl speed

Sometimes desktop tools become the bottleneck.

Command-line crawlers can be much faster.

GNU Wget

Can recursively crawl sites and report errors.

Pros

• Fast
• Scriptable
• Lightweight

Cons

• Minimal reporting
• Poor redirect visualization

HTTrack

Useful for mirroring websites.

Can identify:

• missing resources
• broken links
• redirect behavior

Not ideal for SEO reporting.

Custom crawler

If you're comfortable programming, building your own crawler can outperform general-purpose tools for targeted checks.

Popular libraries include:

• Crawlee (JavaScript/TypeScript)
• Scrapy (Python)
• Colly (Go)
• Heritrix (Java)

With a focused crawler that only performs HEAD or lightweight GET requests and follows redirects up to a fixed limit, you can often scan millions of URLs much faster than a full SEO crawler.

For redirect-chain analysis specifically

Look for reports showing:

• 301 → 301 → 301 → 200
• 302 chains
• redirect loops
• HTTP → HTTPS
• www ↔ non-www
• trailing slash redirects
• lowercase/uppercase redirects
• canonical mismatches

The best tools for this are:

1. Screaming Frog
2. JetOctopus
3. Lumar
4. Sitebulb

For websites with over 10 million URLs

I would generally recommend:

• JetOctopus for cloud-based crawling at massive scale.
• Screaming Frog for deep investigations of subsets of the site or if you have a high-spec workstation and need maximum control.
• A custom distributed crawler (using Crawlee, Scrapy, or Go/Colly) if you need to scan tens or hundreds of millions of URLs regularly and only care about HTTP status codes, redirect chains, and broken links. This approach can be significantly faster and more cost-effective than full SEO suites because it avoids extracting and analyzing unnecessary page content.

That combination gives you both comprehensive reporting and the flexibility to handle very large websites efficiently.

For very large websites (hundreds of thousands to millions of URLs), the best crawling tools depend on whether you need SEO auditing, developer-grade crawling, distributed crawling, or continuous monitoring. The main challenges at scale are crawl speed, memory usage, JavaScript rendering, URL deduplication, redirect-chain detection, and handling crawl budgets.

Here are the strongest options:

Best overall choices

1. Best practical SEO crawler: Screaming Frog

Screaming Frog SEO Spider

Use it when you need:

• broken internal links
• 404/410 detection
• redirect chains and loops
• canonical issues
• sitemap validation
• hreflang problems
• response-code analysis

For very large sites:

• enable database storage mode
• run on a high-memory machine
• split crawls by subdomain, directory, or sitemap
• crawl from XML sitemaps rather than discovery alone

A 10M URL crawl is possible with the right setup, but it becomes a resource-management exercise.

2. Best enterprise-scale crawler: Botify / Oncrawl / Lumar

For sites like:

• marketplaces
• news sites
• ecommerce catalogs
• documentation platforms
• international sites with millions of pages

cloud crawlers are usually better because they handle:

• distributed crawling
• URL queues
• retries
• throttling
• historical comparisons
• scheduled monitoring

They also combine crawl data with:

• Google Search Console
• server logs
• analytics
• rankings

This is useful because a "broken link" is not always just a 404 problem — a redirect chain wasting crawl budget or a page blocked by robots rules can be equally damaging.

3. Best engineering solution: Scrapy

Scrapy Documentation

If you need a custom crawler, Scrapy is usually the first choice.

A scalable architecture might look like:

URL frontier
     |
     v
Redis/Kafka queue
     |
     v
Scrapy workers
     |
     +--> HTTP fetchers
     |
     +--> Redirect tracker
     |
     +--> Link extractor
     |
     v
PostgreSQL / ClickHouse
     |
     v
Reports + dashboards

You can efficiently detect:

• broken links:
  • 404
  • 410
  • connection failures
  • timeout errors

• redirect problems:
  • 301 → 302 → 301 loops
  • chains longer than N hops
  • redirects to irrelevant destinations

For redirect-chain detection specifically

A good crawler should record the entire redirect path:

Example:

/old-page
   |
   301
   v
/temporary-page
   |
   302
   v
/new-page
   |
   200

Flag:

• chains > 2 hops
• mixed 301/302 sequences
• redirect loops
• redirects to 404 pages
• internal links pointing to redirected URLs

Tools particularly strong here:

1. Screaming Frog
2. Lumar
3. Botify
4. Custom Scrapy crawler

Open-source stack for very large sites

If you need to crawl tens or hundreds of millions of URLs without paying enterprise SaaS:

Crawler

• Scrapy

Queue

• Redis
• Apache Kafka

Storage

• ClickHouse (excellent for crawl analytics)
• PostgreSQL
• Elasticsearch/OpenSearch

Scheduling

• Airflow
• Kubernetes CronJobs

Reporting

• Grafana
• Metabase

This setup can outperform many commercial tools, but requires engineering effort.

My recommendations by site size

<500k URLs

Use:

• Screaming Frog
• Sitebulb

500k–10M URLs

Use:

• Screaming Frog with database storage
• Lumar
• Oncrawl
• Botify

10M–100M+ URLs

Use:

• Botify
• Oncrawl
• Lumar
• custom Scrapy/Nutch pipeline

Continuous monitoring

Use:

• Lumar
• Botify
• scheduled Screaming Frog crawls
• custom crawler + alerts

For most teams, the sweet spot is Screaming Frog for audits + a cloud crawler (Botify/Oncrawl/Lumar) for ongoing monitoring. If you are building your own crawling infrastructure, Scrapy + ClickHouse + a queue system is the most flexible high-scale approach.

If you're crawling very large websites (100k–10M+ URLs), the best tool depends on whether your priority is speed, scale, reporting, or automation. For broken links and redirect chains specifically, these are the strongest options.

1. Screaming Frog (best value)

This is still the benchmark for technical SEO.

Strengths:

• Crawls millions of URLs in database storage mode.
• Detects:
  • 404s
  • 410s
  • 5xx errors
  • redirect loops
  • redirect chains
  • mixed redirects
  • orphan URLs (when combined with logs/sitemaps)
• Can crawl XML sitemaps and compare against live URLs.
• Supports JavaScript rendering.
• Command-line automation.

Weaknesses:

• Limited by your hardware.
• Very large sites (>10M URLs) require careful tuning.

Ideal for:

• ecommerce
• publishers
• universities
• SaaS documentation

2. OnCrawl

Excellent for enterprise websites.

Advantages:

• Distributed cloud crawler
• Huge crawl capacity
• Crawl budget analysis
• Log file integration
• Internal PageRank calculations
• Redirect visualization

Especially good when working with:

• millions of URLs
• multiple subdomains
• international sites

3. Botify

Probably the strongest enterprise solution.

Highlights:

• Extremely fast cloud crawling
• Server log analysis
• Redirect chain reporting
• Broken internal links
• JavaScript rendering
• Crawl budget optimization

Common among Fortune 500 SEO teams.

4. Lumar

Focused on continuous website monitoring.

Good for:

• scheduled crawls
• accessibility
• SEO
• redirect monitoring
• broken asset detection

Useful if you need recurring audits rather than one-off crawls.

5. Sitebulb

Very user-friendly.

Best if you want:

• visual reports
• easy redirect diagrams
• explanations of issues
• prioritization

It isn't quite as scalable as the enterprise cloud platforms but is excellent for small to large sites.

If you want maximum crawling speed

For custom engineering work:

• Apache Nutch
• Scrapy
• Heritrix
• StormCrawler

These can crawl tens or hundreds of millions of URLs if deployed across multiple machines, but they don't provide SEO-focused reports out of the box. You would need to build your own reporting for broken links and redirect chains.

Best for redirect chains

The tools that provide the clearest redirect-chain analysis are:

1. Screaming Frog
2. Botify
3. OnCrawl
4. Sitebulb

They typically report chains like:

A
 ↳ 301
B
 ↳ 302
C
 ↳ 301
D (200)

along with chain length and opportunities to update internal links to point directly to the final destination.

Best for finding broken internal links

The most useful workflow is to crawl from the site's homepage and record:

• Source URL
• Destination URL
• HTTP status
• Anchor text
• Redirect hops
• Inlinks count
• Outlinks count

Prioritize:

• internal 404s with many inlinks
• redirect chains longer than one hop
• redirect loops
• links to redirected URLs that could be updated to the final URL
• broken image, CSS, and JavaScript assets

Recommendation by website size

• Up to 500k URLs: Screaming Frog is usually sufficient and offers excellent value.
• 500k–10M URLs: Screaming Frog (well-configured) or Sitebulb for desktop workflows; OnCrawl if you want cloud scale.
• 10M+ URLs: OnCrawl, Botify, or Lumar are generally better suited because they distribute crawling and provide ongoing monitoring without depending on a single workstation.

For most SEO teams, Screaming Frog remains the best balance of cost, features, and flexibility. If you're managing a truly massive enterprise site with millions of pages and need continuous monitoring, Botify or OnCrawl are typically the strongest choices.

For crawling very large websites (hundreds of thousands to millions of URLs) to find broken links, redirect chains, redirect loops, and crawl issues, the best tools depend on whether you want a desktop crawler, cloud-scale platform, or developer-focused pipeline.

Best large-scale website crawlers

Best choices by use case

1. Maximum crawl control (technical teams)

Best: Screaming Frog SEO Spider

Use it when you need:

• Broken internal links (404/410)
• Redirect chains (301 → 302 → 200, etc.)
• Redirect loops
• Canonical issues
• Orphan URL discovery
• Custom regex extraction
• Sitemap crawling
• JavaScript-rendered pages

For very large sites:

• Run in database storage mode
• Increase RAM allocation
• Crawl in batches by sitemap sections
• Export crawl data to a database

2. Millions+ of URLs with minimal infrastructure

Best: Botify or Lumar

These are better when:

• You crawl continuously
• Multiple teams need access
• You need scheduled monitoring
• You have huge ecommerce/news sites
• You need historical comparisons

They handle:

• Distributed crawling
• Queue management
• Crawl prioritization
• API access
• Large exports

3. Developer-built crawler pipelines

If you want to build your own system:

Python

Good options:

• Scrapy — high-performance crawling framework
• aiohttp — async HTTP requests
• httpx — modern async HTTP client

Typical architecture:

URL Queue
    ↓
Async Workers
    ↓
HTTP Fetcher
    ↓
Status/Headers Parser
    ↓
Redirect Resolver
    ↓
URL Database
    ↓
Broken Link + Chain Reports

For very large sites, store:

• URL
• HTTP status
• Final URL
• Redirect hops
• Anchor text
• Referring page
• Crawl timestamp
• Content type
• Response time

A database like PostgreSQL, ClickHouse, or BigQuery is usually better than flat CSV files.

Fastest approach for broken links specifically

If your only goal is "find every broken link and redirect chain", you do not need a full SEO crawler.

A high-performance workflow:

1. Crawl all URLs from:
   • XML sitemaps
   • Internal links
   • Previous crawl databases

2. Send asynchronous HEAD/GET requests.

3. Record:

Example:

/old-product
    ↓ 301
/products/old-product
    ↓ 302
/products/new-product
    ↓ 200

Flag:

• Chain length > 1
• 404/410 responses
• 5xx errors
• Redirects to irrelevant pages
• External broken links

Tools optimized for this:

• Screaming Frog (easy)
• Scrapy + custom pipeline (maximum scale)
• Botify/Lumar (enterprise)

My shortlist

Small–medium site (<500k URLs):

1. Screaming Frog
2. Sitebulb

Large site (500k–50M URLs):

1. Screaming Frog with database storage
2. Lumar
3. Botify

Huge site (50M+ URLs):

1. Botify
2. Lumar
3. Custom crawler infrastructure

Engineering teams wanting automation:

1. Scrapy
2. Async Python crawler + database
3. Scheduled CI/CD crawl jobs

For most organizations doing a serious technical SEO audit, Screaming Frog is the default starting point, while Botify/Lumar are the enterprise choices when crawl scale and continuous monitoring matter.