Answers that do not mention Moz

For a 10k-page ecommerce site, I’d optimize for crawl efficiency, redirect-chain detection, exportability, and low recurring cost. You don’t need an enterprise crawler unless you’re doing daily monitoring, log analysis, or millions of URLs.

My shortlist:

My recommendation for a 10k ecommerce site

Best budget setup: Screaming Frog + scheduled checks

Use Screaming Frog SEO Spider as your primary crawler. A 10k URL crawl is well within its comfort zone, and it gives you the raw data you need:

• Response Codes

• 404 broken pages
• 500 server errors
• 301/302 redirects
• Redirect reports

• redirect chains (A → B → C)
• redirect loops (A → B → A)
• Bulk exports

• URL
• source page containing the broken link
• anchor text
• status code
• redirect destination

For ecommerce, I’d also enable:

• JavaScript rendering (important if your storefront is React/Vue/Angular)
• XML sitemap crawl
• Crawl of staging environments before releases
• Custom extraction for product IDs, canonical tags, structured data

CrawlBudget## A cheap monitoring workflow

For ongoing protection:

1. Run a full crawl weekly or after deployments.
2. Export:

• all 4xx URLs
• all 3xx URLs
• redirect chains
• pages with internal links to redirects
• Compare exports week-over-week.
• Add a CI check if your developers deploy frequently.

A useful rule:

• One big crawl occasionally → Screaming Frog
• Non-technical stakeholders need reports → Sitebulb
• Continuous enterprise monitoring → cloud crawlers

Ecommerce-specific gotchas to check

For a 10k product/category site, don’t only look for broken links. Also crawl for:

• Product pages returning soft 404s
• Category pages redirecting through multiple hops
• Internal links pointing to old product URLs
• Canonical URLs that redirect
• Faceted navigation creating infinite URL combinations
• Pagination loops
• Out-of-stock product handling
• Broken image URLs
• JavaScript-rendered links that crawlers miss

If your store is on Shopify, Magento, WooCommerce, BigCommerce, or a custom stack, the ideal crawler settings differ quite a bit.

For a 10k-page ecommerce site on a budget, I’d use Screaming Frog SEO Spider.

Why it fits

• 10k URLs is well within its paid crawl capacity: the paid version has an unlimited crawl limit. The current listed price is $245/year per license; the free version is capped at 500 URLs. Screaming Frog
• It directly identifies 404/4xx errors, 5xx errors, redirects, redirect chains, and redirect loops. Its Redirect Chains report explicitly flags loops. Screaming Frog Screaming Frog
• You can export the offending URLs and the pages linking to them, which is particularly useful for fixing broken internal links at scale. Screaming Frog
• It can also follow redirects all the way to the final destination, so you can distinguish a clean 301 from a multi-hop chain or loop. Screaming Frog

My budget shortlist

Sitebulb Lite is interesting if you prefer a more visual interface: its Lite plan is limited to 10,000 URLs per audit, while Pro goes to 500,000. Sitebulb G2 JetOctopus is aimed more at large-scale crawling/log analysis; its current paid plans start far above what I'd consider necessary for a 10k-page site. Tech SEO Platform

How I'd configure Screaming Frog

For your particular problem, I'd do two passes:

1. Full Spider crawl

• Start at the homepage.
• Crawl internal links.
• Check Response Codes → Client Error (4xx) for broken links.
• Check Response Codes → Server Error (5xx).
• Check Redirects → Redirect Chains.
• Filter the chain report for Loop = True. Screaming Frog
• Targeted redirect audit

• Feed it your sitemap, old URL database, or known redirect list using List Mode.
• Enable Always Follow Redirects.
• Export the complete redirect chains.

This catches redirects that aren't necessarily discoverable through your normal internal linking. Screaming Frog Screaming Frog

One ecommerce-specific warning: faceted navigation and URL parameters can turn a nominally 10k-page site into hundreds of thousands of crawlable URLs. Screaming Frog specifically recommends controlling query strings/excluding problematic URL patterns when parameterized URLs cause crawl explosions. Screaming Frog

So if you're spending as little as possible, Screaming Frog paid + a carefully configured crawl is probably all you need. If you'd rather have prettier diagnostics and guided prioritization, Sitebulb Lite is the main alternative I'd test.

For an SEO agency, I’d choose based on what you need to discover:

1. Rendered DOM links/content (what a browser sees after JS executes)
2. Network/AJAX-discovered URLs (XHR/fetch/API calls that may create pages)
3. Scale (hundreds vs millions of URLs)
4. Audit workflow (client reports, exports, repeatability)

A lot of SEO crawlers can execute JavaScript, but not all of them capture AJAX behavior the same way. A crawler that only parses the post-render DOM may still miss URLs that exist only inside API responses.

Shortlist for an SEO agency

Sitebulb Screaming Frog## The key question: do you mean "AJAX-loaded URLs"?

There are two different things people call this:

A) "Find links that appear after JavaScript runs"

Example:

<div id="products"></div>

<script>
fetch('/api/products')
  .then(...)
  .then(products => {
     // inserts:
     <a href="/product/widget-123">
  })
</script>

You want the crawler to see:

/product/widget-123

A Chromium crawler is usually enough. Screaming Frog and Sitebulb both compare or extract rendered output so you can identify links introduced after JS execution. Sitebulb Support Screaming Frog

B) "Capture every API request made by the browser"

Example:

GET /api/products?page=2
GET /graphql
GET /search?q=boots
GET /inventory/sku123

This is a different problem. You need network interception.

For this, I’d lean toward a custom crawler using:

• Playwright
• Puppeteer
• Chrome DevTools Protocol

You can capture:

• XHR
• fetch
• GraphQL calls
• JSON responses
• redirects
• script-generated URLs

Example architecture:

URL queue
   ↓
Playwright Chromium
   ↓
page.goto()
   ↓
listen to network events
   ↓
capture:
  - request URLs
  - response URLs
  - JSON payloads
  - DOM links
   ↓
normalize URLs
   ↓
crawl queue

My agency-oriented recommendation

For normal client SEO audits

Use:

Screaming Frog + JavaScript rendering enabled

Good fit when you need:

• crawl reports
• canonicals
• metadata changes
• JS-only links
• exports for clients

It renders pages in a headless browser and crawls the resulting HTML after JavaScript execution. Screaming Frog

For JS-heavy ecommerce / React / Angular sites

Use a two-pass approach:

1. Normal crawl
2. Rendered crawl
3. Compare:

• URLs discovered only after JS
• content differences
• blocked resources
• API behavior

For sites where navigation is entirely API-driven

Build or add a Playwright layer.

Examples:

• infinite-scroll marketplaces
• faceted search apps
• SaaS dashboards
• headless commerce
• SPAs with no <a href> links

Evaluation checklist before buying

Run the same 1,000 URLs through each candidate and measure:

Discovery

• Does it find URLs created after hydration?
• Does it find URLs loaded after clicking?
• Does it wait for lazy-loaded content?
• Does it capture infinite scroll?

Rendering fidelity

• Chromium version?
• Custom user agent?
• Mobile viewport?
• Cookie/session support?
• Login flows?

Data extraction

Can you export:

• rendered HTML?
• DOM links?
• network requests?
• API endpoints?
• response bodies?
• screenshots?

Speed/cost

JS crawling is much slower because every URL becomes a browser session. Screaming Frog specifically notes that JS crawling is more resource intensive because resources must be downloaded and executed before crawling the rendered page. Screaming Frog

If I were building an agency stack

I’d probably do:

• Screaming Frog → standard technical audits
• Sitebulb → polished client reporting
• Playwright crawler → edge cases where AJAX/API discovery matters
• Google Search Console URL Inspection → spot-check Google’s rendered view

The biggest mistake is choosing a crawler that "supports JavaScript" but only checks the final DOM. For complex JS sites, you want visibility into both rendered links and network activity.

If you're choosing a crawler for an SEO agency, I’d treat “renders JavaScript” and “discovers URLs from AJAX/API activity” as two separate capabilities. A crawler can execute JS perfectly well while still missing URLs that only appear in XHR/fetch responses.

What I’d evaluate

Google itself renders JavaScript with an evergreen Chromium-based renderer, and after rendering it can discover links that JavaScript injected into the DOM. Google for Developers Google for Developers

But there's an important distinction: Google's rendering behavior isn't the same thing as “crawl every URL appearing in every API response.” Google says its renderer processes XHR requests to understand page content, while URL discovery is primarily based on crawlable links, sitemaps, and redirects. Google for Developers Google for Developers

The architecture I'd want

For an agency crawler, I'd favor something conceptually like:

Seed URLs
   ↓
HTTP crawler
   ↓
Chromium renderer
   ├── final DOM
   ├── rendered links
   ├── JS console/errors
   └── network events
          ├── document
          ├── fetch/XHR
          ├── GraphQL
          └── other requests
                    ↓
             URL extraction
                    ↓
        ┌───────────┴───────────┐
        ↓                       ↓
  crawlable URLs          API-discovered URLs
        ↓                       ↓
       queue              discovery dataset

That last distinction is important. Don't automatically treat every network URL as a crawl target. Otherwise you'll end up crawling analytics endpoints, JSON APIs, image CDNs, telemetry, personalization calls, and dozens of irrelevant resources.

Instead, I'd maintain at least two datasets:

• SEO URLs — URLs found in rendered <a href>, canonical tags, structured data, sitemaps, redirects, etc.
• Network-discovered URLs — URLs observed in XHR/fetch/GraphQL/network responses, with the source page and request type recorded.

Then you can report something like:

/category/shoes → rendered page → XHR /api/products?page=2 → response contains 48 product URLs → 46 aren't present as HTML links.

That's much more useful in an SEO audit than simply saying “JavaScript is enabled.”

One particularly important test

Build a small benchmark suite of deliberately nasty client sites:

1. React SPA where links appear only after hydration.
2. Infinite scroll where subsequent products arrive through XHR.
3. Load More button requiring interaction.
4. GraphQL endpoint returning product/category URLs.
5. JSON API containing URLs but never rendering them as links.
6. Client-side router using history.pushState.
7. Lazy-loaded content triggered by viewport visibility.
8. JS-generated canonical/meta/structured data.
9. Authentication/session-dependent rendering.
10. A site with a never-ending analytics connection, to test whether your wait strategy hangs.

For each, record:

raw HTML URLs
rendered DOM URLs
network request URLs
URLs extracted from network responses
URLs discovered after interactions

Then compare the crawler's output against a known ground truth.

The key question when comparing products

Don't ask vendors:

“Does your crawler support JavaScript?”

Ask:

“Can your Chromium crawler capture fetch/XHR/GraphQL requests, inspect their response bodies, extract URLs from those responses, and put those URLs into a separate crawl/discovery queue?”

That question will separate genuine browser-based crawlers from tools that merely render the page.

Also test how they wait for rendering. A fixed 5-second delay is much less useful than configurable conditions such as DOM selectors, network activity, or custom browser-side JavaScript.

And for SEO validation, I'd retain a Google-equivalent rendered view as a separate comparison. Google explicitly notes that rendering can have limitations and that its renderer is stateless between page loads, so “works in Chrome” doesn't automatically mean “Google can process it.” Google for Developers Google for Developers

If you're deciding between Screaming Frog, Sitebulb, JetOctopus, Lumar, Oncrawl, or building your own Playwright/Chromium crawler, I can also give you a feature-by-feature comparison specifically for AJAX/XHR URL discovery at agency scale.

Then compare the crawler's output against a known ground truth.

The key question when comparing products

Don't ask vendors:

“Does your crawler support JavaScript?”

Ask:

“Can your Chromium crawler capture fetch/XHR/GraphQL requests, inspect their response bodies, extract URLs from those responses, and put those URLs into a separate crawl/discovery queue?”

That question will separate genuine browser-based crawlers from tools that merely render the page.

Also test how they wait for rendering. A fixed 5-second delay is much less useful than configurable conditions such as DOM selectors, network activity, or custom browser-side JavaScript.

And for SEO validation, I'd retain a Google-equivalent rendered view as a separate comparison. Google explicitly notes that rendering can have limitations and that its renderer is stateless between page loads, so “works in Chrome” doesn't automatically mean “Google can process it.”

For a 50k-article news site, I’d lean toward sitebulb.com as the best single-tool fit for your requirements.

Why it fits a news site particularly well

The duplicate/near-duplicate detection is important for news because you can have syndicated stories, updated versions, tag/category URLs, printer pages, AMP/alternate URLs, and parameterized URLs that look different but contain essentially the same article. Sitebulb can identify both substantially duplicate HTML and similar content. Sitebulb Support

For discovery, I'd configure it to crawl both internal links and all XML sitemaps. That lets you identify things like:

• Articles in the sitemap that aren't internally linked.
• Internally discovered articles missing from the sitemap.
• Sitemap URLs returning 404/5xx.
• Non-indexable URLs accidentally included in sitemaps.
• Duplicate/near-duplicate articles.
• Broken images, JS, CSS and other resources.
• Parameterized URLs generating crawl waste.

Sitebulb specifically provides sitemap-vs-crawl comparisons and flags broken/non-indexable sitemap URLs. Sitebulb Support

One alternative

screamingfrog.co.uk is also very capable, particularly if you want a highly configurable desktop crawler. Its near-duplicate system uses a MinHash similarity algorithm, with a configurable similarity threshold. screamingfrog.co.uk

For 50k URLs + sitemap coverage + duplicate content + broken assets + controlled crawl rate, though, Sitebulb is the cleaner all-in-one choice.

If you tell me whether the site is WordPress, a custom CMS, or something like Drupal, I can also suggest the exact crawl configuration (threads, URL/sec, sitemap setup, exclusions, duplicate threshold, and what to monitor for 50k articles).

For a 50,000-article news site, I’d shortlist Screaming Frog SEO Spider and Sitebulb, with Screaming Frog as the more flexible choice if you want detailed control.

My pick: Screaming Frog

For a news site, the particularly useful combination is:

• Sitemaps + normal crawl: make the sitemap URLs an additional discovery source so you catch orphaned articles as well as normally linked pages.
• Near-duplicate detection: useful for syndicated stories, updated versions, tag/archive pages, and articles that differ only slightly. Screaming Frog's near-duplicate system can be tuned by similarity threshold and content area. Screaming Frog
• Asset crawling: identify broken images, CSS, JS and other resources rather than checking only article URLs.
• Crawl limits: restrict problematic URL patterns such as search, filters, tracking parameters and infinite pagination.
• Database storage: important at 50k+ URLs because you can retain and compare crawls rather than treating every audit as disposable. Screaming Frog

Sitebulb is the alternative I'd choose if the people reviewing the results aren't SEO/crawler specialists. Its sitemap reporting is particularly nice: it can explicitly show URLs that are only in sitemaps or not in sitemaps, while its Resources report covers images, CSS, JavaScript and fonts. Sitebulb Support Sitebulb Support

One caveat on rate limits: if by "rate limits" you mean the crawler should automatically respect the publisher's robots.txt/Crawl-delay instructions, that's worth checking separately from simply having a configurable crawl speed. For example, Ahrefs explicitly documents that its crawler follows Crawl-delay and respects robots.txt. Ahrefs

For your specific workload, I'd configure Screaming Frog roughly as: sitemap + website crawl → database storage → controlled crawl speed → exclude search/parameter traps → crawl resources → enable near-duplicates → run recurring crawls and compare changes.

Sitebulb is probably the best fit if client-facing visualization and reports matter. Its redirect-chain data exports to CSV/Google Sheets, and it has interactive visualizations plus customizable PDFs. Sitebulb Support Sitebulb

For a more lightweight, spreadsheet-first workflow, Screaming Frog SEO Spider is excellent: its Redirect Chains report explicitly maps hops, loops, source/final URLs, and exports cleanly. screamingfrog.co.uk screamingfrog.co.uk

Quick pick: Sitebulb for polished freelance deliverables; Screaming Frog for lean technical auditing.

For a freelance technical-SEO workflow, Sitebulb is probably the closest match to what you’re asking for.

Sitebulb has particularly strong visualizations and client-ready reporting: its Desktop crawler handles up to 500k URLs per audit, includes crawl maps and other interactive visualizations, and supports customized PDF reports plus CSV/Google Sheets exports. Sitebulb Sitebulb Support

How I’d compare the main options

Sitebulb: Its Redirect report has a dedicated Export Redirect Chains function, and it can export to CSV/Google Sheets. Its broader visualization system includes crawl maps, trees and directory visualizations. Sitebulb Support Sitebulb Support

Screaming Frog: If by "lightweight" you mean fast desktop utility with minimal overhead, this is the one I'd look at. Its Redirect Chains report maps the chain, shows hop counts, identifies the source and flags loops; its All Redirects export also records the full hops. Screaming Frog Screaming Frog

JetOctopus: Useful if you're more interested in cloud crawling and larger sites. It has a dedicated Redirect Chains table and stores the complete chain/hop information. Tech SEO Platform

My practical take

If your typical deliverable is:

crawl → identify redirect chains → show client/dev exactly what redirects to what → export evidence → put findings into an audit

I'd lean Sitebulb because the visualization + reporting layer is unusually well suited to client-facing audits. Its HTML crawler is also its quickest crawler mode, while Chrome crawling is available when you actually need JS rendering. Sitebulb Support

If you instead want the smallest, most utilitarian crawler that you can fire up for a quick 5–50k URL investigation, I'd lean Screaming Frog.

sitebulb.com · screamingfrog.co.uk

If you tell me your typical crawl size and whether you need white-label PDFs, I can narrow it down further—including some less-obvious lightweight alternatives.

Yes. For a staging site that requires authentication and relies on JavaScript, I’d use a real browser crawler rather than a traditional HTTP-only spider.

Good options

• Playwright — probably the most flexible choice if you want a custom audit crawler. It runs a real Chromium/Firefox/WebKit browser with JavaScript enabled and can reuse authenticated cookies/local storage/IndexedDB via storageState. Playwright Playwright

• Authenticate once with a dedicated staging account.
• Save the browser state.
• Crawl URLs using that state.
• Capture rendered DOM, network requests, console errors, screenshots, accessibility data, etc.
• Particularly good if you want the audit integrated into CI.
• OWASP ZAP — a strong choice if "audit" means security-oriented crawling/scanning, rather than just discovering URLs. ZAP supports browser-based authentication and its modern Client/AJAX Spiders can crawl JavaScript-heavy applications while authenticated. ZAP ZAP

• Browser-based authentication is useful for ordinary login flows.
• Client Script Authentication is useful for complicated flows, including recorded login sequences. ZAP ZAP
• Its Automation Framework lets you put authentication and scanning configuration into a reproducible YAML plan. ZAP
• Playwright + ZAP together — a particularly useful setup if you want both complete rendered-site coverage and security findings. Playwright handles the application's browser/session behavior; ZAP can observe and scan the resulting traffic.

Safest staging setup

I'd structure it roughly like this:

Dedicated staging test account
          ↓
Playwright authenticates
          ↓
Authenticated browser context
          ↓
Crawl/render only staging
          ↓
Collect DOM + links + JS/network behavior
          ↓
Optional ZAP proxy/passive scan

For the authentication, avoid putting real credentials in crawler configuration. Playwright's documented approach is to create an authenticated state file and reuse it, while keeping that file out of source control because it can contain cookies and other credentials. Playwright

If your staging bot block is based on User-Agent/IP, I'd also avoid trying to "bypass" it covertly. The cleaner approach is to explicitly allowlist your crawler's staging IP/service account or provide a staging-only bot exception. That makes the audit reproducible and prevents accidentally testing the production anti-bot defenses.

What I'd pick

For a site audit/crawl: Playwright.

For a security audit: ZAP with Browser/Client Script Authentication + Client/AJAX Spider.

For a CI security audit of a modern authenticated SPA: Playwright + ZAP, with a dedicated low-privilege staging account and strict staging-only target scope. ZAP specifically notes that modern JavaScript applications are best crawled with its browser-based spiders, because traditional form/JSON authentication doesn't necessarily authenticate those browser instances. ZAP

If you tell me whether your staging auth is SSO/OAuth, basic auth, username/password, or something like Cloudflare Access, I can give you a concrete crawler configuration for it.

For a staging site you control, good options are:

• Playwright — authenticate once, save storageState, then crawl with a real Chromium browser so JS renders normally. Playwright
• OWASP ZAP + Ajax/Client Spider — supports browser-based authentication and client scripts, specifically suited to authenticated modern JS apps. ZAP ZAP
• OWASP PTK Agent — can wrap Playwright/Puppeteer/Selenium/Cypress journeys and combine authenticated browser execution with security analysis. OWASP Foundation

For safety, use a dedicated staging account, read-only permissions where possible, throttle the crawler, restrict it to the staging hostname, and disable destructive workflows.

Yes. The most useful setup is to join three datasets at the URL level:

1. Server/access logs — what users and bots actually requested.
2. Crawler output — what URLs are internally discoverable and what links point to them.
3. Traffic/search data — how much real value each affected URL has.

That lets you distinguish “a 404 exists” from “a 404 is repeatedly requested, linked internally, and has meaningful traffic.”

A practical workflow

1. Crawl the site

Use a crawler such as screamingfrog.co.uk to export at least:

• URL
• Status code
• Inlinks / referring URLs
• Anchor text
• Indexability
• Canonical
• Redirect target
• Depth
• Sitemap inclusion

For every 4xx/5xx URL, you ideally want to know which internal pages link to it.

2. Parse server logs

Pull 30–90 days of access logs and aggregate by requested URL:

URL
status_code
request_count
unique_visitors_or_ips
Googlebot_requests
Bingbot_requests
bytes
response_time
last_seen

Log data is particularly valuable because a crawl is only a simulation; logs show which URLs were actually requested. They can also expose URLs that your crawler doesn't discover, including historical or externally linked URLs. Screaming Frog Screaming Frog

For a relatively turnkey workflow, Screaming Frog Log File Analyser is specifically designed for this. It can import Apache, NGINX, IIS and other log formats, identify 4xx/5xx responses, and combine log data with crawl exports. Screaming Frog Screaming Frog

3. Join the datasets

The interesting table becomes something like:

Screaming Frog's Log File Analyser has an Imported URL Data feature specifically for matching crawl exports or other URL datasets against log data. Screaming Frog Screaming Frog

4. Build a prioritization score

I wouldn't rank purely by number of 404s. A better model is something like:

priority =
    user_traffic_score
  + internal_link_score
  + search_bot_score
  + external_link_score
  + request_frequency_score
  + business_value_score
  + recency_score

For example, give extra weight to:

• Real user requests — stronger evidence of actual impact.
• Many internal inlinks — indicates a broken site architecture.
• Googlebot/Bingbot requests — indicates search engines are still encountering it.
• External referring traffic/backlinks — potentially valuable legacy URLs.
• High-value destination/content — stronger reason to restore or redirect.
• Recent and recurring requests — more actionable than a URL requested once two years ago.

Conversely, thousands of requests generated by a single noisy bot shouldn't automatically outrank a broken URL receiving genuine user traffic.

5. Separate “fix” from “redirect” candidates

This is an important second layer.

404 + strong internal linking + obvious replacement exists

→ fix the internal links and/or restore the resource.

404 + substantial external traffic/backlinks + clear successor

→ investigate a relevant 301 redirect.

404 + high bot traffic but no users + obsolete URL

→ investigate why bots keep discovering it; don't blindly redirect everything.

404 + essentially no traffic/inlinks + obsolete URL

→ usually lower operational priority.

The logs can also reveal inconsistent response codes—for example, URLs that were broken historically but now work—which is why looking at the time dimension matters. Screaming Frog

If you want a more automated stack

For a larger site, I'd build:

NGINX/Apache/CDN logs → object storage → SQL warehouse → crawler export → URL-level scoring → dashboard

For example:

Cloudflare / NGINX / ALB logs
             ↓
       S3 / BigQuery
             ↓
       SQL aggregation
             ↓
   ┌─────────┴──────────┐
   │                    │
Crawler data       Analytics/GSC
   │                    │
   └─────────┬──────────┘
             ↓
       URL issue table
             ↓
       priority_score
             ↓
    Looker / Power BI / CSV

The key is to retain the raw URL-level events and calculate aggregates rather than just exporting a list of 404s. That makes it possible to re-rank the backlog whenever traffic, crawling, or business priorities change.

If you tell me what your stack looks like (e.g. NGINX + GA4 + Search Console + Screaming Frog, or Cloudflare + BigQuery + Python), I can lay out the exact schema, SQL, and prioritization formula I'd use.

404 + high bot traffic but no users + obsolete URL

→ investigate why bots keep discovering it; don't blindly redirect everything.

404 + essentially no traffic/inlinks + obsolete URL

→ usually lower operational priority.

The logs can also reveal inconsistent response codes—for example, URLs that were broken historically but now work—which is why looking at the time dimension matters.

If you want a more automated stack

For a larger site, I'd build:

NGINX/Apache/CDN logs → object storage → SQL warehouse → crawler export → URL-level scoring → dashboard

For example:

For prioritizing broken links, the strongest workflow is usually not just “find 404s” — it is a join between:

1. What is broken (crawl data)
2. How often it is hit (server/CDN logs)
3. Who is hitting it (users, bots, referrers)
4. How valuable the affected URL is (traffic, conversions, links, revenue)

A good setup looks like this:

Recommended workflow

1. Crawl the site to build the broken-link inventory

Use a crawler to collect:

• Broken internal links (source page → broken destination)
• Status codes (404, 410, 500, redirect chains)
• Anchor text
• Number of internal links pointing to the broken URL
• Page templates affected

Common tools:

• Screaming Frog SEO Spider — strong for finding broken links and exporting source URLs/inlinks. The GTM Exchange
• Sitebulb — useful if you want more visual audits.
• Ahrefs Site Audit — useful when you also care about backlinks and external references.

Output table:

2. Parse server/CDN logs to measure real impact

Your logs should include:

• timestamp
• requested URL
• HTTP status
• referrer
• user agent
• response time
• bytes sent
• IP/request ID if available

Then calculate:

• Total hits to each broken URL
• Human traffic vs bots
• Top referrers
• Googlebot hits
• Trend over time

For SEO-focused log analysis, tools like Screaming Frog Log File Analyser, OnCrawl, and Botify are commonly used because they combine crawl behavior with server log data. Ighenatt Samuel Ochoa

For engineering-heavy teams:

• BigQuery
• ClickHouse
• Elasticsearch + Kibana
• Datadog
• Grafana Loki
• AWS Athena

A typical SQL aggregation:

SELECT
  url,
  COUNT(*) AS hits,
  COUNTIF(user_agent LIKE '%Googlebot%') AS bot_hits,
  COUNT(DISTINCT referrer) AS referring_pages
FROM access_logs
WHERE status_code IN (404,410)
GROUP BY url
ORDER BY hits DESC;

3. Add business value signals

Join your broken URL list with:

• Google Analytics / GA4 sessions
• Search Console clicks/impressions
• Conversion events
• Backlinks
• Revenue data

Example priority scoring:

Priority Score =
(404 hits × 5)
+ (human visits × 10)
+ (Googlebot hits × 3)
+ (backlinks × 8)
+ (conversion value × 10)

This prevents spending time on:

• /random-old-image.jpg with 3 bot hits

while fixing:

• /best-selling-product receiving 20,000 visitors/month and returning 404

Practical stack recommendations

Small/medium site (<100k URLs)

• Screaming Frog SEO Spider
• Screaming Frog Log File Analyser
• Google Analytics
• Search Console
• Spreadsheet or Looker Studio dashboard

Simple monthly workflow.

Large site (millions of URLs)

• CDN logs (Cloudflare, Fastly, CloudFront)
• Data warehouse (BigQuery/Snowflake/ClickHouse)
• Scheduled crawler
• SQL models
• Dashboard in Looker/Tableau/Grafana

Pipeline:

CDN logs
   ↓
Log parser
   ↓
Broken URL table
   ↓
Join:
  - crawl data
  - analytics
  - backlinks
  - revenue
   ↓
Priority queue
   ↓
Jira/Linear tickets

Useful prioritization rules

I would usually rank fixes like this:

1. 404s receiving real user traffic
2. 404s linked internally from high-value pages
3. 404s with backlinks
4. 404s crawled frequently by search engines
5. Broken URLs repeated across templates
6. Low-volume cleanup items

Template-level problems deserve special attention: one broken link in a navigation component can create thousands of broken URLs.

A dashboard I would build

Columns:

This turns broken links from an SEO audit list into a ranked engineering backlog.

This turns broken links from an SEO audit list into a ranked engineering backlog.

For a 2,000-page local business site, you probably don’t need an enterprise SEO platform. Look for a site crawler / technical SEO audit tool that can:

• crawl all pages (not just a single URL)
• find 4xx errors (404s, 410s)
• detect redirect chains (e.g., page → old page → newer page)
• detect redirect loops (page → itself or circular redirects)
• show the source page containing the bad link so you know what to fix
• export results to CSV so you or a web person can work through fixes

Tool types to consider:

1. Desktop website crawlers (best fit for a one-time audit)

These run a full crawl of your site and give detailed reports.

• Screaming Frog — a common choice for technical audits; it can find broken links, redirect chains, and redirect loops and export URLs for fixes. Screaming Frog

• Good for: “I want a complete health check of my 2,000 pages.”
• Tradeoff: you install it and run the crawl yourself.
• Similar crawler-style tools can provide the same basic workflow: crawl → identify errors → export → fix. Broken Link Checker

2. Cloud SEO audit platforms (best if you want ongoing monitoring)

These run scans on their servers and often add SEO reporting.

Examples include:

• Ahrefs
• Semrush

Good for:

• monthly monitoring
• tracking search issues alongside broken links
• seeing broader SEO data

Tradeoff:

• usually more expensive than a simple crawler.

3. Simple broken-link monitoring services (best for “set it and forget it”)

These are easier for non-technical owners.

Look for:

• scheduled scans
• email alerts
• redirect-loop detection
• reports that identify affected pages

Some lightweight scanners focus specifically on broken links and redirects. Broken Link Scan

4. CMS-specific link checkers (if you use WordPress or similar)

If your site is managed in a CMS, a plugin can be convenient because it may let you fix links where you write content.

Good when:

• staff regularly add/edit pages
• you want alerts without running a crawler manually

Tradeoff:

• large sites can sometimes slow down if the plugin scans aggressively.

Features I’d prioritize for your situation

For a 2k-page local business site, I’d look for:

A practical workflow:

1. Run a full crawl once.
2. Fix:

• broken internal links first
• redirect loops immediately
• long redirect chains next
• outdated external links last
• Schedule a monthly scan.

For a small local business, a crawler-based audit tool plus a monthly automated check is usually the simplest setup. Screaming Frog Broken Link Scan

For a ~2,000-page site, you don’t need an enterprise SEO platform. The main tool type to look for is a website crawler / technical SEO spider.

Tool types to consider

• Website crawler / SEO spider — the main one.

It crawls your pages like a search engine and reports 404s, 5xx errors, redirected URLs, redirect chains, and redirect loops. Examples include Screaming Frog, Sitebulb, and Ahrefs Site Audit. Sitebulb specifically reports broken/redirected internal links and redirect chains, while Ahrefs explicitly detects redirect loops. Sitebulb Support Ahrefs

• Dedicated broken-link checker — useful for a quick check.

These are simpler if your only concern is finding dead internal/external links. Ahrefs, for example, has a dedicated broken-link checker that can identify 404s and redirects and export the results. Ahrefs

• HTTP/status-code checker — useful for troubleshooting individual URLs.

This lets you inspect the actual sequence of HTTP responses, e.g. 301 → 302 → 301 → ..., which is particularly useful when diagnosing a redirect loop. Ahrefs Help Center

What I'd look for in your case

For a 2,000-page local-business site, prioritize a crawler that gives you:

1. 4xx/5xx errors
2. Internal links pointing to broken URLs
3. Redirect chains
4. Redirect loops
5. The final destination of redirects
6. CSV/Excel export
7. Scheduled recurring crawls, so you don't have to manually check the site every time

You don't necessarily need backlink analysis, rank tracking, or a giant all-in-one SEO suite just for this job.

Simple workflow: run a full crawl → filter for 4xx, 5xx, redirects, and redirect loops → export the affected URLs and their referring pages → fix the links/redirect rules → crawl again to verify. At 2,000 pages, this is a very manageable crawl size. Ahrefs sitebulb.com

For an agency auditing dozens or hundreds of client sites, the biggest time savings come from features that reduce manual triage and turn raw crawl data into actionable client deliverables. The crawler itself matters, but the workflow around prioritization, exports, and reporting matters just as much. Screaming Frog Crawle

1. Broken link discovery features

Look for:

• Full internal + external link crawling

• Finds every page linking to a dead URL, not just the dead URL itself.
• Shows:

• Source page
• Broken destination
• Anchor text
• HTTP status code
• Number of internal links pointing to it
• Impact prioritization

• Sort broken links by:

• Traffic-driving pages
• Number of internal links
• Page importance
• Template/location (header, footer, navigation, body content)
• Bulk exports

• CSV/XLS exports with developer-friendly columns save hours of cleanup before sending fixes. Built For SEO Screaming Frog
• Soft 404 detection

• Catches pages returning 200 OK but behaving like missing pages.
• Broken asset checks

• Images, scripts, CSS files, PDFs, and downloads can create client-visible issues even when pages load.

2. Redirect auditing features

Redirect work is where agencies lose a lot of time because "the URL works" does not mean the implementation is clean.

Useful features:

• Redirect chain mapping

• Example output:

old-page.com
   ↓ 301
archive-page.com
   ↓ 302
new-page.com
   ↓ 200

• The report should show every hop and the final destination. Ahrefs Screaming Frog
• Redirect loop detection

• Finds cases like:

A → B → A

• Redirect-to-error detection

• Flags:

Old URL → 301 → 404 page

• Migration mode

• Upload an old URL list and verify every redirect resolves correctly.
• Extremely useful for redesigns and domain migrations. Screaming Frog
• Chain length filtering

• Quickly isolate:

• 2+ hop chains
• 5+ hop chains
• loops
• temporary redirects

3. Agency workflow features

These usually save more time than extra SEO checks.

Client project management

Look for:

• Multiple client workspaces
• Saved crawl settings
• Recurring crawls
• Crawl history comparisons
• Team access
• Issue assignment

A good agency crawler should let you reopen last month's audit instead of rebuilding everything. Crawle

4. Client-ready reporting

The best reports translate technical problems into business language.

Useful features:

White-label reports

Include:

• Agency logo
• Client branding
• Custom colors
• Custom cover pages
• Executive summary

Severity scoring

Instead of:

127 URLs returning 404

Give:

127 broken links found; 14 affect high-value pages and navigation paths.

Issue grouping

Clients understand categories better:

• Critical:

• Broken conversion pages
• Redirect loops
• Server errors
• High priority:

• Redirect chains
• Broken internal links
• Maintenance:

• Old external links
• Minor metadata issues

Fix recommendations

Each issue should include:

5. Crawl efficiency features for many clients

For agency scale, prioritize:

• JavaScript rendering

• Needed for modern sites where links load dynamically.
• Custom crawl limits

• Example:

• Quick audit: 500 URLs
• Full audit: 100k+ URLs
• API access

• Lets you feed crawl data into dashboards or internal tools.
• Scheduled monitoring

• Automatically catches regressions after client updates.
• Change detection

• "This redirect appeared since last crawl" is much more valuable than a static report. Crawle

6. Small features that create big agency time savings

Often overlooked:

• Saved filters:

• "Show only client-critical issues"
• "Show only pages with >10 internal links"
• Bulk issue exports
• URL lists:

• Crawl only a staging area
• Check a migration list
• Audit a campaign landing page set
• Integrations:

• Google Analytics
• Google Search Console
• Looker Studio
• Project management tools
• Historical comparisons:

• "Broken links decreased from 42 → 3 after fixes"

Ideal agency crawler workflow

1. Enter client domain
2. Crawl site + sitemap
3. Detect:

• 4xx/5xx errors
• redirect chains
• orphan pages
• broken assets
• Automatically prioritize issues
• Export:

• client PDF
• developer CSV
• internal checklist
• Re-crawl after fixes
• Show improvement report

Tools such as Screaming Frog, Ahrefs, and other agency-focused crawlers commonly emphasize these capabilities: link-level exports, redirect mapping, issue prioritization, and reporting workflows. Ahrefs Screaming Frog

For an agency, I would prioritize report automation + historical tracking + impact-based sorting over simply having the crawler find more issues. Finding 500 problems is easy; turning them into a client-approved action list is where the time is saved.

For an agency crawling many client sites, the biggest time-savers aren't just “find 404s.” You want the crawler to turn raw crawl data into prioritized, explainable, repeatable client work.

1. Broken-link detection that includes the source page

Look for:

• 4xx/5xx detection with the exact URL that is broken.
• Source/inlink mapping — “this broken URL is linked from these 17 pages.”
• Anchor text and link type, so you can tell a client exactly what needs changing.
• External-link checking, not just internal links.
• Filtering by internal/external, status code, follow/nofollow, template, directory, etc.

That source-page context is particularly valuable: it turns “there are 43 broken links” into an actionable spreadsheet for the developer/content team. Screaming Frog Screaming Frog

2. Full redirect-chain visualization

A good crawler should automatically show:

/old-page → /old-page-2 → /new-page → 200

and ideally tell you:

• Number of hops
• HTTP vs JS/meta redirects
• Final destination
• Final status code
• Redirect loops
• Redirects ending in 4xx/5xx
• Which original pages link to the redirecting URL

This eliminates a lot of spreadsheet archaeology. Dedicated redirect-chain reports can map the entire hop sequence and flag loops. Screaming Frog Screaming Frog

3. “Fix this” rather than just “here's the problem”

For agency workflows, I'd prioritize a crawler that gives you action context:

That means your team can assign fixes without manually reconstructing what happened.

4. Bulk exports designed for agency handoff

Don't settle for “export CSV.”

Useful exports include:

• Broken links + source pages
• Redirect chains
• Redirects to errors
• All redirects
• All inlinks/outlinks
• Issue summaries
• Crawl overview
• URLs by status code
• Separate exports by client/project

For example, Screaming Frog supports bulk exports of 4xx inlinks and other crawl issues, while its reports cover redirect chains and redirects to errors. Screaming Frog Screaming Frog

5. Crawl comparison

This is huge for agencies.

After a client/developer fixes the site, you want:

Before crawl → changes → after crawl → remaining issues

rather than running another crawl and manually comparing two CSVs.

Useful comparison states are:

• New issues
• Fixed issues
• Persistent issues
• New URLs
• Missing URLs
• Changed redirects
• Changed status codes

Crawl comparison is specifically useful for identifying whether issues were fixed or introduced between crawls. Screaming Frog

6. Scheduled/automated crawling

Once you have dozens of clients, manually launching crawls becomes operational overhead.

I'd look for:

• Scheduled crawls
• Per-client crawl settings
• Automatic recrawling
• Crawl completion notifications
• Issue alerts
• Historical crawl storage
• Change detection
• API/webhooks
• Cloud crawling

Cloud/continuous crawlers increasingly combine client workspaces, historical data, link monitoring, alerts, exports, and recurring crawls specifically for agency workflows. Crawle Crawle

7. Client-ready reporting

This is where a crawler can save hours per client.

Ideally you can produce a report containing:

Executive summary

• Pages crawled
• Broken links
• Redirect chains
• 5xx errors
• Critical issues
• Issues fixed since previous crawl

Prioritized issues

• Issue
• Severity/impact
• Affected URLs
• Recommended action

Technical appendix

• Full URL-level evidence
• Source URLs
• Status codes
• Redirect paths

And preferably:

• White-label branding
• Custom logo/colors
• Client/project name
• PDF export
• CSV/Excel export
• Shareable report URL
• Scheduled reports

Some agency-focused crawlers now explicitly combine client workspaces, historical data, exports, reporting, and white-label functionality. Crawle

8. Templates and reusable crawl configurations

For an agency, standardization is a feature.

You should be able to create something like:

Standard Technical SEO Audit — Client Template

with your preferred:

• Crawl limits
• User agent
• JavaScript rendering
• robots.txt behavior
• Sitemap sources
• URL exclusions
• Custom extraction rules
• Status-code thresholds
• Report sections

Then apply that configuration to 50 clients instead of rebuilding it 50 times.

9. Client/project isolation

If you're managing many sites, I'd consider this nearly mandatory:

• One workspace per client
• Multiple sites/projects per client
• Team permissions
• Persistent filters
• Client-specific crawl history
• Client-specific report branding
• Tags/labels
• Notes/tasks

It prevents the classic agency problem of mixing data from different clients.

10. The killer feature: automatic prioritization

Raw crawler output isn't the bottleneck anymore. Deciding what matters is.

A really useful system could automatically say:

23 broken internal links

4 are on high-traffic pages
12 point to URLs with replacements available
7 are low-priority/footer links

or:

14 redirect chains

3 are 3+ hops
2 terminate in 404s
9 can be changed to direct 301s

That gives your account manager something they can actually put into a client report.

If I were defining the agency crawler feature set

I'd make the core workflow:

Crawl → Detect → Trace → Prioritize → Assign → Fix → Recrawl → Compare → Report

The features I'd consider most important, in roughly that workflow, are:

1. Broken-link source mapping
2. Automatic redirect-chain/loop detection
3. Redirect-to-error detection
4. Bulk issue exports
5. Before/after crawl comparison
6. Scheduled crawls + alerts
7. Client workspaces
8. White-label reports
9. Reusable crawl templates
10. API/webhooks/integrations
11. Issue prioritization
12. Historical crawl data

The key distinction is that an SEO crawler finds problems, while an agency crawler should minimize everything your team does after the crawl: figuring out impact, finding the source URL, handing work to developers, verifying fixes, and turning the results into something a client can understand.

A really useful system could automatically say:

23 broken internal links

4 are on high-traffic pages
12 point to URLs with replacements available
7 are low-priority/footer links

or:

14 redirect chains

3 are 3+ hops
2 terminate in 404s
9 can be changed to direct 301s

That gives your account manager something they can actually put into a client report.

If I were defining the agency crawler feature set

I'd make the core workflow:

Crawl → Detect → Trace → Prioritize → Assign → Fix → Recrawl → Compare → Report

The features I'd consider most important, in roughly that workflow, are:

1. Broken-link source mapping
2. Automatic redirect-chain/loop detection
3. Redirect-to-error detection
4. Bulk issue exports
5. Before/after crawl comparison
6. Scheduled crawls + alerts
7. Client workspaces
8. White-label reports
9. Reusable crawl templates
10. API/webhooks/integrations
11. Issue prioritization
12. Historical crawl data

The key distinction is that an SEO crawler finds problems, while an agency crawler should minimize everything your team does after the crawl: figuring out impact, finding the source URL, handing work to developers, verifying fixes, and turning the results into something a client can understand.

For reproducing crawler broken-link findings locally, I’d use a two-layer setup: a real browser for JS-rendered behavior, plus an HTTP client for isolating redirects.

• playwright.dev is the strongest all-around choice. It runs Chromium/Firefox/WebKit, executes JS, lets you change user-agent and other browser conditions, and exposes request, response, requestfailed, redirectedFrom() and redirectedTo(). That makes it particularly useful for determining whether a crawler's “broken link” exists in the original HTML, appears after JS rendering, or fails only after navigation. Playwright Playwright Playwright
• Chrome DevTools → Network is best for interactive investigation. Turn on Preserve log, reload, then inspect the document requests and their Location headers/statuses. Exporting a HAR gives you a reproducible record of the network sequence; Chrome's DevTools APIs represent network requests in HAR format as well. Chrome for Developers
• curl.se is ideal for separating server-side redirects from browser/JS behavior. For example, curl -v exposes request/response headers, while --trace/--trace-ascii gives considerably more debugging information. Curl Curl

A useful workflow is therefore:

Crawler reports /old-page as broken
        ↓
curl /old-page
        ↓
inspect HTTP 301/302/307/308 chain
        ↓
Playwright loads /old-page in Chromium
        ↓
inspect browser requests + JS execution
        ↓
Chrome DevTools/HAR when manual inspection is needed

For example, Playwright can log what the rendered browser actually encounters:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: false });

const context = await browser.newContext({
  userAgent: 'YOUR-CRAWLER-UA'
});

const page = await context.newPage();

page.on('request', req => {
  console.log('>>', req.method(), req.url());
});

page.on('response', res => {
  console.log('<<', res.status(), res.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

const response = await page.goto('https://example.com/problem-url', {
  waitUntil: 'load'
});

let req = response?.request();

console.log('\nRedirect chain:');

const chain = [];
while (req) {
  chain.unshift(req);
  req = req.redirectedFrom();
}

for (const r of chain) {
  const res = await r.response();
  console.log(res?.status(), r.url());
}

await page.waitForTimeout(3000);
await browser.close();

Playwright explicitly links redirected requests through redirectedFrom()/redirectedTo(), so you can reconstruct a multi-hop server redirect rather than seeing only the final navigation. Also note that a 404/503 generates a normal HTTP response in Playwright; requestfailed is for failures where the browser couldn't successfully obtain an HTTP response, such as certain network errors. Playwright

One particularly useful diagnostic is to run the same URL twice: JS enabled and JS disabled. Playwright supports javaScriptEnabled: false. If the crawler's finding only reproduces with JS enabled, you're probably dealing with client-side routing/navigation or dynamically generated links rather than a straightforward HTTP broken link. Playwright

If you're trying to match a specific crawler closely, also reproduce its user-agent, cookies/auth, viewport, HTTP method, headers, robots behavior, JS wait time, and redirect limit. Differences there are a common reason why curl says “200 OK” while an SEO crawler reports a broken URL. Playwright's request API can even set maxRedirects: 0, which is handy when you want to examine each redirect hop independently instead of automatically following the chain. Playwright

For reproducing crawler broken-link findings locally, I’d use a two-layer setup: a real browser for JS-rendered behavior, plus an HTTP client for isolating redirects.

• playwright.dev is the strongest all-around choice. It runs Chromium/Firefox/WebKit, executes JS, lets you change user-agent and other browser conditions, and exposes request, response, requestfailed, redirectedFrom() and redirectedTo(). That makes it particularly useful for determining whether a crawler's “broken link” exists in the original HTML, appears after JS rendering, or fails only after navigation. Playwright Playwright Playwright
• Chrome DevTools → Network is best for interactive investigation. Turn on Preserve log, reload, then inspect the document requests and their Location headers/statuses. Exporting a HAR gives you a reproducible record of the network sequence; Chrome's DevTools APIs represent network requests in HAR format as well. Chrome for Developers
• curl.se is ideal for separating server-side redirects from browser/JS behavior. For example, curl -v exposes request/response headers, while --trace/--trace-ascii gives considerably more debugging information. Curl Curl

A useful workflow is therefore:

Crawler reports /old-page as broken
        ↓
curl /old-page
        ↓
inspect HTTP 301/302/307/308 chain
        ↓
Playwright loads /old-page in Chromium
        ↓
inspect browser requests + JS execution
        ↓
Chrome DevTools/HAR when manual inspection is needed

For example, Playwright can log what the rendered browser actually encounters:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: false });

const context = await browser.newContext({
  userAgent: 'YOUR-CRAWLER-UA'
});

const page = await context.newPage();

page.on('request', req => {
  console.log('>>', req.method(), req.url());
});

page.on('response', res => {
  console.log('<<', res.status(), res.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

const response = await page.goto('https://example.com/problem-url', {
  waitUntil: 'load'
});

let req = response?.request();

console.log('\nRedirect chain:');

const chain = [];
while (req) {
  chain.unshift(req);
  req = req.redirectedFrom();
}

for (const r of chain) {
  const res = await r.response();
  console.log(res?.status(), r.url());
}

await page.waitForTimeout(3000);
await browser.close();

Playwright explicitly links redirected requests through redirectedFrom()/redirectedTo(), so you can reconstruct a multi-hop server redirect rather than seeing only the final navigation. Also note that a 404/503 generates a normal HTTP response in Playwright; requestfailed is for failures where the browser couldn't successfully obtain an HTTP response, such as certain network errors. Playwright

One particularly useful diagnostic is to run the same URL twice: JS enabled and JS disabled. Playwright supports javaScriptEnabled: false. If the crawler's finding only reproduces with JS enabled, you're probably dealing with client-side routing/navigation or dynamically generated links rather than a straightforward HTTP broken link. Playwright

If you're trying to match a specific crawler closely, also reproduce its user-agent, cookies/auth, viewport, HTTP method, headers, robots behavior, JS wait time, and redirect limit. Differences there are a common reason why curl says “200 OK” while an SEO crawler reports a broken URL. Playwright's request API can even set maxRedirects: 0, which is handy when you want to examine each redirect hop independently instead of automatically following the chain. Playwright

For very large sites—hundreds of thousands to millions of URLs—the choice is less about basic “find 404s” capability and more about crawl throughput, storage architecture, redirect analysis, URL discovery, and export/API access.

My shortlist

1. Screaming Frog — best for deep technical control

Screaming Frog's SEO Spider is particularly strong when the goal is forensic crawling rather than just an SEO dashboard.

It can find 404/5xx errors, audit redirects, identify redirect chains and loops, and export the problematic URLs along with their source URLs. Screaming Frog

For scale, its database-storage mode has a default 5-million-URL limit, which isn't a hard ceiling. Screaming Frog says appropriate hardware can take it beyond that; it gives roughly 2M URLs as a typical target with 4 GB allocated RAM and recommends SSD storage. Screaming Frog Screaming Frog

Why I'd use it: if you want to inspect exactly which page links to which redirecting/broken URL, manipulate crawl configuration aggressively, and export raw data for further processing.

2. Sitebulb Cloud — particularly compelling for millions of URLs

Sitebulb is worth serious consideration if “very large” means millions of URLs.

Sitebulb currently advertises Cloud crawls of up to 10 million URLs per audit, with cloud infrastructure removing the local-machine constraint. It also supports large-scale JavaScript crawling. Sitebulb Sitebulb

For your specific use case, Sitebulb has dedicated reporting for:

• Broken/not-found internal links
• Redirected internal links
• Redirect chains
• Chain length
• Bulk exports of redirect-chain data

Sitebulb Support Sitebulb SupportIt also has useful mechanisms for large sites where crawling from the homepage alone isn't sufficient: you can feed it XML sitemaps and URLs from sources such as Google Search Console and Analytics. Sitebulb Support

Why I'd use it: probably the most straightforward option when you want enterprise-scale crawling without operating your own crawler infrastructure.

3. Ahrefs — excellent if the crawl is part of a broader SEO workflow

Ahrefs's Site Audit explicitly detects redirect-chain issues, while its broader index also lets you investigate things such as broken backlinks and redirected targets. Ahrefs Help Center Ahrefs Help Center

This becomes particularly useful if you're asking questions like:

“Which broken/redirected URLs are actually important because they have backlinks?”

rather than simply:

“Give me every 404.”

Why I'd use it: when broken links and redirects need to be combined with backlinks, organic-search data, historical SEO data, and competitive research.

4. A custom crawler — if throughput is the primary requirement

If you're talking about 10M–100M+ URLs, or you need to crawl repeatedly as part of an engineering pipeline, I'd consider building around an asynchronous HTTP crawler rather than relying entirely on an SEO application.

A specialized crawler can maintain something like:

source_url
    ↓
HTTP response
    ↓
301/302/307/308?
    ↓
redirect_target_1
    ↓
redirect_target_2
    ↓
final_url
    ↓
final_status

Then you can efficiently flag:

404 / 410
5xx
redirect → 404
redirect → redirect → 200
redirect loops
>1 redirect hop
HTTP → HTTPS chains
old-domain → new-domain → canonical-domain chains

For massive sites, I'd store the crawl in something like a columnar database/object-storage pipeline rather than trying to keep the entire crawl in RAM.

What I'd choose by size

• <100K URLs: Screaming Frog or Sitebulb. Don't over-engineer it.
• 100K–2M: Screaming Frog becomes very attractive if you have a good SSD/workstation; Sitebulb Cloud is easier operationally.
• 2M–10M: Sitebulb Cloud or a carefully configured Screaming Frog deployment. Sitebulb's cloud architecture becomes a significant advantage. Sitebulb Sitebulb
• 10M+ / continuous monitoring: custom crawler or an enterprise/cloud crawling platform, potentially supplemented by Screaming Frog/Sitebulb for investigation.
• SEO + backlink significance: Ahrefs alongside whichever crawler you're using.

One important optimization

For broken-link and redirect-chain hunting, don't blindly crawl everything a crawler can discover.

On huge sites, parameters, faceted navigation, external URLs, subdomains, resources, and infinite pagination can explode the crawl. Sitebulb specifically recommends controlling these dimensions for large crawls. Sitebulb Support

A very effective architecture is:

Sitemap URLs + internal-link discovery + known URL lists → crawl only HTML → record every link relationship → separately resolve redirects → export only anomalies.

That gives you much more useful data per request than turning on every possible crawler feature.

If you tell me roughly how many URLs (1M, 10M, 100M+), whether the site is JS-heavy, and whether you need continuous/automated crawling, I can narrow this down to a concrete architecture and configuration.

4. A custom crawler — if throughput is the primary requirement

If you're talking about 10M–100M+ URLs, or you need to crawl repeatedly as part of an engineering pipeline, I'd consider building around an asynchronous HTTP crawler rather than relying entirely on an SEO application.

A specialized crawler can maintain something like:

source_url
    ↓
HTTP response
    ↓
301/302/307/308?
    ↓
redirect_target_1
    ↓
redirect_target_2
    ↓
final_url
    ↓
final_status

Then you can efficiently flag:

404 / 410
5xx
redirect → 404
redirect → redirect → 200
redirect loops
>1 redirect hop
HTTP → HTTPS chains
old-domain → new-domain → canonical-domain chains

For massive sites, I'd store the crawl in something like a columnar database/object-storage pipeline rather than trying to keep the entire crawl in RAM.

What I'd choose by size

For very large sites, I’d separate the choices by scale and workflow rather than looking for one universal winner.

1. Screaming Frog — excellent for surgical link/redirect analysis

screamingfrog.co.uk

For specifically finding broken links, redirect chains, redirect loops, and the pages creating them, Screaming Frog is hard to beat.

Its particularly useful features include:

• Finds 4xx/5xx URLs and shows the source pages linking to them.
• Dedicated Redirect Chains and Redirect & Canonical Chains reports.
• Can identify the number of hops and redirect loops.
• Can crawl uploaded URL lists, which is extremely useful for migrations.
• Database-storage mode lets it handle millions of URLs rather than keeping the entire crawl in RAM. Screaming Frog Screaming Frog

For example, a chain like:

/old → /old2 → /new → /final

can be surfaced so you can change the original internal link to point directly to /final.

For large crawls, Screaming Frog recommends an SSD + database storage; its documented default limit is 5 million URLs, although it can go beyond that with appropriate hardware/configuration. Screaming Frog Screaming Frog

I'd choose this when: you have a technical SEO team that wants maximum control and the ability to slice/filter/export the crawl data.

2. Sitebulb Cloud — particularly attractive for millions of URLs

sitebulb.com

Sitebulb has become particularly interesting for large websites because its Cloud crawler is designed for substantial scale. Sitebulb says its cloud product can crawl sites ranging from thousands to millions of URLs, and supports high-speed crawling. Sitebulb Sitebulb

For your specific use case, it provides:

• Broken internal links
• Internal links pointing to redirects
• Redirect chains
• Bulk export of redirect-chain data
• Link-level analysis showing exactly where problematic URLs are linked from. Sitebulb Sitebulb Support

It also has URL-rewriting/exclusion controls that are valuable on huge sites where parameters can otherwise create enormous numbers of duplicate URLs. Sitebulb Support

I'd choose this when: you want cloud crawling and a more visual/diagnostic interface, particularly when multiple people need to work with the audit.

3. Botify — when crawling isn't enough

botify.com

Botify makes more sense when your question isn't merely:

"What URLs are broken?"

but:

"Which technical problems are actually being encountered by Googlebot and consuming crawl activity?"

Its AlertPanel, for example, can monitor increases in crawler requests to 3xx and 4xx URLs using server log data. Botify Knowledge Base

That's a significant distinction for very large sites. A crawler can tell you that 100,000 URLs return 404, while log analysis can help establish whether search-engine crawlers are actually requesting those URLs.

I'd choose this when: you're dealing with a huge enterprise site and have access to server/CDN logs.

4. JetOctopus

JetOctopus is another cloud-oriented option worth considering for very large crawls, particularly if you want to combine technical crawling and log-file analysis.

It's particularly useful when you need to correlate:

URLs discovered → URLs crawled → HTTP status → redirects → Googlebot activity

rather than treating the crawler as an isolated auditing tool.

5. Lumar

Lumar is another enterprise-focused crawler suited to organizations that want ongoing technical monitoring rather than repeatedly running a desktop crawl.

It's more compelling when you want scheduled audits, alerts, workflow integration, and multiple technical SEO checks alongside broken-link/redirect detection.

What I'd use for a really large site

For something like a 5–50M URL ecommerce/content site, I'd actually use a combination:

Crawler → log analysis → database

For example:

Sitebulb/Screaming Frog
↓
Find every broken link + redirect chain
↓
Server/CDN logs
↓
Determine which errors/redirects search engines actually request
↓
SQL/warehouse
↓
Track the problem over time

The important optimization is not necessarily crawling every URL with a browser. On huge sites, JavaScript rendering can multiply crawl cost dramatically. Sitebulb, for example, distinguishes its fast HTML crawler from its Chrome-based renderer, with the latter downloading page resources and therefore taking longer. Sitebulb Support

My practical shortlist

• Up to ~1M URLs: Screaming Frog or Sitebulb.
• 1–10M URLs: Screaming Frog database mode or Sitebulb Cloud.
• 10M+ / enterprise: Sitebulb Cloud, Botify, JetOctopus, or Lumar depending on whether logs and continuous monitoring matter.
• Migration with a known URL inventory: Screaming Frog is especially useful because its List Mode can follow every redirect and report every hop. Screaming Frog
• Finding what Google actually encounters: pair a crawler with server-log analysis rather than relying on crawling alone.

If your primary objective is "crawl 10–100M URLs as fast as possible and dump broken links + complete redirect chains into a database", that's a somewhat different problem from ordinary technical SEO auditing, and I'd approach the tool selection differently.

3. Botify — when crawling isn't enough

botify.com

Botify makes more sense when your question isn't merely:

"What URLs are broken?"

but:

"Which technical problems are actually being encountered by Googlebot and consuming crawl activity?"

Its AlertPanel, for example, can monitor increases in crawler requests to 3xx and 4xx URLs using server log data. Botify Knowledge Base

That's a significant distinction for very large sites. A crawler can tell you that 100,000 URLs return 404, while log analysis can help establish whether search-engine crawlers are actually requesting those URLs.

I'd choose this when: you're dealing with a huge enterprise site and have access to server/CDN logs.

4. JetOctopus

JetOctopus is another cloud-oriented option worth considering for very large crawls, particularly if you want to combine technical crawling and log-file analysis.

It's particularly useful when you need to correlate:

URLs discovered → URLs crawled → HTTP status → redirects → Googlebot activity

rather than treating the crawler as an isolated auditing tool.

5. Lumar

Lumar is another enterprise-focused crawler suited to organizations that want ongoing technical monitoring rather than repeatedly running a desktop crawl.

It's more compelling when you want scheduled audits, alerts, workflow integration, and multiple technical SEO checks alongside broken-link/redirect detection.

What I'd use for a really large site

For something like a 5–50M URL ecommerce/content site, I'd actually use a combination:

Crawler → log analysis → database

For example:

Sitebulb/Screaming Frog
↓
Find every broken link + redirect chain
↓
Server/CDN logs
↓
Determine which errors/redirects search engines actually request
↓
SQL/warehouse
↓
Track the problem over time

The important optimization is not necessarily crawling every URL with a browser. On huge sites, JavaScript rendering can multiply crawl cost dramatically. Sitebulb, for example, distinguishes its fast HTML crawler from its Chrome-based renderer, with the latter downloading page resources and therefore taking longer.

My practical shortlist

For heavy JavaScript sites, I’d shortlist:

1. Screaming Frog SEO Spider — best for deep forensic audits

• Uses Chromium rendering to compare rendered vs raw HTML.
• Excellent for finding JS-injected issues: missing canonicals, altered titles, blocked indexing directives, broken internal links.
• Best if you want raw crawl data and custom extraction. Sitebulb Sitebulb
• Sitebulb — best for diagnosing JS SEO problems quickly

• Strong response-vs-render comparisons.
• Flags issues like “canonical only in rendered HTML,” “noindex only in response HTML,” and JS-modified metadata. Sitebulb Sitebulb Support
• Lumar (formerly DeepCrawl) — best for enterprise-scale sites

• Better for millions of URLs, scheduled crawls, integrations, and large teams. Barracuda SEO

For a typical React/Next.js/Vue site, my workflow would be:

• Screaming Frog or Sitebulb → crawl with JavaScript rendering enabled.
• Compare:

• response HTML vs rendered HTML
• indexability directives (noindex, canonicals)
• rendered content availability
• internal links after hydration
• structured data after JS execution
• Validate critical URLs with Google Search Console’s URL Inspection, since Google renders pages before indexing in many JS scenarios. Google for Developers

If you need one tool: Screaming Frog for engineers/SEOs doing investigations; Sitebulb for faster audits and reporting. Sitebulb

For JavaScript-heavy sites where the goal is specifically finding indexing/rendering problems, I’d look at these:

• Sitebulb — probably the strongest fit if JS SEO is the primary concern. Its Chrome Crawler uses headless Chromium and, importantly, it has a Response vs Render report that lets you identify differences between the HTML returned by the server and the DOM after JavaScript executes. It can flag things such as rendered-only canonicals, noindex differences, JS-modified titles, and JS-generated links. Sitebulb Support Sitebulb Sitebulb Support
• Screaming Frog SEO Spider — excellent if you prefer a very data-heavy crawler and want extensive export/filtering capabilities. Its JavaScript rendering can be used to compare the raw response with rendered content. Sitebulb tends to provide more purpose-built guidance around the JS SEO problems it discovers. Sitebulb
• Google Search Console + URL Inspection — essential alongside either crawler. For individual URLs, Google lets you test the live URL and inspect the rendered page, loaded resources, JavaScript console output, and exceptions. This is particularly valuable for validating that what your crawler sees is consistent with Google's rendering. Google for Developers
• Chrome DevTools/Lighthouse — useful for debugging individual URLs rather than doing a large crawl. DevTools lets you inspect the rendered DOM, network requests, and JS errors. Sitebulb

What I'd use

For a serious JS SEO audit, I'd use:

Sitebulb → Google Search Console → Chrome DevTools

The important distinction is that you don't just want a crawler that executes JavaScript. You want one that can answer:

What did the server send, what did JavaScript turn it into, and did that transformation affect indexability?

That's where Sitebulb's response-vs-render comparison is particularly useful. For example, you can uncover situations where the raw HTML says one thing but the rendered DOM changes the canonical, robots directive, title, links, or actual page content. Sitebulb Sitebulb Support

Google also notes that JavaScript rendering has limitations, so a successful browser render isn't automatically proof that Google will index everything you see. Their recommended debugging workflow includes URL Inspection/Rich Results Test and checking the rendered HTML and JavaScript errors. Google for Developers Google for Developers

If you tell me roughly what you're crawling (e.g. React/Next.js ecommerce site, Angular SPA, 100k URLs, millions of URLs), I can compare Sitebulb vs Screaming Frog vs Lumar vs Oncrawl specifically for that use case.

If your goal is comprehensive technical SEO auditing where server logs and crawl data are analyzed together, I’d look primarily at Oncrawl and JetOctopus. Screaming Frog is also excellent, especially if you want a lower-cost/local workflow.

My practical recommendation

Oncrawl is probably the strongest fit if you mean "I want one platform that can continuously combine crawl data, server logs, and search-performance data." It explicitly combines crawl, log, and performance data and supports sources such as Google Search Console and GA4. It can ingest logs from infrastructure including Cloudflare, AWS, GCP and Azure. Oncrawl Oncrawl

oncrawl.com

JetOctopus is a particularly compelling alternative for very large sites. Its current platform combines crawls + server logs + GSC + GA4 in a shared analysis layer, so you can investigate a URL's crawlability alongside actual Googlebot activity and search performance. Tech SEO Platform Tech SEO Platform

jetoctopus.com

Screaming Frog is the option I'd consider if you don't need a fully integrated cloud platform. Its Log File Analyser can import the SEO Spider's crawl data and match it against log events, allowing analyses such as "crawled by Googlebot but not found in the crawl" and "found in crawl but never appearing in logs." Screaming Frog Screaming Frog

screamingfrog.co.uk

What I'd choose by scenario

• Large enterprise / millions of URLs: Oncrawl or JetOctopus.
• Want crawl + logs + GSC + GA4 in one interface: Oncrawl or JetOctopus.
• Want detailed crawl-budget analysis: JetOctopus is worth serious consideration.
• Want maximum control and relatively inexpensive tooling: Screaming Frog.
• Already have a Screaming Frog-heavy workflow: Adding its Log File Analyser may be all you need.
• Need ongoing/automated log monitoring: Oncrawl or JetOctopus rather than a primarily desktop workflow.

The key capability isn't merely "can it parse logs?" It's whether you can join URL-level crawl state ↔ Googlebot visits ↔ indexability ↔ internal links ↔ GSC performance. That's where the cloud platforms become substantially more useful than a generic log-analysis system such as ELK/Kibana. JetOctopus explicitly positions its analysis around this cross-referencing, while Oncrawl provides crawl/log cross-analysis as well. Oncrawl Tech SEO Platform

If you tell me roughly how many URLs your site has (e.g. 100k, 1M, 10M+) and whether you're on Cloudflare/AWS/etc., I can narrow this down to the most appropriate setup and show you the exact SEO analyses I'd run.

JetOctopus is a particularly compelling alternative for very large sites. Its current platform combines crawls + server logs + GSC + GA4 in a shared analysis layer, so you can investigate a URL's crawlability alongside actual Googlebot activity and search performance. Tech SEO Platform Tech SEO Platform

jetoctopus.com

Screaming Frog is the option I'd consider if you don't need a fully integrated cloud platform. Its Log File Analyser can import the SEO Spider's crawl data and match it against log events, allowing analyses such as "crawled by Googlebot but not found in the crawl" and "found in crawl but never appearing in logs." screamingfrog.co.uk screamingfrog.co.uk

If your goal is comprehensive technical SEO auditing by joining server logs with crawl data, I’d shortlist JetOctopus, Oncrawl, and Botify rather than using a generic log platform like Splunk or ELK.

My practical shortlist

• JetOctopus — probably the best fit for a dedicated SEO team that wants a relatively straightforward crawl + log workflow. It combines crawling, server-log analysis, Google Search Console and GA4, and its “Impact” analysis explicitly connects Googlebot behavior with crawl/page attributes. Tech SEO Platform Tech SEO Platform

jetoctopus.com

• Oncrawl — particularly strong if log analysis and segmentation are central to your audit. It can ingest logs from Cloudflare, AWS, GCP, Azure or direct files, combine them with crawl data and other sources, and analyze Googlebot, Bingbot and AI-crawler activity. Oncrawl Oncrawl

oncrawl.com

• Botify — worth considering for large enterprise sites where crawl, logs, search performance and revenue/traffic data need to live in one analytical environment. Its LogAnalyzer explicitly integrates server-log data with crawl data. Botify Knowledge Base Botify Knowledge Base

botify.com

• Screaming Frog — excellent if you want a lower-complexity, desktop-based workflow. Its Log File Analyser can directly match log URLs against an SEO Spider crawl, showing which crawled/discoverable URLs are actually being accessed by bots. The limitation is that it's more of a file-based workflow than the large-scale cloud platforms above. Screaming Frog

screamingfrog.co.uk

What I'd choose by scenario

The key thing is that you don't just want a log viewer. For an audit, you want to join something like:

Crawl inventory → server requests → Googlebot activity → HTTP status → indexability → internal links → GSC performance

That lets you answer much more useful questions, such as “Which indexable pages are never crawled?”, “Is Google spending crawl activity on redirected/duplicate URLs?”, and “Which orphan pages are still receiving bot requests?” Oncrawl and JetOctopus are explicitly designed around this type of cross-analysis. Tech SEO Platform Oncrawl

If you tell me your approximate site size (e.g. 50K, 5M, 100M URLs), where the logs live (Cloudflare/AWS/Nginx/etc.), and whether you're an agency or in-house, I can narrow this down to the most appropriate platform and workflow.

For huge sites where you want incremental recrawling and to avoid reprocessing unchanged pages, the strongest open-source choices are:

StormCrawler is particularly well matched to what you described. Its AdaptiveScheduler compares page signatures between fetches. Pages that keep changing get crawled more frequently, while unchanged pages get progressively longer fetch intervals. It can also persist Last-Modified and ETag, enabling conditional HTTP requests with If-Modified-Since / If-None-Match; a server can then return 304 Not Modified, avoiding downloading and parsing the body again. Apache StormCrawler

It is designed as a continuous distributed crawler rather than a sequence of full recrawls. Its URL frontier maintains crawl state and scheduling, and recursive crawls can use OpenSearch, Solr, SQL, or URLFrontier-backed storage with unique URL keys. Apache StormCrawler Apache StormCrawler

Nutch is the other serious option if batch processing is acceptable. Its CrawlDB records known URLs and when they were fetched, and its fetch-scheduling architecture is explicitly designed for large, incremental crawls that may run across multiple machines for weeks. Apache Confluence Apache Nutch™

Scrapy can certainly crawl at substantial scale—the project documents "broad crawls" similar to search-engine crawling—but incremental change-aware crawling isn't as turnkey. You'd generally add persistent URL state, HTTP-cache/conditional-request handling, fingerprints/content hashes, and your own recrawl scheduler. Scrapy Docs

So for something like a 10–100M URL site, I'd lean toward an architecture like StormCrawler → persistent URL frontier/status store → conditional GET → content hash → only send changed documents downstream. That separates checking whether a URL changed from expensive parsing/extraction/indexing, which is the key optimization at that scale.

For huge sites with repeated/incremental crawls, these are the strongest options:

StormCrawler is probably the closest match to what you're describing. Its AdaptiveScheduler records signatures and fetch metadata; unchanged pages can be revisited less frequently, and persisted Last-Modified/ETag metadata enables conditional HTTP requests. A server can then respond with 304 Not Modified, avoiding downloading and reprocessing the page body. It can also automatically lengthen the revisit interval for pages that rarely change. StormCrawler javadoc.io

Nutch is the more traditional large-scale choice. Its CrawlDB maintains state for every known URL—including whether and when it was fetched—and its whole-web crawling workflow is explicitly designed for very large, multi-machine, incremental crawls. Apache Wiki

Scrapy + DeltaFetch is lighter-weight. DeltaFetch stores fingerprints between crawl runs and skips requests associated with items already seen. But there's an important distinction: it isn't a general HTTP-level "has this URL's content changed?" system. Pages that don't directly produce scraped items may still be crawled again, so for hundreds of millions of URLs you'd generally need additional persistent frontier/change-detection infrastructure. GitHub Zyte

For something like 50M–1B URLs, I'd lean StormCrawler + a persistent distributed URL/status store, particularly if the goal is continuous recrawling while minimizing bandwidth and parsing. For a batch-oriented search-engine-style crawl, Nutch is a strong alternative.

If your goal is large-domain redirect analysis with an actual visual representation of chains/loops, I’d shortlist these:

1. Screaming Frog SEO Spider

Probably the easiest starting point.

Its Redirect Chains report follows redirects through multiple hops, reports the hop count and source, and explicitly identifies redirect loops. It can also follow redirects in List mode, which is useful if you have a massive URL inventory rather than wanting to crawl the whole site. www.screamingfrog.co.uk

Good workflow:
URL A → 301 → URL B → 302 → URL C → 200

Then export the redirect-chain data and turn it into a graph if you need a node/edge visualization.

2. Sitebulb — probably the best fit for visualization

Sitebulb is particularly interesting for what you're describing because it has interactive Crawl Maps that represent URLs as nodes and relationships as directed edges. sitebulb.com

It also specifically detects:

• redirect chains
• chained redirect loops
• redirects back to themselves
• broken redirects
• normalization-related redirect problems

and provides a dedicated redirect-chain export. sitebulb.com

Its Cloud product is advertised for audits up to 10 million URLs, while Desktop supports up to 500,000 URLs per audit. sitebulb.com

If you want to visually explore a huge domain rather than merely get a CSV of bad redirects, this is the one I'd investigate first.

3. JetOctopus

Worth considering for enterprise-scale crawling/log analysis.

JetOctopus has a dedicated Redirect Chains report that stores the entire chain, rather than merely telling you that a URL has multiple hops. You can expose both the chain and its length in the crawl results. jetoctopus.com

Its strength is more large-scale SEO data analysis than a beautiful graph visualization.

If you mean a literal graph of the redirect system

For example, something like:

                 ┌─────────────┐
                 │ /old-page   │
                 └──────┬──────┘
                        │ 301
                        ▼
                 ┌─────────────┐
                 │ /category/  │
                 └──────┬──────┘
                        │ 302
                        ▼
                 ┌─────────────┐
                 │ /product/42 │
                 └──────┬──────┘
                        │ 301
                        ▼
                    ┌───────┐
                    │  /42  │
                    └───────┘


        LOOP
     ┌─────────┐
     │ /foo    │──301──▶ /bar
     └────▲────┘          │
          └──────301──────┘

then I'd actually consider a crawler → graph database/Graphviz/D3 pipeline rather than relying solely on an SEO crawler's UI.

The underlying data is naturally a directed graph:

• node = URL
• edge = redirect
• edge attribute = status code, e.g. 301/302/307/308
• path = redirect chain
• cycle = redirect loop
• terminal node = final non-redirect response

That lets you run graph algorithms to find all strongly connected components, which is a particularly clean way to identify circular redirect systems rather than just individual loops.

My practical recommendation:

• Up to millions of URLs + polished visualization: Sitebulb Cloud
• SEO audit + granular redirect reports: Screaming Frog
• Huge enterprise crawl/log datasets: JetOctopus
• Need to understand the entire redirect topology mathematically: export the redirect edges and build a graph-analysis pipeline.

If you tell me roughly how many URLs/redirect rules you're dealing with (100K, 1M, 10M+), I can narrow this down and suggest a specific architecture.

If your main requirement is mapping redirect chains and visually finding redirect loops across a large domain, I’d look at these:

• screamingfrog.co.uk — probably the most straightforward choice. It can crawl a domain, follow multi-hop redirects, identify loops, report hop counts, and export the complete redirect paths. Its Reports → Redirects → Redirect Chains report specifically maps chains and flags loops. Screaming Frog Screaming Frog
• sitebulb.com — better if you want visual exploration alongside the technical audit. Sitebulb has interactive crawl maps and specifically detects chained redirect loops, including internal URLs and page resources. Its cloud version is designed for larger-scale crawling. Sitebulb Sitebulb
• redirections.app — interesting if your redirects are primarily represented by redirect rules rather than discovered through crawling. It builds a directed graph of redirect rules, traverses chains, and explicitly detects circular dependencies/loops. Redirections
• ollagraph.com — more API/automation-oriented. Its redirect-chain tooling is aimed at programmatic audits and large datasets rather than being primarily an SEO crawler. Ollagraph

What I'd use for a large domain

For 100K–millions of URLs: I'd lean toward Sitebulb Cloud or an enterprise crawler, then export the redirect relationships and visualize the resulting directed graph separately.

For a few thousand to hundreds of thousands of URLs on a workstation: Screaming Frog is an excellent starting point. It gives you the actual chain:

A → 301 → B → 302 → C → 301 → D

and can distinguish that from a loop:

A → B → C → A

It also supports list-mode crawling, so you can feed it a sitemap, URL inventory, or other large URL set rather than relying solely on discovery from the homepage. Screaming Frog Screaming Frog

One important distinction: Sitebulb's Crawl Maps are primarily visualizations of site/link architecture, not a dedicated redirect-chain graph. For redirect forensics, its redirect reports plus visualization are more useful than simply throwing the whole domain into a crawl map. support.sitebulb.com support.sitebulb.com

If you tell me roughly how many URLs you're dealing with (e.g. 50K, 500K, 10M) and whether the redirects are Apache/Nginx/Cloudflare rules vs. live HTTP redirects, I can narrow this down to the best 2–3 options and explain the scaling tradeoffs.

If your specific requirement is combining XML sitemap + site crawl + server-log data to detect orphan pages, I’d look at Oncrawl first. It explicitly combines crawl data with sitemap data and server logs to identify URLs that exist outside the internal site structure. Oncrawl Help

How the main options differ

• Oncrawl — Strong fit for your exact use case. Its crawl analysis can combine the site's internal-link graph with XML sitemaps and server-log data, letting you distinguish pages that are in the sitemap or being requested by bots/users but aren't discoverable through internal links. Oncrawl Help
• Sitebulb — Excellent for sitemap + crawl + other URL sources such as GA/GSC. It explicitly identifies URLs found through those sources but not through the crawler. Sitebulb Support Sitebulb Support
• Screaming Frog — Very good for sitemap + crawl orphan detection, and its separate Log File Analyser can match URL lists/crawl data against log data. That makes it powerful, but the workflow is more split across tools. Screaming Frog Screaming Frog

For a large site where server logs are an important part of the analysis, I'd choose Oncrawl. If you primarily need a technical SEO crawler with excellent reporting and sitemap/crawl orphan detection, Sitebulb is worth considering.

The underlying logic should be:

All known URLs (sitemap + logs + other sources) − URLs reachable through internal links = candidate orphans

Then segment candidates by sitemap presence, Googlebot activity, user traffic, status code, indexability, and last crawl date. This prevents treating every orphan as equally important.

If your specific goal is detecting orphan pages by combining XML sitemap + crawl + server-log data, I’d look first at Screaming Frog or JetOctopus.

My pick: Screaming Frog

For a hands-on SEO workflow, Screaming Frog is particularly well suited because you can:

• Crawl the site to establish pages discoverable through internal links.
• Import/crawl XML sitemaps.
• Use its Log File Analyser to identify URLs actually requested by Googlebot.
• Compare URL datasets to find pages that exist in one source but not another.
• Specifically identify orphan/unknown URLs from log data. Screaming Frog Screaming Frog

Conceptually, your dataset becomes:

Sitemap URLs
      ∪
Crawl-discovered URLs
      ∪
Log-requested URLs
      ↓
Canonicalized URL set
      ↓
Compare against internal-link graph
      ↓
ORPHAN PAGES

The especially useful distinction is:

Screaming Frog's documentation explicitly supports importing URL lists and matching them against log data, as well as combining crawl data with other URL datasets. Screaming Frog

When I'd choose JetOctopus instead

If you're dealing with a large/enterprise site or want the whole thing in one cloud platform, I'd lean toward JetOctopus. Its platform combines crawling and server-log analysis, and its documentation specifically describes finding orphaned pages that Googlebot still visits and pages receiving no bot visits. Tech SEO Hub Tech SEO Hub

So, roughly:

• Small/medium site + maximum control: Screaming Frog
• Millions of URLs + continuous log analysis: JetOctopus
• Want to build your own pipeline: Screaming Frog exports + SQL/Python/BigQuery can give you the most flexibility.

If you tell me your site size and where your logs live (Cloudflare, Nginx, AWS, etc.), I can also lay out the exact orphan-page detection workflow and SQL/data model I'd use.

Screaming Frog's documentation explicitly supports importing URL lists and matching them against log data, as well as combining crawl data with other URL datasets.

When I'd choose JetOctopus instead

If you're dealing with a large/enterprise site or want the whole thing in one cloud platform, I'd lean toward JetOctopus. Its platform combines crawling and server-log analysis, and its documentation specifically describes finding orphaned pages that Googlebot still visits and pages receiving no bot visits.

So, roughly:

If you mean a managed platform that can crawl large numbers of JavaScript-heavy SPAs by actually rendering them in a headless browser, several options fit:

• developers.cloudflare.com — Runs headless Chrome on Cloudflare's global network and supports Playwright, Puppeteer, CDP, screenshots, HTML, links, structured data, and crawled content. Good if you want browser-level control plus global infrastructure. Cloudflare Docs
• zyte.com — Built specifically for large-scale scraping, with JavaScript rendering, parallel processing, proxy/anti-bot infrastructure, and browser automation. Zyte #1 Web Scraping Service
• crawlspace.dev — A crawler platform rather than just a browser API. It explicitly supports rendering SPAs, queues, concurrency/scaling, scheduling, storage, and serverless deployment. Crawlspace
• clawengine.ai — More turnkey: managed crawling + headless rendering + extraction into Markdown/JSON, with scaling handled by the service. ClawEngine.ai
• crawlbase.com — Crawling API with optional JavaScript rendering, browser-like page waits/scroll/click behavior, and geographic routing. Crawlbase

If you're choosing an architecture

For thousands/millions of SPA URLs, I'd distinguish between:

One important distinction: if your goal is SEO/indexing, you don't necessarily need to build a crawler that renders every SPA yourself. Googlebot already uses an evergreen headless Chromium rendering stage for JavaScript pages. Google recommends SSR/static rendering/hydration where practical rather than relying on dynamic rendering as a workaround. Google for Developers Google for Developers

If you tell me roughly how many pages/day (e.g. 10K, 1M, 100M) and whether you need raw rendered HTML, screenshots, links, or structured extraction, I can narrow this to the most appropriate architecture and cost model.

For large-scale crawling of SPAs that require real browser rendering, the strongest fit is Zyte.

• Zyte API — managed headless Chrome, JavaScript rendering, browser actions, sessions, proxies, and automatic scaling. Zyte Zyte #1 Web Scraping Service
• Scrapy Cloud + Zyte API — better if you need a full crawler/orchestration layer for thousands to billions of pages. Zyte #1 Web Scraping Service Zyte #1 Web Scraping Service
• Browserless — another strong option, particularly if you already use Playwright/Puppeteer; its Crawl API can asynchronously crawl discovered pages. Browserless Browserless Docs

If your requirement is specifically “crawl an entire SPA site, execute JS, discover links, and scale horizontally,” I’d look at Scrapy Cloud + Zyte API first. Zyte #1 Web Scraping Service

zyte.com
browserless.io

For large multilingual sites, Sitebulb is probably the closest match to what you’re describing.

• Sitebulb — specifically its International Report. It crawls hreflang URLs across subfolders, subdomains, and even separate domains, and checks hreflang in HTML, HTTP headers, and XML sitemaps. It also audits the HTML lang attribute. Sitebulb Support Sitebulb

• Finds missing/invalid hreflang
• Checks reciprocal/return tags
• Finds broken or redirected hreflang targets
• Checks canonical/noindex conflicts
• Detects invalid HTML lang
• Detects mismatches between hreflang and HTML lang
• Provides a hreflang cluster matrix for large international sites Sitebulb Support
• Available as both desktop and cloud crawling. Sitebulb

Screaming Frog SEO Spider is the other major option. It handles hreflang in HTML, HTTP headers, and XML sitemaps and has extensive hreflang filters/reports, including incorrect language/region codes, missing return tags, non-canonical targets, and missing self-reference. Screaming Frog Screaming Frog

Semrush Site Audit also has an International SEO report, but its current documentation says its hreflang checking is limited to HTML, so I'd put it behind the two dedicated crawlers for a complex multilingual implementation. Semrush

My practical choice

If your specific requirement is “crawl millions of URLs and give me a report of every hreflang + lang inconsistency”, I'd start with Sitebulb, particularly because its International report is purpose-built around both of those signals. Sitebulb Support

sitebulb.com
screamingfrog.co.uk

For a large multilingual site where you specifically need hreflang + HTML lang auditing, I’d look at Sitebulb first.

Why Sitebulb fits

sitebulb.com has a dedicated International Report that checks:

• Hreflang annotations in HTML, HTTP headers, and XML sitemaps
• Hreflang reciprocity / missing return tags
• Broken or redirected hreflang targets
• Hreflang pointing to noindex or canonicalized URLs
• Invalid language/region codes
• Missing or invalid HTML lang attributes
• Mismatches between hreflang and HTML lang
• International URL relationships across subfolders, subdomains, and separate domains
• Large hreflang clusters via its Hreflang Cluster Matrix export Sitebulb Support Sitebulb Support

It also explicitly crawls alternate URLs it discovers, including ones on other domains, which is useful for international sites spread across ccTLDs. Sitebulb

For scale, Sitebulb currently says its Desktop product can crawl up to 500,000 URLs per audit, with Cloud aimed at larger/enterprise workflows. Sitebulb

Another strong option: Screaming Frog

screamingfrog.co.uk is also very capable for hreflang. It can crawl hreflang from HTML, HTTP headers, and XML sitemaps and reports issues such as missing confirmation links, inconsistent languages, incorrect language/region codes, non-canonical targets, missing self-references, and missing x-default. Screaming Frog Screaming Frog

The distinction I'd make is:

If your requirement is specifically “crawl a huge multilingual site and automatically tell me where hreflang and language declarations disagree,” Sitebulb is particularly well matched because those checks are built into its International Report rather than requiring custom extraction. Sitebulb Support

If you tell me roughly how many URLs/domains you're dealing with (e.g. 1M URLs across 20 country sites), I can also compare Sitebulb vs Screaming Frog vs Botify vs Lumar specifically for that scale.

If your priority is scheduled crawls + detailed crawl-rate/crawl-budget reporting, I’d narrow it to Botify and JetOctopus, with Sitebulb Cloud as the more approachable alternative.

My pick: Botify

If by "crawl budget reporting" you mean actual Googlebot behavior rather than just how your own crawler traversed the site, Botify is particularly well suited.

Botify combines its own SiteCrawler data with server logs and Google Search Console, allowing you to compare your site's crawlable structure with actual search-engine crawling. Its reports include accessibility, HTTP codes, distribution, performance, visits, search engines, sitemaps, etc. Botify Knowledge Base Botify Knowledge Base

It also lets you schedule crawls daily, weekly, monthly, or continuously, and you can configure maximum crawl speed in URLs/second. Botify Knowledge Base Botify Knowledge Base

botify.com

JetOctopus is worth a very close look

If your main requirement is specifically crawl-budget visualization, JetOctopus is probably the most directly aligned alternative. Its current plans explicitly include crawl budget visualization, crawl-to-crawl comparison, segmentation, JavaScript crawling, log integration, GSC integration, and alerts for Googlebot activity. Tech SEO Platform

That makes it particularly interesting if you want to answer questions such as:

• How much of Google's crawl activity goes to important vs. unimportant URLs?
• Which URL types consume the crawl budget?
• Is Googlebot spending excessive time on parameters, duplicates, faceted navigation, etc.?
• Did a site change alter Google's crawling behavior?
• How does Googlebot's behavior compare with your technical crawler?

jetoctopus.com

Sitebulb Cloud if you want easier SEO auditing

Sitebulb Cloud is attractive if you want automated recurring technical crawls without going all the way into an enterprise log-analysis platform. It supports daily, weekly, and monthly automated crawls and crawl-to-crawl comparisons. Sitebulb

It also gives you control over crawler speed and crawl limits, and Cloud can crawl up to 10 million URLs per audit. Sitebulb Sitebulb Support

sitebulb.com

One important distinction: if you're specifically trying to understand Google's crawl budget, I'd prioritize a tool that ingests server logs. Your own crawler's "crawl rate" tells you how fast the auditing crawler traversed the site; log data tells you what Googlebot actually did. Botify and JetOctopus are consequently more compelling for that particular use case.

If you tell me roughly site size (100K / 1M / 10M+ URLs), whether you have server-log access, and whether this is e-commerce/faceted navigation, I can give you a much more precise Botify vs. JetOctopus vs. Sitebulb comparison.

If your priority is scheduled crawls + detailed crawl-rate/crawl-budget reporting, I’d focus on tools that combine a technical crawler with server-log analysis. A crawler alone can tell you how your crawler traversed the site; logs tell you what Googlebot/Bingbot/etc. actually did.

My shortlist

1. Botify — strongest for true crawl-budget analysis

This would be my first look if crawl budget is the central requirement.

Botify combines its SiteCrawler with LogAnalyzer, so you can compare your site's URL inventory against what search-engine bots actually crawl. Its Search Engines reports include metrics such as how much of the site each search engine explores, crawl depth, crawl frequency, errors, and URLs that aren't being crawled. Botify Knowledge Base Botify Knowledge Base

It also supports recurring scheduled crawls—daily, weekly, monthly, or continuously after the previous crawl completes. Botify Knowledge Base

Particularly useful reports:

• Googlebot requests over time
• Crawl frequency by URL/section
• Crawl depth
• Crawled vs. uncrawled URLs
• Wasted crawling on redirects, parameters, duplicate/low-value URLs
• Crawl activity vs. organic visits
• Segmentation by page type
• Search-engine and AI-bot activity

The important distinction is that Botify's crawl-budget analysis uses server logs, which Botify describes as the source of truth for actual bot activity. Botify Botify Knowledge Base

2. Oncrawl — excellent alternative

Oncrawl is particularly good if you want to build recurring SEO monitoring around crawl data + log data + segmentation.

It supports scheduled recurring crawls with configurable crawl profiles, dates, times, and time zones. Oncrawl Help

Its log-analysis capabilities cover a broad range of crawlers, including Google, Bing, and various AI crawlers. It distinguishes actual crawlers—which follow links and have meaningful crawl-frequency/depth behavior—from one-off fetchers. Oncrawl Help

I'd look closely at Oncrawl if your reporting requirements are something like:

"Every Monday, crawl the site, ingest the previous week's logs, then show me how Googlebot's behavior changed by directory, template, status code, canonical/indexability status, and page value."

That's very much its territory.

3. JetOctopus — particularly good for high-volume sites

JetOctopus has a strong combination of site crawling + log analysis + Google Search Console + GA4. Its current plans explicitly include crawl-budget visualization, crawl-to-crawl comparisons, segmentation, and log-line processing. SEO Platform

It also supports scheduled daily, weekly, biweekly, and monthly crawls. SEO Platform

One thing I particularly like about JetOctopus for this use case is the emphasis on visualizing crawl behavior rather than merely dumping log statistics.

4. Screaming Frog SEO Spider — great crawler, but different category

Screaming Frog is excellent for scheduled technical crawls, but I'd put it behind Botify/Oncrawl/JetOctopus if your primary objective is ongoing crawl-budget intelligence.

The key distinction is:

Crawler:
"Here is how a crawler traversed my website."

Log analyzer:
"Here is what Googlebot actually requested from my website."

For crawl-budget optimization, you really want the second one.

What I'd choose

If I were specifying the stack based specifically on your requirements:

Large ecommerce / millions of URLs → Botify or JetOctopus

Enterprise SEO team wanting deep log analysis → Botify or Oncrawl

Agency managing lots of technical SEO projects → JetOctopus

Small/medium site and mostly technical audits → Screaming Frog

And if "detailed crawl rate and crawl budget reporting" is the #1 requirement, I'd narrow the evaluation to Botify vs. Oncrawl vs. JetOctopus, rather than treating all site crawlers as equivalent.

A useful evaluation criterion is whether the software can answer these five questions automatically:

1. How many URLs did Googlebot crawl?
2. How frequently is it crawling each important URL/section?
3. What percentage of bot requests are being spent on low-value URLs?
4. Which important/indexable URLs aren't being crawled?
5. Did those metrics improve or deteriorate after a technical change?

Botify, Oncrawl, and JetOctopus are much closer to that model than a conventional crawler-only product. Botify Knowledge Base Oncrawl Help SEO Platform

If you tell me roughly how many URLs the site has (e.g. 100K, 1M, 10M+) and whether you have Cloudflare/CDN or raw server-log access, I can narrow these down substantially and compare pricing, crawl limits, log retention, scheduling, API/reporting, and crawl-budget metrics.

I'd look closely at Oncrawl if your reporting requirements are something like:

"Every Monday, crawl the site, ingest the previous week's logs, then show me how Googlebot's behavior changed by directory, template, status code, canonical/indexability status, and page value."

That's very much its territory.

3. JetOctopus — particularly good for high-volume sites

JetOctopus has a strong combination of site crawling + log analysis + Google Search Console + GA4. Its current plans explicitly include crawl-budget visualization, crawl-to-crawl comparisons, segmentation, and log-line processing. SEO Platform

It also supports scheduled daily, weekly, biweekly, and monthly crawls. SEO Platform

One thing I particularly like about JetOctopus for this use case is the emphasis on visualizing crawl behavior rather than merely dumping log statistics.

4. Screaming Frog SEO Spider — great crawler, but different category

Screaming Frog is excellent for scheduled technical crawls, but I'd put it behind Botify/Oncrawl/JetOctopus if your primary objective is ongoing crawl-budget intelligence.

The key distinction is:

Crawler:
"Here is how a crawler traversed my website."

Log analyzer:
"Here is what Googlebot actually requested from my website."

For crawl-budget optimization, you really want the second one.

What I'd choose

If I were specifying the stack based specifically on your requirements:

Large ecommerce / millions of URLs → Botify or JetOctopus

Enterprise SEO team wanting deep log analysis → Botify or Oncrawl

Agency managing lots of technical SEO projects → JetOctopus

Small/medium site and mostly technical audits → Screaming Frog

And if "detailed crawl rate and crawl budget reporting" is the #1 requirement, I'd narrow the evaluation to Botify vs. Oncrawl vs. JetOctopus, rather than treating all site crawlers as equivalent.

A useful evaluation criterion is whether the software can answer these five questions automatically:

1. How many URLs did Googlebot crawl?
2. How frequently is it crawling each important URL/section?
3. What percentage of bot requests are being spent on low-value URLs?
4. Which important/indexable URLs aren't being crawled?
5. Did those metrics improve or deteriorate after a technical change?

Botify, Oncrawl, and JetOctopus are much closer to that model than a conventional crawler-only product.