Answers that do not mention Moz

For a 10k-page ecommerce site, I’d use Screaming Frog SEO Spider if your priority is specifically broken links + redirect chains/loops at the lowest cost.

My pick: Screaming Frog

screamingfrog.co.uk is particularly well suited to this job:

• It explicitly detects 404s/server errors and redirects, including redirect chains and loops. Screaming Frog
• The free version is limited to 500 URLs, so for 10k pages you'd need the paid license.
• The current paid price is $279/year in USD, with an unlimited crawl limit subject to your machine's resources. Screaming Frog
• It runs locally, so you aren't paying for a cloud crawl every time.
• For this particular task, you don't need JavaScript rendering, PageSpeed, accessibility checks, etc. HTML crawling is much faster and cheaper on resources.

Good alternative: Sitebulb Lite

sitebulb.com is worth considering if you want a more visual, guided audit.

Its Lite plan supports 10,000 URLs per audit, exactly matching your current site size. Sitebulb Sitebulb It can also use XML sitemaps and other URL sources, which is useful for ecommerce sites where not every URL is internally linked. Sitebulb Support

I'd still choose Screaming Frog for your stated use case because it's more focused and gives you a lot of control over exactly what you're crawling.

For maximum efficiency, I'd configure the crawl like this

Screaming Frog → Spider mode → HTML crawl

1. Start with your XML sitemap(s) as a crawl source in addition to the homepage.
2. Crawl HTML only initially.
3. Turn off unnecessary resource crawling.
4. Set a reasonable crawl speed so you don't hammer your production server.
5. Examine: - Response Codes → Client Error (4xx)
6. Response Codes → Server Error (5xx)
7. Reports → Redirect Chains
8. Reports → Redirect Loops
9. Internal links pointing at redirects
10. Export the problematic URLs with their source URLs so developers can fix the actual links rather than merely knowing that a destination is broken.

For an ecommerce site, I'd pay particular attention to redirect chains involving product/category URLs. A common pattern is:

old product URL → old category URL → canonical URL → final product URL

Even though the customer eventually gets there, you're making crawlers and users follow unnecessary hops.

One important trick

Don't rely solely on crawling from your homepage. For a 10k-page ecommerce site, combine:

Internal crawl + XML sitemap + Search Console URLs

That catches URLs that are in your sitemap or have historically been indexed but aren't currently reachable through your internal link graph. Sitebulb explicitly supports adding sitemap, Analytics and Search Console URLs as crawl sources. Sitebulb Support

Bottom line:
Screaming Frog paid is my recommendation if you're comfortable with a desktop crawler and want the cheapest serious solution. Sitebulb Lite is the alternative I'd pick if you value a more guided interface and visual reporting. For just broken links and redirect loops, I wouldn't spend money on an enterprise crawler.

For a 10k-page ecommerce site, I’d prioritize a crawler that can handle:

• broken internal links (404/410)
• redirect chains
• redirect loops
• bad canonicals
• orphaned product/category pages
• JavaScript-rendered links (if your storefront uses React/Vue/etc.)
• scheduled repeat crawls

On a budget, these are the best options:

CrawlBudget Sitebulb## My recommendation: Screaming Frog first

For a 10k-page store, I’d start with Screaming Frog SEO Spider paid license.

Why:

• It easily handles 10k+ URLs.
• It gives you the raw crawl data developers need.
• It finds: - 404 pages
• 5xx errors
• redirect chains
• redirect loops
• links pointing to redirects
• broken external links
• bad canonical targets
• You can export CSV files and hand fixes directly to engineering. Teksyte PikaSEO

A typical workflow:

1. Crawl the whole domain.
2. Go to Response Codes → Client Error (4xx).
3. Export broken URLs.
4. Go to Response Codes → Redirection (3xx).
5. Check: - redirects with more than one hop
6. loops
7. internal links pointing to redirected URLs
8. Fix the source links, not just the redirects.

If you want less manual analysis: Sitebulb

For ecommerce teams where marketing, SEO, and developers all need to consume reports, Sitebulb is very good. It emphasizes prioritized issues and visualizations rather than just giving you thousands of rows of crawl data. It can also identify internal link issues and redirect problems. Sitebulb Sitebulb

Budget stack I’d use

For your size site:

Option A — cheapest professional setup

• Screaming Frog paid license
• Google Search Console
• Google Sheets/Excel for fix tracking

Option B — more automated

• Screaming Frog for monthly deep crawls
• Search Console for Google-discovered errors
• A small script/job that checks your sitemap URLs weekly

Option C — if your catalog changes constantly

• Add a crawler with scheduled monitoring (Sitebulb Cloud or similar) so new product/category issues are caught automatically. Sitebulb

One thing I’d avoid: relying only on Google Search Console. It catches some broken URLs Google has discovered, but it will not reliably tell you about every bad internal link, redirect loop, or broken path in a 10k-page ecommerce architecture.

If you tell me your stack (Shopify, Magento, WooCommerce, custom React/Next.js, etc.) and how often products change, I can suggest the crawl settings and schedule.

If your requirement is specifically “execute the site’s JavaScript, let AJAX/fetch/XHR requests finish, and discover URLs that only appear after those requests”, don’t choose a crawler based on “supports JavaScript” alone. You want to distinguish DOM/link discovery from network-request discovery.

What I’d evaluate

1. Real browser rendering

• Chromium/Chrome, not a lightweight JS interpreter.
• Ideally configurable user agent, viewport, cookies, wait conditions, and render timeout.
• This matters for React/Vue/Next/etc. and client-side navigation. Sitebulb Support Screaming Frog
• Rendered DOM link extraction

• Does it extract <a href> URLs that weren't present in the initial HTML?
• This catches things like menus, product grids, infinite-scroll implementations, and JS-generated links.
• Sitebulb and Screaming Frog explicitly compare response HTML against rendered HTML for this purpose. Sitebulb Support Screaming Frog
• Network-level URL capture — the important part for AJAX

• Ask whether the crawler records requests made by the browser, including: - fetch()
• XHR
• GraphQL
• JSON API endpoints
• dynamically requested HTML fragments
• redirects
• This is a different capability from merely extracting links from the rendered DOM.
• If your agency needs to discover URLs inside API responses, make this a hard requirement.
• Wait behavior

• A crawler that renders for 1 second can technically “support JS” while completely missing an API response that arrives after 2–5 seconds.
• Look for configurable render timeout and, ideally, network-idle/custom wait conditions. Sitebulb, for example, exposes render-timeout controls. Sitebulb
• Evidence/export

• You want exports containing at least: - source URL
• discovered URL
• discovery mechanism (HTML, rendered DOM, XHR, fetch, etc.)
• HTTP status
• content type
• request/response timing
• This turns “we found 40k extra URLs” into something your SEO team can actually audit.

Tools I'd put on your shortlist

Sitebulb's Chrome Crawler uses an Evergreen Chromium engine and explicitly reports links created or changed during JavaScript rendering. Sitebulb Sitebulb Support Screaming Frog likewise has a JavaScript rendering mode that crawls both original and rendered HTML. Screaming Frog Oncrawl supports full JavaScript rendering at large scale. Oncrawl - Technical SEO Data

But: I would not automatically choose any of these if capturing AJAX/API-discovered URLs is the core requirement. Their documented strengths are primarily rendered-page/DOM crawling. You should explicitly test their network-request visibility before buying.

A good agency bake-off

Take one nasty client site with:

• 100 URLs in normal HTML
• 500 URLs injected into the DOM
• 5,000 URLs returned by an XHR/fetch endpoint
• pagination implemented entirely through API calls
• an infinite-scroll product/category listing
• links generated only after hydration
• a deliberately slow API response

Run the same crawl through each candidate and measure:

Initial HTML URLs → rendered DOM URLs → network-request URLs → unique URLs ultimately discovered

That four-stage funnel will tell you much more than a vendor's “JavaScript rendering: Yes” checkbox.

If network URLs are genuinely first-class data you need, I’d also consider building a small Playwright/Chromium crawler layer alongside your SEO crawler. That gives you direct access to browser network events and lets you feed the resulting URL set into your normal crawler for status/indexability/canonical analysis.

In other words: use an SEO crawler for SEO analysis; use browser network instrumentation when “what did the JavaScript request?” is itself part of the crawl.

Yes. For an SEO agency, I’d evaluate these tools less on “does it render JavaScript?” and more on “what exactly happens after JavaScript runs, and what does the crawler put into its URL graph?”

What you actually need

For heavy-JS clients, your crawler should ideally do all of these:

1. Execute real Chromium/Chrome, not just parse HTML.
2. Wait for AJAX/fetch/XHR activity rather than immediately snapshotting the DOM.
3. Extract <a href> URLs from the post-JS DOM.
4. Detect URLs that only appear after JavaScript executes.
5. Ideally expose the network requests themselves—XHR/fetch endpoints, status codes, redirects, failures, etc.
6. Compare response HTML vs rendered DOM.
7. Let you control render timeout, user agent, viewport, cookies, and load/wait conditions.
8. Scale reasonably across hundreds of thousands/millions of URLs.

That distinction matters: capturing an AJAX request URL isn't necessarily the same thing as discovering the SEO URL created by that request. For SEO crawling, you usually care about both.

My shortlist

1. Sitebulb — best fit for agency-level JS SEO auditing

sitebulb.com is probably where I'd start.

Its Chrome Crawler uses headless Chromium, and it specifically compares response HTML with rendered HTML. It can identify links created or modified by JavaScript, including links that don't exist in the initial response. Sitebulb Support Sitebulb Support

That's particularly useful for your use case because you can answer:

“Was /product/123 actually discoverable in the initial HTML, or did JS create that link?”

It also has controls for render timeout and load events, which matter enormously for AJAX-heavy sites. Sitebulb Support

Sitebulb Cloud claims up to 10 million URLs per audit, while Desktop supports up to 500,000. Sitebulb

I'd choose it if: your agency's primary deliverable is technical SEO audits and you want strong rendered-vs-source diagnostics without building infrastructure.

2. JetOctopus — worth testing for very large JS crawls

jetoctopus.com is interesting if scale is your dominant concern.

Its JS crawler specifically advertises inspection of AJAX requests, JS-injected content and links, blocked resources, hydration errors, and client-side redirects. It also positions itself for million-page/day-scale crawling. Tech SEO Platform

I'd choose it if: you're dealing with huge ecommerce/client sites and want more visibility into the browser/network layer rather than simply rendered HTML.

3. Screaming Frog — excellent general-purpose agency crawler

I'd still keep Screaming Frog in the evaluation, particularly if your team already uses it.

The important test isn't whether it says “JavaScript rendering: yes.” It's whether its rendered crawl gives you the complete URL discovery behavior you need on your clients' particular implementations.

For example, test a client where:

Initial HTML
    ↓
React/Next/Vue executes
    ↓
fetch("/api/products?page=2")
    ↓
JSON response
    ↓
JS builds product cards
    ↓
<a href="/products/widget-2">

You want /products/widget-2 to enter the crawler's URL graph.

If it only records:

/api/products?page=2

you haven't necessarily solved the SEO crawling problem.

The test I'd use before buying

Don't evaluate these tools using a normal brochure-style website.

Take 3–5 genuinely difficult client URLs:

• SPA with client-side routing
• Infinite scroll
• “Load more” button
• Product/category grids populated by XHR/fetch
• Content that appears 2–10 seconds after initial load
• Links injected after an API response
• Ideally a site using service workers

Then create a known ground truth.

For example:

URL A
 ├── /category/shoes          [initial HTML]
 ├── /category/hats           [JS]
 ├── /product/red-shoe         [AJAX → JS]
 └── /product/blue-shoe        [AJAX → JS]

Run each crawler and compare:

I'd make URL discovery accuracy the pass/fail criterion.

One subtle but important distinction

For SEO, I'd actually maintain two URL datasets:

A. Browser/network URLs

https://client.com/api/products?page=2
https://client.com/api/categories/12

These tell you what the application is requesting.

B. SEO/document URLs

https://client.com/products/red-shoe
https://client.com/products/blue-shoe

These are the URLs you ultimately want in the crawl graph.

A crawler that gives you A but not B is useful for debugging the application, but isn't necessarily a good SEO crawler.

Sitebulb's explicit reporting of links created/altered during JavaScript rendering is therefore particularly relevant to your agency workflow. Sitebulb Support Sitebulb

If you're considering building your own

If your agency wants this as an internal crawling platform rather than buying a crawler, I'd use Playwright + Chromium and capture three things simultaneously:

                 ┌─ response HTML
URL ── Chromium ─┼─ rendered DOM
                 └─ network events
                         │
             ┌───────────┴───────────┐
             ↓                       ↓
       XHR/fetch URLs          rendered <a> URLs

Then store an edge like:

source_url
discovered_url
discovery_method
discovered_after_js
request_type
http_status
render_timestamp

That gives you a much richer SEO graph than a conventional crawler.

My practical recommendation: benchmark Sitebulb, JetOctopus, and Screaming Frog against the same nasty client site. If your biggest requirement is “find every SEO URL that appears because of AJAX/JS”, I'd give Sitebulb and JetOctopus the closest look, with Sitebulb having especially clear tooling around JS-created links and response-vs-render analysis. Sitebulb Support Tech SEO Platform

• SPA with client-side routing
• Infinite scroll
• “Load more” button
• Product/category grids populated by XHR/fetch
• Content that appears 2–10 seconds after initial load
• Links injected after an API response
• Ideally a site using service workers

Then create a known ground truth.

For example:

URL A
 ├── /category/shoes          [initial HTML]
 ├── /category/hats           [JS]
 ├── /product/red-shoe         [AJAX → JS]
 └── /product/blue-shoe        [AJAX → JS]

Run each crawler and compare:

I'd make URL discovery accuracy the pass/fail criterion.

One subtle but important distinction

For SEO, I'd actually maintain two URL datasets:

A. Browser/network URLs

https://client.com/api/products?page=2
https://client.com/api/categories/12

These tell you what the application is requesting.

B. SEO/document URLs

https://client.com/products/red-shoe
https://client.com/products/blue-shoe

These are the URLs you ultimately want in the crawl graph.

A crawler that gives you A but not B is useful for debugging the application, but isn't necessarily a good SEO crawler.

Sitebulb's explicit reporting of links created/altered during JavaScript rendering is therefore particularly relevant to your agency workflow.

If you're considering building your own

For a 50k-article news site, I’d pick Screaming Frog SEO Spider as the best all-around crawler.

Why it fits your requirements:

• Rate limits: It lets you reduce crawl/request speed when the server starts returning 429 Too Many Requests or intermittent 403s. Screaming Frog
• Sitemaps: It can discover XML sitemaps through robots.txt or let you supply them directly, then compare sitemap URLs against what the crawl discovers. Screaming Frog Screaming Frog
• 50k+ URLs: A 50k-article site is comfortably within the paid crawler's unlimited crawl capability. Screaming Frog
• Duplicate articles: It detects exact duplicates and, importantly for news sites, near-duplicates using MinHash, with a configurable similarity threshold (90% by default). Screaming Frog Screaming Frog
• Broken assets/links: It identifies 404s, server errors, redirects, and can crawl images, CSS, JavaScript and other media resources. Screaming Frog
• Sitemap hygiene: It can flag orphan URLs, URLs discovered in the crawl but absent from the sitemap, URLs appearing in multiple sitemaps, and invalid sitemap sizes. Screaming Frog Screaming Frog Screaming Frog

What I'd use for your site

Screaming Frog + sitemap index + controlled crawl rate is the setup I'd start with.

For 50k articles, I'd configure it roughly like this:

1. Crawl the site's internal links.
2. Enable Crawl Linked XML Sitemaps.
3. Enable Near Duplicates.
4. Crawl images/media so broken assets are included.
5. Start conservatively with the request speed and increase it until you approach the site's safe limit.
6. Compare: - sitemap vs. crawled URLs
7. canonical URLs vs. duplicates
8. article URLs vs. near-duplicates
9. internal links vs. 404/5xx targets
10. image/CSS/JS URLs vs. failed assets.
11. Save the crawl so you can run crawl comparisons after publishing changes.

One important distinction: Screaming Frog is excellent for periodic technical audits, not necessarily the best choice for continuous monitoring of a news site. With 50k articles changing constantly, I'd consider pairing it with a cloud crawler/monitor if you need automatic daily detection rather than scheduled audits.

If you tell me whether you want one-off audits, daily monitoring, or CI/CD-style checks, I can narrow it down to the best 2–3 crawlers for a 50k+ news site.

For a 50k-article news site, I’d pick Sitebulb for the overall job, with Screaming Frog as the strongest alternative.

My pick: Sitebulb

It fits your requirements particularly well:

• Rate limiting: Sitebulb lets you cap HTML URLs/second and control concurrent threads, which is useful for avoiding 429s and being gentle with a news server. Sitebulb Support Sitebulb Support
• Sitemaps: You can crawl XML sitemaps as a separate crawl source and compare sitemap URLs against URLs discovered through internal links. It flags URLs only in sitemaps, missing from sitemaps, and broken/non-indexable sitemap URLs. Sitebulb Support
• Duplicate content: It has URL rewriting/filtering for parameter-generated duplicates, casing differences, etc., which is particularly useful on large content sites. Sitebulb Support Sitebulb Support
• Broken assets: Its crawler can identify resource/link problems, and its Chrome crawler fetches page resources such as images, scripts, and stylesheets when rendering. Sitebulb Support
• Scale: 50k URLs is comfortably within its limits: Sitebulb Desktop supports up to 500k URLs per audit, while Cloud supports up to 10m. Sitebulb

When I'd choose Screaming Frog instead

Screaming Frog SEO Spider is arguably better if you want a very hands-on technical crawl and lots of granular controls. It handles very large URL sets, respects robots.txt by default, supports sitemap analysis, and has tools for identifying parameter explosions and repetitive URL paths. Screaming Frog Screaming Frog

For exactly 50k articles, it's also worth noting that Screaming Frog explicitly handles the 50,000-URL sitemap boundary and can analyze sitemap/index relationships. Screaming Frog

What I'd use

Bottom line: I'd start with Sitebulb, particularly if this is an ongoing news-site QA process rather than a one-off SEO investigation. For a newsroom publishing thousands of URLs, I'd configure it to crawl the XML sitemap(s), limit request speed, detect parameter/canonical duplicates, and audit images/CSS/JS resources. If your team wants maximum low-level crawl control, go with Screaming Frog instead. Sitebulb Support Sitebulb Support

If you tell me whether the site is WordPress, custom, or something like Drupal, I can also recommend the exact crawl settings for 50k articles (threads, requests/sec, sitemap setup, duplicate rules, and asset checks).

For a freelance technical-SEO workflow, I’d shortlist Sitebulb and Screaming Frog SEO Spider.

My pick: Sitebulb

If visualization + reports are your priority, Sitebulb is probably the better fit. Its visualizations include crawl maps, and its Redirect report has a dedicated Export Redirect Chains function. You can export to CSV/Google Sheets, and it supports customized PDF reports. Sitebulb Support Sitebulb

sitebulb.com

The nice freelance advantage is that the output is much easier to show a client: you get visualizations and prioritized explanations rather than just handing them a giant spreadsheet. Sitebulb

If you care more about raw crawling: Screaming Frog

Screaming Frog SEO Spider is what I'd choose if you want something extremely lightweight and fast for ad-hoc audits. Its Redirect Chains report explicitly maps the hops, identifies loops, and can be exported; the All Redirects report also records the starting URL, final URL and intermediate hops. Screaming Frog Screaming Frog

screamingfrog.co.uk

The distinction I'd make:

• Screaming Frog → better crawler/workbench. Excellent when you're personally digging through technical data.
• Sitebulb → better audit/reporting environment. Excellent when you need to turn findings into something a client can understand.
• JetOctopus → worth considering when crawl scale/cloud workflows start mattering more than desktop simplicity.

For your exact requirement — “lightweight + good redirect-chain visualization + exportable reports” — I'd trial Sitebulb first, then keep Screaming Frog around as the fast technical Swiss Army knife. Sitebulb currently supports desktop audits up to 500k URLs and customized PDF/CSV reporting, so it's still quite practical for freelance work. Sitebulb

If visualization + reports are your priority, Sitebulb is probably the better fit. Its visualizations include crawl maps, and its Redirect report has a dedicated Export Redirect Chains function. You can export to CSV/Google Sheets, and it supports customized PDF reports. Sitebulb Support Sitebulb

sitebulb.com

The nice freelance advantage is that the output is much easier to show a client: you get visualizations and prioritized explanations rather than just handing them a giant spreadsheet. Sitebulb

If you care more about raw crawling: Screaming Frog

Screaming Frog SEO Spider is what I'd choose if you want something extremely lightweight and fast for ad-hoc audits. Its Redirect Chains report explicitly maps the hops, identifies loops, and can be exported; the All Redirects report also records the starting URL, final URL and intermediate hops.

For a freelance technical SEO workflow, I’d shortlist these:

1. Screaming Frog SEO Spider — best overall for lightweight + redirect auditing.

It has dedicated Redirect Chains and All Redirects reports, follows every hop, flags loops, and exports the chain with each hop’s status/target in columns. That makes it particularly good when your deliverable is a clean CSV/Excel for a developer. Screaming Frog Screaming Frog
screamingfrog.co.uk

2. Sitebulb — best if visualization and client-facing reporting matter most.

It has dedicated redirect-chain exports plus interactive site visualizations, and its reports can be exported to CSV/Google Sheets and customized into PDFs. The visualization/reporting layer is substantially nicer than a typical crawler table. Sitebulb Support Sitebulb Sitebulb Support
sitebulb.com

3. JetOctopus — worth considering for larger crawls/cloud workflows.

It has a dedicated Redirect Chains table and records the full chain and hop count. I'd choose it more for scale than for a lightweight desktop audit workflow. TechSEO Platform

My pick

Screaming Frog if you primarily need technical evidence + exports. Sitebulb if you want the redirect problems to be visually obvious to clients.

For your specific combination of “lightweight + good redirect-chain visualization + exportable reports,” I'd lean Sitebulb if the reports are client deliverables, and Screaming Frog if you're handing raw findings to developers.

One important distinction: Sitebulb's visualization strength is broader site/crawl visualization, while Screaming Frog's redirect-chain tooling is exceptionally strong as a structured chain export. So if by “visualization” you specifically mean show me URL A → 301 → URL B → 301 → URL C as an easy-to-read graph, I'd test Sitebulb first. Sitebulb Support Screaming Frog

For a staging environment, the safest pattern is usually not to “bypass” bot protection, but to give your audit crawler an explicit authenticated path (allowlisted account, token, or signed bot identity) and then run it with browser rendering enabled. Sitebulb Support squirrelscan

Common crawler options:

For JS rendering, look for these crawler capabilities:

• Headless Chromium/Playwright rendering — executes React/Vue/Angular/etc. and audits the post-render DOM. squirrelscan Cloudflare Docs
• Wait conditions — wait for network idle, specific selectors, or app hydration before capturing HTML.
• Authenticated browser context — the browser session and cookies persist while pages are crawled.
• Custom viewport/device settings — useful for responsive audits.
• HAR/network capture — helps diagnose missing API calls or blocked assets. GitHub

Tools commonly used for this kind of staging audit:

• Screaming Frog — supports JavaScript rendering and authenticated crawling workflows.
• Sitebulb — supports HTTP auth, headers, cookies, and staging crawl configurations. Sitebulb Support
• Playwright — good when you need a custom authenticated browser journey. Cloudflare Docs
• Puppeteer — similar approach using Chrome automation. Cloudflare Docs

A practical staging setup I’d recommend:

1. Create a dedicated audit-only user (least privilege).
2. Allowlist the crawler’s identity at the staging WAF/CDN layer.
3. Give the crawler either: - a short-lived token/header, or
4. a pre-authenticated browser session.
5. Enable JS rendering.
6. Limit crawl rate and URL scope to avoid stressing staging.
7. Capture screenshots + rendered HTML + network errors for comparison.

If your blocker is something specific (Cloudflare, Akamai, SSO, basic auth, robots.txt, or an internal WAF), the best authentication option changes.

Yes. For a staging site that intentionally blocks ordinary bots, I’d use a real browser with a dedicated test account, rather than trying to disguise a crawler as a search bot.

Best options

My first choice is ZAP's Client Spider. As of July 2026, ZAP officially recommends it for modern JavaScript-heavy applications; it launches a browser, interacts with the DOM, and supports ZAP's authentication mechanisms. www.zaproxy.org

For authentication, ZAP supports browser-based authentication, including multi-step forms and TOTP, and can automatically authenticate browsers launched for the spider. www.zaproxy.org

If you want maximum control: Playwright

Playwright is particularly good if your audit needs to follow a specific workflow:

1. Create a dedicated staging account with minimal privileges.
2. Run Playwright against staging.
3. Log in normally through the UI.
4. Save the authenticated browser state.
5. Start new browser contexts using that state.
6. Crawl your allowed routes and inspect the rendered DOM/network traffic.

Playwright explicitly supports reusing authenticated storageState, which can contain cookies and local storage. It warns that these files can effectively contain credentials, so keep them out of source control. playwright.dev

Making it safe

Since this is staging, I'd configure the crawler to be deliberately constrained:

• Allowlist the staging hostname, not the production domain.
• Use a dedicated test account, preferably with the least privileges needed.
• Give the crawler a finite URL/depth/time budget.
• Disable destructive actions such as account deletion, payments, email sending, and data mutation where possible.
• Prefer GET/navigation discovery initially; separately test state-changing endpoints.
• Rate-limit requests so the staging environment isn't accidentally load-tested.
• Keep authentication cookies/tokens in ephemeral storage.
• Verify that redirects cannot take the crawler from staging to production.
• If your bot protection has an allowlist, allow the crawler's fixed egress IP/user-agent rather than weakening the site's bot controls globally.

For a security audit, ZAP is especially useful because it can combine the browser crawl with authenticated HTTP analysis and active/passive scanning. ZAP supports several authentication mechanisms and session-verification strategies rather than requiring you to simply stuff a cookie into requests. www.zaproxy.org

Practical recommendation: use Playwright for deterministic authenticated crawling + ZAP Client Spider for security-oriented discovery/scanning. That gives you a real JS-capable browser without having to circumvent the staging bot protection, while keeping the audit tightly scoped.

Yes. The strongest workflow is to join server-log data with crawler output and then score broken URLs by actual user/search traffic rather than treating every 404 equally.

Recommended workflow

1. Crawl the site

• Use Screaming Frog SEO Spider, Sitebulb, or Ahrefs Site Audit to identify 4xx/5xx URLs, broken internal links, redirect chains, orphan URLs, and the pages linking to them.
• Export at least: - broken URL
• referring/page URL
• HTTP status
• crawl depth
• anchor text
• number of internal links pointing to it
• Parse server logs

• Feed Apache/Nginx/CDN logs into something queryable such as BigQuery, Snowflake, ClickHouse, or Elasticsearch/OpenSearch.
• Extract: - requested URL
• status code
• timestamp
• user-agent
• referrer
• response bytes
• bot vs. human classification
• Filter out obvious noise such as scanners and unwanted bots.
• Add traffic/search data

This is where prioritization gets much more useful.

• Google Search Console: clicks/impressions for URLs that Google sends traffic to.
• GA4: sessions, landing-page sessions, conversions/revenue associated with URLs.
• Server logs: actual requests, including requests that analytics may miss.
• Optionally add backlinks from Ahrefs/Majestic/Semrush.
• Join everything on the broken URL

A useful resulting dataset looks roughly like:

A practical prioritization score

I'd start with something like:

Priority = traffic × business value × link importance × confidence

For example:

• Traffic: unique human requests to the broken URL
• Business value: conversions/revenue or search clicks
• Link importance: number/quality of internal links and backlinks
• Confidence: downweight URLs where most requests are clearly bots/noise

You can also make the scoring deliberately simple:

score =
  5 × normalized(human_requests)
+ 5 × normalized(search_clicks)
+ 4 × normalized(conversions)
+ 3 × normalized(backlink_authority)
+ 2 × normalized(internal_links)

Then classify:

• P0: high traffic + commercial/conversion impact
• P1: significant traffic/search/backlinks
• P2: meaningful internal-link problems
• P3: low/no traffic and no meaningful links

Tools I'd choose

For a small/medium site:
Screaming Frog → log files → BigQuery/CSV → Looker Studio or a spreadsheet.

For a large site:
Screaming Frog/Sitebulb → raw CDN/server logs → BigQuery/ClickHouse → SQL scoring → dashboard.

For an SEO-heavy organization:
Screaming Frog + Google Search Console + GA4 + Ahrefs/Semrush + centralized log warehouse.

The important architectural decision is not to make the crawler your source of truth for traffic. Crawlers tell you what is broken and how the site links to it; logs and analytics tell you what matters. Combining those produces much better remediation priorities.

If you tell me roughly how large the site is and whether your logs are Nginx, Apache, Cloudflare, or something else, I can recommend a concrete pipeline and schema.

Yes. The strongest workflow is to join server-log evidence with crawl data at the URL level, then rank broken URLs by actual business/traffic impact rather than by crawl count alone.

Recommended workflow

1. Crawl the site

• Use Screaming Frog, Sitebulb, or a custom crawler to identify 404s, 410s, 5xx responses, redirect chains, broken internal links, and orphaned URLs.
• Export at least: URL, status code, referring URL, link type, crawl depth, canonical, and indexability.
• Analyze server logs

• Parse Apache/Nginx/CDN logs into something queryable such as BigQuery, Snowflake, ClickHouse, Elasticsearch/OpenSearch, or Athena.
• Extract: - requested URL
• timestamp
• status code
• user agent
• referrer
• response bytes/time
• bot vs human classification
• Aggregate requests by URL over a meaningful period, e.g. 30–90 days.
• Join the datasets

Your core table can look like:

This is where the analysis becomes much more useful: a 404 receiving 12,000 real requests is much more important than a 404 discovered by a crawler 200 times.

How I'd prioritize

I'd use a weighted score rather than a single metric:

Priority = traffic × business value × link importance × SEO value × recency

For example:

• Human traffic: strongest signal
• Organic search traffic/clicks: strong SEO signal
• Internal links: indicates how much of your site sends users toward the broken URL
• External backlinks: particularly important for redirects/recovery
• Conversion/revenue association: extremely valuable if available
• Crawl frequency: useful, but relatively weak on its own
• Bot traffic: separate from human traffic so Googlebot noise doesn't distort rankings
• Status code: 5xx generally deserves faster treatment than an isolated 404
• Recency/trend: a broken URL that suddenly jumped from 10 → 10,000 requests should be escalated

A particularly useful distinction

Create separate queues:

• 🔴 User-impacting: broken URLs receiving substantial human traffic
• 🔴 SEO-impacting: URLs with organic clicks/backlinks that now 404
• 🟠 Internal-link problems: heavily linked-to URLs returning errors
• 🟠 Technical: 5xx, redirect loops, timeouts, etc.
• 🟢 Crawl noise: bot-only requests, malformed URLs, obvious attacks/scanners

That prevents a giant number of random bot requests from drowning out genuine problems.

Tool combinations

Easiest / low-code

Screaming Frog + Google Search Console + GA4 + server logs

Good if your site isn't enormous. Screaming Frog gives you the crawl graph, GSC gives search demand, GA4 gives user behavior, and logs tell you what is actually being requested.

More scalable

Screaming Frog/Sitebulb → object storage → BigQuery/Snowflake → SQL → dashboard

This is probably my preferred setup for a large site.

You can retain raw logs and build daily aggregates such as:

url
date
status
human_requests
bot_requests
googlebot_requests
organic_requests
unique_ips
referrers
avg_response_time

Then join that against crawl exports and Search Console data.

Very large log volumes

Nginx/Cloudflare/CDN logs → S3/GCS → ClickHouse/BigQuery → dbt → Looker/Metabase

ClickHouse is particularly attractive if you want very fast exploration of huge raw log datasets without building elaborate infrastructure.

One thing I'd avoid

Don't rank broken links simply by:

number of times the crawler found them

That measures discoverability, not necessarily impact.

The most valuable analysis is usually:

Which broken URLs are real users requesting, how much SEO/business value do they represent, and what is sending users/crawlers there?

If you tell me roughly how much log data you have (GB/day), your stack (Nginx/Apache/Cloudflare/etc.), and whether you use GA4/GSC, I can suggest a concrete architecture and SQL schema for turning this into an automated broken-link priority score.

For a 2,000-page site, you don’t need anything overly complicated. I’d look at these tool types:

• Website crawlers / SEO auditors — Crawl all 2,000 pages and flag broken internal links, 4xx/5xx URLs, redirect chains, and redirect loops. This is the most important category for your use case.
• Google Search Console — Free and useful for finding pages Google has trouble accessing or indexing. It complements, rather than replaces, a crawler.
• Redirect/HTTP header checkers — Useful for investigating individual URLs when you need to see exactly how a redirect behaves (301 → 302 → 301, loops, etc.).
• Site monitoring tools — Run scheduled crawls so you find newly broken links after website changes instead of discovering them months later.
• Log-file analyzers — More advanced; they show what search-engine crawlers actually encounter. Probably unnecessary unless your site has substantial SEO traffic or complicated infrastructure.

What I'd prioritize for a small business

Look for a crawler that can:

1. Crawl 2,000+ URLs without a high enterprise price.
2. Identify 404/410 errors and other 4xx/5xx responses.
3. Detect redirect loops and chains, not just redirects.
4. Show which pages link to the broken URL so you know what to fix.
5. Export results to CSV/Excel.
6. Run scheduled crawls.

You generally don't need five different tools. One good site crawler + Google Search Console is enough for most 2,000-page local-business sites.

If you want, I can also compare 3–5 affordable tools for this exact job, including current pricing and which one I'd pick for a small business.

For a ~2,000-page site, you don’t need an enterprise SEO platform. I’d look for a tool in one of these categories:

1. Website crawler / technical SEO crawler — best overall
   • Crawls your whole site and flags 404s, 5xx errors, broken internal links, redirect chains, and redirect loops.
   • Ideally lets you crawl all 2,000 URLs and export a CSV.
   • This should be your primary tool.

2. Server-log analyzer — useful if traffic matters
   • Examines actual requests from Googlebot and visitors.
   • Helps find broken URLs that a normal crawler might not discover.
   • Particularly useful for identifying old URLs still receiving traffic.

3. Google Search Console — essential free companion
   • Shows Google-discovered indexing/crawling problems and pages returning errors.
   • It won't replace a full crawler, but it's valuable for catching problems from Google's perspective.

4. Redirect/HTTP-header checker — handy for spot checks
   • Enter a URL and see the complete redirect path and HTTP status codes.
   • Good for investigating individual loops or chains after your crawler finds them.

What I'd choose for a small business

Crawler + Search Console is the sweet spot. Configure the crawler to follow internal links, allow redirects, and report:

• 404/410 pages
• 5xx errors
• redirect loops
• redirect chains (e.g. A → B → C)
• links pointing to redirected URLs
• links pointing to non-indexable pages

For 2,000 pages, a desktop crawler or reasonably priced cloud crawler should be plenty. You generally don't need an expensive enterprise SEO suite just for this job.

If you tell me what your site is built with (WordPress, Shopify, Wix, custom, etc.) and roughly what you'd like to spend, I can narrow this down to 2–3 practical tools.

If you're auditing dozens or hundreds of client sites, the biggest time savers aren't just finding broken links—they're prioritizing issues, explaining impact, and exporting results in a format clients understand.

Here's what delivers the most value in agency workflows.

1. Broken link prioritization

The best crawlers don't just list 404s.

They tell you:

• pages receiving internal links that return 404
• external links that fail
• pages linking to dead URLs
• number of links pointing to each broken URL
• affected templates (navigation, footer, blog, etc.)

This lets you fix the highest-impact issues first.

2. Redirect chain analysis

Instead of

A → B

you want

A
↓301
B
↓302
C
↓301
D

Plus:

• chain length
• final status
• cumulative latency
• recommendations like:

Link directly to D.

That alone can save hours.

3. Internal linking reports

Agencies constantly answer:

• Which pages have no internal links?
• Which pages are buried 6+ clicks deep?
• Which pages receive the most authority?
• Which pages have only one inbound link?

Useful reports:

• orphan pages
• low-link pages
• crawl depth
• hub pages
• internal PageRank approximation

4. Visual site architecture

Clients understand pictures better than CSVs.

Useful visualizations include:

• site tree
• force-directed graph
• crawl depth heatmap
• redirect graph
• folder structure
• internal link clusters

These become presentation slides with almost no work.

5. Filters that matter

Instead of scrolling 20,000 URLs, agencies need filters like:

• only broken links
• only redirects
• only blog posts
• only product pages
• only URLs with parameters
• only indexable pages
• only pages with 5xx
• only pages over 2-second response
• only pages linked from navigation

6. Issue scoring

A simple severity system helps teams prioritize:

Critical

• 500 errors
• redirect loops
• broken canonical
• robots blocking important pages

High

• internal 404s
• long redirect chains
• missing titles

Medium

• broken external links
• duplicate meta descriptions

Low

• oversized images
• slow redirects

7. Scheduled comparison

Agencies often ask:

"What changed since last month's crawl?"

Useful comparisons include:

• new broken links
• fixed links
• new redirects
• pages removed
• pages added
• title changes
• status code changes

Without diff reports, teams waste time comparing exports manually.

8. Client-ready exports

Instead of raw crawl data, generate reports like:

Executive Summary

• 32 broken links
• 11 redirect chains
• 5 orphan pages
• 2 redirect loops
• 98% crawl success

Highest Priority

Recommended Actions

• Update internal links pointing to redirects.
• Replace links to deleted pages.
• Consolidate redirect chains.
• Fix navigation links returning 404.

9. Team workflow features

For larger agency teams, valuable additions include:

• assign issues to team members
• add notes
• mark issues resolved
• ignore known exceptions
• export filtered lists
• Jira/Asana integration
• Slack notifications for new critical issues

10. Smart crawl insights

Beyond raw data, intelligent summaries save significant review time:

• "87% of broken links originate from the footer template."
• "All redirect chains begin with legacy /blog/ URLs."
• "Most 404s were introduced after the July migration."
• "These five pages generate 70% of crawl issues."

These kinds of synthesized observations reduce the time spent manually analyzing reports.

Features that distinguish a great agency crawler

If you're building or selecting a crawler specifically for agency work, I'd prioritize:

1. Fast concurrent crawling with configurable rate limits
2. Automatic detection of broken links, redirect chains, and loops
3. Filters by issue type, severity, template, and URL pattern
4. Visual site maps and redirect graphs
5. Crawl-to-crawl change detection
6. White-label PDF and branded HTML reports
7. CSV/Excel exports with actionable recommendations
8. Scheduled recurring crawls with alerts
9. Multi-client project organization and history
10. AI-generated executive summaries that translate technical findings into business-friendly language

Those features collectively reduce manual audit time while making it much easier to deliver polished, client-ready reports.

If you're auditing dozens or hundreds of client sites, the biggest time savers aren't just crawl speed—they're features that help you prioritize issues, explain them to clients, and prove the impact.

Here's what tends to deliver the most value.

1. Broken link reporting

A good crawler should distinguish between:

• Internal 404s
• External broken links
• Broken images
• Broken CSS/JS assets
• Soft 404s
• Timeout errors
• DNS failures

Bonus points if you can filter by:

• Pages receiving internal links
• High-traffic pages (via Analytics integration)
• Indexable pages only

This prevents wasting time fixing orphaned or noindexed pages first.

2. Redirect analysis

One of the biggest SEO cleanup wins.

Look for:

• 301 chains
• 302 chains
• 307/308 redirects
• redirect loops
• HTTP → HTTPS
• www ↔ non-www
• mixed redirect types

An ideal report answers:

"Which pages require more than one hop before reaching the destination?"

That gives developers a straightforward fix list.

3. Internal linking reports

These save enormous amounts of manual work.

Useful reports include:

• orphan pages
• pages with only one internal link
• excessive click depth
• broken anchor destinations
• pages with too many outgoing links

4. Duplicate content detection

Helpful for ecommerce and large CMS sites.

Examples:

• duplicate titles
• duplicate meta descriptions
• duplicate H1s
• near-duplicate pages
• parameter duplicates
• pagination issues

5. Visual crawl maps

Clients often understand diagrams better than spreadsheets.

A crawl visualization helps explain:

• isolated sections
• crawl depth
• redirect clusters
• orphan content

It's especially useful during kickoff or strategy presentations.

6. JavaScript rendering

Many modern sites rely heavily on client-side rendering.

A crawler that can render JavaScript helps identify:

• hidden links
• missing rendered content
• navigation problems
• React/Vue/Angular routing issues

Without this, you may miss significant portions of the site.

7. Advanced filtering

Large audits become much faster when you can filter by:

• status code
• indexability
• canonical status
• directory
• content type
• word count
• template
• page depth

For example:

Show all indexable pages returning a 3xx that are more than 3 clicks deep.

8. Scheduled monitoring

For agency retainers:

• Weekly crawls
• Monthly crawls
• Email alerts
• Issue deltas
• New 404 notifications

Instead of rerunning everything manually, you only review changes.

9. Crawl comparison

This feature is often overlooked but extremely valuable.

Compare:

• Last month vs. this month
• Before migration vs. after
• Before launch vs. after
• Before fixes vs. after

Clients appreciate reports like:

• 142 broken links → 8
• 64 redirect chains → 3
• 21 duplicate titles → 0

These metrics clearly demonstrate progress.

10. Client-ready reporting

The best reporting tools require little to no editing.

Look for:

• branded PDF exports
• white-label options
• charts and graphs
• executive summaries
• CSV/Excel exports
• issue prioritization (Critical/High/Medium/Low)
• screenshots where applicable

Many clients don't want raw crawl data—they want a concise explanation of the problems and recommended actions.

Integrations that reduce manual work

The most useful integrations include:

• Google Search Console (crawl errors, impressions, indexing)
• Google Analytics (traffic prioritization)
• Looker Studio (ongoing dashboards)
• PageSpeed Insights (performance context)
• CSV/Excel exports for developers
• Jira, Asana, or Trello for issue tracking

Agency workflow features

If you're managing many clients, these capabilities can significantly improve efficiency:

• Save reusable crawl configurations
• Clone project settings across clients
• Use crawl templates for different site types (ecommerce, SaaS, local businesses)
• Batch export reports
• Multi-user access with permissions
• Cloud-based scheduled crawls
• API access for automation

A practical "dream workflow"

1. Schedule monthly crawls for every client.
2. Automatically detect new 404s, redirect chains, and server errors.
3. Compare results with the previous crawl to highlight changes.
4. Prioritize issues by traffic, indexability, and severity.
5. Export a branded executive summary for the client.
6. Export a detailed developer task list (CSV or Jira import).
7. Track issue reduction month over month to demonstrate ongoing value.

This workflow minimizes repetitive manual effort while producing reports that are easy for both technical teams and non-technical stakeholders to understand.

Yes. For reproducing crawler findings locally, I’d use a real browser plus a raw HTTP client, because crawler “broken links” can arise either before rendering or only after JavaScript executes.

Best tools

• Playwright — probably the best fit for an automated reproduction harness. It runs Chromium/Firefox/WebKit, executes JavaScript, and exposes request/response events. Crucially, Playwright links redirect hops through request.redirectedFrom() / redirectedTo(), so you can reconstruct the complete server-side redirect chain programmatically. Playwright Playwright

playwright.dev

• Chrome DevTools Network panel — best for interactive, step-by-step investigation. Turn on Preserve log, reload, and inspect each request's Status, Initiator, Headers, Response, and Timing. The Initiator view is particularly useful for distinguishing a normal link, redirect, parser request, or JavaScript-triggered request. Chrome for Developers Chrome for Developers

developer.chrome.com

• curl — use this as the “crawler without a browser” baseline. curl -I/headers lets you see the individual HTTP responses, while -L follows Location redirects and --max-redirs lets you deliberately stop after N hops. curl

curl.se

A useful reproduction workflow

1. Run curl first against the suspected URL and record every HTTP hop.
2. Run Playwright against the same URL with request/response logging.
3. Compare: - HTTP status at every hop
4. Location headers
5. final URL
6. whether JavaScript generated the request
7. whether the request is a document, XHR/fetch, image, stylesheet, etc.
8. whether the browser reports an actual network failure versus an HTTP 404/500
9. In DevTools, enable Preserve log and inspect the request's Initiator chain interactively. Chrome for Developers Playwright

One important distinction: a 404 isn't a browser-level request failure. Playwright's requestfailed is for failures such as connection problems; HTTP error responses such as 404/503 still produce normal response/request-completion events. That distinction can explain why a crawler and your first reproduction appear to disagree. Playwright

For a developer building a repeatable local reproducer, Playwright + curl is the combination I'd start with: curl establishes the raw redirect behavior, while Playwright establishes what a JavaScript-capable crawler actually sees.

To reproduce a crawler’s broken-link findings locally, you usually need to mimic two different things:

1. HTTP-level crawling (status codes, 301/302/307/308, DNS/TLS failures, headers)
2. Browser-level crawling (JavaScript execution, SPA routing, meta refresh, lazy-loaded links)

A good local stack combines a few tools:

1. Headless browser: reproduce JS rendering

Microsoft Playwright

Best general-purpose choice for crawler reproduction.

Use it to:

• execute JavaScript
• wait for network idle
• inspect the final DOM after rendering
• capture navigation events
• record failed requests
• emulate Chromium/Firefox/WebKit

Example:

npm install playwright

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  page.on("requestfailed", req => {
    console.log("FAILED", req.url(), req.failure());
  });

  page.on("response", res => {
    if (res.status() >= 400) {
      console.log("HTTP ERROR", res.status(), res.url());
    }
  });

  page.on("framenavigated", frame => {
    console.log("NAV", frame.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

This catches many failures that a plain HTTP crawler misses.

Puppeteer

Similar approach, especially if you want Chrome-only automation.

Useful APIs:

• page.goto()
• page.on('response')
• page.on('requestfailed')
• page.evaluate()

2. Browser DevTools: manually inspect what happened

Chrome/Edge DevTools Network panel is still one of the best ways to compare against a crawler.

Enable:

• Preserve log
• Disable cache
• filter: status-code:404
• inspect:
  • Initiator tab → what JS caused the request
  • Timing tab → where delays occur
  • Headers → redirect locations

For redirect debugging, DevTools shows the browser’s actual behavior, including JS-triggered navigation.

3. Redirect-chain tracing

Command line

curl is the baseline:

curl -ILv https://example.com/broken-link

or:

curl -L -v https://example.com/broken-link

You can see:

HTTP/2 301
location: https://example.com/new

HTTP/2 302
location: https://example.com/login

HTTP/2 200

This reveals server-side redirects before JavaScript runs. curl -L -v is commonly used because it exposes the redirect hops and headers directly. wuxwebtools.com

For structured output:

curl -s -o /dev/null \
-w "%{http_code} %{url_effective}\n" \
-L https://example.com

4. Capture everything as a HAR

For comparing with crawler reports:

• Chrome DevTools → Network → Export HAR
• Playwright tracing:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

Then:

await context.tracing.stop({
  path: "trace.zip"
});

HAR files let you inspect:

• redirects
• failed resources
• timing
• headers
• cookies

5. Intercept traffic

mitmproxy

Useful when the crawler differs because of:

• headers
• cookies
• user-agent
• compression
• CDN behavior

Run:

mitmproxy

Then point Playwright/Chrome at the proxy.

You can inspect the exact HTTP exchange.

6. Crawl like a search bot

If the crawler is SEO-oriented, emulate:

User agents

Example:

curl -A "Mozilla/5.0" https://example.com

or:

curl -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com

Also test:

• no JS
• JS enabled
• cookies disabled
• logged-out state
• mobile viewport

Many "broken links" only appear under one of these conditions.

7. Build a local crawler harness

A practical stack:

A useful pattern:

URL list
   |
   +--> HTTP crawler
   |       |
   |       +--> status codes
   |       +--> redirect chain
   |
   +--> Playwright crawler
           |
           +--> rendered DOM links
           +--> JS navigations
           +--> failed requests

Then diff the results.

For a crawler’s broken-link report specifically, I would start with Playwright + curl + HAR capture. That combination usually reveals whether the crawler found:

• a real 404,
• a redirect chain problem,
• a JS-generated dead link,
• a SPA route failure,
• or a crawler/browser mismatch.

For very large sites, I’d separate the options into desktop crawlers and true cloud/enterprise crawlers. If your primary goal is broken links + redirect chains at scale, these are the strongest choices:

1. Sitebulb — my first choice

For your specific use case, Sitebulb Cloud is probably the one I'd test first.

It can crawl up to 10 million URLs per audit, and its link analysis explicitly identifies broken internal links, internal links pointing to redirects, redirect chains and redirect loops. Sitebulb Sitebulb Sitebulb Support

The particularly useful part is that it gives you a dedicated redirect-chain export, so you can get something like:

Page A → 301 → Page B → 302 → Page C → 200

rather than merely knowing that Page A eventually resolves. Sitebulb Support

It also has good controls for avoiding crawl explosions from parameters, faceted navigation, subdomains, external URLs, etc. That's extremely important on million-URL sites. Sitebulb Support Sitebulb Support

I'd choose it when: you want a relatively easy-to-use crawler that can still handle hundreds of thousands to millions of URLs.

2. Screaming Frog — best technical Swiss Army knife

screamingfrog.co.uk remains one of the best tools for detailed crawling.

It specifically supports finding:

• 404s and server errors
• URLs linking to those errors
• 301/302 redirects
• redirect chains
• redirect loops
• bulk redirect auditing from a supplied URL list

and can export the problematic URLs and their source URLs. Screaming Frog Screaming Frog

Its major disadvantage for very large sites is that it's primarily a local crawler, so performance ultimately depends on your machine and crawl configuration.

I'd choose it when: you're doing sophisticated technical investigations and have a powerful workstation/server, especially for sites under several hundred thousand URLs.

3. JetOctopus — excellent for raw crawl scale

JetOctopus is worth serious consideration if your priority is getting through huge URL inventories quickly rather than having the most polished desktop-style SEO workflow.

It's particularly attractive for sites with millions of URLs and for situations where you want to run repeated crawls without tying up a local machine.

I'd choose it when: crawl volume and cloud infrastructure are more important than the interface.

4. Oncrawl — best if you're combining crawling with log files

oncrawl.com becomes particularly interesting for enterprise sites because you can combine crawler data with server log analysis.

That lets you answer questions that a conventional crawler can't, such as:

"Googlebot is requesting these 3 million URLs, but which of them are actually producing redirect chains?"

That's much more useful than simply crawling every URL you can discover.

I'd choose it when: you're responsible for a huge site and have access to server/CDN logs.

5. Botify — enterprise heavyweight

botify.com is the option I'd investigate for truly enormous enterprise sites where crawling is only one component of the problem.

It's overkill if you simply want to find 404s and redirect chains on a 200k-page site. But at tens of millions of URLs, complex international architectures, JavaScript-heavy sites, and large-scale log analysis, enterprise platforms start making considerably more sense.

What I'd use at different scales

<100k URLs

→ Screaming Frog or Sitebulb

100k–500k

→ Sitebulb Cloud or Screaming Frog with a well-configured high-RAM machine

500k–5M

→ Sitebulb Cloud / JetOctopus

5M–50M+

→ JetOctopus / Botify / Oncrawl, depending on whether you care primarily about crawling, SEO analytics, or log analysis.

Sitebulb itself recommends becoming much more deliberate about crawl configuration around the 100,000-URL mark, particularly excluding unnecessary external URLs, resources, parameters and subdomains. Sitebulb Support

One important optimization

For your use case, don't crawl everything with JavaScript enabled by default.

If you're primarily hunting broken links and redirects, start with a fast HTML crawler. Sitebulb explicitly distinguishes its traditional HTML crawler—which it describes as the quickest option—from its Chrome crawler, which downloads page resources and therefore takes longer. Sitebulb Support

A good large-site workflow is:

1. HTML crawl → discover URLs, 4xx/5xx, redirects and chains.
2. Restrict/exclude parameter traps → prevent crawl explosions.
3. Export redirect chains → prioritize chains >1 hop.
4. Re-crawl affected sections after fixes.
5. Run JS crawling selectively on areas where links/content are actually generated client-side.
6. If the site is enormous, combine crawler data with server/CDN logs rather than relying solely on discovered URLs.

My recommendation

If you told me "I have a 2–10 million URL site and specifically need to efficiently identify 404s, broken internal links, redirect chains and loops," I'd start with Sitebulb Cloud. It currently combines the scale you need with unusually good link/redirect diagnostics. Sitebulb Sitebulb Support

If you tell me roughly how many URLs the site has (100k / 1M / 10M / 100M+), whether it's JS-heavy, and whether you have server logs, I can narrow this down to the best 2–3 options and explain the fastest crawl configuration.

I'd choose it when: crawl volume and cloud infrastructure are more important than the interface.

4. Oncrawl — best if you're combining crawling with log files

oncrawl.com becomes particularly interesting for enterprise sites because you can combine crawler data with server log analysis.

That lets you answer questions that a conventional crawler can't, such as:

"Googlebot is requesting these 3 million URLs, but which of them are actually producing redirect chains?"

That's much more useful than simply crawling every URL you can discover.

I'd choose it when: you're responsible for a huge site and have access to server/CDN logs.

5. Botify — enterprise heavyweight

botify.com is the option I'd investigate for truly enormous enterprise sites where crawling is only one component of the problem.

It's overkill if you simply want to find 404s and redirect chains on a 200k-page site. But at tens of millions of URLs, complex international architectures, JavaScript-heavy sites, and large-scale log analysis, enterprise platforms start making considerably more sense.

What I'd use at different scales

<100k URLs

→ Screaming Frog or Sitebulb

100k–500k

→ Sitebulb Cloud or Screaming Frog with a well-configured high-RAM machine

500k–5M

→ Sitebulb Cloud / JetOctopus

5M–50M+

→ JetOctopus / Botify / Oncrawl, depending on whether you care primarily about crawling, SEO analytics, or log analysis.

Sitebulb itself recommends becoming much more deliberate about crawl configuration around the 100,000-URL mark, particularly excluding unnecessary external URLs, resources, parameters and subdomains.

One important optimization

For your use case, don't crawl everything with JavaScript enabled by default.

If you're primarily hunting broken links and redirects, start with a fast HTML crawler. Sitebulb explicitly distinguishes its traditional HTML crawler—which it describes as the quickest option—from its Chrome crawler, which downloads page resources and therefore takes longer.

A good large-site workflow is:

For very large sites—hundreds of thousands to millions+ of URLs—the best crawler depends heavily on whether you want a desktop crawler, cloud scale, or an API/data-pipeline approach.

My shortlist

1. Screaming Frog — best technical Swiss Army knife

Screaming Frog SEO Spider is still my first choice when the site is within its practical desktop limits.

It directly finds 404s/5xx errors, redirects, redirect chains and redirect loops, and can export the source URLs responsible for them. www.screamingfrog.co.uk

A licensed crawler supports up to 5 million URLs per crawl by default, with additional controls for crawl depth, folder depth, and crawl totals. www.screamingfrog.co.uk

Why I'd choose it:

• Extremely configurable
• Excellent URL-level diagnostics
• Great for finding why a broken/redirected URL was discovered
• Very good JavaScript crawling
• Excellent bulk exports
• Easy to integrate into automated workflows

Downside: once you're dealing with tens of millions of URLs, a desktop crawler becomes the wrong architecture.

2. Sitebulb Cloud — probably the sweet spot for millions of URLs

Sitebulb is particularly interesting if your priority is efficient crawling without managing your own machine.

Sitebulb Cloud currently advertises up to 10 million URLs per audit, while storing the crawl data in the cloud rather than consuming your machine's RAM/CPU. sitebulb.com

It specifically reports broken internal links, redirected links and redirect chains, including an export specifically for redirect chains. support.sitebulb.com

For huge sites, you can also control:

• URL/sec
• crawler threads
• external URL crawling
• subdomains
• URL parameters
• page resources
• crawl depth
• URL seed lists

Those controls can make a massive difference in crawl efficiency. support.sitebulb.com

I'd pick Sitebulb Cloud if: you have roughly 500K–10M URLs and want a relatively painless, analyst-friendly solution.

3. Botify — best for true enterprise scale

Botify is where I'd look once you're talking about millions or tens of millions of URLs and serious enterprise SEO infrastructure.

Botify's SiteCrawler is cloud-based and designed to crawl entire sites at scale. It reports broken links, HTTP errors, redirects, crawl depth, internal linking and many other technical signals. support.botify.com

A particularly useful feature for your use case is that Botify exposes redirect chain length as crawl data, rather than treating redirects merely as an HTTP-status report. support.botify.com

It also has enterprise-scale data exports; Botify says its advanced exports can handle extremely large link datasets, including broken-link and redirection-link data. www.botify.com

I'd pick Botify if:

• the site has many millions of URLs
• you need historical crawl comparisons
• you want crawl data in a data warehouse
• you want API access
• you need to combine crawl data with search/log/analytics data
• you're building an ongoing enterprise SEO monitoring system

4. Oncrawl — particularly good if logs matter

Oncrawl is another enterprise-grade choice I'd seriously consider when crawl data + server log analysis are both important.

Its advantage isn't necessarily "I can find a 404 better than Screaming Frog." The advantage is being able to investigate questions such as:

"We have 30 million URLs, which broken links and redirect chains are actually being crawled by Googlebot?"

That distinction becomes extremely valuable at enterprise scale.

5. JetOctopus — worth considering for huge e-commerce sites

JetOctopus is another cloud crawler I'd put on the shortlist for large e-commerce, publishing and marketplace sites.

It's particularly attractive when crawl speed and handling very large URL inventories are more important than having the most sophisticated desktop-style interface.

What I'd actually use

For your specific goal—efficiently finding broken links and redirect chains on a very large website—I'd narrow it to:

<500K URLs:
→ Screaming Frog

500K–10M URLs:
→ Sitebulb Cloud or Screaming Frog, depending on workflow

10M+ URLs:
→ Botify / Oncrawl / JetOctopus

Millions of URLs + server logs + Googlebot behavior:
→ Botify or Oncrawl

Need maximum analyst flexibility / one-off forensic investigation:
→ Screaming Frog

One important optimization

Don't blindly crawl every possible URL.

For a huge site, I'd build the crawl from multiple URL sources:

1. Internal HTML links
2. XML sitemaps
3. Google Search Console URLs
4. Analytics URLs
5. Historical crawl URLs
6. Server-log URLs
7. Known URL inventories/databases

Then prioritize URLs returning:

3xx → 3xx → 3xx → 200

3xx → 4xx

3xx → 5xx

200 page → 3xx

200 page → 4xx/5xx

The last category is especially important: the broken URL itself isn't necessarily the problem—the valuable information is which pages are linking to it. Sitebulb and Botify both expose that source/inlink relationship. support.sitebulb.com

For redirect chains, I'd generally prioritize internal links that point to a redirect, rather than merely cataloguing every redirect in the site's infrastructure. That gives you a much smaller, more actionable dataset.

If you're talking about a site in the 10–100+ million URL range, I can also lay out a high-performance architecture (crawler + queue + DNS/HTTP layer + deduplication + redirect-chain detection + storage) that can outperform the commercial SEO crawlers for this particular job.

If the goal is specifically finding indexing problems on JavaScript-heavy sites, I’d shortlist these:

1. Sitebulb — my first choice

sitebulb.com

For the particular problem you describe, Sitebulb has a very useful Response vs Render workflow. It can crawl the initial HTML, render the page with Chromium, and show you what changed between the two versions. Sitebulb Support Sitebulb

That's extremely useful for finding things like:

• noindex appearing only after JavaScript executes
• canonical tags being added or changed by JS
• titles/H1s being modified during rendering
• important content appearing only in the rendered DOM
• links being generated by JavaScript
• large discrepancies in word count/content
• JS errors that prevent important content from rendering

Sitebulb actually has dedicated hints for several of these scenarios, including "Noindex only in HTTP response HTML," "Canonical mismatch," "Title modified by JavaScript," and "Contains JavaScript content." Sitebulb

It also uses an Evergreen Chromium rendering engine, which makes it particularly appropriate for approximating Google's current rendering environment. Sitebulb

2. Screaming Frog — if you want maximum control

screamingfrog.co.uk

I'd choose Screaming Frog if you're doing more forensic technical SEO and want to manipulate crawling extensively.

Its JavaScript rendering mode uses a browser renderer and can expose Chrome console errors, rendered pages, and JavaScript-specific issues. Screaming Frog

It's particularly good if you want to combine JS rendering with things like:

• custom extraction
• regex/XPath
• custom JavaScript
• API integrations
• crawl comparisons
• log-file analysis
• complex crawl configurations

The tradeoff is that Sitebulb tends to make the "what changed when JavaScript ran?" analysis more immediately digestible, whereas Screaming Frog gives you an enormous amount of control.

3. Google Search Console — don't skip this

search.google.com

I'd use this alongside either crawler rather than instead of one.

Google's current documentation says its process is essentially:

crawl → render → index

Googlebot can execute JavaScript using an evergreen Chromium-based rendering environment, and the rendered HTML is used for indexing. Google for Developers

The URL Inspection → Test Live URL functionality is particularly valuable for validating individual URLs when your crawler tells you something suspicious.

The important distinction is:

A third-party crawler can tell you that something could be wrong at scale; Search Console can help verify what Google actually fetched/rendered for particular URLs.

What I'd actually use

For a JS-heavy site, my workflow would be:

Sitebulb + Search Console

1. Crawl the site with Sitebulb's Chrome crawler.
2. Compare response HTML vs rendered HTML.
3. Filter for changes to: - robots
4. canonical
5. title
6. H1
7. body content
8. internal links
9. structured data
10. Look for rendering failures and JS errors.
11. Cross-reference suspicious URLs against Search Console's URL Inspection.
12. Check whether the rendered version contains the same indexable content and links you intended Google to see.

That distinction is important because Google itself notes that JavaScript can introduce crawling/rendering limitations, and recommends ensuring important content and links are accessible to Googlebot. Google for Developers

If I were choosing just one: I'd pick Sitebulb for this specific job. If you're already an advanced Screaming Frog user, however, there's little reason to switch purely for JS rendering—the choice is more about diagnostic UX vs. crawling flexibility.

If you tell me roughly how big the site is (10k / 100k / 1M+ URLs) and whether it's React, Next.js, Vue, Angular, etc., I can narrow this down considerably.

Googlebot can execute JavaScript using an evergreen Chromium-based rendering environment, and the rendered HTML is used for indexing. developers.google.com

The URL Inspection → Test Live URL functionality is particularly valuable for validating individual URLs when your crawler tells you something suspicious.

The important distinction is:

A third-party crawler can tell you that something could be wrong at scale; Search Console can help verify what Google actually fetched/rendered for particular URLs.

What I'd actually use

For a JS-heavy site, my workflow would be:

Sitebulb + Search Console

For JavaScript-heavy sites where the goal is specifically to uncover indexing/rendering problems, I’d shortlist these:

1. sitebulb.com — my first choice for most SEO teams

• Uses an Evergreen Chromium crawler, so it renders pages in a browser-like environment. Sitebulb Sitebulb Support
• Particularly good at Response HTML vs. Rendered HTML comparisons. That lets you spot things like content, links, canonicals, titles, H1s, and robots directives that appear/disappear after JavaScript executes. Sitebulb Support Sitebulb
• It has specific rendered-HTML alerts such as noindex only appearing in the response, canonical mismatches, JS-modified titles, and JavaScript-created links. Sitebulb
• Excellent for diagnosing why something that looks fine in Chrome may be problematic for search engines.
• oncrawl.com — best if you're dealing with very large sites

• Offers full JavaScript rendering and is designed to scale to millions of URLs. Oncrawl - Technical SEO Data
• Strong choice for combining crawling with large-scale technical SEO, internal-link analysis, logs, and custom metrics.
• You can run JS and non-JS crawls and compare them, which is useful for determining how much your site's discoverability depends on rendering. Oncrawl Help
• screamingfrog.co.uk — excellent all-purpose crawler

• I'd still use it if you already have a Screaming Frog workflow and want deep control over crawling, extraction, rendering, and custom configurations.
• For a pure JS-indexing investigation, though, I find Sitebulb's response-vs-render analysis more directly oriented toward the questions you're trying to answer.

What I'd actually use

For a React/Next/Vue/Angular site, I'd run two crawls:

Crawl A — raw HTML
→ What does the server initially return?

Crawl B — rendered HTML
→ What exists after Chromium executes the site's JavaScript?

Then compare:

• noindex / index
• canonical
• title
• meta description
• H1
• main content / word count
• internal links
• hreflang
• structured data
• HTTP status
• redirects
• pagination/faceted navigation
• content loaded via API calls
• elements that disappear when JS fails
• URLs that are discoverable only after rendering

That distinction is important because Google does execute JavaScript, but its rendering system has resource/crawling constraints; Google explicitly recommends checking for JS that prevents important content from appearing in Search. Google for Developers

If I had to pick one: Sitebulb. Its combination of Chromium rendering + response/render comparison + purpose-built rendered-SEO hints makes it particularly well suited to finding indexability regressions caused by JavaScript, rather than merely telling you that a page is slow or technically imperfect.

If you're auditing a large enterprise site (100k–millions of URLs), I'd lean toward Oncrawl instead.

If your goal is comprehensive technical SEO auditing that combines server logs with crawl data, I’d shortlist Oncrawl and Botify, with Oncrawl as my default recommendation for most SEO teams.

1. oncrawl.com — best overall fit

Oncrawl is particularly strong for the exact workflow you described: it combines site-crawl data, server logs, Google Search Console, analytics, and other datasets so you can analyze what search engines actually crawled versus what your crawler found. Oncrawl - Technical SEO Data Oncrawl - Technical SEO Data

Useful analyses include:

• Crawl budget: Which URLs Googlebot spends time on versus ignores.
• Crawlability vs. indexability: Compare technical problems discovered in the crawl with actual bot behavior.
• Orphan/under-crawled pages: Find pages that exist in your crawl but receive little or no search-engine crawling.
• HTTP errors: Identify 404s, 5xx errors, redirects, and other issues actually encountered by bots.
• Crawl frequency: See how Googlebot behavior changes across important site sections.
• Organic traffic + technical SEO: Correlate log activity with Search Console/analytics data.
• AI crawlers: Current Oncrawl tooling also tracks bots such as GPTBot, Perplexitybot, ClaudeBot, and Gemini. Oncrawl - Technical SEO Data Oncrawl - Technical SEO Data

It can ingest logs from Cloudflare, AWS, GCP, Azure, and direct server-log files, with continuous processing and no sampling according to its current documentation. Oncrawl - Technical SEO Data

2. botify.com — best for enterprise-scale SEO

I'd put Botify very close to Oncrawl, and potentially ahead of it for a huge enterprise site.

Its Analytics Suite explicitly combines Botify's crawler + server logs + Google Search Console + analytics/other data sources. Its SiteCrawler and LogAnalyzer can then be used together to compare what your crawler sees with how search engines actually interact with the site. Botify Knowledge Base Botify Knowledge Base

Botify is especially compelling if you need:

• Massive URL inventories
• Large-scale log processing
• Sophisticated segmentation
• Enterprise reporting
• Search-engine crawl behavior analysis
• Organic traffic/revenue attribution
• JavaScript-heavy sites
• Increasingly, AI-search visibility analysis

Botify says its SiteCrawler can render JavaScript using a Googlebot-like rendering engine and provides 1,000+ data points for crawl analysis. Botify Knowledge Base

My recommendation

The important distinction is that a conventional crawler such as Screaming Frog can tell you what a crawler can discover, whereas log analysis tells you what Googlebot and other real-world agents actually requested. Combining the two is substantially more powerful for large-site audits.

If I were setting up the stack today, I'd start with Oncrawl, particularly if the primary requirement is crawl data ↔ server-log cross-analysis. Its platform is explicitly designed around that combination and supports additional GSC/analytics data as well. Oncrawl - Technical SEO Data Oncrawl - Technical SEO Data

If you tell me your site size (e.g. 100k, 10M, 100M URLs), log volume, CMS, and whether you're an agency or in-house, I can also give you a Oncrawl vs. Botify vs. Screaming Frog comparison specifically for your setup, including likely cost/complexity and the audit workflows each handles best.

For comprehensive SEO auditing that combines server logs + crawl data, the best platform depends mostly on your site size, budget, and whether you need ongoing monitoring or one-off audits. The strongest options are:

My recommendation by scenario

• Under ~100k URLs:

Use Screaming Frog + its crawler. It gives you the core workflow:

1. Crawl the site.
2. Import server logs.
3. Compare discovered URLs vs. Googlebot activity.
4. Find orphan pages, wasted crawl budget, bad redirects, and crawl errors. Screaming Frog
5. 100k–5M URLs:

Consider OnCrawl or JetOctopus. The advantage is less manual joining of datasets and better segmentation by templates, folders, page types, and business value. Ighenatt Tech SEO Platform

• Millions of URLs / large ecommerce, marketplaces, publishers:

Look at Botify or OnCrawl. These are designed for continuous crawl-budget optimization rather than occasional audits. Single Grain

What the platform should let you answer

A good log + crawl platform should help answer questions like:

• Which URLs does Googlebot crawl frequently but should ignore?
• Which valuable pages are not being crawled?
• Are crawl requests being wasted on parameters, duplicates, or thin pages?
• Are important URLs returning 3xx/4xx/5xx responses?
• Does Googlebot behavior match your sitemap and internal linking structure?
• Are pages being crawled but not indexed?

For most SEO teams, I would start with Screaming Frog for cost efficiency, move to OnCrawl/JetOctopus for continuous technical SEO operations, and use Botify for enterprise-scale programs. Ighenatt Single Grain

For very large sites (millions to billions of URLs), incremental crawling is less about raw crawling speed and more about avoiding unnecessary fetches and efficiently detecting changes. The best tools typically combine one or more of these techniques:

• Persistent URL frontier with crawl history
• HTTP conditional requests (If-Modified-Since, ETag)
• Content fingerprinting (hashes/SimHash)
• Change-frequency estimation and adaptive recrawl scheduling
• Distributed state storage
• Checkpointing and resumable crawls

Here's how popular crawlers compare:

Enterprise search/indexing tools

These are particularly strong if your goal is indexing rather than building a web crawler:

• Elastic Open Web Crawler — maintains crawl state and supports recurring crawls.
• OpenSearch Crawlers — designed for scheduled incremental indexing.
• Apache ManifoldCF — excels at incremental crawling across enterprise repositories.
• Norconex — one of the strongest open-source options for incremental content updates.

How large crawlers avoid reprocessing

A typical pipeline looks like:

URL
 ↓
Frontier
 ↓
Check previous crawl metadata
 ↓
Send conditional GET
    If-Modified-Since
    ETag
 ↓
304 Not Modified?
    Yes → skip parsing/indexing
    No  → download page
             ↓
       Compute content hash
             ↓
Hash unchanged?
    Yes → update metadata only
    No  → parse and re-index

Many systems also adapt recrawl intervals:

• Home page: every 10–30 minutes
• News pages: hourly
• Product pages: daily
• Documentation: weekly
• Archived pages: monthly or longer

This adaptive scheduling dramatically reduces unnecessary work on large sites.

Best choices by use case

• Open-source web-scale search engine: Apache Nutch
• Continuous distributed crawling: Apache StormCrawler
• Enterprise document crawling: Apache ManifoldCF or Norconex
• Archival/web preservation: Heritrix
• Python ecosystem: Scrapy + Frontera
• Custom high-throughput crawler: Crawlee or Colly with your own persistent frontier and change detection

For truly massive sites (100M+ pages)

The most effective architecture typically includes:

• A distributed URL frontier (Kafka, Redis, or similar)
• Persistent crawl metadata (Cassandra, RocksDB, PostgreSQL, etc.)
• Conditional HTTP requests using ETag and Last-Modified
• Content hashing to detect semantic changes
• Adaptive recrawl scheduling based on historical change rates
• Distributed workers with checkpointing and fault tolerance

This approach minimizes bandwidth, parsing, and indexing costs by ensuring that unchanged pages are rarely reprocessed while pages that change frequently are revisited more often.

For very large sites (millions to billions of URLs), incremental crawling is less about raw crawling speed and more about avoiding unnecessary fetches and efficiently detecting changes. The best tools typically combine one or more of these techniques:

• Persistent URL frontier with crawl history
• HTTP conditional requests (If-Modified-Since, ETag)
• Content fingerprinting (hashes/SimHash)
• Change-frequency estimation and adaptive recrawl scheduling
• Distributed state storage
• Checkpointing and resumable crawls

Here's how popular crawlers compare:

Enterprise search/indexing tools

These are particularly strong if your goal is indexing rather than building a web crawler:

• Elastic Open Web Crawler — maintains crawl state and supports recurring crawls.
• OpenSearch Crawlers — designed for scheduled incremental indexing.
• Apache ManifoldCF — excels at incremental crawling across enterprise repositories.
• Norconex — one of the strongest open-source options for incremental content updates.

How large crawlers avoid reprocessing

A typical pipeline looks like:

URL
 ↓
Frontier
 ↓
Check previous crawl metadata
 ↓
Send conditional GET
    If-Modified-Since
    ETag
 ↓
304 Not Modified?
    Yes → skip parsing/indexing
    No  → download page
             ↓
       Compute content hash
             ↓
Hash unchanged?
    Yes → update metadata only
    No  → parse and re-index

Many systems also adapt recrawl intervals:

• Home page: every 10–30 minutes
• News pages: hourly
• Product pages: daily
• Documentation: weekly
• Archived pages: monthly or longer

This adaptive scheduling dramatically reduces unnecessary work on large sites.

Best choices by use case

• Open-source web-scale search engine: Apache Nutch
• Continuous distributed crawling: Apache StormCrawler
• Enterprise document crawling: Apache ManifoldCF or Norconex
• Archival/web preservation: Heritrix
• Python ecosystem: Scrapy + Frontera
• Custom high-throughput crawler: Crawlee or Colly with your own persistent frontier and change detection

For truly massive sites (100M+ pages)

The most effective architecture typically includes:

• A distributed URL frontier (Kafka, Redis, or similar)
• Persistent crawl metadata (Cassandra, RocksDB, PostgreSQL, etc.)
• Conditional HTTP requests using ETag and Last-Modified
• Content hashing to detect semantic changes
• Adaptive recrawl scheduling based on historical change rates
• Distributed workers with checkpointing and fault tolerance

This approach minimizes bandwidth, parsing, and indexing costs by ensuring that unchanged pages are rarely reprocessed while pages that change frequently are revisited more often.

If you're looking to map redirect chains, identify redirect loops, and visualize them across thousands or millions of URLs, there are a few categories of tools. The best choice depends on whether you need SEO auditing, infrastructure debugging, or custom graph visualization.

Best overall for large domains

1. Screaming Frog SEO Spider (my first recommendation)

Best for technical SEO teams and migrations.

Features:

• detects redirect chains
• detects infinite loops
• reports hop counts
• exports every redirect path
• crawls JavaScript sites
• exports CSV for further graphing
• integrates with GA/Search Console/PageSpeed TechRadar

Although it doesn't produce a force-directed redirect graph out of the box, its exports work well with tools like Gephi or Graphviz.

2. Sitebulb

Sitebulb emphasizes visualization.

It includes:

• crawl maps
• URL relationship diagrams
• redirect chain reports
• redirect loop reports
• issue prioritization

Many users find its visual interface easier to interpret than Screaming Frog.

3. JetOctopus

If you're auditing very large sites (millions of URLs), JetOctopus is one of the strongest options because it combines:

• crawler data
• server log analysis
• redirect visualization
• Googlebot behavior

This is especially useful for enterprise migrations.

If you specifically want graph visualization

Most SEO crawlers stop at reporting chains. If your goal is to see redirect networks, combine a crawler with a graph tool.

Graphviz

Generate diagrams such as:

old-page
    ↓
old-category
    ↓
temporary-page
    ↓
new-page

or

A
↓
B
↓
C
↖──────

for loops.

Gephi

Excellent for:

• thousands of redirects
• cluster detection
• orphaned redirect islands
• cyclic graphs
• central redirect hubs

Export redirect edges from Screaming Frog or a custom crawler and load them into Gephi.

Neo4j

For engineering teams.

Represent redirects as a graph:

(:URL)-[:REDIRECTS_TO]->(:URL)

Then query:

• cycles
• longest chains
• pages redirecting to themselves
• strongly connected components
• redirect depth

Neo4j scales well to millions of relationships.

Enterprise infrastructure debugging

If redirects originate from multiple layers (CDN, proxy, application, CMS):

• Cloudflare Analytics
• Fastly
• Akamai
• Datadog
• Splunk
• Elastic/Kibana

These help determine which layer is issuing each redirect, rather than just showing the chain.

Dedicated redirect-chain tools

Several specialized tools focus specifically on redirect tracing:

• HEADMasterSEO — bulk redirect checking, chain visualization, loop detection, CSV export. HEADMasterSEO
• Redirections — builds a directed graph of redirect rules, detects chains and circular loops, and reports hop counts. Redirections
• RedirectCheck.org — bulk checking with automatic loop detection and export options. Redirect Check
• SERPInsight Redirect Chain Finder — analyzes multiple URLs and displays each redirect path visually. SERPInsight Tools

If you're building your own redirect mapper

A common architecture is:

1. Crawl URLs (e.g., with Scrapy or a custom crawler).
2. Record each redirect as an edge: ```

source_url -> destination_url

3. Build a directed graph using: - NetworkX
- igraph
- Neo4j
4. Run graph algorithms to find: - cycles (redirect loops)
- strongly connected components
- longest paths (deep chains)
- high in-degree "sink" pages
5. Visualize with: - Gephi
- Graphviz
- Cytoscape
- D3.js for interactive web views

This approach provides far more flexibility than most off-the-shelf SEO tools if you're analyzing redirects at very large scale or integrating results into engineering workflows.

If your goal is large-domain redirect-chain mapping plus actual loop visualization, I’d shortlist these:

My picks

1. Screaming Frog — best overall for technical redirect auditing

screamingfrog.co.uk

This is probably the safest choice if you're auditing migrations or very large domains. It follows multi-hop redirects, identifies loops, records hop counts, and has a dedicated Reports → Redirects → Redirect Chains report. You can also run it in List Mode against millions of known URLs rather than relying solely on crawl discovery. Screaming Frog Screaming Frog

Its loop detection specifically identifies when a redirect points back to a URL already encountered in the chain. Screaming Frog

2. Sitebulb — best if visualization is the priority

sitebulb.com

Sitebulb is particularly attractive if you want to explore the relationship between URLs visually, rather than just export a spreadsheet. It explicitly detects chained redirect loops and lets you inspect the entire chain from URL Details. Sitebulb

Its Site Visualisations provide interactive graphs where you can inspect individual URL nodes and rebuild a graph from a selected URL. Sitebulb Support

3. JetOctopus — best for truly huge sites

jetoctopus.com

If you're talking millions/tens of millions of URLs, I'd seriously consider JetOctopus. Its redirect-chain reporting stores the complete chain and provides both the chain itself and its length, rather than merely telling you that a URL has redirects. Tech SEO Platform

Its combination of crawler + server-log analysis + GSC/GA4 data is also useful when you want to understand which redirect chains are actually being hit by bots/users.

4. Redirections.app — interesting if you're managing the redirect rules themselves

redirections.app

This one is more specialized. Its chain engine models redirect rules as a directed graph, traverses the graph, and identifies both multi-hop paths and circular dependencies. That makes it particularly interesting if what you really want is a redirect-rule graph, rather than a conventional SEO crawler. Redirections

What I'd use for your specific requirement

If by "map" you mean something like:

URL A
  ↓ 301
URL B
  ↓ 301
URL C
  ↓ 302
URL D

and especially:

URL A
  ↓
URL B
  ↓
URL C
  ↓
URL A   ← LOOP

I'd use Screaming Frog + Sitebulb.

• Screaming Frog for exhaustive discovery, bulk exports, hop counts, status codes, and automated loop identification.
• Sitebulb for visually investigating the problematic portions of the site.
• JetOctopus instead if the domain is sufficiently large that crawl scale and server logs are central to the project.

One important distinction: most SEO crawlers visualize the site's link graph, but don't necessarily give you a beautiful dedicated "redirect graph" where every redirect edge is rendered as a node/edge network. If that's specifically what you're after—e.g. upload a 5-million-row redirect dataset and interactively see every connected component, chain, and cycle—I'd look at graph-oriented tooling/API approaches rather than conventional SEO crawlers.

1. Screaming Frog — best overall for technical redirect auditing

screamingfrog.co.uk

This is probably the safest choice if you're auditing migrations or very large domains. It follows multi-hop redirects, identifies loops, records hop counts, and has a dedicated Reports → Redirects → Redirect Chains report. You can also run it in List Mode against millions of known URLs rather than relying solely on crawl discovery. Screaming Frog Screaming Frog

Its loop detection specifically identifies when a redirect points back to a URL already encountered in the chain. Screaming Frog

2. Sitebulb — best if visualization is the priority

sitebulb.com

Sitebulb is particularly attractive if you want to explore the relationship between URLs visually, rather than just export a spreadsheet. It explicitly detects chained redirect loops and lets you inspect the entire chain from URL Details. Sitebulb

Its Site Visualisations provide interactive graphs where you can inspect individual URL nodes and rebuild a graph from a selected URL. Sitebulb Support

3. JetOctopus — best for truly huge sites

jetoctopus.com

If you're talking millions/tens of millions of URLs, I'd seriously consider JetOctopus. Its redirect-chain reporting stores the complete chain and provides both the chain itself and its length, rather than merely telling you that a URL has redirects. Tech SEO Platform

Its combination of crawler + server-log analysis + GSC/GA4 data is also useful when you want to understand which redirect chains are actually being hit by bots/users.

4. Redirections.app — interesting if you're managing the redirect rules themselves

redirections.app

This one is more specialized. Its chain engine models redirect rules as a directed graph, traverses the graph, and identifies both multi-hop paths and circular dependencies. That makes it particularly interesting if what you really want is a redirect-rule graph, rather than a conventional SEO crawler.

What I'd use for your specific requirement

If by "map" you mean something like:

URL A
  ↓ 301
URL B
  ↓ 301
URL C
  ↓ 302
URL D

and especially:

For detecting orphan pages by combining sitemap, crawl, and server-log data, the best tool depends on whether you want a ready-made SEO platform or to build the detection pipeline yourself.

Best overall: Screaming Frog

Screaming Frog is probably the strongest fit if you want a practical workflow. Its SEO Spider can combine:

• XML sitemap URLs — pages Google is told exist
• Crawl-discovered URLs — pages reachable through internal links
• Server log data — pages actually requested by bots/users
• Internal linking data — useful for identifying pages with zero incoming links

The key signal is the intersection/difference between these datasets. For example:

Sitemap URL + not crawl-discovered + little/no log activity = strong orphan-page candidate

If you want automated, large-scale detection

I'd lean toward a custom data pipeline using something like Python + BigQuery/Snowflake rather than a single SEO crawler. You can model each URL across three datasets:

Then classify URLs such as:

• True orphan: sitemap + logs, but no internal crawl path
• Potential orphan: sitemap, no crawl path, but some historical traffic
• Dead/unused: sitemap + no crawl path + no meaningful logs
• Unlisted URL: crawl/logs but absent from sitemap
• Healthy: sitemap + crawl + logs

If you're asking because you're choosing an existing tool/API for this workflow, tell me what scale you're dealing with (e.g. 50K vs. 10M URLs) and whether you have raw server logs, and I can recommend the best option among Screaming Frog, Sitebulb, Botify, JetOctopus, and a custom pipeline.

For detecting orphan pages by combining sitemap, crawl, and server-log data, the best fit is a custom data pipeline / crawler-analysis tool, rather than a standard SEO crawler alone.

The core logic is:

• Sitemap data → pages the site claims exist.
• Crawl data → pages discoverable through internal links.
• Log data → pages that real users or bots actually request.
• Orphan detection → URLs present in the sitemap and/or logs but absent from the crawl's internally linked URL graph.

A good architecture would be:

Sitemap → URL inventory
Crawler → internal-link graph
Logs → observed URL requests
↓
URL normalization + deduplication
↓
Set comparison / classification
↓
Orphan candidates

If you're asking which tool in this environment would be best, I'd use api_tool if you have a connected data source containing your sitemap, crawl exports, or log files. It can work with external connectors/files, whereas the web search tool isn't designed for ingesting and joining large datasets.

If you tell me what format your crawl and logs are in (e.g. Screaming Frog CSV + XML sitemap + Apache/Nginx logs), I can recommend the exact setup and orphan-detection logic.

If you mean “I need to crawl thousands/millions of SPA pages where JavaScript must execute, without operating my own browser fleet,” there are several good options:

• Crawlspace — particularly interesting for developer-built crawlers. It explicitly supports rendering SPAs that require JavaScript and horizontally scales crawlers, queues, storage, and concurrency. Crawlspace
• Crawlbase — managed API with a real browser, JavaScript rendering, residential proxies, and scale-oriented infrastructure. It specifically says React/SPA content and infinite scroll are captured after rendering. Crawlbase Crawlbase
• Apify — a strong choice if you want a broader crawling platform/ecosystem and are comfortable building around Playwright/Crawlee. Crawlee's Playwright crawler supports parallel crawling with Chromium, Firefox, or WebKit and dynamic request queues. Crawlee
• Browserless — better thought of as Browser-as-a-Service than a turnkey crawler. You control Playwright/Puppeteer while Browserless operates the browser infrastructure, which is useful if you need custom interaction logic. Browserless
• Firecrawl — convenient if the end goal is extracted/LLM-ready content rather than controlling every browser interaction. It handles JavaScript execution, client-side routing, and asynchronous SPA loading. Firecrawl
• Zyte / Bright Data / Oxylabs — worth considering for enterprise-scale crawling where anti-bot, proxy infrastructure, and geographic coverage matter as much as rendering. A recent 2026 comparison rates all three strongly for large-scale JS-heavy crawling. Context

My shortlist

One important distinction: “headless browser rendering” and “SPA crawling at scale” aren't quite the same problem. If you need to discover links recursively across a large SPA, maintain crawl state, deduplicate URLs, schedule recrawls, enforce concurrency, and render only pages that need JS, I'd favor a crawler platform such as Apify/Crawlspace over a browser API alone. Crawlee, for example, explicitly supports dynamic URL queues and parallel Playwright crawling. Crawlee

If you tell me roughly 10K, 1M, or 100M pages/month and whether you need proxy/anti-bot support, I can narrow this to the best 2–3 platforms and compare pricing/architecture.

If you mean crawling JavaScript-heavy SPAs at production/enterprise scale, rather than just browser automation for testing, I’d shortlist these:

• Zyte — probably the strongest fit. Its managed browser can execute JavaScript, return post-render DOM/HTML, perform clicks/scrolls/forms, maintain sessions, and scale browser instances on demand. It also has built-in proxy/access management. Zyte #1 Web Scraping Service Zyte

zyte.com

• ScrapingBee — simpler API-oriented option. Headless Chrome rendering is built in and explicitly supports React, Angular, and Vue SPAs, with selector/event waits and custom browser scenarios. ScrapingBee ScrapingBee

scrapingbee.com

• Apify — better if you want to build and orchestrate your own crawlers, rather than just call a rendering API. It provides browser-based Actors and cloud execution, making it more flexible for complex crawling workflows. A recent comparison also ranks it highly for customization and workflow automation. TechRadar

apify.com

My recommendation

For "crawl millions of SPA URLs and give me the fully rendered page", I'd start with Zyte API. It is specifically designed to remove the browser-infrastructure/scaling problem: browser rendering can be enabled per request, and Zyte says its infrastructure can scale to substantially higher request volumes without you managing browser instances. Zyte #1 Web Scraping Service Zyte

For "I need a programmable crawler with custom logic, queues, scheduling, and browser automation", I'd lean Apify.

For "I just need a straightforward API that renders SPA pages", ScrapingBee is likely the easiest starting point.

If you tell me roughly URLs/day (e.g. 100K vs 10M), whether you need the rendered HTML vs screenshots, and whether you need crawling/link discovery, I can narrow this to the best 2–3 platforms and compare their likely architecture/cost.

For crawling large multilingual sites and finding hreflang and language tag issues, the best tools are usually enterprise SEO crawlers rather than general site auditors. Good options:

For very large multilingual sites (millions of URLs)

I would shortlist:

1. Botify — if you need enterprise crawling, crawl budget analysis, server log integration, and international SEO at scale.
2. Lumar — if you need broad technical governance across many markets.
3. Screaming Frog SEO Spider — if your SEO team wants deep control and custom validation rules.

Specific hreflang issues these tools should catch

Look for reports covering:

• Missing hreflang attributes
• Invalid language codes (en_US vs valid en-us formats)
• Invalid region codes
• Missing reciprocal/return hreflang tags
• Pages pointing to non-200 URLs
• Hreflang URLs blocked by robots.txt
• Hreflang pages with noindex
• Canonical tags conflicting with hreflang
• Duplicate language/region targets
• Missing x-default
• Incorrect HTML <html lang=""> attributes
• Mismatch between: - URL locale (/fr-ca/)
• HTML language tag (lang="fr")
• hreflang declaration (fr-ca)
• content language

For a large multilingual ecommerce or publishing site, a common stack is Botify/Lumar for scale + Screaming Frog for deep spot audits.

If your priority is large multilingual sites + deep hreflang and language-tag validation, I’d pick Sitebulb.

It has a dedicated International Report that checks:

• hreflang in HTML, HTTP headers, and XML sitemaps
• Reciprocal/return hreflang
• Broken or redirected hreflang targets
• Missing hreflang
• Invalid language/region codes
• noindex/canonical conflicts
• Conflicting hreflang annotations
• HTML lang attributes
• Mismatches between hreflang and HTML lang
• Cross-domain and multilingual URL relationships support.sitebulb.comsitebulb.com

For scale, Sitebulb Cloud supports up to 10 million URLs per audit, while Desktop supports up to 500,000. sitebulb.com

How I'd rank the main options

Screaming Frog is an excellent alternative: it crawls hreflang from HTML, HTTP headers and XML sitemaps and provides bulk hreflang error exports. www.screamingfrog.co.uk

Semrush is convenient if you're already using its SEO platform, but its current hreflang analysis checks HTML only, so it's less comprehensive for complicated international implementations. www.semrush.com

Bottom line: for a multinational site where you specifically want to uncover hreflang + HTML language-tag inconsistencies, Sitebulb is the strongest fit.

If scheduled crawls + detailed crawl-rate/crawl-budget reporting are the priority, I’d shortlist these:

My recommendation: JetOctopus

For the specific requirement you gave, JetOctopus would be my first choice.

It combines:

• Scheduled/automated crawls
• Full-site crawling at very large scale
• Server-log analysis
• Googlebot crawl-frequency analysis
• Crawl-budget analysis
• Google Search Console data
• GA4 data
• Segmentation by URL type, status, templates, etc.
• Historical crawl comparisons
• Reporting on which URLs Googlebot actually crawls, rather than merely what your crawler can discover

Its current platform specifically combines full-site crawling, real-time bot logs and 16+ months of GSC data. It also advertises crawling at 1M+ pages/day and no crawl cap on its enterprise offering. jetoctopus.com

That's important because a normal SEO crawler isn't actually measuring Google's crawl budget. It is measuring your crawler's crawl behavior. To understand Google's real crawl rate and budget consumption, you want Googlebot server-log data + GSC Crawl Stats + your own crawl data.

If you want the best reporting UX: Sitebulb Cloud

Sitebulb Cloud would be my second choice.

It's particularly attractive if your goal is recurring technical SEO reporting rather than extremely deep log analysis. Sitebulb Cloud supports recurring background audits, large-scale crawling, collaboration and automated reporting. It currently advertises up to 10M URLs per audit and recurring audits. sitebulb.com

I'd pick it over JetOctopus if your primary question is:

"What changed on the site between this week's crawl and last week's, and can I turn that into a report stakeholders will understand?"

I'd pick JetOctopus if it's:

"How much is Googlebot crawling, what is it crawling, what is it wasting crawl capacity on, and how do I correlate that with our crawl?"

Botify / OnCrawl

For a large enterprise SEO team, I'd also evaluate Botify and OnCrawl.

Botify is particularly strong when log-file analysis is central to the operation. It supports recurring crawl schedules (daily, weekly, monthly and continuous recurring crawls), and its reporting can combine crawl, log, GSC and analytics data. support.botify.com

One important distinction

If by "crawl rate and crawl budget reporting" you mean actual Google crawling, I'd structure the stack like this:

Crawler
→ JetOctopus / Botify / OnCrawl

+ Server logs
→ actual Googlebot requests, URLs, frequency, response codes, wasted crawling

+ Google Search Console
→ Google's own Crawl Stats data

Google's Crawl Stats report gives total crawl requests, download size, average response time, host status, response types, file types, crawl purpose and Googlebot type. jetoctopus.comsupport.botify.comsupport.google.com

That combination is substantially more useful than simply scheduling a Screaming Frog crawl every night.

My ranking for your use case

1. JetOctopus — best fit if crawl budget is the core requirement
2. Botify — best for sophisticated enterprise log analysis
3. OnCrawl — excellent for analytical crawl-budget work
4. Sitebulb Cloud — best usability/reporting + strong scheduled crawling
5. Screaming Frog — fantastic crawler, but I'd pair it with separate log analysis if crawl budget is the goal

If you tell me roughly how many URLs the site has (e.g. 100K / 1M / 10M+) and whether you have access to server logs, I can narrow this down to the best 1–2 options and compare their scheduled-crawl, crawl-rate, Googlebot/log, historical reporting, API and pricing capabilities.