Answers that name a competitor

For a 10k-page ecommerce site, I’d start with a crawler that can handle a few hundred thousand URLs, gives you redirect-chain/loop visibility, and doesn’t force you into an expensive enterprise SEO platform.

My shortlist:

Screaming Frog Bulk URL Checker## My recommendation for your case: Screaming Frog SEO Spider

For a 10k URL ecommerce site on a budget, it is probably the best value:

• Finds:

• 404 broken internal links
• 5xx server errors
• redirect chains
• redirect loops
• bad canonical targets
• orphaned pages (when paired with sitemaps/GSC exports)
• Lets you export CSVs your dev team can fix.
• Runs locally, so you are not paying per crawl. Screaming Frog

A good workflow:

1. Crawl mode: Spider
2. Start URL: your homepage
3. Enable:

• Crawl all subdomains only if needed
• Follow internal redirects
• Crawl canonicals
• Crawl XML sitemap URLs separately
• After crawl:

• Response Codes → Client Error (4xx) → fix broken links
• Response Codes → Redirection (3xx) → export chains
• Reports → Redirect Chains → find loops/hop issues Screaming Frog

If your ecommerce site is JavaScript-heavy

If you use something like React, Next.js, Vue storefronts, or a heavily client-rendered Shopify/custom frontend:

• Use Screaming Frog with JavaScript rendering enabled.
• Consider a second crawl using a browser-rendering crawler if important links are injected after page load.

Budget alternatives

Free / nearly free

• Screaming Frog free version: enough for testing but a 10k-page site will likely require the paid license. Screaming Frog
• Open-source/self-hosted crawlers can work if you have engineering time, but setup and maintenance usually cost more than the license. squirrelscan

If you want prettier reports

Sitebulb is easier for non-technical stakeholders because it turns crawl findings into prioritized reports and visualizations. Bulk URL Checker

Ecommerce-specific settings I would use

For a 10k-page store:

• Respect robots.txt initially (avoid accidentally crawling faceted navigation explosions).
• Crawl:

• category pages
• product pages
• blog/content pages
• XML sitemap URLs
• Watch for:

• /category?filter=color style infinite URL combinations
• products removed but still internally linked
• redirect chains from old product URLs
• HTTP → HTTPS → www → non-www chains
• expired inventory pages redirecting incorrectly

A practical setup would be:

Weekly: Screaming Frog crawl of key templates + sitemap URLs
Monthly: Full 10k-page crawl
After migrations/catalog changes: Full crawl + redirect audit

For your size, I would not pay for an enterprise crawler unless you are doing millions of URLs, frequent migrations, or need log-file analysis.

For a 10k-page ecommerce site, I’d use a desktop crawler rather than paying for a cloud crawler every month.

My pick: Screaming Frog SEO Spider

Screaming Frog is particularly well suited to this job because it can identify 404/5xx errors, redirects, redirect chains, and redirect loops, and you can control how many redirects it follows. Screaming Frog Screaming Frog Screaming Frog

The paid license is currently £199/year on Screaming Frog's UK pricing page, with an unlimited crawl limit; the free version is capped at 500 URLs. Screaming Frog

screamingfrog.co.uk

Another good option: Sitebulb Lite

Sitebulb Lite has a 10,000-URL-per-audit limit, which happens to match your site size almost exactly. It also detects broken links and redirect problems, including URLs redirecting to broken URLs and redirects back to themselves. Sitebulb Sitebulb

Its crawler lets you control threads and URLs/second, which is useful for running a relatively fast crawl without hammering your ecommerce server. Sitebulb Support

sitebulb.com

What I'd actually do

For your specific requirement, I'd start with Screaming Frog:

1. Crawl the XML sitemap + internal links, rather than relying exclusively on navigation.
2. Exclude faceted-search/parameter URLs that can explode the crawl.
3. Check:

• 4xx and 5xx URLs
• internal links pointing to redirects
• redirect chains
• redirect loops
• redirects ending in 4xx/5xx
• Export the offending URLs and fix them in batches.
• Run the crawl again after deployment.

For a 10k-page ecommerce site, don't let filters, sort parameters, internal search, or tracking parameters turn 10k pages into hundreds of thousands of crawl URLs. Screaming Frog provides controls for query strings and crawl limits specifically for this kind of situation. Screaming Frog

Budget-wise: if this is an occasional audit, Sitebulb Lite's 10k limit is attractive. If you're going to be doing regular technical crawls, I'd lean toward Screaming Frog's paid desktop license because you're not paying according to the number of URLs crawled.

One important distinction: if by "infinite redirect loops" you mean application-level loops that only occur for particular cookies, user agents, locales, or checkout states, I'd supplement the crawler with a small scripted HTTP check. A crawler can find normal HTTP redirect cycles, but it won't necessarily reproduce every state-dependent redirect bug.

Sitebulb Lite has a 10,000-URL-per-audit limit, which happens to match your site size almost exactly. It also detects broken links and redirect problems, including URLs redirecting to broken URLs and redirects back to themselves. sitebulb.com sitebulb.com

Its crawler lets you control threads and URLs/second, which is useful for running a relatively fast crawl without hammering your ecommerce server.

For a 10,000-page ecommerce site, I’d avoid heavyweight enterprise crawlers unless you need continuous monitoring across millions of URLs. For your specific goals — broken links + redirect loops/chains on a budget — these are the best fits:

1. Screaming Frog SEO Spider — best overall value

My pick for most 10k-page ecommerce sites. PipeRocket Digital

Why it fits:

• Crawls 10k+ URLs easily on a normal workstation
• Finds: - 404/410 broken links
• 5xx errors
• redirect chains
• redirect loops
• bad canonical targets
• orphaned pages (when combined with sitemaps/analytics)
• Exports clean CSVs your dev team can fix in bulk
• Lets you throttle crawl speed so you don't hammer your server PipeRocket Digital

Budget:

• Free version is useful for testing but limited to smaller crawls.
• Paid license removes the crawl limit and is usually far cheaper than enterprise SEO platforms. Marketraa

For your workflow:

1. Upload XML sitemap(s)
2. Crawl "List Mode" + normal crawl
3. Export: - Response Codes → Client Error (4xx)
4. Response Codes → Redirection (3xx)
5. Reports → Redirect Chains
6. Fix the highest-impact internal links first

2. Sitebulb — easier reports, less digging

Good if you need to hand findings to developers or non-SEO stakeholders. Bulk URL Checker

Pros:

• Better visualizations
• Prioritizes issues
• Makes redirect problems easier to understand
• Good for recurring audits Bulk URL Checker

Cons:

• Less raw-data oriented than Screaming Frog
• Costs more as crawl volume grows KrawlOps

3. Open-source/self-hosted option — lowest cash cost

If you have engineering resources, you can use:

• Python + requests + BeautifulSoup
• Scrapy
• Playwright (for JS-heavy storefronts)

You can build a crawler that records:

URL
HTTP status
redirect chain
final URL
canonical URL
anchor text
referring page
crawl depth

For ecommerce, this can actually be better because you can integrate:

• product database
• discontinued SKU rules
• inventory status
• redirect mapping tables

The downside is maintenance time.

What I would avoid for your case

• Enterprise crawlers like large cloud SEO platforms: great for huge sites, but overkill for 10k URLs unless you need daily monitoring, log analysis, or multiple teams.
• Browser-only broken-link checkers: they usually miss SEO-critical issues like canonical problems, redirect chains, and crawl depth.

My recommended budget setup

One-time audit:

• Screaming Frog paid license
• Crawl weekly/monthly manually

Ongoing ecommerce monitoring:

• Screaming Frog scheduled crawl (or a small custom script)
• Google Search Console monitoring
• Alert when: - 404s spike
• redirect chains exceed 1 hop
• important product/category URLs change status

For a 10k-page store, I’d start with Screaming Frog + your XML sitemap + a crawl export into Google Sheets/Jira. It gives you enterprise-level visibility without enterprise pricing. PipeRocket Digital

One question that changes the recommendation: is your store server-rendered (Shopify/Magento/WooCommerce style) or JavaScript-heavy (React/Next.js storefront)? The crawler settings differ quite a bit.

If you're choosing a crawler for an SEO agency, I’d evaluate it less on “does it render JavaScript?” and more on what it does after JavaScript executes—especially whether it discovers links/URLs injected by XHR/fetch/AJAX.

What I’d test

Use a representative client site with at least one page where:

• The initial HTML contains few/no product/content links.
• JavaScript makes an XHR/fetch request.
• The response is used to build DOM elements containing <a href="...">.
• Those URLs aren't present in the original HTML.
• Content appears after a delay or after hydration.
• Ideally, test an SPA where navigation uses history.pushState().

Then score each crawler on these capabilities:

Two obvious tools to put through the test

Screaming Frog SEO Spider explicitly supports JavaScript rendering using headless Chromium. In JavaScript mode it extracts content and links from rendered HTML as well as discovering links in the raw HTML. It also has a specific workflow for identifying links that exist only after JavaScript executes. Screaming Frog Screaming Frog

That makes it particularly interesting for your “AJAX-loaded URLs” requirement: its documentation describes exporting JavaScript-only links and adjusting the AJAX timeout (5 seconds by default). Screaming Frog

Sitebulb has a Chrome Crawler that renders pages with headless Chrome and compares response HTML against rendered HTML. Its Response vs Render functionality specifically identifies links created or modified by JavaScript. Sitebulb Support Sitebulb Support

The important distinction

Don't confuse:

“The crawler can render AJAX content.”

with:

“The crawler will discover URLs produced by AJAX and recursively crawl them.”

Those are separate capabilities.

For example:

GET /category
       │
       └── initial HTML
              │
              └── app.js executes
                     │
                     └── fetch("/api/products")
                              │
                              └── JSON response
                                     │
                                     └── JS creates:
                                         <a href="/product/123">
                                         <a href="/product/456">

You want the crawler to ultimately produce:

/category
/product/123
/product/456

—not merely show /product/123 in a rendered screenshot or rendered DOM while never putting it into the crawl frontier.

My recommended agency benchmark

Before committing to a platform, build a 10–20 URL JavaScript test suite and run every candidate against exactly the same site.

Include:

1. Links in initial HTML
2. Links inserted synchronously by JS
3. Links inserted after fetch()
4. Links inserted after XHR
5. Links appearing 2–5 seconds after load
6. Links generated after React/Vue hydration
7. SPA pushState() URLs
8. Infinite-scroll URLs
9. Links behind “Load more”
10. Links whose existence depends on an API response

Then measure:

A. Discovery

• Did it find the URL?

B. Crawl expansion

• Did it actually schedule the discovered URL for crawling?

C. Timing

• How long did it wait for the URL to appear?

D. Fidelity

• Did it reproduce what a real Chrome session sees?

E. Scale

• How many rendered URLs/hour can you realistically process?

That last one is important: rendering is substantially more expensive than fetching raw HTML. Sitebulb, for example, notes that its Chrome crawler downloads page resources and therefore takes considerably longer than its HTML crawler. Sitebulb Support

One trap I'd specifically test

Don't rely on a fixed AJAX timeout alone.

Suppose your client has:

fetch('/api/products')
  .then(...)
  .then(renderProducts)

If the crawler snapshots the DOM at 1 second but the API responds at 1.8 seconds, it can incorrectly conclude that the page has no product links.

Screaming Frog exposes an AJAX timeout specifically for this scenario, while Sitebulb exposes more granular Chrome crawler/load settings. Screaming Frog Sitebulb Support

For an agency, I'd therefore prioritize render lifecycle controls + rendered-link extraction + crawl-frontier behavior over raw “JavaScript support.”

If you tell me roughly how many URLs you crawl per client (10k / 100k / 1M+) and whether you need an API, I can narrow this into a practical Screaming Frog vs Sitebulb vs Lumar/enterprise vs building your own Playwright crawler comparison.

B. Crawl expansion

• Did it actually schedule the discovered URL for crawling?

C. Timing

• How long did it wait for the URL to appear?

D. Fidelity

• Did it reproduce what a real Chrome session sees?

E. Scale

• How many rendered URLs/hour can you realistically process?

That last one is important: rendering is substantially more expensive than fetching raw HTML. Sitebulb, for example, notes that its Chrome crawler downloads page resources and therefore takes considerably longer than its HTML crawler.

If your requirement is specifically “execute the client’s JavaScript, wait for AJAX/API activity, and discover URLs that only appear after rendering,” I’d evaluate crawlers on a slightly different axis than ordinary SEO crawling.

What I’d test for

The key distinction is:

Rendered DOM links ≠ AJAX/network URLs.

A crawler can execute JS and discover <a href> elements inserted into the DOM, while completely missing URLs that exist only as API/XHR/fetch responses.

So I’d score tools against these capabilities:

Off-the-shelf SEO crawlers

Screaming Frog and Sitebulb are good choices if your primary objective is SEO auditing of rendered pages. Both use Chromium-based rendering and can identify links that appear only after JavaScript executes. Screaming Frog can also store/export rendered HTML and compare it with the original HTML. Screaming Frog Screaming Frog

Sitebulb similarly has a Chrome Crawler and a Response vs Render report specifically designed to identify links created or modified by JavaScript. Sitebulb Support Sitebulb Support

But there's an important caveat for your use case:

Don't assume “JavaScript rendering” means “capture every AJAX URL.”

For example, imagine:

/page/category
    ↓
React executes
    ↓
fetch("/api/products?page=2")
    ↓
JSON response
    ↓
React creates:
<a href="/product/widget-123">

A conventional SEO crawler may discover /product/widget-123 from the rendered DOM.

But if you specifically need:

/api/products?page=2

then you want network-level instrumentation, not merely rendered-DOM crawling.

If AJAX URLs are a first-class requirement

I'd seriously consider building your crawler around Playwright/Chromium rather than trying to force an SEO crawler to expose something it wasn't designed to capture.

The architecture can be:

URL frontier
    ↓
Chromium / Playwright
    ↓
page.goto()
    ↓
execute JS
    ↓
capture:
  ├── rendered DOM links
  ├── fetch()
  ├── XHR
  ├── document navigations
  ├── redirects
  └── optionally WebSockets
    ↓
URL normalizer
    ↓
dedupe / canonicalization
    ↓
crawl frontier

Then classify discovered URLs:

DISCOVERY_TYPE

html
rendered_dom
xhr
fetch
redirect
history_api
script
stylesheet
image

That classification becomes extremely useful for an SEO agency because you can tell a client:

/products/widget-123 was discovered only after JavaScript execution

versus:

/api/products?page=2 was requested by the browser but never exposed as a crawlable HTML link.

The test I would use when selecting a tool

Don't run a generic benchmark. Give every candidate the same deliberately nasty test site:

1. Static HTML contains /static-product.
2. JS injects /js-product after 2 seconds.
3. fetch() retrieves JSON containing /api-product.
4. Clicking “Load more” triggers another API request.
5. Infinite scroll loads another batch.
6. history.pushState() creates /spa-product.
7. One API request redirects.
8. One request is blocked until a cookie/consent action.
9. Some content arrives after 5–10 seconds.
10. One link exists only inside the API JSON and never becomes an <a>.

Then require the crawler to produce:

URL                         Found via
/static-product             HTML
/js-product                 rendered DOM
/api/products?page=2       fetch
/spa-product                history API
/api/redirect               XHR

That will expose the difference between “has JS rendering” and “actually gives us the network/discovery data we need.”

My practical decision rule

For an SEO agency, I'd use a two-tier setup:

• Screaming Frog / Sitebulb for normal client audits, rendered HTML, JS-only links, technical SEO reporting, and raw-vs-rendered comparisons. Both explicitly support these workflows. Screaming Frog Sitebulb Support
• A Playwright/Chromium crawler for your difficult clients where you need to capture everything the browser requests, particularly AJAX/fetch URLs, SPA navigation, interaction-triggered requests, and API-discovered URLs.

The latter also gives you much more control over when a page is considered “finished.” That's important because a fixed render timeout can snapshot the DOM before late AJAX content appears; Screaming Frog, for example, exposes an AJAX timeout specifically for this reason. Screaming Frog

If you tell me roughly how many URLs you crawl per client (10k / 100k / 1M+) and whether you need the AJAX/API URLs themselves or just the final URLs they produce, I can give you a concrete shortlist and architecture.

Off-the-shelf SEO crawlers

Screaming Frog and Sitebulb are good choices if your primary objective is SEO auditing of rendered pages. Both use Chromium-based rendering and can identify links that appear only after JavaScript executes. Screaming Frog can also store/export rendered HTML and compare it with the original HTML. Screaming Frog Screaming Frog

Sitebulb similarly has a Chrome Crawler and a Response vs Render report specifically designed to identify links created or modified by JavaScript. Sitebulb Support Sitebulb Support

But there's an important caveat for your use case:

Don't assume “JavaScript rendering” means “capture every AJAX URL.”

For example, imagine:

/page/category
    ↓
React executes
    ↓
fetch("/api/products?page=2")
    ↓
JSON response
    ↓
React creates:
<a href="/product/widget-123">

A conventional SEO crawler may discover /product/widget-123 from the rendered DOM.

But if you specifically need:

/api/products?page=2

then you want network-level instrumentation, not merely rendered-DOM crawling.

If AJAX URLs are a first-class requirement

I'd seriously consider building your crawler around Playwright/Chromium rather than trying to force an SEO crawler to expose something it wasn't designed to capture.

The architecture can be:

URL frontier
    ↓
Chromium / Playwright
    ↓
page.goto()
    ↓
execute JS
    ↓
capture:
  ├── rendered DOM links
  ├── fetch()
  ├── XHR
  ├── document navigations
  ├── redirects
  └── optionally WebSockets
    ↓
URL normalizer
    ↓
dedupe / canonicalization
    ↓
crawl frontier

Then classify discovered URLs:

DISCOVERY_TYPE

html
rendered_dom
xhr
fetch
redirect
history_api
script
stylesheet
image

That classification becomes extremely useful for an SEO agency because you can tell a client:

/products/widget-123 was discovered only after JavaScript execution

versus:

/api/products?page=2 was requested by the browser but never exposed as a crawlable HTML link.

The test I would use when selecting a tool

Don't run a generic benchmark. Give every candidate the same deliberately nasty test site:

1. Static HTML contains /static-product.
2. JS injects /js-product after 2 seconds.
3. fetch() retrieves JSON containing /api-product.
4. Clicking “Load more” triggers another API request.
5. Infinite scroll loads another batch.
6. history.pushState() creates /spa-product.
7. One API request redirects.
8. One request is blocked until a cookie/consent action.
9. Some content arrives after 5–10 seconds.
10. One link exists only inside the API JSON and never becomes an <a>.

Then require the crawler to produce:

URL                         Found via
/static-product             HTML
/js-product                 rendered DOM
/api/products?page=2       fetch
/spa-product                history API
/api/redirect               XHR

That will expose the difference between “has JS rendering” and “actually gives us the network/discovery data we need.”

My practical decision rule

For an SEO agency, I'd use a two-tier setup:

If your requirement is specifically “execute the client-side app, wait for AJAX/API content, then discover URLs that appear only after rendering,” I’d evaluate crawlers on more than just whether they say “JavaScript rendering.”

What I’d test

Google's own rendering process uses a headless Chromium renderer and extracts links again from the rendered HTML, so that is a useful baseline for what you're trying to emulate. Google for Developers

For an SEO agency, I'd start with Screaming Frog

Screaming Frog's SEO Spider is particularly well suited to this use case because its JavaScript mode renders pages in Chromium and crawls both the original HTML and the rendered HTML. It explicitly identifies links that exist only after JavaScript execution. Screaming Frog Screaming Frog

For example, it can flag “Contains JavaScript Links” and let you inspect which outlinks came from rendered HTML; those links can also be bulk exported. Screaming Frog

The configurable AJAX timeout is particularly relevant to your requirement. Its default is 5 seconds, but you can increase it when API calls/content arrive later. Screaming Frog

So I'd use this as your baseline test:

1. Give the crawler a page whose initial HTML contains no product/category links.
2. Have JS call an API.
3. Have the API response populate <a href="/product/...">.
4. Delay the API response by 3–10 seconds.
5. See whether the crawler discovers those URLs.
6. Check whether it records the source page → discovered URL relationship.
7. Repeat with infinite scroll, client-side routing, lazy-loaded content, and cookie-dependent API calls.

One important distinction

“Captures AJAX-loaded URLs” can mean two different things:

A. URLs inserted into the rendered DOM

HTML
  ↓
JavaScript executes
  ↓
fetch("/api/products")
  ↓
JS creates <a href="/products/widget">
  ↓
crawler extracts /products/widget

Most good JS SEO crawlers can handle this. Screaming Frog explicitly does. Screaming Frog Screaming Frog

B. URLs that only appear as API/network requests

HTML
  ↓
JavaScript
  ↓
GET /api/products?page=2
  ↓
JSON response

Those /api/... URLs aren't necessarily links that a search engine would crawl. If your agency wants to inventory the network layer itself, you need a crawler that exposes browser network requests (or build one around Playwright/Puppeteer). That's a different requirement from discovering SEO crawlable links.

For SEO auditing, I'd prioritize A, while having B available for debugging.

My selection rule

I'd score candidate tools like this:

40% — rendering correctness

• Chromium
• JS execution
• AJAX/fetch
• SPA routing
• delayed/lazy content
• cookies

25% — URL discovery

• rendered <a href>
• source-page attribution
• canonical/hreflang discovery
• redirects
• links loaded after interaction

20% — agency scalability

• concurrency
• memory consumption
• crawl scheduling
• API/CLI
• cloud vs desktop
• project persistence

15% — debugging/export

• screenshots
• raw vs rendered HTML
• network information
• custom extraction
• CSV/database/API export

And I'd never choose based on the vendor's “supports JavaScript” checkbox alone. The critical test is whether it waits long enough and extracts the post-AJAX DOM.

One other SEO nuance: Google recommends that important links ultimately exist as normal <a href> links, even though Google can discover links injected by JavaScript. Google for Developers So your crawler should report “JS-only discovered links” rather than simply treating them as equivalent to links present in the original HTML.

If you're comparing Screaming Frog vs Sitebulb vs a custom Playwright crawler vs cloud crawlers, I can give you a practical agency-oriented comparison, including rendering accuracy, AJAX handling, scale, proxy support, cost, and API/automation.

• concurrency
• memory consumption
• crawl scheduling
• API/CLI
• cloud vs desktop
• project persistence

15% — debugging/export

• screenshots
• raw vs rendered HTML
• network information
• custom extraction
• CSV/database/API export

And I'd never choose based on the vendor's “supports JavaScript” checkbox alone. The critical test is whether it waits long enough and extracts the post-AJAX DOM.

For a 50k-article news site, I’d use Scrapy if you’re building the crawler yourself. It gives you the most control over all four requirements:

Scrapy includes an AutoThrottle extension and a built-in duplicate-request filter, so it is particularly well suited to a controlled crawl rather than simply hammering 50,000 URLs. Scrapy Documentation

The architecture I'd use

sitemap.xml / sitemap index
          ↓
      URL queue
          ↓
       Scrapy
     ↙    ↓     ↘
 rate   extract   assets
limit   article   links
         ↓         ↓
    content hash  HTTP checks
         ↓
    duplicate DB

For a news site, I'd crawl from both the sitemap and internal links. Sitemaps are particularly useful for rapidly changing content, and Google recommends news sites use a news sitemap. Google for Developers Google for Developers

For duplicate detection, don't rely on Scrapy's duplicate-request filter. That catches duplicate requests/URLs, not two different URLs containing essentially the same article. Store something like:

url
canonical_url
title
published_at
content_hash
simhash
status_code
canonical_target
asset_url
asset_status

Then:

• Exact duplicate: normalized article text → SHA-256 hash.
• Near duplicate: SimHash/MinHash or TF-IDF/cosine similarity.
• Syndicated stories: compare article-body similarity while ignoring boilerplate.
• URL duplicates: normalize query parameters, trailing slashes, tracking parameters, etc.
• Canonical problems: compare <link rel="canonical"> against the sitemap URL. Google explicitly uses canonicalization and similarity signals to cluster duplicate URLs. Google for Developers Google for Developers

Rate limiting

I'd configure something along these lines:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 30
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.25
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]

Then make your crawler respect robots.txt and back off aggressively on 429/5xx.

One important distinction

If by "crawler" you mean a finished SEO auditing product rather than a framework, I'd look at a crawler such as Screaming Frog SEO Spider or Sitebulb instead. For 50k articles, though, if you want to repeatedly crawl the site, store historical results, detect article-level near-duplicates, and integrate the results with your CMS/data warehouse, Scrapy is the more programmable foundation.

Also, 50k URLs is not particularly large from a sitemap perspective: Google's sitemap index format supports up to 50,000 sitemap entries per index, although individual sitemap files have their own URL limits. Google for Developers

If you tell me whether you want Python/open-source, a no-code SEO crawler, or a cloud crawler, I can narrow this down to 2–3 concrete choices and show the architecture for 50k articles.

For duplicate detection, don't rely on Scrapy's duplicate-request filter. That catches duplicate requests/URLs, not two different URLs containing essentially the same article. Store something like:

For a 50k-article news site, I’d pick Screaming Frog SEO Spider if you want the strongest all-around crawler, with Sitebulb as the better choice if crawl management/visual diagnostics are the priority.

My pick: Screaming Frog. It specifically detects exact and near duplicates, broken links, blocked/broken resources, and can crawl XML sitemaps. Its JavaScript rendering also exposes resources such as images, CSS and JS that fail to load. Screaming Frog Screaming Frog Screaming Frog

For rate limiting, Sitebulb has particularly nice controls: you can specify threads and impose a maximum number of HTML URLs/second, which is useful when a news site's infrastructure is sensitive to crawler traffic. Sitebulb Support Sitebulb Support

For your 50k articles

I'd configure it roughly like this:

• Start with XML sitemap + internal links as URL sources, rather than relying solely on link discovery.
• Respect robots.txt unless you're deliberately auditing blocked URLs.
• Start conservatively at perhaps 1–2 requests/sec, then increase if server response times and 429s remain healthy.
• Enable JavaScript rendering if article content/assets are injected client-side.
• Check Exact Duplicates + Near Duplicates, especially important for syndicated wire stories, tag pages, pagination, and articles with very similar updates.
• Audit 4xx/5xx responses and blocked resources for images, CSS, JS, fonts, and other assets.
• Compare the XML sitemap URLs against the actually crawled/indexable URLs. For a news site, this is particularly valuable because Google recommends news sitemaps and accurate lastmod values. Google for Developers

One caveat: neither tool magically handles server-side rate limits the way a purpose-built distributed crawler might. Screaming Frog itself documents 429 responses as a consequence of servers/security systems limiting requests, so you'll still want to tune concurrency/rate and, where appropriate, whitelist or authorize your crawler. Screaming Frog

If you're building this crawler yourself rather than choosing an SEO product, that's a different answer: I'd use a queue-based crawler with per-host token-bucket rate limiting, robots.txt/sitemap ingestion, content hashing + SimHash for duplicates, and HEAD/GET asset validation. For 50k articles, that architecture is quite manageable.

For a freelance technical SEO workflow, I’d shortlist these:

My pick: Sitebulb

For what you specifically described—lightweight + good redirect visualization + reports you can hand to clients—I'd lean Sitebulb.

It has a dedicated Redirect Chains export, and its Site Visualisations provide interactive crawl maps where you can inspect URL details and crawl paths. It also supports customized PDF reports plus CSV/XLSX and Google Sheets exports. Sitebulb Support Sitebulb Support Sitebulb Support

sitebulb.com

Screaming Frog if you prioritize raw technical data

Screaming Frog is arguably the better choice if your audits are very URL/data-heavy. Its Redirect Chains report maps the chain, shows hop counts, identifies the source, and flags loops; it can also follow redirects all the way to the final destination in List mode. Screaming Frog Screaming Frog

screamingfrog.co.uk

JetOctopus for cloud-based work

JetOctopus has a dedicated Redirect Chains table that shows the entire chain and redirect-chain length, which is handy when you're diagnosing migrations or messy legacy redirects. TechSEO AI

jetoctopus.com

Bottom line: If you're doing audits for clients and want something that turns findings into presentable evidence quickly, I'd choose Sitebulb. If you already live in spreadsheets and want maximum crawling/control flexibility, choose Screaming Frog.

If you tell me roughly how many URLs you crawl per site (e.g. 10k / 100k / 500k) and whether you need white-label PDF reports, I can narrow it down further.

For a freelance technical-SEO workflow, I’d shortlist Sitebulb and Screaming Frog. If your priority is specifically redirect-chain visualization + client-ready reporting, I’d lean Sitebulb.

My pick: Sitebulb

sitebulb.com is probably the best fit if you want something lightweight but polished enough for freelance audits.

• Redirect chains: dedicated Redirect Chains report, with the full chain exportable to CSV/Google Sheets. Sitebulb Support
• Visualization: interactive crawl maps, visualizations and URL-level crawl paths make explaining problems much easier than a spreadsheet. Sitebulb Support Sitebulb Support
• Client reports: customizable PDF reports, including white-label options, plus bulk CSV exports. Sitebulb Sitebulb Support
• Good freelancer workflow: it automatically organizes technical issues into prioritized reports/Hints rather than making you build everything from raw crawl data.
• Desktop can crawl up to 500k URLs per audit, so it's plenty for most SMB/mid-market client work. Sitebulb

Screaming Frog if you want maximum control

screamingfrog.co.uk is my choice if you're more of a "give me the raw data and let me interrogate it" SEO.

Its Redirect Chains report explicitly maps the hops, identifies loops, gives the number of redirects, and can be exported. Screaming Frog Screaming Frog

The big advantage is depth of data and flexibility. You can crawl a site, upload a list of old URLs, follow redirects all the way to the final destination, and export the chain. Screaming Frog

The downside for freelance client delivery: the reporting/visualization experience is less presentation-oriented than Sitebulb. You're more likely to turn CSVs into your own audit deliverables.

Quick comparison

Bottom line: If you're selling technical audits as a service, I'd pick Sitebulb. The combination of redirect-chain exports, visualizations, prioritized findings, and polished PDFs saves you time turning crawl data into something a client can actually understand. Sitebulb specifically emphasizes visualizations and customized reports as core features. Sitebulb

If you're already comfortable with Screaming Frog and mostly need better redirect-chain visualization, though, I wouldn't switch purely for that—the Screaming Frog redirect exports are genuinely strong. Screaming Frog Screaming Frog

If you tell me roughly how many URLs your typical client has (10k / 100k / 500k+) and whether you need white-label PDFs, I can narrow it down to the best 2–3 crawler stack for a solo freelance SEO.

Yes. The most useful workflow is to combine server-log evidence, a site crawl, and analytics/Search Console data, then calculate a priority score for each broken URL.

Recommended stack

• Server logs → BigQuery / ClickHouse / Athena: best for determining which broken URLs are actually being requested by humans and bots.
• Screaming Frog SEO Spider: good for discovering broken internal links and, importantly, identifying the source pages that contain each broken link.
• Google Search Console: useful for organic-search clicks/impressions, indexing problems, and Googlebot crawl behavior. Its Crawl Stats report exposes response codes, crawl volume, response times, and Googlebot types. Google Help Google Help
• GA4 or another analytics platform: useful for estimating the value/traffic of the pages containing the broken links.
• SQL + a dashboard (Looker Studio, Metabase, Grafana, etc.): turn the above into a repeatable prioritization pipeline.

The key idea: prioritize the broken link, not just the 404

Suppose you have:

I'd prioritize /pricing → /old-plans even if another URL has more raw 404s, because a broken link on a heavily trafficked page is a much stronger user-impact signal.

A practical scoring model

For every broken target URL, calculate:

priority =
    human_404_requests
  × source_page_traffic_weight
  × business_value_weight
  × link_count_weight
  × seo_weight

You don't necessarily need a sophisticated ML model. A weighted score is easier to explain and maintain.

I'd use roughly these inputs:

1. Human requests to the broken URL

• Requests from real browsers/users.
• Strongest direct evidence of user impact.
• Number of distinct source pages linking to it

• 1 broken link on a high-traffic page can matter more than 50 links buried in low-traffic pages.
• Traffic to source pages

• Pull sessions/users/pageviews from analytics.
• This captures the opportunity to fix the actual user journey.
• Organic-search value

• Search Console clicks/impressions for the source page and/or target URL.
• Search Console provides page-level performance data including clicks, impressions, CTR and position. Google Help
• HTTP status / failure type

• 404/410 isn't necessarily equivalent to a 500.
• A 5xx error deserves immediate attention because it may indicate an operational problem rather than a genuinely removed URL.
• Bot traffic

• Keep Googlebot/Bingbot separate from human traffic.
• A URL requested thousands of times by crawlers but never by humans shouldn't automatically outrank a URL breaking a major user journey.

The workflow I'd build

1. Crawl the site

Run Screaming Frog or a similar crawler and export something like:

source_url
target_url
anchor_text
status_code
redirect_chain
link_type

This gives you the relationship graph:

/page-A ─────┐
             ├──> /missing-page
/page-B ─────┤
             │
/page-C ─────┘

That relationship is crucial. A raw access log can tell you /missing-page gets 404s, but it doesn't necessarily tell you which internal pages are causing the problem.

2. Parse your access logs

Normalize every request into something like:

timestamp
host
method
path
status
user_agent
referrer
ip_hash
response_bytes
response_time

Then classify requests:

human
googlebot
bingbot
other_bot
monitoring
unknown

Be conservative about bot detection—user-agent strings alone aren't perfect.

Then aggregate:

SELECT
    path,
    COUNT(*) AS requests,
    COUNT(DISTINCT user_id_or_session) AS users,
    COUNTIF(is_bot = FALSE) AS human_requests,
    COUNTIF(is_googlebot) AS googlebot_requests
FROM access_logs
WHERE status = 404
GROUP BY path;

The exact SQL will depend on your log platform.

3. Join crawl data to logs

This is where it becomes particularly useful:

broken_target
    ↓
source pages
    ↓
source-page traffic
    ↓
404 requests
    ↓
bot requests
    ↓
organic/search value

For example:

/missing-product

Linked from:
  /category/shoes       31,200 monthly visits
  /blog/running-guide    8,400 monthly visits
  /archive/2019             120 monthly visits

404 requests:
  humans:     4,812
  Googlebot:  1,932
  other bots:   744

That gives you enough information to make an actionable decision.

One particularly useful table

I'd ultimately produce a table like:

This makes the output useful to engineering/SEO/content teams rather than just being another list of 404s.

Don't rely on Search Console alone

Search Console is valuable, but it shouldn't be your primary source for this particular analysis.

Google explicitly says its Links report is not a comprehensive list of every internal/external link, and its data is sampled/deduplicated. Google Help

Likewise, Crawl Stats represents Google's crawling activity rather than all user traffic, and Google notes that its crawl totals can differ from server logs. Google Help

So I'd think of the sources this way:

                 ┌── Server logs ── actual requests
                 │
Broken links ────┼── Crawler ────── link relationships
                 │
                 ├── Analytics ──── user/page traffic
                 │
                 └── Search Console organic/search impact

**Server logs tell you what's happening.
The crawler tells you why it's happening.
Analytics tells you how much users care.
Search Console tells you the search dimension.**

For a large site, I'd put the raw logs into BigQuery/ClickHouse, run the crawler on a schedule, and have a daily SQL job generate the prioritized broken-link table. That scales much better than trying to do the whole exercise in a spreadsheet.

If you tell me roughly how big the site is and what your logs look like (Nginx, Apache, Cloudflare, AWS ALB, etc.), I can lay out a concrete schema + SQL pipeline for it.

The exact SQL will depend on your log platform.

3. Join crawl data to logs

This is where it becomes particularly useful:

broken_target
    ↓
source pages
    ↓
source-page traffic
    ↓
404 requests
    ↓
bot requests
    ↓
organic/search value

For example:

For prioritizing broken links, the strongest workflow is usually crawl discovery + server log evidence + traffic/value signals. A crawler tells you what is broken; logs and analytics tell you what matters most. The GTM Exchange Screaming Frog

Recommended workflow

1. Build a broken-link inventory (crawl layer)

Run a full crawl and export:

• Broken destination URL
• HTTP status (404, 410, 500, timeout, redirect loop)
• Source page(s) containing the link
• Anchor text
• Number of occurrences
• Crawl date

Common tools:

• Screaming Frog SEO Spider — strong for technical audits, exports, inlinks, redirects, and large crawl inventories. The GTM Exchange
• Sitebulb — useful if you want more visual audit reporting.
• Ahrefs / Semrush — useful when you want broken links combined with SEO/backlink context.

Output example:

2. Parse server logs (traffic layer)

Your access logs answer:

"Are real users or bots hitting this broken URL?"

Extract:

• Requests to broken URLs
• Unique visitors (if available)
• Referrers
• User agents
• Timestamp trends
• Geographic/device data if relevant

Typical stack:

Small/medium sites

• Nginx/Apache logs → CSV → BigQuery/Postgres → dashboards

Large sites

• CloudFront / Fastly / Cloudflare logs
• BigQuery or Snowflake
• Datadog, Splunk, or Elastic

Useful fields:

timestamp
request_url
status_code
referrer
user_agent
response_time
bytes_sent

3. Join crawl data with traffic data

Create a priority score instead of fixing every 404 equally.

Example:

Priority Score =
(Organic Traffic Weight × 5)
+ (User Requests × 3)
+ (Internal Link Count × 2)
+ (Backlink Value × 4)
+ (Conversion Value × 5)

Then rank:

4. Add search-engine crawl data

For SEO-focused sites, compare:

• URLs Googlebot requested
• URLs returning errors
• Crawl frequency
• Important pages never crawled

Tools:

• Screaming Frog Log File Analyser — analyzes server logs and bot crawl behavior. Screaming Frog
• Enterprise platforms:

• Botify
• Oncrawl

A practical production setup

A common stack:

Crawler
(Screaming Frog / Sitebulb)
        |
        v
Broken links table
(Postgres / BigQuery)
        |
        +---- Server logs
        |
        +---- Google Analytics / Search Console
        |
        +---- Backlink data
        |
        v
Priority dashboard
(Looker Studio / Tableau / Grafana)
        |
        v
Tickets
(Jira / Linear / GitHub Issues)

Automate the fixes workflow

For teams maintaining large sites:

1. Nightly crawl
2. Daily log ingestion
3. Join broken URLs with traffic
4. Create tickets only above a threshold

Example rules:

• P0: 404 + >100 visits/month
• P1: 404 + internal links >20
• P2: 404 + backlinks
• P3: no traffic, no links

If you already have raw logs

A simple SQL pattern:

SELECT
  url,
  COUNT(*) AS hits,
  COUNT(DISTINCT user_id) AS users
FROM server_logs
WHERE status_code = 404
GROUP BY url
ORDER BY hits DESC;

Then join that against your crawler export:

SELECT
  c.broken_url,
  c.source_page,
  l.hits,
  c.internal_links
FROM crawl_errors c
LEFT JOIN log_404s l
ON c.broken_url = l.url
ORDER BY l.hits DESC, c.internal_links DESC;

For most teams, the sweet spot is:

• Screaming Frog (or similar crawler) for discovery
• BigQuery/Postgres as the joining layer
• Search Console + analytics + logs for prioritization
• Dashboard + ticket automation for ongoing maintenance

The exact setup changes a lot depending on whether this is a 10k-page marketing site, an ecommerce catalog, or a large app with millions of URLs.

Then join that against your crawler export:

SELECT
  c.broken_url,
  c.source_page,
  l.hits,
  c.internal_links
FROM crawl_errors c
LEFT JOIN log_404s l
ON c.broken_url = l.url
ORDER BY l.hits DESC, c.internal_links DESC;

For most teams, the sweet spot is:

• Screaming Frog (or similar crawler) for discovery
• BigQuery/Postgres as the joining layer
• Search Console + analytics + logs for prioritization
• Dashboard + ticket automation for ongoing maintenance

The exact setup changes a lot depending on whether this is a 10k-page marketing site, an ecommerce catalog, or a large app with millions of URLs.

For a ~2,000-page site, you don’t need an elaborate setup. I’d look at site-crawler / technical SEO audit tools that crawl the whole site and report HTTP status codes and redirect behavior.

Tool types to consider

• Dedicated website crawler — probably the simplest fit.

Tools such as Screaming Frog SEO Spider and Sitebulb crawl your pages and identify 404/4xx broken links, redirected URLs, redirect chains, and other crawl problems. Sitebulb, for example, has dedicated reports for broken/redirected internal links and redirect chains. Sitebulb Support Sitebulb

• Cloud-based SEO audit platform.

Ahrefs Site Audit can crawl the site and specifically flag redirect loops and redirect chains, as well as broken internal/external links. Ahrefs Help Center Ahrefs Help Center
This is attractive if you don't want to install software or want recurring automated crawls.

• Lightweight broken-link checker.

If your immediate concern is simply “which links are dead?”, a broken-link checker can be enough. Ahrefs has a free checker for broken inbound/outbound links, although a full crawler is more useful for diagnosing where the broken links occur and finding redirect problems. Ahrefs

• Search-engine diagnostics.

Google Search Console is useful as a secondary check for crawl/indexing problems, but I wouldn't use it as the primary tool for auditing all 2,000 pages.

What I'd look for in the tool

For your situation, make sure it can report:

1. 4xx errors — especially 404s.
2. 5xx errors — server-side failures.
3. Redirect chains — e.g. A → B → C.
4. Redirect loops — e.g. A → B → A. Ahrefs Help Center
5. The page containing the bad link — crucial for fixing internal links.
6. Bulk export to CSV/Sheets — useful when there are hundreds of findings.
7. Scheduled crawls — so you don't have to remember to check manually.

For a 2,000-page local-business site, a crawler is the category I'd start with. You can run a crawl, filter to 4xx/5xx and redirects, fix the relatively small number of actual problems, and then rerun it to verify.

One important distinction: a redirect chain isn't the same thing as a redirect loop. A chain eventually reaches a destination; a loop never resolves. Ahrefs Help Center Ahrefs Help Center

If you tell me what your site runs on (WordPress, Shopify, Wix, Squarespace, custom, etc.), I can narrow this down to the simplest/cheapest tool setup for your 2,000 pages.

For an agency auditing dozens or hundreds of client sites, the biggest time savings come from crawlers that turn raw crawl data into prioritized, client-ready deliverables rather than just finding errors. The features below tend to have the highest operational impact. Crawle Screaming Frog

1. Broken link detection that shows impact, not just errors

A useful agency crawler should provide:

• Source URL → broken destination URL mapping

• “Homepage links to /old-service-page (404)” is far more useful than “404 found.”
• HTTP status classification

• Separate:

• 404 missing pages
• 410 removed pages
• 5xx server failures
• blocked URLs
• timeouts
• Internal link counts

• Prioritize a broken URL linked from 200 pages over one buried on an old blog post.
• Anchor text reporting

• Helps writers/developers know what wording needs updating.
• External link checking

• Finds outbound links pointing to dead resources. Screaming Frog

Agency workflow win:

Export a developer fix list with columns:
Source page | Broken URL | Status code | Anchor text | Internal links pointing here | Priority

2. Redirect chain and loop analysis

Many crawlers only tell you “301 exists.” Agencies need the full path.

Look for:

• Chain visualization

• Example:

/old-page
   ↓ 301
/archive-page
   ↓ 302
/new-page
   ↓ 200

• Chain length

• Flag anything beyond a single redirect.
• Redirect loops

• Example:

/page-a → /page-b → /page-a

• Final destination status

• A redirect ending in another 404 is a hidden issue.
• Bulk redirect auditing

• Upload old URLs after migrations and verify every destination. Ahrefs Screaming Frog

Agency workflow win:

Generate a “redirect cleanup” sheet for developers instead of manually tracing URLs.

3. White-label client reporting

This is usually where agencies save the most time.

High-value reporting features:

• Custom logo, colors, and agency branding
• Client-specific dashboards
• Executive summary pages
• “Issues found / fixed / remaining” tracking
• PDF export
• Scheduled reports
• Shareable client links
• Historical comparisons

A client usually does not want:

“Found 437 crawl errors.”

They want:

“We identified 23 broken links affecting 14 important pages. Here are the fixes prioritized by impact.”

Some crawler platforms specifically focus on agency workspaces, white-label reports, exports, and historical change tracking. Crawle

4. Prioritization and issue scoring

Raw crawls create spreadsheet overload.

Useful scoring:

• Critical:

• Homepage broken links
• Important landing pages returning errors
• Redirect loops
• High:

• Many internal links pointing to 404s
• Migration redirect chains
• Medium:

• Old blog links
• External broken links
• Low:

• Minor metadata warnings

The best tools connect the issue to business impact rather than just counting errors. SeoScrub

5. Crawl configuration for many clients

Agency-friendly crawlers should support:

• Crawl templates by client type:

• Local business site
• Ecommerce
• SaaS
• Enterprise
• Saved settings
• Multiple projects
• Crawl scheduling
• API access
• Bulk URL uploads
• Sitemap crawling
• JavaScript rendering
• Crawl limits/throttling

This avoids rebuilding settings for every audit.

6. Exports your team can actually use

Look for:

• CSV exports
• Google Sheets compatibility
• API access
• Jira/Trello/Asana integrations
• Developer-friendly exports

Ideal exports:

Broken Links Report

URL
Issue
Status
Linked From
Anchor Text
Recommended Fix

Redirect Report

Old URL
Redirect Type
Chain Steps
Final URL
Final Status
Action Needed

7. Historical comparison

Very useful for retainers:

• Compare this month vs last month
• Show issues resolved
• Detect regressions after site releases
• Prove ongoing SEO maintenance value

A “before/after” report is often more valuable to clients than a single audit snapshot. Crawle

8. Multi-client agency management

For a large portfolio, look for:

• Client folders/workspaces
• User permissions
• Notes/comments on issues
• Assign issues to team members
• Client-specific branding
• Crawl quotas
• Audit history

A practical agency crawler checklist

For an agency at scale, I would prioritize report automation + issue prioritization + client workspaces over simply having the fastest crawler. A crawler that finds 10,000 issues but requires an SEO specialist to manually turn them into a client report becomes a bottleneck.

If you’re auditing many client sites, the biggest time savings come from features that reduce the work after crawling—not just from crawling faster.

Highest-value crawler features for an agency

• Automatic broken-link discovery

• Find 4xx/5xx URLs and, crucially, show which pages link to them.
• Separate internal vs. external broken links.
• Export source URL → broken destination → status code so a developer can fix it immediately.
• This avoids manually opening pages or stitching together spreadsheets. Sitebulb Support Screaming Frog
• Redirect-chain and loop detection

• Don't settle for "this URL redirects."
• You want the crawler to expose the full path: A → B → C → final URL, number of hops, redirect types, and loops.
• Ideally, it also identifies redirects that ultimately lead to errors. Screaming Frog and Sitebulb both provide this type of analysis. Screaming Frog Sitebulb Support
• Source-page context

• For every bad URL, immediately answer: "Where is this being linked from?"
• Even better: show anchor text, link location, HTTP status, indexability, and the affected page's importance.
• This turns a crawler finding into a developer-ready ticket rather than an investigation task. Sitebulb
• Prioritization instead of raw error dumps

• Automatically classify issues by severity/impact.
• For example, a broken link from the homepage should rank above one buried in an old blog post.
• Sitebulb's Hints system is a good example: it categorizes and prioritizes issues rather than leaving the auditor with thousands of rows to interpret. Sitebulb
• Crawl comparisons

• Store previous crawls and show new / fixed / unchanged issues.
• This is hugely valuable for recurring client work because you can immediately see whether the developer actually fixed the problems from last month's audit. Sitebulb
• Scheduled/automated crawls

• Set weekly or monthly crawls per client rather than remembering to run them manually.
• The best agency workflow is: crawl automatically → compare with previous crawl → review only meaningful changes → report. Sitebulb
• White-label reporting

• Custom logo, colors, client name, executive summary, charts, prioritized issues and recommendations.
• Ideally you can create different outputs for different audiences: - Client: 5–10 important findings with plain-English explanations.
• Developer: URLs, status codes, sources and exact fixes.
• Internal SEO: full crawl dataset.
• Sitebulb, for example, supports customizable PDF reports and exports designed for different stakeholders. Sitebulb
• One-click exports

• CSV/Excel/Google Sheets exports for: - broken links
• redirect chains
• redirecting internal links
• orphan pages
• affected source URLs
• This makes developer handoff dramatically faster. Sitebulb specifically provides redirect-chain exports and bulk link data. Sitebulb Support
• Client-friendly visualizations

• Charts showing issue counts, severity, trends and site structure are much easier to put into a client deck than raw crawler tables.
• Crawl maps, timelines and visual issue summaries are particularly useful for explaining technical problems to non-SEOs. Sitebulb
• Integrations with GSC/GA4

• This is an underrated agency feature: combine crawl errors with actual organic traffic/search data.
• A broken URL receiving 2,000 organic visits is much more urgent than a broken URL nobody visits.
• Sitebulb supports GA and Search Console integrations for this kind of analysis. Sitebulb

The ideal agency workflow

I'd look for a crawler that can effectively turn this:

Crawl → 1,200 issues → manually investigate → spreadsheet → write report

into:

Crawl → automatically prioritize → compare against last crawl → export developer fixes → generate client report

For example:

Redirect chain: /old-page → /page-2 → /final-page
3 hops
Source: /blog/article
Impact: 14 internal links
Recommendation: Update internal links directly to /final-page
Status: New since previous crawl

That level of context is where an agency actually saves hours.

If you're evaluating crawlers specifically

I'd score them on these 10 agency criteria:

1. Broken-link detection
2. Redirect-chain/loop detection
3. Source-page and anchor-text visibility
4. Automatic prioritization
5. Historical crawl comparison
6. Scheduled crawling
7. Bulk developer exports
8. White-label/client reporting
9. GSC/GA4 integration
10. Multi-client/cloud collaboration

For an agency, #4–#8 are often more important than raw crawl speed. A crawler that saves 10 minutes during crawling but costs you two hours turning the output into a client deliverable isn't actually the faster tool.

Crawl → automatically prioritize → compare against last crawl → export developer fixes → generate client report

For example:

Redirect chain: /old-page → /page-2 → /final-page
3 hops
Source: /blog/article
Impact: 14 internal links
Recommendation: Update internal links directly to /final-page
Status: New since previous crawl

That level of context is where an agency actually saves hours.

If you're evaluating crawlers specifically

I'd score them on these 10 agency criteria:

1. Broken-link detection
2. Redirect-chain/loop detection
3. Source-page and anchor-text visibility
4. Automatic prioritization
5. Historical crawl comparison
6. Scheduled crawling
7. Bulk developer exports
8. White-label/client reporting
9. GSC/GA4 integration
10. Multi-client/cloud collaboration

For an agency, #4–#8 are often more important than raw crawl speed. A crawler that saves 10 minutes during crawling but costs you two hours turning the output into a client deliverable isn't actually the faster tool.

For very large sites, especially when your main goals are broken-link discovery, redirect-chain detection, and efficient recrawling, I’d shortlist these:

1. Screaming Frog — my first choice for hands-on technical crawling

Screaming Frog is particularly good when you want raw crawl data and precise control rather than a heavily abstracted SEO dashboard.

Its current SEO Spider supports database storage specifically for large crawls. Screaming Frog says the default limit is 5 million URLs, with substantially larger crawls possible depending on hardware; it gives an example of roughly 10M URLs on a machine with a 500GB SSD and 16GB RAM. Screaming Frog Screaming Frog

For your use case, I'd configure it to:

• Crawl HTML only initially.
• Use database storage on an SSD.
• Start from the XML sitemap(s) and the homepage/internal links.
• Export all 3xx responses.
• Examine the redirect destination of every 3xx.
• Identify chains such as A → B → C → 200.
• Identify loops such as A → B → A.
• Find internal links pointing to redirects rather than the final URL.
• Separately crawl external links if that's important.

Screaming Frog also supports restricting crawls by subdirectory, URL patterns, query parameters, and resource type, which can dramatically reduce wasted crawling on huge sites. Screaming Frog

Best fit: technical SEO teams that want maximum control and don't mind running the crawler themselves.

screamingfrog.co.uk

2. Sitebulb Cloud — easiest choice for large cloud crawls

Sitebulb is particularly attractive if you don't want your laptop/workstation to be the bottleneck.

Sitebulb Cloud advertises crawls of up to 10 million URLs per audit, while its desktop version is aimed at substantially smaller crawls. Its crawler can also use multiple threads/Chrome instances and lets you control URL-per-second limits. Sitebulb Sitebulb Support

It's also good when you need to turn crawl data into something humans can quickly understand. Sitebulb provides prioritized technical issues and visualizations rather than simply dumping millions of rows into a spreadsheet. Sitebulb

One particularly useful feature for large sites is combining crawl sources: Sitebulb recommends using XML sitemaps, Google Search Console/Analytics URLs, and seed lists in addition to ordinary link discovery. That's important because a pure link crawl won't necessarily discover orphaned URLs. Sitebulb Support

Best fit: large sites where you want cloud infrastructure + strong reporting without building your own crawler.

sitebulb.com

3. Lumar / Botify / Oncrawl — when "very large" means enterprise

If you're talking about tens of millions of URLs, multiple domains, huge international sites, or needing crawl data alongside server logs, I'd look beyond desktop crawlers.

These enterprise platforms become more interesting because the problem stops being merely:

"Can I crawl this site?"

and becomes:

"How do I efficiently crawl, segment, store, compare and analyze billions of URLs/request relationships?"

For example, combining crawler data + server logs + Search Console data + sitemaps can reveal things a conventional link crawl can't: URLs Googlebot actually visits, URLs users encounter, orphan URLs, crawl-budget patterns, and redirect behavior at scale.

The important distinction: finding redirects vs. finding bad redirects

For your particular objective, I wouldn't just look for a crawler that reports 301 and 302.

You want a crawl that lets you construct something like:

Internal link
     ↓
/old-product
     ↓ 301
/product
     ↓ 301
/products/product-name
     ↓ 200

and distinguish it from:

/old-product → /product → 200

The first is a redirect chain; the second is a single redirect.

I'd also look for these separately:

• 4xx links — especially internally linked 404/410 pages
• 5xx responses
• redirect loops
• chains >1 hop
• internal links to 3xx URLs
• redirects to irrelevant destinations
• redirects to another redirect
• redirects ending in 4xx/5xx
• canonical URLs that redirect
• sitemap URLs that redirect
• hreflang URLs that redirect
• redirects caused by HTTP→HTTPS or www/non-www normalization
• external links returning 3xx/4xx
• orphan URLs that aren't discoverable through normal internal linking

What I'd actually use

For a 100K–5M URL site, I'd probably start with Screaming Frog + database storage. It gives you unusually granular control and is very capable at this scale. Screaming Frog Screaming Frog

For a 1M–10M+ URL site, particularly if repeated crawls are needed, I'd lean toward Sitebulb Cloud or an enterprise crawler, because moving the crawl workload and storage off your workstation becomes increasingly valuable. Sitebulb Sitebulb

For a 10M–100M+ URL site, I'd seriously consider an enterprise crawler or a custom distributed crawler, rather than simply throwing more RAM at a desktop application.

And for a site where redirect problems are the primary objective, I'd use a two-stage approach:

1. Discover URLs cheaply from sitemaps, internal links, historical crawl exports, GSC, etc.
2. Run a focused HTTP crawl against those URLs and follow redirects, rather than rendering every page and downloading every resource.

That can reduce the crawl workload enormously while giving you a much cleaner redirect dataset.

If you tell me roughly whether you're dealing with 100K, 1M, 10M, or 100M+ URLs, I can give you a concrete crawling architecture—including Screaming Frog/Sitebulb settings, concurrency, SSD/RAM requirements, and a SQL/Python approach for detecting every redirect chain and loop.

Best fit: large sites where you want cloud infrastructure + strong reporting without building your own crawler.

sitebulb.com

3. Lumar / Botify / Oncrawl — when "very large" means enterprise

If you're talking about tens of millions of URLs, multiple domains, huge international sites, or needing crawl data alongside server logs, I'd look beyond desktop crawlers.

These enterprise platforms become more interesting because the problem stops being merely:

"Can I crawl this site?"

and becomes:

"How do I efficiently crawl, segment, store, compare and analyze billions of URLs/request relationships?"

For example, combining crawler data + server logs + Search Console data + sitemaps can reveal things a conventional link crawl can't: URLs Googlebot actually visits, URLs users encounter, orphan URLs, crawl-budget patterns, and redirect behavior at scale.

The important distinction: finding redirects vs. finding bad redirects

For your particular objective, I wouldn't just look for a crawler that reports 301 and 302.

You want a crawl that lets you construct something like:

Internal link
     ↓
/old-product
     ↓ 301
/product
     ↓ 301
/products/product-name
     ↓ 200

and distinguish it from:

/old-product → /product → 200

The first is a redirect chain; the second is a single redirect.

I'd also look for these separately:

• 4xx links — especially internally linked 404/410 pages
• 5xx responses
• redirect loops
• chains >1 hop
• internal links to 3xx URLs
• redirects to irrelevant destinations
• redirects to another redirect
• redirects ending in 4xx/5xx
• canonical URLs that redirect
• sitemap URLs that redirect
• hreflang URLs that redirect
• redirects caused by HTTP→HTTPS or www/non-www normalization
• external links returning 3xx/4xx
• orphan URLs that aren't discoverable through normal internal linking

What I'd actually use

For a 100K–5M URL site, I'd probably start with Screaming Frog + database storage. It gives you unusually granular control and is very capable at this scale.

For very large sites, I’d separate the problem into two needs: high-throughput crawling and accurate HTTP/redirect analysis. The strongest options are:

1. Sitebulb Cloud — best all-around choice

For a site in the hundreds of thousands to several million URLs, I'd look closely at Sitebulb Cloud. Its current documentation says Cloud supports up to 10 million URLs per audit, while its enterprise offering is designed for much larger sites. It also lets you control crawl speed and concurrency rather than simply throwing requests at the server. Sitebulb Sitebulb

It's particularly useful for your use case because you can:

• identify 404/5xx URLs
• find internal links pointing to redirects
• identify redirect chains
• distinguish direct redirects from multi-hop chains
• control URL/second limits
• exclude unnecessary resources/parameters
• crawl JavaScript-rendered sites when necessary

For external links, Sitebulb can perform HTTP status checks without actually following external sites, which is a useful safety/performance optimization for large crawls. Sitebulb Support

sitebulb.com

2. JetOctopus — strongest option when "very large" means tens of millions+

JetOctopus is particularly interesting if you're talking about 10M+ URLs, huge ecommerce catalogs, or sites where crawling needs to be combined with server-log analysis.

Its enterprise offering currently advertises crawling 10M, 50M, 100M+ pages, and its platform combines crawler data with server logs, Google Search Console and analytics data. Tech SEO Hub Tech SEO Hub

That makes it useful for a more sophisticated workflow:

crawl → find redirects/404s → compare against actual Googlebot activity → prioritize fixes

Its log analyzer specifically surfaces things such as unintended redirect chains and URL patterns that inflate crawlable URL counts. Tech SEO Hub

JetOctopus also advertises a JavaScript crawler capable of 1M+ pages/day, although actual throughput will depend heavily on the site and rendering requirements. Tech SEO Hub

jetoctopus.com

3. Screaming Frog — best if you want maximum control

Screaming Frog remains excellent if you're comfortable running the crawler yourself and have substantial hardware.

Its current FAQ says the SEO Spider can crawl millions of URLs, with its database-storage mode designed to reduce RAM pressure. It recommends roughly <2M URLs for typical use, but says considerably larger crawls are possible with sufficient hardware/memory. Screaming Frog

I'd particularly choose it when you want to:

• customize exactly what gets crawled
• control URL discovery/exclusions very precisely
• export huge datasets
• integrate the crawl into your own technical SEO workflow
• inspect individual redirect chains in detail

screamingfrog.co.uk

4. Ahrefs Site Audit — excellent if you're already using Ahrefs

Ahrefs is less compelling purely as a maximum-throughput crawler, but very useful when broken links are part of a broader SEO operation.

Its Site Audit explicitly detects redirect chains, while its broader link database lets you investigate broken backlinks and redirected targets—not just links discovered during your own crawl. Ahrefs Help Center Ahrefs

That distinction matters: a conventional crawler answers:

"What broken/redirected links exist on my site?"

Ahrefs can additionally help answer:

"Which external links point to those broken or redirected URLs?"

ahrefs.com

What I'd use at different scales

• <500K URLs: Screaming Frog or Sitebulb Desktop
• 500K–5M: Sitebulb Cloud or Screaming Frog with appropriately configured hardware
• 5M–10M: Sitebulb Cloud / JetOctopus
• 10M–100M+: JetOctopus or another enterprise/cloud crawler
• Need actual bot behavior too: JetOctopus + server logs
• Need backlink impact of broken URLs: Ahrefs alongside your crawler

The important thing is that crawl size isn't the only bottleneck. For redirect-chain work, you want a crawler that records the entire HTTP sequence, e.g.:

/old-page → 301 → /intermediate → 302 → /final-page → 200

rather than merely reporting that /old-page eventually returns 200. Ahrefs explicitly treats this as a redirect-chain issue, for example. Ahrefs Help Center

For millions of URLs, I'd also avoid blindly crawling every image, JS file, parameter combination, external domain, and faceted URL. Restricting those can make the difference between a useful crawl and an enormous crawl full of noise; Sitebulb's large-site guidance specifically recommends controlling external URLs, resources, parameters, and crawl scope. Sitebulb Support

If you tell me roughly how big the site is (1M, 10M, 100M+ URLs), whether it's JS-heavy, and whether you need the crawl to run from your own infrastructure or cloud, I can narrow this to the most suitable architecture and settings.

3. Screaming Frog — best if you want maximum control

Screaming Frog remains excellent if you're comfortable running the crawler yourself and have substantial hardware.

Its current FAQ says the SEO Spider can crawl millions of URLs, with its database-storage mode designed to reduce RAM pressure. It recommends roughly <2M URLs for typical use, but says considerably larger crawls are possible with sufficient hardware/memory.

I'd particularly choose it when you want to:

For very large websites (hundreds of thousands to millions+ of URLs), the best crawler depends on whether you prioritize raw crawl depth, cloud scale, automation, or reporting. The tools below are the ones most commonly used for finding broken links (4xx/5xx), redirect chains, redirect loops, bad canonicals, orphan URLs, and crawlability problems. SEO Automation Club theStacc

Best overall large-site crawlers

SEO Automation Club CrawlRaven## My recommendations by site size

Under ~500,000 URLs

Use:

1. Screaming Frog SEO Spider - Best value.
2. Finds: - broken internal links
3. external broken links
4. redirect chains
5. redirect loops
6. unnecessary 301 hops
7. bad canonical targets
8. Has a dedicated redirect-chain report and bulk redirect checking workflow. Screaming Frog

Typical setup:

• Run in database storage mode.
• Use a machine with 32–64 GB RAM for large crawls.
• Export CSVs into your own analysis pipeline.

500,000–5 million URLs

Use:

• JetOctopus if crawl speed and automation matter.
• Sitebulb Cloud if humans need to consume the audits.
• Screaming Frog on a dedicated server if you want maximum control.

JetOctopus is particularly strong when you need repeated scheduled crawls and want crawl data combined with server log analysis. SEO Automation Club AI Search Tools

5M+ URLs

Look primarily at:

• JetOctopus
• Lumar
• Botify

At this scale, the problem changes from “can the crawler find broken links?” to:

• Can it finish before the site changes?
• Can it crawl continuously?
• Can multiple teams access the data?
• Can it combine crawl data with logs and analytics?

Astro SEO Blog AI Search Tools## If your main goal is broken links + redirect chains

My ranking would be:

1. Screaming Frog — best technical accuracy

Best when you need to answer:

• “Which pages link to this 404?”
• “How many hops does this redirect chain have?”
• “Which internal URLs should be updated?”

Example workflow:

1. Crawl entire site.
2. Export: - Response Codes → Client Errors (4xx)
3. Response Codes → Redirection (3xx)
4. Reports → Redirect Chains
5. Fix internal links pointing at redirected or dead URLs.

Screaming Frog### 2. JetOctopus — best at scale

Better when:

• you crawl daily/weekly
• you have millions of URLs
• you need API access
• you want server log correlation

SEO Automation Club### 3. Sitebulb — best for explaining problems

Good if you need:

• executive reports
• visual crawl maps
• prioritized recommendations

Astro SEO Blog## For automated monitoring (not just audits)

For a large production site, I would build a pipeline like:

1. Scheduled crawler

• JetOctopus / Lumar / Botify
• Nightly URL checks

• Crawl sitemap URLs
• Check status codes
• Store: - URL
• status
• final URL
• redirect hops
• response time
• Alert rules

• New 404 spike
• Redirect chain > 2 hops
• Redirect loops
• 5xx increase
• Sitemap URL returning non-200
• Data warehouse

• BigQuery / Snowflake / PostgreSQL
• Dashboard

• Looker Studio / Grafana / internal dashboard

If you are building your own crawler

For millions of URLs, avoid simple scripts with one HTTP client. Use:

• async HTTP fetching (aiohttp, httpx)
• URL frontier queue
• Bloom filters for deduplication
• robots.txt handling
• per-domain rate limiting
• persistent crawl state
• redirect graph storage

A good architecture is closer to a search-engine crawler than a link checker.

Short answer

• Best overall for enterprise: JetOctopus
• Best technical SEO crawler: Screaming Frog
• Best enterprise platform: Lumar/Botify
• Best reporting: Sitebulb
• Best DIY stack: custom crawler + database + scheduled checks

For a site with millions of pages and a primary goal of finding broken links and redirect chains, I would usually choose JetOctopus for continuous monitoring and keep Screaming Frog for deep investigations. SEO Automation Club Screaming Frog

For JavaScript-heavy sites where the goal is specifically finding indexing/SEO problems, I’d narrow the field to Sitebulb and Screaming Frog, with enterprise tools becoming relevant at very large scale.

My shortlist

Sitebulb

For your particular use case, Sitebulb is probably the first tool I'd test. Its Chrome Crawler renders pages with headless Chrome and, importantly, compares the response HTML against the rendered HTML. That lets you identify things like:

• Canonicals changed by JavaScript
• noindex/robots directives introduced after rendering
• Titles and meta descriptions modified by JS
• Internal links injected by JavaScript
• Content that exists only after rendering
• Missing content caused by rendering failures
• Pages where the rendered DOM differs substantially from the server response Sitebulb Support Sitebulb Support Sitebulb Support

That response-vs-render comparison is particularly useful for indexing audits because a normal browser can make a site look perfectly healthy while the initial HTML contains very little useful content.

Sitebulb also has controls for things such as render timeout, cookies, infinite scrolling, and troubleshooting pages that aren't rendering correctly. Sitebulb Support

sitebulb.com

Screaming Frog

I'd choose Screaming Frog SEO Spider if you want a more traditional technical-SEO crawler and expect to do lots of custom extraction, URL filtering, regex/XPath work, and detailed investigation.

It can render JavaScript with Chromium and expose the differences between source and rendered pages. It's especially useful when you're combining JS rendering with a broader crawl containing things like status codes, canonicals, hreflang, structured data, internal links, redirects, etc. Sitebulb itself notes Screaming Frog as another option for JavaScript SEO auditing. Sitebulb

screamingfrog.co.uk

One important distinction

For indexing problems, don't just run a rendered crawl.

I'd actually run two passes:

1. Raw/response HTML crawl
2. Chromium-rendered crawl

Then look specifically for differences in:

<title> → canonical → robots → headings → body content → internal links → structured data

For example:

Response HTML
    ↓
canonical = /product/123
robots = index,follow
links = 12

        ↓ JavaScript execution

Rendered HTML
    ↓
canonical = /product/123?variant=red
robots = noindex
links = 47

That kind of discrepancy is much more valuable than simply knowing that "JavaScript is enabled."

Also, Google does render JavaScript, so "Google can render JS" doesn't mean a JS-heavy implementation is automatically safe for indexing. Rendering can still expose differences in canonicals, robots directives, links, and content. Sitebulb

If you tell me roughly how big the site is (e.g. 50k, 500k, 10M URLs) and whether it's React/Next.js/Vue/etc., I can give you a more specific crawler setup and the exact checks I'd run.

For JavaScript-heavy sites (React, Next.js, Angular, Vue, SPAs) where the goal is to find indexing problems, I would avoid relying on a normal HTML crawler alone. You want a tool that can compare:

• raw server HTML vs rendered DOM
• rendered links vs crawlable links
• rendered content vs indexed content
• JavaScript errors/resource failures
• canonical/meta/schema changes after JS execution

Google itself processes JavaScript in separate crawling/rendering stages, so a site can look fine in a browser while still having crawl/indexing issues. Google for Developers Google for Developers

My recommended stack

1. Screaming Frog SEO Spider — best overall starting point

Best for: most technical SEO audits

Why I like it for JS sites:

• Chromium-based JavaScript rendering
• Compare HTML crawl vs rendered crawl
• Find: - missing titles/meta after JS
• blocked JS resources
• orphaned pages
• links injected only after rendering
• incorrect canonicals
• missing structured data
• Very customizable extraction rules

Use it with:

• Configuration → Spider → Rendering → JavaScript
• Crawl once with JS off
• Crawl again with JS on
• Diff the results

This is usually the first tool I would put in an SEO engineer's hands. TechWench | All Things Tech Search Engine Land

2. Sitebulb — best for explaining problems

Best for: teams that need actionable reports

Strengths:

• Strong visualization of crawl paths
• Good JavaScript rendering support
• Easier for non-technical stakeholders to understand
• Better "why this matters" reporting than many crawlers

Good if you need to present findings to developers or executives. AI Agents Directory

3. Lumar (formerly DeepCrawl) — enterprise option

Best for: very large sites

Consider it if you have:

• millions of URLs
• frequent releases
• international sites
• SEO QA integrated into workflows

Useful features:

• cloud crawling
• rendered page analysis
• recurring monitoring
• enterprise reporting

AI Agents Directory

4. JetOctopus or Oncrawl — best when indexing diagnosis is the priority

Best for: answering "why isn't Google crawling/indexing these pages?"

Pair rendering data with:

• Googlebot logs
• crawl budget analysis
• indexation trends
• sitemap analysis

A rendered crawl alone tells you what could happen; log data tells you what Googlebot actually did. AI Agents Directory

Developer-focused option: build your own crawler checks

If your site is a large app and you have engineering resources, I would also consider:

• Playwright
• Puppeteer

You can create automated SEO tests such as:

• "Does /product/123 contain an <h1> after rendering?"
• "Does JSON-LD exist after hydration?"
• "Are internal links present without user interaction?"
• "Does a page render within Google's likely timeout window?"

This is especially valuable in CI/CD so SEO regressions are caught before deployment. W3Corner

I would combine tools like this

Google Search Console should remain the source of truth for individual URL rendering/indexing checks because it shows what Google actually sees. Google for Developers

For most teams, I would start with Screaming Frog + Google Search Console + server logs. Add an enterprise crawler only when scale or monitoring requirements justify it.

Oncrawl is probably the best fit if your priority is combining server logs + crawl data into one SEO auditing workflow. It explicitly cross-references crawl, log, Search Console, and analytics data and supports large-scale sites. Oncrawl Oncrawl

• Oncrawl: Best for integrated, large-scale technical SEO analysis.
• Botify: Strong alternative, particularly for enterprise sites and detailed segmentation across crawler, log, GSC, and traffic data. Botify Knowledge Base Botify Knowledge Base
• Screaming Frog Log File Analyser: Best budget/local option, but less comprehensive; free up to 1,000 log events. screamingfrog.co.uk

For a comprehensive audit, I'd shortlist Oncrawl vs. Botify rather than using a standalone log analyzer.

If your main requirement is server-log analysis + crawl data in the same SEO workflow, I’d shortlist JetOctopus, Oncrawl, and Botify.

JetOctopus

JetOctopus is particularly well aligned with what you're describing. It combines crawler data, server logs, Google Search Console and GA4 in a shared analysis layer, allowing you to look at things like:

• URLs that Googlebot crawls vs. URLs your crawler discovers
• Crawl frequency by directory/template/content type
• Orphaned or poorly internally linked pages that Google rarely visits
• 3xx/4xx/5xx URLs receiving crawler attention
• Crawl depth vs. actual Googlebot activity
• Crawl behavior alongside GSC impressions/clicks
• Pages Googlebot crawls heavily but that generate little organic traffic
• Historical bot behavior and segmentation

JetOctopus also supports log ingestion from infrastructure such as Nginx, Cloudflare, AWS/CloudFront, Fastly, Vercel and others. jetoctopus.comjetoctopus.com

JetOctopus

Oncrawl

Oncrawl is the other platform I'd investigate seriously. Its core proposition is explicitly combining crawl, log and performance data. It supports server logs from Cloudflare, AWS, GCP, Azure and direct log files, alongside GSC and GA4 integrations. www.oncrawl.comjetoctopus.com

It's especially useful if you want to do sophisticated segmentation and analyze relationships such as:

crawl frequency → indexation → organic visits → SEO performance

rather than simply producing a list of technical errors.

Oncrawl

Botify

Botify is worth considering if you're dealing with a very large enterprise site. Its Analytics suite combines its crawler with server-log data and other organic-search datasets. Its LogAnalyzer specifically overlays bot activity with crawl information. support.botify.com

Botify

My practical take

For the specific workflow you described, I'd evaluate them in this order without treating that as a quality ranking:

JetOctopus → if you want hands-on technical SEO analysis, powerful segmentation, and a relatively direct crawl + logs + GSC/GA4 workflow.

Oncrawl → if your SEO team is particularly data-oriented and you want extensive cross-analysis and integrations.

Botify → if you're operating at very large enterprise scale and need a broader SEO data/analytics platform.

The important distinction is that Splunk/ELK/Datadog-type tools aren't substitutes for these. They can analyze server logs extremely well, but SEO platforms add the semantic layer: which URLs are indexable, canonical, orphaned, internally linked, ranking, receiving impressions, etc., and then correlate that with what Googlebot actually did. JetOctopus and Oncrawl explicitly provide this cross-source analysis. www.oncrawl.comjetoctopus.com

If you tell me roughly how many URLs the site has (e.g. 100k, 5M, 50M), what your log source is (Cloudflare/Nginx/AWS/etc.), and whether you're agency or in-house, I can narrow this down and outline the exact audit workflow I'd use.

If your goal is comprehensive technical SEO auditing by combining server logs with crawl data, I’d shortlist three platforms:

If your goal is comprehensive technical SEO auditing by combining real server logs with crawl data, I’d shortlist these:

My recommendation: Oncrawl

For the specific workflow you described, Oncrawl is probably the strongest choice. It is designed to cross-reference crawl data, server-log data, and performance/SEO data, rather than treating log analysis as a separate tool. Oncrawl Oncrawl

That lets you answer questions such as:

• Which URLs does Googlebot actually crawl vs. which URLs your crawler discovers?
• Are important pages being crawled frequently enough?
• Which crawl budget is being wasted on parameters, redirects, duplicates, or low-value URLs?
• Which pages are in the crawl but never appearing in server logs?
• Which URLs appear in logs but aren't discoverable in your crawl?
• How does crawl behavior differ by page type, directory, template, status code, etc.?
• How do Googlebot and AI crawlers interact with the site?

Oncrawl also supports large-scale log processing and integrations for bringing logs in from sources such as S3, Google Cloud Storage, and Akamai. Oncrawl

When I'd choose something else

Choose Botify if you're auditing a huge enterprise/e-commerce site and want to combine server logs with crawl, organic traffic, GSC, and other SEO datasets. Botify explicitly integrates its crawler with web-server logs and other data sources, and its Analytics suite is built around URL/segment-level analysis. Botify Knowledge Base Botify Knowledge Base

Choose Screaming Frog if you're an SEO consultant or working on smaller sites and don't need a cloud-scale platform. Its Log File Analyser has a particularly useful feature: you can import your Screaming Frog crawl and have the tool match crawl URLs against log events, showing "Not In Log File" and "Not In URL Data" URLs. Screaming Frog

Choose Semrush Enterprise if you're already heavily invested in Semrush. Its 2026 Bot Analytics capability combines server-log analysis with technical SEO workflows and covers both search-engine and AI crawlers. Semrush

The practical choice

I'd use this rule of thumb:

• <100k URLs: Screaming Frog + Log File Analyser
• 100k–several million URLs: Oncrawl
• Very large enterprise / multiple domains / sophisticated data warehouse needs: Botify
• Already standardized on Semrush Enterprise: Semrush's log analysis is worth evaluating before adding another platform.

If you tell me roughly how many URLs the site has, how many log lines/month you have, and whether you're auditing one site or many clients, I can recommend the best platform and setup for your situation.

Choose Screaming Frog if you're an SEO consultant or working on smaller sites and don't need a cloud-scale platform. Its Log File Analyser has a particularly useful feature: you can import your Screaming Frog crawl and have the tool match crawl URLs against log events, showing "Not In Log File" and "Not In URL Data" URLs.

If you mean large-domain crawling where you want both a machine-readable redirect-chain report and an actual visual map of loops, I’d look at these:

• Screaming Frog SEO Spider — probably the strongest all-around choice. It follows multi-hop redirects, explicitly detects loops, and has a Redirect Chains report showing the source, every hop, hop count, and loop status. It can also crawl in Spider mode or process huge URL lists in List mode. Screaming Frog Screaming Frog

screamingfrog.co.uk

• Sitebulb — best if visualization is the priority. Its interactive Crawl Maps let you visualize URL relationships, while its Redirects reporting identifies chained redirects and specifically flags chained redirect loops. It can export redirect-chain data for further analysis. Sitebulb Support Sitebulb Sitebulb Support

sitebulb.com

• JetOctopus — particularly interesting for very large sites. Its Redirect Chains report stores the complete chain and exposes both the chain itself and its length, making it useful for bulk analysis. Tech SEO Platform

jetoctopus.com

• Yetty Redirect Chain Visualizer — useful for individual chains or pasted redirect rules, rather than replacing a large-site crawler. It draws chains hop-by-hop and identifies loops and forks. Yetty

yetty.ai

For your specific use case

I'd choose Screaming Frog + Sitebulb if the goal is something like:

100,000+ URLs
       │
       ▼
   Crawl domain
       │
       ├── /old-a → /old-b → /new-a       ← 2-hop chain
       ├── /foo → /bar → /baz → /foo      ← LOOP
       ├── /x → /y                         ← single redirect
       └── /a → /b → /c → /d → /e        ← long chain

Screaming Frog is better for extracting and fixing the redirect relationships at scale; Sitebulb is better when you want to see site structure and relationships graphically. Screaming Frog's redirect report specifically exposes loops rather than merely showing generic crawl relationships. Screaming Frog Screaming Frog

If you're dealing with millions of URLs or multiple domains, I'd lean toward a cloud crawler such as JetOctopus/Sitebulb Cloud and then feed the redirect edges into Neo4j, Gephi, or Graphviz for a true graph visualization. That approach lets you identify not just obvious A→B→A loops, but large strongly connected components where dozens or hundreds of URLs participate in a redirect cycle.

For combining XML sitemap + site crawl + server log data, the best fit is usually:

1. Screaming Frog SEO Spider + Log File Analyser — best overall for technical SEO audits. It can compare crawl-discovered URLs against sitemap URLs, Analytics/Search Console sources, and log files to identify orphan or uncrawled pages. Screaming Frog Screaming Frog
2. Botify — better for very large sites (millions of URLs) because it combines crawling with log analysis and enterprise-scale reporting. First Page SG
3. Ahrefs Site Audit or Semrush Site Audit — easier SaaS options, but log-file depth is generally less specialized than dedicated crawl + log tools. First Page SG

A typical workflow:

• Sitemap URLs = pages you claim exist.
• Crawler URLs = pages reachable through internal links.
• Log URLs = pages search engines actually requested.
• The gaps reveal:

• sitemap-only pages → likely orphan candidates
• log-only pages → forgotten/hidden URLs
• crawl-only pages → sitemap coverage issues

For most SEO teams, Screaming Frog + Log File Analyser is the practical choice; for enterprise-scale sites, Botify is usually the stronger option. Screaming Frog First Page SG

If you specifically want to combine XML sitemap URLs + a site crawl + server log data, I’d pick screamingfrog.co.uk.

• SEO Spider gives you the URLs actually discoverable through internal links and can incorporate XML sitemap URLs. Screaming Frog
• Log File Analyser lets you import your crawl/URL data and match it against real Googlebot activity, identifying URLs that are in your known URL set but aren't being crawled, as well as URLs appearing in logs that you didn't otherwise know about. Screaming Frog Screaming Frog
• This gives you a much richer orphan-page diagnosis than simply checking "in sitemap, but not in crawl."

A useful way to classify the results

Sitebulb is arguably easier if your priority is finding and visualizing orphan pages from sitemap/crawl/analytics sources, and its orphan-page workflow is excellent. Sitebulb Support Sitebulb But for your specific requirement of combining crawl + sitemap + actual server-log behavior, I'd favor Screaming Frog because its Log File Analyser is purpose-built for that third data source. Screaming Frog

If you tell me the site size (e.g. 50k vs 10M URLs), I can also recommend the best setup/workflow for doing this at scale.

For large multilingual sites, I’d look first at sitebulb.com.

Why Sitebulb fits

• Hreflang: crawls and validates hreflang from HTML, HTTP headers, and XML sitemaps, including alternate URLs on other domains/subdomains. Sitebulb Support Sitebulb
• Reciprocity: detects missing return tags and inconsistent hreflang relationships. Sitebulb Support
• Language tags: specifically audits the HTML lang attribute and flags invalid values. Sitebulb Support
• Cross-checking: identifies mismatches between hreflang and HTML lang declarations. Sitebulb Support
• Large-scale crawling: Sitebulb Cloud supports up to 10 million URLs per audit, while Desktop supports up to 500,000. Sitebulb
• International reporting: its Hreflang Cluster Matrix makes it easier to spot missing language/region variants across large URL sets. Sitebulb Support

Screaming Frog SEO Spider is the other major option. It also crawls hreflang in HTML, HTTP headers and XML sitemaps and reports issues such as invalid language/region codes, missing return links, non-canonical URLs, and noindex hreflang targets. screamingfrog.co.uk

If your specific requirement is “crawl millions of URLs + systematically diagnose both hreflang and HTML lang problems,” Sitebulb is the more directly aligned choice based on its dedicated International report and cloud-scale crawling. Sitebulb Support Sitebulb

For large multilingual sites, I’d look first at Sitebulb.

• Sitebulb — particularly strong for international SEO. Its International Report audits hreflang and HTML lang attributes, follows alternate URLs even across different domains/subdomains, and can identify language-declaration mismatches. It also expects the whole site to be crawled so it can validate relationships between language versions. Sitebulb Sitebulb Support

sitebulb.com

• Screaming Frog SEO Spider — excellent if you want a very powerful, configurable crawler and detailed exports. It checks hreflang in HTML, HTTP headers, and XML sitemaps, crawls referenced hreflang URLs, and reports issues such as invalid language/region codes and non-200 alternate URLs. Screaming Frog Screaming Frog

screamingfrog.co.uk

• Semrush Site Audit — easier cloud-based option. Its International SEO report detects incorrect hreflang codes, conflicts, and potentially missing hreflang relationships. Business-tier Site Audit can crawl up to 100,000 pages per audit. Semrush Semrush

semrush.com

If your requirement is specifically “large + multilingual + hreflang + HTML lang”

Sitebulb is probably the closest match. Its documentation explicitly says it checks every hreflang variation and also checks the HTML lang tag, including mismatches between the declarations. Sitebulb

If you're dealing with millions of URLs, though, I'd look at enterprise-scale crawlers or a custom crawling pipeline rather than assuming any of these desktop/cloud limits will fit.

For large multilingual sites, you’ll want a crawler rather than a simple hreflang checker. The strongest options are:

My shortlist by use case

• Millions of URLs / global enterprise: Botify or Oncrawl.
• Hundreds of thousands of URLs with detailed hreflang debugging: Sitebulb or Screaming Frog.
• Agency technical audits: Screaming Frog + Sitebulb combination.
• Quick validation of a sitemap or smaller locale set: a dedicated hreflang validator can work, such as the Hreflang Testing Tool, which checks return tags, canonicals, language/region codes, x-default, and URL health. Hreflang Tool

For a serious multilingual audit, I would look for these specific reports:

• Missing reciprocal hreflang (“return tag” errors)
• Invalid ISO language or region codes (en-UK vs en-GB, etc.)
• Missing self-referencing hreflang
• Missing x-default
• Hreflang pointing to 3xx/4xx/5xx URLs
• Hreflang pointing to canonicalized or noindex pages
• HTML <html lang=""> mismatches
• Conflicts between sitemap hreflang and HTML hreflang
• Locale pages that exist but are not connected in the hreflang cluster

If you tell me the approximate size (e.g., 50k URLs vs 10M URLs) and whether locales are subfolders (/fr/), subdomains (fr.example.com), or ccTLDs (example.fr), I can narrow it down.

If your main requirement is scheduled crawls plus detailed measurement of how search engines are crawling the site, I’d separate the tools into two categories:

My recommendation

If “crawl budget reporting” is the key requirement, I'd look first at Botify, Lumar, or OnCrawl rather than a conventional site-audit crawler.

The distinction is important: a crawler such as Sitebulb tells you how your crawler traversed the site. Actual Google crawl behavior is better measured using Google Search Console's Crawl Stats plus server logs. Search Console reports Googlebot's crawl requests, response times, response types, file types, crawl purpose, and Googlebot type. Google Help

Botify is particularly interesting for a large site because its SiteCrawler is designed around search-engine crawling and combines crawl data with broader SEO analysis; it also supports recurring crawl schedules. Botify Knowledge Base Botify Knowledge Base

Lumar/OnCrawl are worth evaluating if you specifically want an enterprise platform centered on technical SEO and log/crawl-budget analysis.

Sitebulb Cloud is what I'd consider if your requirement is more like “crawl my sites every week, compare the crawls, and produce detailed technical SEO reports.” It supports automated recurring crawls and can handle very large crawls in its cloud product. Sitebulb Sitebulb

One caveat: don't equate your crawler's configured crawl rate with Google's crawl rate. Those are different measurements. For actual Googlebot behavior, I'd ideally build the reporting stack around:

Server logs → Googlebot crawl activity → Google Search Console Crawl Stats → crawler crawl data → dashboards/trends

That gives you things such as:

• Googlebot requests/day
• Googlebot requests by URL type
• Googlebot requests by response code
• Average response time
• Bytes downloaded
• Crawl activity by directory/template
• Crawl waste from redirects, errors, duplicate URLs, parameters, etc.
• Changes in crawl activity after technical SEO releases
• Your own crawler's crawl rate and discovered/crawled URL inventory

If you tell me roughly how large the site is (e.g. 100k, 1M, 10M+ URLs) and whether you have access to server/CDN logs, I can narrow this down considerably and compare Botify vs Lumar vs OnCrawl vs Sitebulb specifically for crawl-budget reporting.

My recommendation

If “crawl budget reporting” is the key requirement, I'd look first at Botify, Lumar, or OnCrawl rather than a conventional site-audit crawler.

The distinction is important: a crawler such as Sitebulb tells you how your crawler traversed the site. Actual Google crawl behavior is better measured using Google Search Console's Crawl Stats plus server logs. Search Console reports Googlebot's crawl requests, response times, response types, file types, crawl purpose, and Googlebot type. Google Help

Botify is particularly interesting for a large site because its SiteCrawler is designed around search-engine crawling and combines crawl data with broader SEO analysis; it also supports recurring crawl schedules. Botify Knowledge Base Botify Knowledge Base

Lumar/OnCrawl are worth evaluating if you specifically want an enterprise platform centered on technical SEO and log/crawl-budget analysis.

Sitebulb Cloud is what I'd consider if your requirement is more like “crawl my sites every week, compare the crawls, and produce detailed technical SEO reports.” It supports automated recurring crawls and can handle very large crawls in its cloud product. sitebulb.com sitebulb.com

One caveat: don't equate your crawler's configured crawl rate with Google's crawl rate. Those are different measurements. For actual Googlebot behavior, I'd ideally build the reporting stack around:

Server logs → Googlebot crawl activity → Google Search Console Crawl Stats → crawler crawl data → dashboards/trends

That gives you things such as: