ChatGPT answers, unedited

For a large-site crawl, I’d compare platforms on throughput, hard limits, and effective cost per successfully crawled URL, rather than advertised “pages/minute.”

A practical comparison

The important distinction: RPM vs concurrency

For a crawl, your approximate throughput is:

pages/sec ≈ concurrency ÷ average page latency

So a platform with 25 concurrent browsers isn't necessarily slower than one advertising 3,000 RPM—it depends on whether pages take 1 second or 10 seconds.

Zyte is particularly interesting for high-throughput crawling because its standard limit is 3,000 requests/minute, and its documentation explicitly recommends parallelizing requests. At a 2-second average response time, 3,000 RPM corresponds to roughly 100 concurrent requests. Zyte Zyte

Firecrawl, conversely, exposes concurrency directly. For example, its current Standard plan permits 25 concurrent browsers, while Growth permits 50. Its crawl endpoint has separate RPM limits as well. GitHub Plaud

How I'd benchmark them

Don't benchmark with "how fast can you crawl Google.com." Use a representative slice of your actual site.

Run, say, 10k–100k URLs through each platform and record:

1. Successful URLs/minute
2. Median and p95 latency
3. HTTP vs browser-rendered percentage
4. 429/block rate
5. Retries per 1,000 URLs
6. Content completeness — did JS-rendered pages actually produce the expected content?
7. Total elapsed time
8. Total bill
9. Cost / 1,000 successful URLs
10. Maximum sustainable throughput without harming the target site

For example, if Platform A crawls 100k URLs in 45 minutes at $80 while Platform B takes 90 minutes at $45, the meaningful comparison isn't simply "$/1k pages." It's:

A: 2,222 successful pages/min, $0.80/1k
B: 1,111 successful pages/min, $0.45/1k

Then you can decide whether the additional $35 buys enough reduction in crawl window.

One particularly important cost trap

Separate raw HTTP crawling from browser crawling.

If most of your site is static HTML, browser rendering can multiply costs unnecessarily. For example, Zyte's published pay-as-you-go ranges are roughly $0.13–$1.27/1k HTTP requests versus $1.01–$16.08/1k browser requests, depending on site complexity. Zyte #1 Web Scraping Service

Apify shows the same general pattern: its Website Content Crawler estimates around $0.20/1k pages for raw HTTP versus $0.50–$5/1k with a headless browser. Apify

So for a large site, I'd test a two-tier crawler architecture:

• Tier 1: fast HTTP crawler for ordinary pages
• Tier 2: browser rendering only for URLs that actually require JavaScript
• Aggressive deduplication/canonical handling
• Per-domain concurrency controls
• Persistent URL frontier so failed jobs can resume
• Metrics collected per URL rather than only per crawl job

That will usually give you a much more useful cost/speed comparison than comparing vendor headline limits.

If you tell me roughly how many URLs (e.g. 1M/10M/100M), whether JS rendering is required, and whether you're crawling one domain or thousands, I can turn this into a concrete benchmark matrix with estimated crawl times and costs for Zyte vs Firecrawl vs Apify (and alternatives such as Scrapy Cloud).

If most of your site is static HTML, browser rendering can multiply costs unnecessarily. For example, Zyte's published pay-as-you-go ranges are roughly $0.13–$1.27/1k HTTP requests versus $1.01–$16.08/1k browser requests, depending on site complexity.

Apify shows the same general pattern: its Website Content Crawler estimates around $0.20/1k pages for raw HTTP versus $0.50–$5/1k with a headless browser.

So for a large site, I'd test a two-tier crawler architecture:

How I'd benchmark them

Don't benchmark with "how fast can you crawl Google.com." Use a representative slice of your actual site.

Run, say, 10k–100k URLs through each platform and record:

For large sites, compare actual pages/sec, concurrency/rate limits, maximum crawl volume, and effective $/million URLs—not headline pricing.

• JetOctopus: up to 250 pages/sec; plans include 1M–3M URLs/month, with higher custom tiers. Tech SEO Platform Tech SEO Platform
• Sitebulb Cloud: reports >300 URLs/sec; desktop performance depends heavily on local hardware. Sitebulb Support Sitebulb
• Semrush API: 10 requests/sec and 10 concurrent requests/account; API usage also depends on subscription units. Semrush Developer
• Ahrefs API: monthly API-unit quotas vary by plan, with costs based on returned rows/fields. Ahrefs Help Center

For a fair benchmark, crawl the same 1M-URL representative dataset, with identical concurrency, JS rendering, robots handling, and response-time conditions. Record wall-clock time, sustained pages/sec, server load/error rate, URLs successfully processed, and total monthly cost.

For a large site, “fast crawling” is not just URLs/second. You need to compare throughput, throttles, crawl capacity, rendering cost, infrastructure limits, and total cost per million URLs. A useful framework:

1. Define the crawl workload first

Before comparing vendors, normalize the test:

A crawler that does 1M HTML URLs/hour may behave very differently when JavaScript rendering, screenshots, custom extraction, or log analysis are enabled.

2. Compare crawl speed using measurable metrics

Ask vendors for these numbers:

Example benchmark:

“Crawl 5 million URLs from our staging environment, HTML-only, max 50 req/sec, then repeat with JS rendering enabled.”

Do not compare marketing claims like “blazing fast” unless they specify workload.

Some tools have very different scaling models. For example, desktop crawlers depend on local CPU/RAM, while cloud crawlers can distribute work across infrastructure. Sitebulb notes that its cloud crawler can operate at much higher scale than desktop crawling, while desktop speed depends heavily on machine resources and crawl settings. Sitebulb

3. Compare limits

Create a matrix like this:

Examples:

• Screaming Frog is a desktop crawler where paid usage is generally not capped by URL count, but practical scale depends on machine resources. theStacc
• Sitebulb offers both desktop and cloud options; its documentation distinguishes desktop limits from cloud-scale crawling. Sitebulb
• Enterprise platforms such as Lumar, Botify, and OnCrawl are generally designed for very large cloud crawls and enterprise workflows. CrawlRaven

4. Compare cost correctly

Avoid comparing subscription price alone. Calculate:

Cost per million URLs crawled

Formula:

(monthly price + infrastructure cost + labor cost)
÷
(millions of URLs crawled)

Example:

The cheapest license can become expensive if engineers spend days managing crawl failures.

5. Include hidden costs

Large-site crawls often fail because teams ignore:

Infrastructure

• Proxy costs
• IP reputation management
• Storage
• Database retention
• Export processing

Engineering

• API integration
• Scheduling
• CI/CD integration
• Alerting

SEO requirements

• Log file analysis
• Crawl budget analysis
• Historical comparisons
• Segmentation
• JavaScript rendering

6. Typical platform positioning

7. Run a vendor bake-off

Give every platform the same test:

1. Crawl 1M URLs
2. Crawl 5M URLs
3. Enable JS rendering
4. Export all issues
5. Run a second crawl for comparison
6. Measure:

• elapsed time
• failures
• memory/storage
• API availability
• total cost

A good procurement scorecard:

For a very large site, the key question is usually not “which crawler is fastest?” but “which crawler delivers the required data volume at the lowest operational cost without harming the site.”

For a very large site, the key question is usually not “which crawler is fastest?” but “which crawler delivers the required data volume at the lowest operational cost without harming the site.”

If the site is large enough that crawl throughput is a primary requirement, I’d compare platforms on URLs/sec, maximum crawl size, concurrency, JavaScript cost, monthly URL allowance, and effective $ per million URLs—not just the headline subscription price.

A practical comparison

One important caveat: advertised maximum URLs/sec isn't a useful apples-to-apples benchmark. TTFB, number of crawler threads, JS rendering, server throttling, response size, redirects, and enabled extraction/reporting can dominate actual throughput. Sitebulb, for example, explicitly notes that its URL/sec setting is a ceiling rather than a guaranteed rate. Sitebulb Support Sitebulb Support

How I'd benchmark them

For your site, run the same representative 100k–500k URL sample through each platform and record:

1. Wall-clock crawl time

• Start → last URL fetched
• Separately measure post-crawl processing time if applicable.
• Sustained URLs/sec

• URLs successfully crawled ÷ crawl seconds
• Ignore the first few minutes if there's a ramp-up period.
• Server impact

• Requests/sec at origin
• CPU, database load, bandwidth
• 5xx/429 rate
• Median and p95 TTFB
• Crawl completeness

• URLs discovered
• URLs successfully fetched
• Timeouts
• blocked/throttled URLs
• duplicate/parameter URLs
• Rendering throughput

• Do a second benchmark with JavaScript rendering enabled.
• This can radically change the economics: Sitebulb says Chrome crawling is slower because it downloads and renders page resources. Sitebulb Support
• Economics

• monthly price / included URLs × 1,000,000
• Then calculate the same metric for actual URLs you expect to crawl, including repeat crawls.

The metric I'd use for the buying decision

I'd build a scorecard like:

For a multi-million-page site, I'd put particular emphasis on cost per completed crawl, rather than cost per URL. A platform that costs 30% more but finishes a 5M-page crawl in 6 hours instead of 2 days may be dramatically cheaper operationally.

JetOctopus is particularly interesting if raw throughput is the priority: it advertises up to 250 pages/sec and explicitly supports sites exceeding 100M URLs. TechSEO AI Sitebulb Cloud is attractive if you want predictable cloud costs and repeated large crawls without consuming crawl credits. Sitebulb

If you tell me roughly how many URLs the site has (e.g. 1M, 10M, 100M), whether JS rendering is required, and how often you want to crawl it, I can turn this into a concrete platform-by-platform cost/throughput comparison.

For a large site, I’d compare crawlers on throughput, hard limits, and cost per useful crawl, rather than just headline “URLs/sec.”

A practical comparison

The important part: benchmark them yourself

“Pages/sec” isn't directly comparable because a crawler might count an HTML URL differently from the total HTTP requests it generates. Sitebulb, for example, notes that Chrome crawling can make many additional requests for JS, images, CSS, etc. Sitebulb Support

For your site, I'd run the same benchmark against each platform:

1. 100k URL sample — measure time to completion.
2. 1M URL crawl — measure sustained throughput.
3. Run both: - HTML-only crawl
4. JavaScript-rendered crawl
5. Record: - URLs successfully crawled/hour
6. HTTP requests/sec hitting your origin
7. 2xx/3xx/4xx/5xx rates
8. 429s and timeouts
9. average/p95 response time
10. CPU/RAM consumption if self-hosted
11. crawl cost
12. time from crawl completion to usable reports/export
13. Repeat at 5, 10, 25, 50+ requests/sec until you find the point where your origin starts degrading.

That's important because the fastest theoretical crawler isn't necessarily the fastest safe crawler. Screaming Frog explicitly warns that increasing threads increases HTTP requests and can affect response times; Sitebulb similarly separates available concurrency from its URLs/sec ceiling. Screaming Frog Sitebulb Support

My shortlist

If your priority is “we have tens of millions of URLs and want the fastest possible crawl”, I'd benchmark JetOctopus first, then Sitebulb Cloud.

If the priority is “we crawl a huge site frequently and want predictable monthly spend”, I'd put Sitebulb Cloud at the top of the test.

If you're talking about 10M–100M+ URLs, I'd also include enterprise crawlers such as Botify/Lumar/Oncrawl in the RFP rather than assuming a conventional desktop SEO crawler will scale economically.

If you give me your approximate site size (e.g. 5M, 50M, 500M URLs), crawl frequency, whether JS rendering is required, and your monthly budget, I can turn this into a $/million-URLs + hours-per-crawl comparison and recommend the best architecture.

If the goal is fast full-site crawling at large scale, I’d compare platforms on throughput, concurrency/rate limits, crawl semantics, and effective $/1,000 pages—not headline pricing alone.

Quick comparison

Firecrawl currently advertises 100k credits/$83/month and 25 concurrent requests on Standard, and 500k credits/$333/month and 50 concurrent requests on Growth. Its crawl product is designed to turn a domain into consistent Markdown across thousands of pages. Firecrawl Firecrawl

Apify's Website Content Crawler is compute-priced rather than page-priced. Apify says its own benchmarks are roughly $0.20 per 1,000 pages for raw HTTP and $0.50–$5 per 1,000 pages using a headless browser, with the actual result depending heavily on the site and crawler configuration. Apify

Zyte is particularly interesting if raw request throughput is the bottleneck. Its standard API limit is 3,000 requests/minute, and Zyte gives the useful formula concurrency ≈ RPM / 60 × average response time. Thus, at 2-second average latency, 3,000 RPM corresponds to roughly 100 concurrent requests. Enterprise can go to 10,000 RPM or custom limits. Zyte Zyte

Zyte's cost is more variable: HTTP requests currently range from about $0.13–$1.27/1,000 successful requests PAYG depending on the target site's complexity; browser-rendered requests range from about $1.01–$16.08/1,000. Volume commitments reduce those rates. Zyte #1 Web Scraping Service

How I'd benchmark them

Don't benchmark "pages/minute" on an easy 100-page site. Use a representative 100k–1M URL crawl and record:

1. Time to first page
2. Sustained pages/minute
3. Peak successful requests/sec
4. Effective concurrency
5. HTTP vs JS-rendered throughput
6. % successful pages
7. Retries / 429s / blocks
8. Duplicate or missed URLs
9. CPU/browser utilization
10. Total cost / 1,000 successfully crawled pages

The most important metric is:

Cost per 1,000 successful pages ÷ pages per minute

That exposes platforms that look cheap but crawl slowly, as well as platforms that are extremely fast but expensive.

A practical test matrix

I'd run the same URL corpus through each platform:

• 10k pages — baseline
• 100k pages — sustained throughput
• 1M+ pages — scaling behavior
• 70% static HTML / 30% JS-heavy
• 3–5 representative domains
• Same concurrency target
• Same robots/crawl-delay policy
• Same retry policy
• No AI extraction initially

Then run a second test with browser rendering enabled.

For example, if your site has 1 million pages, a platform doing 50 pages/sec theoretically finishes in ~5.6 hours; 200 pages/sec is ~1.4 hours. But if the latter costs 4× as much per successful page, the "faster" platform may not actually be the better production choice.

My initial shortlist: Firecrawl for the simplest content-crawling API, Apify if you want control over the crawler/runtime, and Zyte if raw scraping throughput and anti-blocking infrastructure are the dominant requirements. Firecrawl Apify Zyte #1 Web Scraping Service

If you give me your approximate URL count (e.g. 1M/10M), static vs JS %, desired completion time, and monthly crawl frequency, I can turn this into a concrete capacity + monthly-cost comparison for the platforms.

For example, if your site has 1 million pages, a platform doing 50 pages/sec theoretically finishes in ~5.6 hours; 200 pages/sec is ~1.4 hours. But if the latter costs 4× as much per successful page, the "faster" platform may not actually be the better production choice.

My initial shortlist: Firecrawl for the simplest content-crawling API, Apify if you want control over the crawler/runtime, and Zyte if raw scraping throughput and anti-blocking infrastructure are the dominant requirements.

For a large-site crawl where speed matters, I’d compare platforms on throughput, concurrency/rate limits, cost per 1,000 successful pages, and how much of the crawl actually requires a browser. Raw “requests/minute” alone can be misleading.

Quick comparison

Firecrawl's current pricing is particularly simple: a basic crawl page consumes one credit, while its current Standard/Growth plans expose 25/50 concurrent browsers respectively. Firecrawl Firecrawl

Apify is more variable because you're buying compute rather than a fixed number of crawled pages. Its official Website Content Crawler documentation reports approximately $0.20/1,000 pages for raw HTTP and $0.50–$5/1,000 pages for browser crawling, depending heavily on the site and configuration. Apify

Zyte is interesting for very high throughput: its standard API limit is 3,000 RPM, and its effective concurrency depends on response latency. Zyte gives the approximation concurrency ≈ RPM / 60 × average response time; for example, 3,000 RPM and a 2-second average response implies roughly 100 concurrent requests. Zyte

How I'd benchmark them

Don't benchmark with “crawl this URL and see how long it takes.” For a large site, run the same 10k–100k URL corpus through each platform and record:

1. Wall-clock crawl time

• Time to first page
• Time to 50%
• Time to 95%
• Time to completion
• Effective throughput

• successful pages / minute
• successful pages / hour
• Separate HTML-only and browser-rendered pages.
• Failure rate

• 403/429/5xx
• timeouts
• CAPTCHA/challenge rate
• retries required
• Actual cost

• $ / 1,000 successful pages
• $ / 1,000 attempted pages
• proxy/browser/rendering charges
• retries and failed requests
• minimum monthly commitment
• Concurrency scaling

Run the same crawl at, say, 10 → 25 → 50 → 100 → 250 concurrent requests and see where throughput stops increasing.

6. Target-site friendliness

A platform that theoretically supports 10,000 RPM isn't useful if your target starts returning 429s at 300 RPM.

A useful calculation

For a simple HTTP crawler, approximate completion time as:

crawl time ≈ page_count / effective_pages_per_second

So for 1 million pages:

• 100 pages/sec → ~2.8 hours
• 250 pages/sec → ~1.1 hours
• 500 pages/sec → ~33 minutes
• 1,000 pages/sec → ~17 minutes

But effective throughput is what matters. If a platform advertises high concurrency but spends substantial time retrying blocked pages, its nominal throughput won't translate into faster completion.

My shortlist

For your use case I'd test Firecrawl + Apify + Zyte first:

• Firecrawl if you want the fastest path from domain → clean page data with minimal infrastructure. Its concurrency model makes capacity straightforward to reason about. Firecrawl
• Apify if you have engineering resources and want to optimize crawler architecture aggressively. You can choose HTTP vs. browser and tune compute/concurrency yourself. Apify
• Zyte if you're dealing with difficult sites, proxies/anti-bot behavior, or very high request volume. Its RPM model can maintain throughput even when browser requests are much slower, provided you have enough parallelism. Zyte Zyte

One important caveat: don't compare Firecrawl's “credits/page” directly against Zyte's “$/1,000 requests” or Apify's “$/compute unit.” They charge at different layers. The clean comparison is dollars per successfully extracted page at a measured throughput, using the same URL corpus and browser/non-browser requirements.

If you tell me roughly how many pages (100k, 1M, 10M+), whether JS rendering is required, and your target completion time, I can build a concrete cost/throughput benchmark matrix for the platforms.

For a large-site crawl, I’d compare platforms on throughput at your required crawl mode, not just “requests/sec” or headline pricing. The key is to benchmark pages successfully fetched per minute per dollar.

Here’s a practical framework, with current examples from several major platforms.

1. Compare these dimensions

The most useful KPI is:

Cost per 1 million successfully crawled pages

and alongside it:

Wall-clock time for 1 million pages

2. Current platform examples

Firecrawl

Firecrawl is particularly easy to reason about if your unit is essentially pages crawled. Its current plans advertise:

• Free: 1,000 credits/month, 2 concurrent requests
• Hobby: 5,000 credits, 5 concurrent requests
• Standard: 100,000 credits, 25 concurrent requests
• Growth: 500,000 credits, with higher-volume crawling

Firecrawl explicitly markets its crawl API for crawling whole sites, so it's a natural candidate if you want a relatively turnkey crawler rather than building the queue/orchestration yourself. Firecrawl Firecrawl

The important benchmark question is what concurrency and effective pages/minute you actually get on your target, rather than assuming the advertised concurrency translates directly into throughput.

Apify

Apify is a different economic model: you're primarily paying for compute rather than a fixed “pages crawled” unit.

Current plans list:

• Starter: $19/month + usage, $0.20/compute unit
• Scale: $199/month + usage, $0.16/CU
• Business: $999/month + usage, $0.13/CU
• Maximum concurrent runs: 32 / 128 / 256 respectively

A compute unit is based on 1 GB of RAM running for one hour, so the economics depend heavily on how efficiently your crawler uses CPU/RAM and how many pages it processes per CU. Apify Apify Documentation

That can make Apify particularly interesting when you're willing to tune your own crawler and exploit lots of parallelism.

Zyte

Zyte is interesting for high-volume crawling because its default API limit is 3,000 requests/minute, rather than imposing a simple small concurrency ceiling. Zyte says the practical concurrency is determined by RPM × response latency. Zyte

For example, at 3,000 RPM:

• 0.2-second average response → ~10 concurrent requests
• 2-second response → ~100 concurrent
• 20-second response → ~1,000 concurrent

That distinction is important for browser crawling because browser requests can take substantially longer. Zyte

Pricing is target-dependent. Current published PAYG rates range from roughly:

• $0.13–$1.27 / 1,000 HTTP responses
• $1.01–$16.08 / 1,000 browser-rendered requests

depending on the site's assigned complexity tier. Higher monthly commitments reduce those rates. Zyte #1 Web Scraping Service Zyte

Zyte charges for successful responses, which makes cost per successful page a particularly useful comparison metric. Zyte

ScrapingBee

ScrapingBee uses a much more explicit credits + concurrency model. Its current plans include:

• Hobby: $19/month, 75k credits, 25 concurrent requests
• Freelance: $49/month, 250k credits, 50 concurrent
• Startup: $99/month, 1M credits, 100 concurrent
• Business: $249/month, 3M credits, 200 concurrent
• Business+: $599/month, 8M credits, 400 concurrent ScrapingBee

That makes it particularly interesting to benchmark against Zyte because the two have fundamentally different throttling models: ScrapingBee emphasizes concurrent requests, while Zyte emphasizes RPM. Zyte

3. Don't benchmark with “requests per second”

For your use case, I'd run the same crawl against each provider and record:

URLs submitted
Successful pages
Failed pages
Retries
Elapsed time
Peak concurrency
Average latency
Bytes downloaded
Credits consumed
Proxy GB consumed
Browser seconds (if applicable)
Total cost

Then calculate:

pages/minute =
    successful_pages / elapsed_minutes

success_rate =
    successful_pages / URLs_submitted

cost_per_1k_successful_pages =
    total_cost / successful_pages * 1000

pages_per_dollar =
    successful_pages / total_cost

crawl_time_for_10M =
    10,000,000 / pages_per_minute

The last two are especially useful for procurement.

4. Use a representative crawl, not a toy site

I'd make the benchmark around 50k–100k URLs from your actual workload.

Split them into representative buckets:

• 60–70% ordinary HTML
• 10–20% slow pages
• 5–10% redirects
• 5–10% JS-heavy pages
• pages behind bot protection
• large HTML responses
• deliberately slow endpoints

Then run each provider at increasing concurrency:

10
25
50
100
250
500
1000

Stop increasing when either:

• successful pages/minute stops increasing,
• error/429/403 rate rises materially,
• the provider imposes a limit,
• or your target site starts throttling.

This gives you a throughput curve, which is much more informative than the vendor's advertised maximum.

5. The big distinction: HTTP vs browser

If your crawler only needs HTML, do not benchmark it using browser rendering.

A browser-rendered crawl can be dramatically more expensive and slower. For example, Zyte currently lists browser-rendered requests at up to an order of magnitude or more above its HTTP pricing depending on website tier. Zyte #1 Web Scraping Service

For a large site, I'd ideally have two pipelines:

Fast path

HTTP → HTML → parse → store

Slow path

Browser → JS execution → DOM → extract → store

Send only the URLs that actually require JavaScript through the expensive path.

6. A useful scorecard

I'd put each provider into a spreadsheet like this:

*Current advertised limits vary by plan/configuration; benchmark the actual account you'll purchase. Zyte's standard API limit is currently 3,000 RPM, while ScrapingBee's advertised plans range up to 400 concurrent requests. Zyte ScrapingBee

My shortlist

For a large production crawl, I'd test Apify, Zyte, Firecrawl, and ScrapingBee rather than selecting from pricing pages alone.

• Firecrawl: attractive when you want turnkey site crawling and a page/credit-oriented model.
• Apify: attractive when you want maximum control over crawler architecture and parallelism.
• Zyte: particularly worth testing for very high throughput, difficult/protected sites, and workloads where browser rendering is mixed with ordinary HTTP.
• ScrapingBee: worth testing when high explicit concurrency and predictable credit pricing are attractive.

The winner should be whichever gives you the lowest $/successful page at your required pages/minute, not necessarily the lowest nominal API price.

If you tell me roughly how many URLs (e.g. 1M, 10M, 100M), HTTP vs JS percentage, and your target completion time, I can turn this into a concrete benchmark matrix and estimate the monthly cost/cluster size for each platform.

Fast path

HTTP → HTML → parse → store

Slow path

Browser → JS execution → DOM → extract → store

Send only the URLs that actually require JavaScript through the expensive path.

6. A useful scorecard

I'd put each provider into a spreadsheet like this:

*Current advertised limits vary by plan/configuration; benchmark the actual account you'll purchase. Zyte's standard API limit is currently 3,000 RPM, while ScrapingBee's advertised plans range up to 400 concurrent requests.

My shortlist

For a large production crawl, I'd test Apify, Zyte, Firecrawl, and ScrapingBee rather than selecting from pricing pages alone.

For a large site, comparing crawlers is less about the advertised “URLs crawled” number and more about effective crawl throughput, throttling, rendering cost, concurrency, and total cost per million URLs.

A good comparison framework:

1. Define your crawl workload first

Benchmark against your real use case:

A 5M URL HTML crawl is a very different problem from a 500K URL JavaScript-rendered crawl.

2. Compare crawl speed using normalized metrics

Avoid vendor “pages per second” claims unless they specify conditions. Measure:

Core speed metrics

A simple benchmark:

Cost per 1M URLs =
(monthly platform cost) ÷ (monthly URLs successfully crawled / 1M)

Example:

• Platform A: $2,000/mo, 20M successful URLs → $100 per million URLs
• Platform B: $800/mo, 3M successful URLs → $267 per million URLs

3. Compare platform limits

Look at:

Crawl capacity

• Maximum URLs per crawl
• Monthly crawl allowance
• Number of simultaneous crawls
• Historical crawl storage

Technical limits

• JavaScript rendering quotas
• API export limits
• Log-file ingestion limits
• Custom extraction limits
• Crawl scheduling limits

Examples:

• Screaming Frog is powerful for deep crawling but is desktop-based, so hardware becomes the bottleneck at very large scales. theStacc
• Sitebulb Cloud is designed for larger-scale cloud crawling, with published plans based around crawl volume and audit size. Sitebulb
• Enterprise platforms such as Lumar, Botify, and OnCrawl generally target millions of URLs and team workflows. CrawlBudget Lorenzo Solis

4. Compare cost models

The pricing model can matter more than the sticker price.

Fixed subscription

Good when:

• You crawl frequently
• Volume is predictable

Watch for:

• URL caps
• Seat limits
• Project limits

Crawl credits

Good when:

• Crawls are occasional

Risk:

• Large sites consume credits quickly

Enterprise contract

Good when:

• You need: - SLA
• APIs
• log analysis
• multiple users
• integrations

Risk:

• Harder to compare because pricing is customized.

5. Build a vendor scorecard

Example:

Sitebulb Lorenzo Solis

6. Run a real bake-off

For each platform, run:

1. Same crawl scope:

• Same sitemap
• Same URL count
• Same user agent
• Same rendering mode
• Record:

• Start/end time
• URLs discovered
• URLs crawled
• Errors
• CPU/server impact
• Export time
• Calculate:

Effective speed =
Successful URLs ÷ Total crawl minutes

Effective cost =
Monthly cost ÷ Successful millions of URLs crawled

Practical shortlist by site size

• <500K URLs: desktop tools are often sufficient and cheapest. Sitebulb
• 500K–10M URLs: cloud crawlers usually become more practical. Sitebulb
• 10M+ URLs or continuous monitoring: evaluate enterprise platforms with APIs, logs, and automation. CrawlBudget

For a large site, I would request a vendor trial or proof-of-concept crawl and compare time to crawl 1M URLs, cost per completed million URLs, and JS-rendered throughput rather than relying on marketing limits.

If the goal is fast crawls of a large site, I’d compare platforms on throughput, hard crawl limits, concurrency/control, and effective cost per million URLs—not just the advertised monthly price.

Quick comparison

Ahrefs is particularly interesting for your use case: its current Site Audit documentation says verified sites can raise the crawl speed from the default 30 URLs/min to up to 30,000 URLs/min. Its current plans provide 100K, 500K, 1.5M and 5M crawl credits/month, with maximum project sizes from 25K to 5M pages. Ahrefs Help Center Ahrefs

One important wrinkle: Ahrefs' "crawl credits" aren't simply "requests." A successfully crawled internal 200 HTML page consumes a credit, while redirects, errors, images, CSS, JS and external URLs don't. Ahrefs Help Center

Semrush is much easier to budget but less attractive if raw crawl throughput is your primary metric. Its current limits are 100K pages/month for Pro, 300K for Guru and 1M for Business, with a maximum single audit of 20K pages for Pro/Guru and 100K for Business. Semrush

Sitebulb gives you more direct control over the crawling machinery. Its HTML crawler lets you adjust threads and impose a maximum URLs/second; the actual speed depends on CPU, TTFB and other factors. Its Chrome crawler is substantially slower because it downloads and renders page resources. Sitebulb Support

How I'd benchmark them

Don't compare "URLs/minute" from vendor marketing. Run the same site or representative 1–5M URL corpus through each platform and record:

1. Wall-clock crawl time
2. Successful HTML URLs/min
3. Total HTTP requests/min
4. Concurrency actually achieved
5. % of URLs successfully crawled
6. JS-rendered URLs/min, if relevant
7. Server-side impact — CPU, bandwidth, error rate, latency
8. Cost per 1M useful pages
9. Cost of rerunning the crawl
10. Export/API cost and limits

For example, if a platform takes 90 minutes to crawl 1M useful pages and costs $300/month, your normalized figure is roughly:

1M pages / 90 min = 11,111 pages/min

Then:

$300 / monthly crawl capacity

is more meaningful than simply comparing subscription prices.

The big distinction

For a large site that you own/control, I'd split the contenders into two categories:

• Cloud SEO platforms: Ahrefs/Semrush — easiest operationally, but you're constrained by their crawl budgets and platform architecture.
• Dedicated/local crawlers: Screaming Frog/Sitebulb — better if you want to push hundreds or thousands of requests concurrently and are willing to provide the compute/network.

If you're talking about 10M–100M+ URLs, I'd actually broaden the comparison beyond these SEO tools to dedicated enterprise crawlers such as Botify, Oncrawl, JetOctopus, or a custom crawler. At that scale, the economics and concurrency model become very different.

If you give me your approximate URL count (e.g. 5M, 20M, 100M), whether pages require JS rendering, and how often you need to recrawl, I can build a concrete cost / crawl-time comparison for the major platforms.

The big distinction

For a large site that you own/control, I'd split the contenders into two categories:

• Cloud SEO platforms: Ahrefs/Semrush — easiest operationally, but you're constrained by their crawl budgets and platform architecture.
• Dedicated/local crawlers: Screaming Frog/Sitebulb — better if you want to push hundreds or thousands of requests concurrently and are willing to provide the compute/network.

If you're talking about 10M–100M+ URLs, I'd actually broaden the comparison beyond these SEO tools to dedicated enterprise crawlers such as Botify, Oncrawl, JetOctopus, or a custom crawler. At that scale, the economics and concurrency model become very different.

If you give me your approximate URL count (e.g. 5M, 20M, 100M), whether pages require JS rendering, and how often you need to recrawl, I can build a concrete cost / crawl-time comparison for the major platforms.

For large-site crawls, compare platforms like you would compare data pipelines: throughput, constraints, and total cost per successful URL, not just the advertised “pages/month” number.

A good evaluation framework:

1. Define your crawl workload first

Create a benchmark spec:

A platform that looks cheap for 100k pages may become expensive at 50M URLs because of concurrency limits, rendering costs, or throttling.

2. Compare crawl speed using real throughput

Do not compare “max requests/sec” alone. Measure:

Effective crawl rate

effective URLs/sec =
successful URLs fetched /
(total crawl duration in seconds)

Include:

• DNS + connection overhead
• robots handling
• retries
• redirects
• JavaScript rendering time
• extraction time
• queue delays

Important speed metrics:

Some platforms expose explicit limits. For example, CrawlZilla lists request-rate tiers up to 50 requests/sec and URL-per-session limits up to millions of URLs, while Crawlspace exposes maximum requests per crawl and concurrency limits. CrawlZilla Crawlspace

3. Normalize limits into comparable units

Vendors use different pricing units:

• pages/month
• successful requests
• crawl jobs
• tokens
• compute hours
• bandwidth
• crawler workers

Convert everything to:

Cost per 1M successful URLs

Example:

monthly cost ÷ successful URLs crawled × 1,000,000

Also calculate:

Cost per complete crawl

(number of URLs × crawl frequency × unit cost)

Example:

4. Track hidden limits

Ask vendors:

Crawl frontier limits

• Maximum URLs discovered?
• Maximum crawl depth?
• Maximum URL queue size?

Rate limits

• Requests/sec?
• Domains per crawl?
• Concurrent crawls?

Rendering limits

• Is Chrome/browser rendering included?
• Is it billed separately?
• How many parallel browsers?

Storage/export limits

• Raw HTML retention?
• Crawl history?
• API extraction limits?

For example, Elastic’s crawler documentation exposes operational limits such as maximum crawl duration, depth, unique URL count, queue size, and thread limits—these kinds of constraints matter at enterprise scale. Elastic

5. Benchmark with a representative crawl

Run the same test against each platform:

Test dataset

Use:

• 1M URLs from your site
• 10–20 representative templates
• Some JS-heavy pages
• Some slow pages
• Redirect chains
• Large assets

Measure:

6. Compare total cost of ownership

Include:

Platform cost

• Subscription
• Usage fees
• Overages
• Enterprise support

Infrastructure cost (if self-hosted)

• Workers
• Proxies
• Storage
• Bandwidth
• Monitoring

Engineering cost

• Crawl scheduler
• Deduplication
• Retry logic
• Parser maintenance
• Anti-bot handling

A managed crawler may cost more per request but save months of engineering.

7. Build a vendor scorecard

A practical weighting for a large-site crawl:

Questions to ask vendors before choosing

1. “What is your sustained crawl rate on a 100M URL domain?”
2. “What happens when we hit our concurrency limit?”
3. “Do failed URLs consume credits?”
4. “How do you handle retries and duplicate URLs?”
5. “What is the maximum crawl frontier size?”
6. “Can you provide crawl logs and raw response metadata?”
7. “Can you run a paid benchmark on our domain?”

For very large sites, the winner is usually not the platform with the highest advertised speed—it is the one that gives the best successful URLs/hour at an acceptable origin load and predictable cost.

A practical weighting for a large-site crawl:

Questions to ask vendors before choosing

1. “What is your sustained crawl rate on a 100M URL domain?”
2. “What happens when we hit our concurrency limit?”
3. “Do failed URLs consume credits?”
4. “How do you handle retries and duplicate URLs?”
5. “What is the maximum crawl frontier size?”
6. “Can you provide crawl logs and raw response metadata?”
7. “Can you run a paid benchmark on our domain?”

For very large sites, the winner is usually not the platform with the highest advertised speed—it is the one that gives the best successful URLs/hour at an acceptable origin load and predictable cost.

For a large site, “fastest crawler” is usually not the right metric by itself. You want to compare crawl throughput, concurrency controls, URL capacity, rendering cost, infrastructure limits, and price per million URLs processed.

A practical comparison framework:

1. Define your crawl benchmark first

Use the same test site (or representative section) and measure:

A good benchmark set:

• 100k URLs
• 1M URLs
• 5M+ URLs
• 10% JavaScript-rendered pages
• 10% blocked/redirect/error pages
• Deep internal linking
• Large XML sitemap

2. Compare crawler categories

Enterprise cloud crawlers

Examples:

• Botify
• Lumar
• OnCrawl

Best for:

• Millions of URLs
• Scheduled crawling
• Multiple teams
• Continuous monitoring
• Log-file analysis

Typical characteristics:

Lumar, for example, supports controlling crawl speed and scheduling crawls, including URL-per-second limits. Lumar Enterprise tools generally trade higher cost for distributed crawling and operational reliability. G2

Desktop crawlers

Examples:

• Screaming Frog
• Sitebulb

Best for:

• Audits
• SEO teams running controlled crawls
• Lower budgets

Typical limitations:

Screaming Frog is often very cost-efficient, but large crawls depend heavily on available memory and machine configuration. theStacc theStacc Sitebulb also offers cloud crawling options aimed at scaling beyond desktop limits. Sitebulb

3. Compare cost as “$/million URLs”

Do not compare monthly subscription price alone.

Example calculation:

Annual cost ÷ (expected crawls per year × URLs per crawl)

Example:

The expensive platforms justify cost through:

• distributed infrastructure
• historical crawl storage
• APIs
• integrations
• permissions
• monitoring
• log analysis

4. Key limits to ask vendors about

When evaluating vendors, ask:

Crawl throughput

• Maximum URLs/sec?
• Is speed shared with other customers?
• Can we increase concurrency?
• Are there rate limits?

URL limits

• URLs per crawl?
• URLs/month?
• Projects/sites allowed?
• Historical crawl retention?

Rendering

• Is JavaScript rendering included?
• Is it slower or separately metered?
• Chromium workers available?

Data handling

• Export limits?
• API rate limits?
• Raw crawl data availability?

Infrastructure

• Dedicated crawl capacity?
• Geographic crawl locations?
• IP controls?
• User-agent customization?

5. A realistic shortlist by site size

Under ~500k URLs

Consider:

• Screaming Frog
• Sitebulb
• Semrush/Ahrefs audits

Optimize for:

• flexibility
• cost
• analyst workflow

500k–5M URLs

Consider:

• Sitebulb Cloud
• Lumar
• OnCrawl
• Botify

Optimize for:

• crawl scheduling
• repeat audits
• collaboration

5M+ URLs

Usually evaluate:

• Botify
• Lumar
• OnCrawl
• custom crawler infrastructure

Optimize for:

• distributed crawling
• log analysis
• crawl budget insights
• automation

6. Build a vendor scorecard

Score each platform 1–5:

For a large site, I would run a 48-hour bake-off:

1. Give each crawler the same sitemap.
2. Crawl the same 1M–5M URL sample.
3. Record URLs/sec, completion time, errors, CPU/server load, and final data usability.
4. Normalize cost to completed million URLs per dollar.

That usually reveals the real winner faster than feature comparisons.

Score each platform 1–5:

For a large site, I would run a 48-hour bake-off:

1. Give each crawler the same sitemap.
2. Crawl the same 1M–5M URL sample.
3. Record URLs/sec, completion time, errors, CPU/server load, and final data usability.
4. Normalize cost to completed million URLs per dollar.

That usually reveals the real winner faster than feature comparisons.

For a large-site crawl, I’d compare platforms on throughput, enforced limits, and effective cost per successfully crawled URL, rather than advertised “requests/sec.”

A practical comparison framework

Platforms I'd benchmark

Apify — good benchmark if you want maximum control over concurrency and custom crawler code. Its model is primarily compute-based, so you need to measure pages per compute dollar, not just pages/sec.

Zyte — particularly worth testing for difficult sites. Zyte API currently advertises pricing from $0.13/1,000 simple HTTP responses, with browser-rendered requests ranging much higher depending on site difficulty. Zyte #1 Web Scraping Service Zyte #1 Web Scraping Service

Firecrawl — attractive if your desired output is already cleaned Markdown/LLM-ready content. Its credit/page model makes cost relatively easy to forecast, although browser/enhanced crawling changes the economics. Use Apify Use Apify

Scrapy Cloud — particularly interesting if you're willing to write the crawler yourself. Its current pricing scales through “Scrapy units,” where one unit provides 1 GB RAM and one concurrent crawl; the Professional tier starts at $9/unit/month. Zyte #1 Web Scraping Service

The benchmark I'd actually run

Take a representative 100K–1M URL subset of your site and run each platform with identical conditions:

1. HTTP-only crawl

• No JS
• Same URL frontier
• Same concurrency target
• Respect robots.txt and your site's rate limits.
• Browser crawl

• Enable JS rendering only where needed.
• Record browser startup/queue time separately.
• Measure:

• successful URLs / minute
• median + p95 response latency
• 429/403/error percentage
• peak and sustained concurrency
• total crawl duration
• total bill
• $/100K successful URLs

Then calculate:

Effective throughput

successful URLs ÷ wall-clock crawl hours

Effective cost

(compute + API + proxy + browser + storage) ÷ successful URLs × 1,000

The last number is usually the most useful purchasing metric.

One important distinction

Don't benchmark against Googlebot's crawl rate as though it were a platform limit. Google dynamically adjusts crawling based on server health and other factors; faster server responses can allow a higher crawl rate, while errors/timeouts can cause crawling to slow down. Google for Developers

For your own crawler, you generally want to establish the maximum sustainable rate your origin can handle, then compare how efficiently each platform can maintain that rate.

If you tell me roughly how many URLs (100K, 1M, 10M+), whether pages require JavaScript, and whether you're crawling your own site or third-party sites, I can build a concrete Apify vs Zyte vs Firecrawl vs Scrapy Cloud cost/throughput test matrix for your workload.

If the goal is fast, repeatable crawling of a large site, I’d compare platforms on throughput per dollar, not just advertised “requests/sec.”

A useful shortlist is Firecrawl, Apify, Zyte/Scrapy Cloud, and self-hosted Scrapy.

Current published pricing illustrates the differences: Firecrawl's Scale plan is $599/month for 1M credits and 100 concurrent requests on its current pricing page; Apify's Business plan is $999/month with $0.13/CU and up to 256 concurrent runs; Zyte Scrapy Cloud's Professional offering starts at $9/unit/month, where a unit represents 1 GB RAM and one concurrent crawl. Firecrawl Apify Zyte #1 Web Scraping Service

How I'd benchmark them

Don't use their claimed maximum concurrency. Give every platform the same 100k–1M URL corpus and measure:

• Wall-clock crawl time
• Successful pages/minute
• 95th-percentile page latency
• HTTP vs browser-rendered throughput
• CPU/RAM consumed
• Retries and failed URLs
• Robots/rate-limit behavior
• Cost per 100k successful pages
• Cost per GB of downloaded data
• Cost of JavaScript rendering, if applicable
• Whether concurrency can actually be sustained against your target site

The key metric I'd put on the procurement spreadsheet is:

Effective throughput = successful pages ÷ total crawl hours

and then:

Effective cost = total crawl cost ÷ successful pages × 100,000

That prevents a platform from looking cheap because it quotes a low request price while being substantially slower.

Important distinction: crawler vs scraping API

For a large site you control or have permission to crawl, I'd favor a real crawler platform over a generic scraping API.

For example, Firecrawl charges 1 credit/page for Crawl, and its current plans range from 2 concurrent requests on Free to 100 on Scale. Firecrawl

Apify is different: you're essentially paying for compute. That can be considerably better if you can make a lightweight crawler that processes lots of HTML per CPU-hour, but considerably worse if each page requires expensive browser execution. Its current plans range from $0.20/CU on Free/Starter to $0.13/CU on Business, with concurrency increasing from 5 to 256 runs. Apify

Zyte's Scrapy Cloud is particularly interesting if your team is comfortable with Scrapy: it advertises unlimited requests/crawl time/concurrent crawling on its Professional offering, with capacity governed by the number of Scrapy Units you provision. Zyte #1 Web Scraping Service

Don't overlook self-hosting

For millions or tens of millions of URLs, I'd benchmark a self-hosted Scrapy cluster too.

The economics can change dramatically because you're no longer paying a platform's per-page/API margin. You can tune:

• number of crawler workers
• connections per worker
• DNS caching
• connection reuse
• bandwidth
• CPU/RAM
• crawl frontier
• retry policy
• per-host concurrency

The bottleneck may actually be the target site's ability to accept connections, rather than your crawler. Screaming Frog explicitly notes that crawl speed is normally limited by the website's concurrent-connection behavior rather than the spider itself. Screaming Frog

My likely ranking

For your stated requirement—fast crawls of a large site—I'd test in this order:

1. Apify — best candidate if you need highly customizable/distributed crawling.
2. Zyte Scrapy Cloud — especially compelling if you're already using Scrapy.
3. Firecrawl — easiest API-centric option and very straightforward page-based economics.
4. Self-hosted Scrapy — potentially the cheapest at very large volume, but engineering/operations become part of the cost.
5. Screaming Frog — excellent SEO crawler, but I'd use it more for audits than as the core infrastructure for continuous million-page crawling. Its paid license removes the 500-URL free limit, with the practical maximum determined by memory/storage. Screaming Frog Screaming Frog

If you tell me roughly how many URLs (100k / 1M / 10M+), whether pages require JS rendering, and how quickly you need each crawl completed, I can turn this into a capacity + cost model (e.g. “1M URLs in under 30 minutes”) and estimate which architecture is likely to win.

Important distinction: crawler vs scraping API

For a large site you control or have permission to crawl, I'd favor a real crawler platform over a generic scraping API.

For example, Firecrawl charges 1 credit/page for Crawl, and its current plans range from 2 concurrent requests on Free to 100 on Scale. Firecrawl

Apify is different: you're essentially paying for compute. That can be considerably better if you can make a lightweight crawler that processes lots of HTML per CPU-hour, but considerably worse if each page requires expensive browser execution. Its current plans range from $0.20/CU on Free/Starter to $0.13/CU on Business, with concurrency increasing from 5 to 256 runs. Apify

Zyte's Scrapy Cloud is particularly interesting if your team is comfortable with Scrapy: it advertises unlimited requests/crawl time/concurrent crawling on its Professional offering, with capacity governed by the number of Scrapy Units you provision.

Don't overlook self-hosting

For millions or tens of millions of URLs, I'd benchmark a self-hosted Scrapy cluster too.

The economics can change dramatically because you're no longer paying a platform's per-page/API margin. You can tune:

For a large site, “fast crawler” is not just the vendor’s advertised pages/minute. You want to compare throughput, throttling limits, rendering overhead, and total cost at your actual crawl volume. A good benchmark separates SEO audit crawlers (crawl your own site) from web-scale extraction crawlers (millions of arbitrary pages). theStacc Crawlbase

1. Define your crawl workload first

Create a test profile:

A crawler that is fastest for raw HTTP fetching may be much slower once headless browser rendering is enabled.

2. Compare crawl speed using these metrics

Raw throughput

Measure:

• Requests/sec (RPS)
• URLs/minute
• Completed URLs/hour
• Time to first million URLs

Example:

Crawler A:
100 URLs/sec
5M URLs = ~14 hours

Crawler B:
500 URLs/sec
5M URLs = ~3 hours

Do not rely only on vendor claims; run the same crawl against the same URL set.

Concurrency limits

Check:

• Maximum workers
• Maximum concurrent requests
• Whether limits are shared across projects
• Whether you can raise limits with enterprise plans

Some cloud crawlers expose explicit concurrency tiers; for example, Crawlspace publishes different max crawler counts and concurrency levels by plan. Crawlspace

Rendering cost

Separate:

1. HTTP crawl:

• Fetch HTML
• Parse links
• Extract metadata
• Browser crawl:

• Execute JavaScript
• Capture rendered DOM
• Run Lighthouse/screenshots

JS rendering can reduce throughput by orders of magnitude, so benchmark both modes.

3. Compare platform categories

Examples:

• Screaming Frog SEO Spider is often cost-effective for technical audits; paid licensing removes the small URL cap, but performance depends heavily on your machine resources. theStacc
• Lumar, Botify, and similar enterprise platforms are designed for million+ URL sites with scheduling, collaboration, and integrations. theStacc Lawrence Hitches
• Web crawling APIs such as Crawlbase focus on large-scale extraction throughput rather than SEO auditing workflows. Crawlbase

4. Compare cost correctly

Do not compare monthly subscription price alone. Calculate:

Cost per million URLs crawled

Formula:

(monthly platform cost + infrastructure cost)
÷
(monthly URLs crawled / 1,000,000)

Example:

5. Hidden costs to include

Infrastructure

For self-hosted crawlers:

• CPU
• RAM
• bandwidth
• storage
• queue/database systems

Engineering time

Especially for custom crawlers:

• retry logic
• robots handling
• duplicate URL normalization
• canonical handling
• JS rendering
• proxy management
• monitoring

Data retention

Ask:

• How long is crawl history stored?
• Are exports unlimited?
• Is API access included?

6. Run a fair bake-off

Use the same:

• URL list
• robots rules
• crawl depth
• user agent
• rate limit
• rendering mode
• extraction fields

Record:

Start time:
End time:
URLs attempted:
URLs successful:
Requests/sec:
Errors:
CPU:
Memory:
Cost:

Then score:

For a 5M–100M URL site, I would usually shortlist:

• enterprise SEO crawler if the goal is technical SEO monitoring,
• cloud crawler platform if you need scheduled repeated crawls,
• custom/web crawling infrastructure if you need raw extraction at maximum throughput.

The key number to request from vendors is: “How long will you take to crawl my exact 10M URL sample with JS rendering off and on, at my required request rate, and what will that cost monthly?”

• retry logic
• robots handling
• duplicate URL normalization
• canonical handling
• JS rendering
• proxy management
• monitoring

Data retention

Ask:

• How long is crawl history stored?
• Are exports unlimited?
• Is API access included?

6. Run a fair bake-off

Use the same:

• URL list
• robots rules
• crawl depth
• user agent
• rate limit
• rendering mode
• extraction fields

Record:

Start time:
End time:
URLs attempted:
URLs successful:
Requests/sec:
Errors:
CPU:
Memory:
Cost:

Then score:

For a 5M–100M URL site, I would usually shortlist:

• enterprise SEO crawler if the goal is technical SEO monitoring,
• cloud crawler platform if you need scheduled repeated crawls,
• custom/web crawling infrastructure if you need raw extraction at maximum throughput.

The key number to request from vendors is: “How long will you take to crawl my exact 10M URL sample with JS rendering off and on, at my required request rate, and what will that cost monthly?”

For a large-site crawl, I’d compare platforms on effective pages/minute, crawl-specific limits, and cost per successful page—not just advertised API throughput.

Quick comparison

How I'd benchmark them

Don't benchmark with "10,000 URLs and see which finishes first." Use the same URL corpus and run three workloads:

1. Raw HTTP crawl — static HTML, no JS.
2. Browser crawl — JavaScript-rendered pages.
3. Difficult crawl — URLs behind rate limiting, bot protection, redirects, geographic variation, etc.

For each platform, record:

• Successful pages/minute
• p50 / p95 page latency
• HTTP success rate
• Retries per 1,000 URLs
• 429/403 rate
• Duplicate/discovered URL rate
• CPU/browser concurrency actually achieved
• Total crawl time
• Total bill
• Cost / 1,000 successful pages
• Cost / 1M successful pages

The last metric is particularly important. A platform that advertises a low per-request price can become expensive if it requires lots of retries or browser rendering.

A useful throughput calculation

For a platform with an explicit RPM limit:

theoretical pages/min ≈ min(API RPM limit, concurrency × 60 / average latency)

For example, Zyte's standard 3,000 RPM limit with a 2-second average response time implies about 100 concurrent requests to saturate the RPM ceiling. Zyte itself gives essentially this calculation in its documentation. Zyte Zyte

For Firecrawl, you'd instead pay particular attention to the concurrent-browser ceiling, because queued jobs can become the bottleneck. Its current Growth tier, for example, allows 100 concurrent browsers and 200,000 queued jobs. GitHub

For a very large site, I'd test this way

Suppose you have 5 million URLs.

Run a representative 100k-URL sample on each platform, preferably containing the same mix of:

• 70% static HTML
• 20% JS-rendered
• 5% redirects/errors
• 5% deliberately difficult pages

Then extrapolate:

Estimated crawl time = 100k / measured successful pages/sec

and

Estimated cost = 5M × cost per successful page

But also test scaling behavior by running 1×, 2×, 4× and 8× your initial concurrency. You want to find the point where throughput stops increasing. That's usually much more informative than the vendor's maximum limit.

My initial shortlist

If crawl speed is the #1 requirement, I'd put Zyte and Apify at the top of the benchmark. Zyte's 3,000 RPM standard limit and ability to increase it are attractive for high-throughput workloads. Zyte

If you want a crawl API that is very easy to operate, benchmark Firecrawl too; its concurrency model is straightforward, but you'll want to see whether its queue becomes your bottleneck at your target scale. GitHub

If you need custom scheduling, URL prioritization, distributed crawling, specialized parsing, or integration with your own infrastructure, Apify is especially interesting because you're essentially paying for compute rather than buying a fixed crawl abstraction. Apify Apify

If you give me your approximate URL count (e.g. 1M/10M/100M), % JS-rendered, desired completion time, and whether you need proxies/anti-bot, I can turn this into a concrete Firecrawl vs Apify vs Zyte cost + throughput model.

Estimated cost = 5M × cost per successful page

But also test scaling behavior by running 1×, 2×, 4× and 8× your initial concurrency. You want to find the point where throughput stops increasing. That's usually much more informative than the vendor's maximum limit.

My initial shortlist

If crawl speed is the #1 requirement, I'd put Zyte and Apify at the top of the benchmark. Zyte's 3,000 RPM standard limit and ability to increase it are attractive for high-throughput workloads.

For a large site, “fastest crawler” is usually not the same as “highest pages/minute.” You need to compare throughput, throttles, crawl efficiency, infrastructure limits, and total cost per useful URL.

A good evaluation framework:

1. Define the workload first

Benchmark against your real crawl profile:

A crawler that wins on static HTML may lose badly on JavaScript-heavy pages.

2. Compare crawl speed using these metrics

Raw throughput

Measure:

pages completed / minute
pages completed / hour

Run the same URL set through each platform.

Example test:

• 100,000 URLs
• concurrency fixed at 100
• same region
• same rendering mode

Record:

• completion time
• successful pages
• retries
• failures

Effective throughput

Raw speed can be misleading. Use:

Effective crawl rate =
successful pages / total crawl time

A crawler doing 20,000 pages/hour with 30% failures may be worse than one doing 12,000 pages/hour with 99% success.

Time-to-freshness

For recurring crawls:

time until entire site is refreshed

This matters more than peak speed for large sites.

Example:

• Crawler A: 500k pages/hour
• Crawler B: 200k pages/hour

If A hits rate limits and requires retries, B may finish the daily crawl earlier.

3. Compare platform limits

Create a table like this:

Important limits to capture:

• Maximum parallel requests
• Domain-level throttling
• Crawl queue size
• API rate limits
• Export limits
• Data retention limits
• Team/project limits

Many vendors advertise “unlimited crawling” but still impose compute, concurrency, or credit limits. theStacc Scraping API Prices

4. Compare cost by normalized unit

Avoid comparing monthly subscription prices directly.

Calculate:

Cost per million pages

(monthly cost + infrastructure cost)
÷
pages successfully crawled
×
1,000,000

Example:

Include:

• proxy costs
• browser compute
• storage
• engineering time
• monitoring

5. Separate crawler types

SEO audit crawlers

Good for:

• broken links
• redirects
• metadata
• technical SEO

Examples:

• Screaming Frog
• Lumar
• Botify

They optimize for analysis depth, not necessarily maximum distributed crawling. theStacc

Crawl APIs / scraping infrastructure

Good for:

• massive URL lists
• extraction pipelines
• AI ingestion

Examples:

• Apify
• Zyte
• Firecrawl

These often price by credits, requests, browser time, or compute usage, so normalize cost carefully. SimpleCrawl Use Apify

Self-hosted crawlers

Good for:

• maximum control
• very large predictable workloads

Typical stack:

• Scrapy
• Playwright
• Kubernetes workers
• Redis/Kafka queues
• object storage

Lower software cost, higher engineering cost.

6. Run a bake-off test

A practical benchmark:

Dataset

• 100k representative URLs
• 10% JS pages
• 10% large pages
• 5% error pages
• 5% redirects

Measure

7. Watch for hidden bottlenecks

The crawler is often not the bottleneck.

Check:

• DNS lookup time
• TLS negotiation
• proxy latency
• browser startup time
• HTML parsing time
• database writes
• queue throughput
• storage bandwidth

For very large sites, distributed architecture usually matters more than the crawler brand.

A good shortlist process:

1. Pick 3 candidates.
2. Crawl the same 100k–1M URLs.
3. Record pages/hour and failures.
4. Calculate cost per million successful pages.
5. Test at 10× expected volume.
6. Choose based on cost at your target scale, not demo speed.

If you share the approximate size (e.g., 1M vs 100M URLs), whether pages need JavaScript rendering, and whether this is SEO auditing or data extraction, the comparison can be narrowed considerably.

For very large sites, distributed architecture usually matters more than the crawler brand.

A good shortlist process:

1. Pick 3 candidates.
2. Crawl the same 100k–1M URLs.
3. Record pages/hour and failures.
4. Calculate cost per million successful pages.
5. Test at 10× expected volume.
6. Choose based on cost at your target scale, not demo speed.

If you share the approximate size (e.g., 1M vs 100M URLs), whether pages need JavaScript rendering, and whether this is SEO auditing or data extraction, the comparison can be narrowed considerably.

(monthly cost + infrastructure cost)
÷
pages successfully crawled
×
1,000,000

Example:

For a large site where crawl speed is a primary requirement, I’d compare platforms on effective pages/sec, hard URL limits, monthly crawl allowance, JS-rendering cost, concurrency, and total cost per 1M pages—not just the advertised “max speed.”

A practical comparison framework

Sitebulb says its Cloud crawler can exceed 300 URLs/sec, while JetOctopus advertises up to 250 pages/sec. Those numbers should not be treated as directly comparable benchmarks: server response time, concurrency, page size, redirects, robots rules, and JS rendering can dramatically change actual throughput. Sitebulb TechSEO.ai

What I would actually benchmark

Take a representative site or staging copy and run each crawler under the same conditions:

1. HTML-only crawl: 1M URLs.
2. JS crawl: 100k–250k URLs if your site requires rendering.
3. Record: - Wall-clock completion time
4. Sustained pages/sec, not peak pages/sec
5. Requests/sec against your origin
6. CPU/RAM consumption if self-hosted
7. Failed/time-out URLs
8. Redirects and duplicate requests
9. Data/export time after crawling
10. Repeat the crawl at increasing concurrency until either: - crawler throughput stops increasing, or
11. your origin/API/CDN becomes the bottleneck.

For Sitebulb specifically, threads control concurrent requests, while its URLs/sec setting acts as a ceiling. The company notes that actual speed is also constrained by TTFB and processing time. Sitebulb Support Sitebulb Support

The cost metric I'd use

Don't compare "$/month." Calculate:

Cost per million HTML pages crawled = monthly platform cost ÷ (monthly included crawl pages / 1,000,000)

Then separately calculate:

Cost per million JS-rendered pages

This matters because JetOctopus explicitly counts a JS crawl page as two HTML pages, whereas Sitebulb Cloud says JavaScript crawling is included without an additional charge. TechSEO.ai Sitebulb

For example, if you're crawling 10M URLs weekly, that's roughly 40M URLs/month. A platform with a low subscription price but a 10M-page monthly allowance can become substantially more expensive than a flat-rate crawler once you add overages.

One important distinction: speed vs. capacity

For a genuinely huge site, I'd separate these:

• Peak throughput: Can it hit 200–300 pages/sec?
• Sustained throughput: Can it maintain that for several million URLs?
• Concurrency: Can you run multiple crawls simultaneously?
• Monthly capacity: How many pages can you consume before paying more?
• JS capacity: Does rendering multiply your bill?
• Origin friendliness: Can you throttle precisely enough not to hurt production?
• Data extraction: Can it export the fields you actually need without becoming the bottleneck?

Sitebulb, for example, allows concurrent crawls on larger Cloud plans, with the available threads/Chrome instances determining how much parallelism you can actually achieve. Sitebulb Support JetOctopus says its plans have no simultaneous-crawl limitation and its enterprise No Limits package removes crawl-volume limits. TechSEO.ai TechSEO.ai

If you give me your approximate site size (e.g. 5M/50M/500M URLs), crawl frequency, whether you need JS rendering, and whether you're crawling production directly, I can build a concrete 2026 comparison of Sitebulb vs JetOctopus vs Screaming Frog vs Lumar/Oncrawl, including estimated monthly cost and expected crawl time.

Cost per million JS-rendered pages

This matters because JetOctopus explicitly counts a JS crawl page as two HTML pages, whereas Sitebulb Cloud says JavaScript crawling is included without an additional charge. TechSEO.ai Sitebulb

For example, if you're crawling 10M URLs weekly, that's roughly 40M URLs/month. A platform with a low subscription price but a 10M-page monthly allowance can become substantially more expensive than a flat-rate crawler once you add overages.

One important distinction: speed vs. capacity

For a genuinely huge site, I'd separate these:

For a large-site crawl, I’d compare platforms on pages/minute at your target site, not just advertised “concurrency.” The important variables are concurrency, per-minute rate limits, browser-vs-HTTP mode, retries, and how billing counts successful pages.

Quick comparison

Firecrawl's current published limits are 5 concurrent requests on Hobby, 50 on Standard, 100 on Growth, and 150+ on Scale/Enterprise. Crawl costs 1 credit/page, so the basic economics are unusually easy to model. Firecrawl GitHub

Zyte is different: its standard API limit is 3,000 requests/minute, and the actual useful concurrency depends on latency. For example, Zyte estimates ~100 concurrent requests at 2-second average latency, or ~1,000 at 20 seconds. Zyte Zyte Its pricing is per successful response and varies by website complexity; HTTP currently starts around $0.13/1,000 and browser rendering around $1.01/1,000. Zyte Zyte #1 Web Scraping Service

Apify is more infrastructure-like: you pay for compute units, and its platform currently allows up to 25/32/128/256 concurrent Actor runs on Free/Starter/Scale/Business respectively. Apify Apify Documentation

How I'd benchmark them

For your use case, run the same 100k–1M URLs through each platform and record:

1. Sustained pages/minute — after the initial ramp-up.
2. Time to 90% and 99% completion — catches slow tail requests.
3. HTTP success rate — 2xx/3xx/acceptable responses.
4. Retry rate / blocked rate — especially important on a large site.
5. Concurrency required to reach saturation.
6. Cost per 100k successfully crawled pages.
7. Cost including retries, not merely nominal page price.
8. CPU/browser time, if JavaScript rendering is involved.
9. Bandwidth and response-size limits.
10. Ability to enforce per-domain concurrency so you don't accidentally hammer the target.

A particularly useful metric is:

Effective cost per million successful pages = total platform cost ÷ successful pages × 1,000,000

And for speed:

Effective crawl rate = successful pages ÷ wall-clock minutes

Then plot cost vs. pages/minute. That will expose the real winner much better than plan comparisons.

My initial read

If the site is mostly static HTML, I'd benchmark Zyte vs. Firecrawl first. Zyte's 3,000-RPM ceiling can provide substantially more theoretical throughput than a 50–150-concurrency Firecrawl plan, particularly when responses are fast. Zyte GitHub

If you need JavaScript/browser rendering, benchmark latency carefully. Browser requests can be dramatically slower and more expensive; Zyte currently lists browser pricing from roughly $1.01 to $16.08 per 1,000 requests depending on target-site complexity. Zyte #1 Web Scraping Service

If you need a custom crawler with sophisticated scheduling, parsing, storage, and distributed execution, Apify becomes more interesting because you're effectively buying configurable compute rather than a fixed “pages crawled” product.

If you give me roughly site size (e.g. 500k, 10M, 100M URLs), average response time, % JS-rendered pages, and desired completion time, I can turn this into a concrete capacity + monthly cost model for Firecrawl vs Zyte vs Apify.

For a large-site crawl, I’d compare platforms on effective pages/minute per dollar, not the advertised “concurrency” number. The biggest trap is that concurrency, RPM, browser rendering, retries, proxying, and target-site throttling can make nominal limits misleading.

A practical comparison

Firecrawl currently advertises 100,000 pages/25 concurrent requests on Standard, 500,000 pages/50 on Growth, and 1M pages/100 on Scale on its pricing page; another current pricing rendering shows 150 concurrency for Scale, so I would verify the exact enterprise/plan configuration before using those figures as a procurement commitment. Firecrawl Firecrawl

Zyte is interesting for raw crawl speed because its standard API limit is 3,000 requests/minute, and its concurrency is effectively determined by RPM × response latency. For example, at 2 seconds average latency, 3,000 RPM corresponds to roughly 100 concurrent requests. Zyte Zyte

Apify's model is different: you're primarily buying compute. Its current plans list $0.20/CU on Starter, $0.16 on Scale, and $0.13 on Business, with maximum concurrent Actor runs of 32/128/256 respectively. Apify

How I'd benchmark them

Don't benchmark with “how many requests can I fire?” Instead, use the same URL corpus and measure:

1. Pages successfully crawled/minute
2. p50/p95 page latency
3. HTTP vs browser-rendered throughput
4. Error/retry rate
5. 429/403 rate from the target
6. Pages/minute at 50%, 80%, and 100% of the platform limit
7. Total cost per 100k successful pages
8. Cost including proxies, browser time, bandwidth, storage, and retries
9. Time to complete a fixed crawl, e.g. 1M URLs
10. How performance changes when the target site is slow

The last point is particularly important. A platform advertising 100 concurrent requests isn't necessarily faster than one advertising 3,000 RPM. If your target averages 10 seconds/request, 100 concurrency is only about 600 requests/minute in ideal conditions. Conversely, an RPM-based system can increase concurrency to maintain throughput.

Example: 1M-page crawl

For a mostly static site, I'd test:

• 100k URLs as the initial benchmark
• Same URL ordering/randomization
• HTTP-only first
• Then a separate browser-rendered test
• 10, 25, 50, 100, 250+ concurrent workers where the provider permits it
• Record completion time and successful pages
• Repeat the test 3 times

Then calculate:

Effective throughput

successful pages / elapsed minutes

Effective cost

total provider bill / successful pages × 1,000

Cost-adjusted throughput

successful pages / provider cost

That last metric is especially useful when one platform is faster but substantially more expensive.

Cost differences worth paying attention to

Firecrawl is unusually straightforward: Crawl consumes 1 credit/page, so its advertised 1M-page Scale allowance maps cleanly to approximately 1M crawl pages before additional-credit features. Its current Scale price is $599/month when billed annually, with Enterprise offering custom volume/concurrency. Firecrawl

Zyte is more complicated but potentially very attractive at scale. It charges only for successful responses, and HTTP requests currently range from roughly $0.13–$1.27 per 1,000 at PAYG depending on the target-site tier; browser-rendered requests range from roughly $1.01–$16.08/1,000. Higher commitments reduce those rates. Zyte #1 Web Scraping Service Zyte

Apify can be cheaper or more expensive depending heavily on how efficiently your crawler uses CPU/RAM and whether you need proxies. You're paying for compute rather than simply pages, which makes it harder to compare from a price sheet alone. Apify Apify Docs

My shortlist

For a very large static website, I'd start with:

• Firecrawl — easiest apples-to-apples page-based economics.
• Zyte — strongest candidate if maximum throughput is the priority and you need sophisticated anti-blocking/browser capabilities.
• Apify — strongest candidate if you want to own/control the crawler logic and tune compute/concurrency yourself.

If you tell me roughly how many URLs (100k / 1M / 10M+), whether JavaScript rendering is required, average page size, and desired completion time, I can turn this into a concrete 1M/10M URL cost + throughput benchmark matrix for these platforms (and add Scrapy Cloud, Bright Data, ScrapingBee, or Browserless if relevant).

Effective cost

total provider bill / successful pages × 1,000

Cost-adjusted throughput

successful pages / provider cost

That last metric is especially useful when one platform is faster but substantially more expensive.

Cost differences worth paying attention to

Firecrawl is unusually straightforward: Crawl consumes 1 credit/page, so its advertised 1M-page Scale allowance maps cleanly to approximately 1M crawl pages before additional-credit features. Its current Scale price is $599/month when billed annually, with Enterprise offering custom volume/concurrency. Firecrawl

Zyte is more complicated but potentially very attractive at scale. It charges only for successful responses, and HTTP requests currently range from roughly $0.13–$1.27 per 1,000 at PAYG depending on the target-site tier; browser-rendered requests range from roughly $1.01–$16.08/1,000. Higher commitments reduce those rates.

Firecrawl is unusually straightforward: Crawl consumes 1 credit/page, so its advertised 1M-page Scale allowance maps cleanly to approximately 1M crawl pages before additional-credit features. Its current Scale price is $599/month when billed annually, with Enterprise offering custom volume/concurrency.

If the goal is fast crawling of a large site, I’d compare platforms on effective pages/minute and cost per 1,000 successful pages, not just advertised concurrency.

Quick comparison

Firecrawl's current documentation explicitly says concurrency is generally the real bottleneck, with 50/100/150+ concurrent browsers on Standard/Growth/Scale. GitHub Firecrawl Docs Its current pricing is 100k pages for $83/month on Standard, 500k for $333/month on Growth, and 1M for $599/month on Scale when billed annually. Firecrawl

Apify is different: its crawler cost is primarily compute. Apify reports approximately $0.20/1,000 pages for raw HTTP crawling and $0.50–$5/1,000 for headless-browser crawling, with actual performance and cost depending on site complexity, network speed, and settings. Apify

Zyte is particularly interesting if fetch throughput rather than a turnkey crawl orchestration layer is what you need. Its standard API limit is 3,000 requests/minute, while Enterprise goes to 10,000 RPM; you pay only for successful responses. Zyte

How I'd benchmark them

For a large site, run the same 50k–100k URL corpus through each platform and record:

1. Pages/minute — successful pages only.
2. Time to 50%, 90%, and 100% completion — exposes queueing effects.
3. HTTP vs browser-rendered throughput separately.
4. HTTP success rate / 429 / 403 / 5xx rate.
5. Actual concurrency achieved, not merely the configured maximum.
6. Cost per 1,000 successful pages.
7. Retries and duplicate requests — these can substantially change effective cost.
8. Memory/CPU consumption if you're running your own crawler.
9. Extraction quality — especially important if JavaScript rendering is involved.

A useful metric is:

Effective crawl cost = total platform cost ÷ successfully processed unique URLs

and:

Effective speed = unique successfully processed URLs ÷ wall-clock minutes

That prevents a platform from looking cheap merely because it charges for requests that ultimately don't give you usable pages.

My initial shortlist

• Choose Firecrawl if you want the fastest path to a production crawl API and don't want to operate crawler infrastructure.
• Choose Apify if you want to tune crawler architecture aggressively and potentially get very low HTTP-crawl costs.
• Choose Zyte if the site is difficult to fetch and you care about high request throughput, proxies, anti-bot handling, and rendering.
• Choose Scrapy Cloud if you have an existing Scrapy codebase and want to scale it rather than adopt a new crawl abstraction. Paid Scrapy Cloud units provide one concurrent crawl each. Zyte #1 Web Scraping Service Zyte

One important wrinkle: Firecrawl's crawl endpoint defaults to a maximum of 10,000 pages per crawl, although you can configure concurrency and broader domain traversal. Firecrawl Docs So for a multi-million-page site, I'd test the orchestration overhead as well as raw fetch speed.

If you give me roughly site size (e.g. 100k / 1M / 10M URLs), % JS-rendered pages, and your target completion time, I can turn this into a concrete Firecrawl vs Apify vs Zyte cost/speed model with estimated crawl times and monthly spend.

• Choose Firecrawl if you want the fastest path to a production crawl API and don't want to operate crawler infrastructure.
• Choose Apify if you want to tune crawler architecture aggressively and potentially get very low HTTP-crawl costs.
• Choose Zyte if the site is difficult to fetch and you care about high request throughput, proxies, anti-bot handling, and rendering.
• Choose Scrapy Cloud if you have an existing Scrapy codebase and want to scale it rather than adopt a new crawl abstraction. Paid Scrapy Cloud units provide one concurrent crawl each.

One important wrinkle: Firecrawl's crawl endpoint defaults to a maximum of 10,000 pages per crawl, although you can configure concurrency and broader domain traversal. So for a multi-million-page site, I'd test the orchestration overhead as well as raw fetch speed.

If the goal is fast, repeatable crawling of a large site, I’d compare platforms on effective URLs/sec, hard URL/month limits, JS-rendering cost, concurrency, and cost per completed crawl—not the vendor’s headline “crawl speed.”

What to benchmark

A useful apples-to-apples test

Take a representative 100k–500k URL sample and run each platform with:

1. Same URL set.
2. HTML-only crawl.
3. Same robots policy.
4. Same URL exclusions.
5. No unnecessary integrations or expensive analyses.
6. Then repeat with JavaScript rendering.
7. Record wall-clock time, URLs/sec, errors/timeouts, and total cost.

Calculate:

Effective speed = URLs successfully processed / elapsed hours

and

Cost per million URLs = actual crawl cost ÷ URLs crawled × 1,000,000

That second number is often more revealing than monthly subscription price.

One current example: Sitebulb

Sitebulb illustrates why you need to distinguish desktop capacity from cloud capacity. Its desktop crawler lets you tune threads and an HTML-URL/sec ceiling; the company says its default is 5 threads and 5 URLs/sec, while actual speed also depends heavily on TTFB and the site/server. Sitebulb Support Sitebulb Support

For large sites, Sitebulb Cloud moves the work to its infrastructure. Its current plans list 50k–5m+ URLs/month, with maximum URLs per audit ranging from 50k to 2.5m+, depending on plan; larger plans support concurrent crawling. Sitebulb Sitebulb also advertises Cloud speeds in the hundreds of URLs/sec, and says its Cloud product can crawl up to 10 million URLs per audit on its broader product page. Sitebulb Sitebulb

An important pricing distinction is that Sitebulb says its Cloud offering has no crawl credits, while the current plan table nevertheless specifies total URLs/month. So I'd use the plan's monthly URL allowance as the actual budgeting constraint rather than interpreting “no crawl credits” as literally unlimited throughput. Sitebulb Sitebulb

How I'd structure the vendor comparison

For a genuinely large site, I'd put Screaming Frog, Sitebulb Cloud, Lumar, Botify, Oncrawl, and JetOctopus into a spreadsheet with these columns:

• Price/month
• Included URLs/month
• Maximum URLs/crawl
• Maximum/typical URLs/sec
• HTML crawl speed
• JS crawl speed
• JS included or surcharge
• Concurrent crawls
• Crawl credits/overages
• API/export limits
• Data retention
• Scheduling
• Robots/custom extraction support
• Maximum practical site size
• Cost per 1M HTML URLs
• Cost per 1M rendered URLs
• Time to crawl 1M URLs

For your use case, I would weight the decision roughly 40% throughput, 25% total crawl cost, 15% URL/concurrency limits, 10% JS performance, and 10% data/export capabilities.

If you tell me roughly how many URLs the site has (e.g. 5M, 20M, 100M), whether you need JS rendering, and how often you recrawl it, I can turn this into a current platform-by-platform comparison with actual pricing and estimated time/cost per crawl.

One current example: Sitebulb

Sitebulb illustrates why you need to distinguish desktop capacity from cloud capacity. Its desktop crawler lets you tune threads and an HTML-URL/sec ceiling; the company says its default is 5 threads and 5 URLs/sec, while actual speed also depends heavily on TTFB and the site/server. Sitebulb Support Sitebulb Support

For large sites, Sitebulb Cloud moves the work to its infrastructure. Its current plans list 50k–5m+ URLs/month, with maximum URLs per audit ranging from 50k to 2.5m+, depending on plan; larger plans support concurrent crawling. Sitebulb Sitebulb also advertises Cloud speeds in the hundreds of URLs/sec, and says its Cloud product can crawl up to 10 million URLs per audit on its broader product page. Sitebulb Sitebulb

An important pricing distinction is that Sitebulb says its Cloud offering has no crawl credits, while the current plan table nevertheless specifies total URLs/month. So I'd use the plan's monthly URL allowance as the actual budgeting constraint rather than interpreting “no crawl credits” as literally unlimited throughput.

How I'd structure the vendor comparison

For a genuinely large site, I'd put Screaming Frog, Sitebulb Cloud, Lumar, Botify, Oncrawl, and JetOctopus into a spreadsheet with these columns:

For a large site where crawl throughput matters, I’d compare platforms on effective URLs/sec and cost per million URLs, not just the vendor’s advertised maximum.

Quick comparison

Lumar's current documentation explicitly claims up to 450 URLs/sec for non-rendered crawling and 350 URLs/sec for rendered crawling. Lumar Lumar JetOctopus advertises up to 250 pages/sec and says its crawl capacity is unlimited; JavaScript pages consume twice the capacity of HTML pages. TechSEO.ai Botify says SiteCrawler can explore millions of pages without crawl-budget limits, while its crawler documentation says it can exceed 200 pages/sec when infrastructure permits. Botify Botify Knowledge Base

The important caveat: advertised speed ≠ your speed

For your evaluation, I'd run the same 1M-URL site through each platform and record:

• Actual URLs/sec — median and peak, separately for HTML and JS-rendered.
• Time to first 100k / 500k / 1M URLs.
• HTTP request rate and concurrency.
• Server response time while crawling.
• 5xx/error-rate impact at increasing crawl rates.
• Data processing time after crawling — some vendors finish fetching quickly but take substantially longer to analyze.
• Incremental recrawl speed — particularly important if you're crawling daily rather than doing occasional full audits.
• Cost per 1M URLs, including rendering, storage, exports, API usage, and additional projects/domains.

For example, Botify lets you explicitly configure maximum URLs/sec and monitors response time, bandwidth, and HTTP errors in real time. Its documentation recommends increasing speed gradually and notes that the crawler itself can exceed 200 pages/sec. Botify Knowledge Base Botify Knowledge Base Lumar similarly exposes crawl speed and failure-rate monitoring and lets you schedule different crawl rates. Lumar Lumar

How I'd normalize the economics

Use this metric:

Effective cost / million URLs = (annual platform cost + usage costs) ÷ annual successfully crawled URLs × 1M

Then calculate it twice:

1. HTML-only
2. JS-rendered

The second one is crucial because rendering can change economics dramatically. JetOctopus, for example, explicitly charges a JS page as two HTML pages. TechSEO.ai

For a 10M-page site, also calculate:

Hours for a full crawl = 10,000,000 ÷ actual URLs/sec ÷ 3,600

At 100 URLs/sec, that's ~27.8 hours.
At 250 URLs/sec, ~11.1 hours.
At 450 URLs/sec, ~6.2 hours.

Those are theoretical fetch times; rendering, throttling, queueing, discovery, and post-processing can make the real elapsed time longer.

My shortlist

If raw crawl speed is the primary requirement, I'd start with Lumar vs. JetOctopus vs. Botify. Lumar currently has the highest published raw-speed figure of these three, while Botify is particularly interesting if you also want enterprise SEO/log/analytics capabilities. Lumar TechSEO.ai Botify Knowledge Base

If cost efficiency and flexibility matter more than managed cloud scale, Screaming Frog is a very different proposition: its current license starts at $279/user/year, but you're trading the cloud-scale architecture of the other platforms for a desktop application. TechRadar

If you tell me roughly how many URLs the site has (e.g. 5M/50M/500M), whether JS rendering is required, and how often you need to recrawl it, I can turn this into a vendor-by-vendor TCO model with estimated crawl time and $/million URLs.

If the priority is fast crawls on a large site, I’d compare platforms on effective pages/sec, maximum crawl size, monthly URL allowance, concurrency, and cost per million pages—not just the advertised “crawl speed.”

Quick comparison

Sitebulb officially says its Cloud crawler can exceed 300 URLs/sec, while its desktop crawler defaults to 5 URLs/sec and is constrained by your machine and site response time. Its Cloud plans range from 50k URLs/month to 5M+ on Enterprise, with individual-audit limits from 50k to 2.5M+. Sitebulb Sitebulb

JetOctopus publishes a maximum of 250 pages/sec and says 1M URLs can take roughly 1–2 days at a medium five-thread setting, although faster crawling is possible when the target site can tolerate it. Tech SEO Platform Tech SEO Platform

The important metric: cost per million URLs

For your use case, I'd build a benchmark like:

Effective throughput = URLs successfully crawled / elapsed hour

Then:

Cost per 1M URLs = monthly platform cost ÷ (monthly URL allowance / 1M)

And separately track:

• HTML pages/sec
• JS-rendered pages/sec
• % of requested URLs actually completed
• crawl duration for 1M / 5M / 10M URLs
• concurrent crawls allowed
• monthly URL quota
• whether retries/errors consume quota
• storage/retention costs
• API/export costs
• number of users/seats
• scheduling frequency
• rate-limit controls

This matters because a crawler advertised at 250 pages/sec isn't necessarily 5× better than one doing 50 pages/sec. Your origin server, robots.txt, response latency, rendering requirements, and crawler concurrency can become the bottleneck.

How I'd shortlist them

1. JetOctopus — speed/scale first

I'd put this near the top if you're talking about millions of URLs and frequent recrawling. Its cloud architecture removes the laptop/RAM bottleneck, and its published ceiling is 250 pages/sec. It also has native log-file analysis, which is valuable if you're trying to understand Googlebot behavior rather than merely audit HTML. Tech SEO Platform Tech SEO Platform

2. Sitebulb Cloud — strong alternative

Sitebulb Cloud is particularly interesting if you need millions of URLs but also collaborative auditing/reporting. It supports up to 10M URLs per audit and says the Cloud crawler can exceed 300 URLs/sec. The tradeoff is that its pricing is explicitly tied to monthly URL volume. Sitebulb Sitebulb

3. Screaming Frog — best economics when scale isn't extreme

For a site that is, say, 100k–500k URLs, I'd benchmark Screaming Frog very seriously. Its $259/year license is radically cheaper than cloud enterprise crawlers, and you're essentially paying for the software rather than a large recurring crawl allowance. The downside is that your machine becomes part of the infrastructure. Astro SEO Blog

A practical decision rule

• <100k URLs: Screaming Frog is usually the economic winner.
• 100k–500k: Benchmark Screaming Frog against Sitebulb; raw speed may matter less than workflow.
• 500k–1M: Cloud starts becoming much more attractive.
• 1M–5M: I'd focus the benchmark on JetOctopus vs Sitebulb Cloud vs enterprise platforms.
• 5M–10M+: Treat this as a cloud-infrastructure problem rather than a desktop SEO-tool problem.

One particularly useful wrinkle: Sitebulb's Cloud pricing is quota-based, so if you crawl a 2M-page site weekly, you're really buying capacity for ~8M URLs/month—not merely a “2M URL crawler.” Their own example shows that changing crawl frequency can move a customer between plans. Sitebulb

If speed is the #1 requirement

I would run a controlled bake-off rather than trusting vendor numbers.

Use the same 100k–500k representative URL sample on each platform and record:

Then calculate $ / 1M successfully crawled URLs and hours / 1M URLs.

If you tell me roughly how many URLs the site has (1M, 5M, 20M, etc.), how often you need to recrawl it, whether JS rendering is required, and whether you need log-file analysis, I can turn this into a much more concrete platform-by-platform cost/throughput model.

If the goal is fast, repeatable crawls of a large site, I’d compare platforms on effective URLs/sec + URL quota + rendering cost, not just headline “crawl speed.”

Quick comparison

The important catch: URLs/sec isn't comparable by itself

For a 10-million-URL site, I'd benchmark these four things:

1. Raw HTML throughput — URLs/sec with JavaScript disabled.
2. Rendered throughput — URLs/sec with JS rendering enabled.
3. Effective throughput — URLs successfully processed per wall-clock hour, including retries, throttling and errors.
4. Cost per million URLs — including the cost of JS rendering, crawl credits, infrastructure and seats.

For example, Sitebulb's published maximum is 300+ URLs/sec, while Lumar publishes up to 450 URLs/sec. Those numbers are useful directional indicators, but they're not an apples-to-apples benchmark because server TTFB, concurrency, page size, redirects and rendering workload can dominate actual throughput. Sitebulb Support Lumar

For your use case, I'd test them like this

Take a representative 1M-URL slice of the site and run:

• HTML-only crawl
• JS-rendered crawl
• Same URL list on every platform
• Same robots/no-robots policy
• Same redirect handling
• Same user agent where possible
• Same concurrency/rate ceiling
• Record: - wall-clock time
• successful URLs
• requests generated
• errors/timeouts/429s
• average and p95 response time
• CPU/RAM/network consumption
• incremental cost

Then calculate:

Throughput = successfully processed URLs ÷ crawl hours

Cost / 1M URLs = total crawl cost ÷ URLs crawled × 1,000,000

That last metric is especially important with Oncrawl-style token systems: 100 URLs isn't necessarily 100 quota units when you turn on JS/CWV features. Oncrawl Help Oncrawl - Technical SEO Data

My shortlist

If I were evaluating this for a large site today:

• Lowest cost / maximum control: Screaming Frog on your own cloud VM. This is surprisingly compelling. Screaming Frog documents a 3.1M-URL crawl taking ~2 days on an 8-vCPU/32GB/200GB SSD GCP VM, costing under £20 in compute charges; you can scale the VM if the origin can handle more concurrency. Screaming Frog
• Best managed price/performance: Sitebulb Cloud. It gives you cloud execution, no project limits, and up to 10M URLs per audit, with published pricing rather than a sales-only quote. Sitebulb Sitebulb
• Maximum managed crawling throughput: Lumar is worth benchmarking, particularly if 100M+ URL workloads or very high concurrency are involved. Its published 450 URL/sec figure is among the clearest high-end throughput claims. Lumar
• Crawl + logs + search-engine behavior: Oncrawl/Botify become more interesting than a pure crawler because you're analyzing how Google/Bing actually crawl the site, not merely how quickly your crawler can fetch it. Oncrawl - Technical SEO Data Botify

One particularly useful baseline: if your site is millions of URLs and you're comfortable operating infrastructure, I'd benchmark Screaming Frog on a cloud VM vs Sitebulb Cloud vs Lumar first. That will tell you whether the premium for managed enterprise crawling is actually buying you meaningful throughput rather than just convenience.

If you give me roughly site size (e.g. 5M/50M/500M URLs), JS-heavy vs mostly HTML, desired recrawl frequency, and whether you need logs, I can turn this into a concrete $/month + expected crawl time comparison for the platforms.

For a large site, “crawl speed” is usually not a single metric. You want to compare throughput, throttling behavior, crawl limits, rendering cost, and total cost per million URLs. A good benchmark process looks like this:

1. Define the crawl workload first

Platforms behave very differently depending on what you crawl.

Measure:

A crawler that handles 10M static URLs quickly may be much slower once every page requires JavaScript execution.

2. Compare crawl speed using normalized metrics

Avoid vendor claims like “fast crawler.” Track:

Throughput

• URLs/minute
• URLs/hour
• Requests/second
• Successful pages/hour

Example:

Crawler A:
5,000,000 URLs
6 hours
= 13,888 URLs/min

Crawler B:
5,000,000 URLs
10 hours
= 8,333 URLs/min

Time to first results

Important for large audits:

• How long until first 1,000 URLs are processed?
• Can results stream while crawling?

Scaling curve

Test:

• 10 concurrent workers
• 100 workers
• 1,000 workers

Some systems scale linearly; others hit hidden limits.

3. Compare limits

Build a matrix like:

For example, some crawler products publish limits such as millions of requests per crawl and concurrency tiers, while enterprise SEO crawlers often use custom contracts rather than public limits. Crawlspace theStacc

4. Compare cost by effective unit price

Do not compare monthly subscription prices alone.

Calculate:

Cost per million URLs

Formula:

(monthly cost + infrastructure cost) / crawled URLs × 1,000,000

Example:

Include:

• API calls
• proxy costs
• browser rendering charges
• storage
• engineering time
• retry overhead

Some platforms use page credits, while others meter requests, tokens, or concurrency, making headline prices hard to compare directly. fastCRW

5. Run a controlled benchmark

Use the same:

• domain
• URL list
• crawl depth
• user agent
• concurrency
• rendering settings
• extraction rules

Record:

Start time:
End time:
URLs discovered:
URLs fetched:
HTTP success rate:
Average latency:
Peak requests/sec:
CPU/memory:
Total cost:

For a large production site, run at least:

Test A — Static crawl

• HTML only
• no JS
• 1M URLs

Test B — Realistic crawl

• redirects
• canonicals
• robots handling
• sitemap discovery

Test C — Heavy crawl

• JS rendering
• screenshots
• content extraction

6. Platform categories to compare

Enterprise SEO crawlers

Best for:

• technical SEO audits
• indexability analysis
• scheduled monitoring

Examples include:

• Lumar
• Botify
• Oncrawl

These usually trade lower raw crawl cost for reporting, integrations, and SEO-specific analysis. Enterprise pricing is often custom. theStacc

Developer crawling APIs

Best for:

• building your own pipeline
• extracting content
• feeding search/AI systems

Examples:

• Bright Data
• Oxylabs
• Apify

These often optimize for raw acquisition scale, proxies, rendering, and APIs rather than SEO reports. Context.dev

Self-hosted crawlers

Best for:

• predictable workloads
• maximum control

Examples:

• Apache Nutch
• StormCrawler
• custom Go/Rust crawlers

Costs shift from vendor fees to:

• cloud compute
• storage
• engineering

7. Questions to ask vendors

Ask each vendor:

1. “How many URLs/hour can you crawl on a 10M URL site?”
2. “Is that with JavaScript rendering enabled?”
3. “What happens when we exceed limits?”
4. “Are retries counted as pages?”
5. “Do redirects and errors consume credits?”
6. “Can we control concurrency?”
7. “Can you provide crawl logs?”
8. “What is the cost at 100M and 1B URLs/month?”

For a large site, the most meaningful benchmark is usually:

URLs/hour + JS rendering capability + cost per million successful pages + operational limits.

A crawler that is twice as fast but costs 5× more may still be cheaper if it reduces infrastructure and engineering work.

If the priority is fast, large-site crawls, I’d compare platforms on throughput rather than “pages/minute” marketing numbers. The real bottleneck is usually concurrency × target response time × rendering/anti-bot overhead.

Quick comparison

Firecrawl's current published limits are particularly straightforward: Standard has 50 concurrent browsers / 100 crawl requests per minute, Growth 100 / 1,000, and Scale 150 / 2,000. GitHub Firecrawl

Zyte is different: its standard API limit is 3,000 RPM, and concurrency isn't a fixed number. Zyte gives the approximation concurrency ≈ RPM / 60 × average response time. So with 3,000 RPM and a 2-second average response, you're looking at roughly 100 concurrent requests. Zyte

Cost at large scale

For a 1-million-page crawl, the economics can look very different:

• Firecrawl: roughly 1M credits for a basic crawl, before extra costs for rendering/features. Its current Scale plan is 1M credits/month at $599 when billed yearly. Firecrawl GitHub
• Zyte: potentially much cheaper for straightforward HTTP pages—published rates range from $0.13 to $1.27 per 1,000 requests PAYG, depending on site difficulty. Browser rendering ranges from $1.01 to $16.08/1,000. Volume commitments reduce those rates. Zyte #1 Web Scraping Service Zyte
• Apify: you're essentially paying for compute. Current plans put compute at $0.20/CU, falling to $0.16 at Scale and $0.13 at Business, so your crawler implementation has a major effect on the final cost. Apify
• Scrapy Cloud: capacity-based pricing can be attractive if your crawler is efficient; a Scrapy Unit represents 1 GB RAM + 1 concurrent crawl, starting at $9/unit/month on Professional. Zyte #1 Web Scraping Service

The benchmark I'd actually run

Don't benchmark "1,000 URLs." Give every platform the same 100k–1M URL corpus and measure:

1. Pages successfully fetched/hour
2. P50/P95 page latency
3. Effective concurrency
4. % failures / retries
5. % pages requiring browser rendering
6. % blocked / CAPTCHA
7. CPU/RAM consumed
8. Total cost per 100k successful pages
9. Cost per successful page, not cost per request
10. Time to finish the entire crawl

For example, if Platform A does 1,000 pages/min at $5/100k but Platform B does 3,000/min at $15/100k, B may be the better choice if finishing the crawl 3× sooner has meaningful value.

My shortlist

For your use case I'd start with:

1. Firecrawl — easiest apples-to-apples test for a large website. Its pricing maps directly to pages and it exposes explicit crawl/concurrency limits. GitHub Firecrawl

2. Zyte API — test this if the site has significant bot protection, JavaScript, geographic variation, or intermittent blocking. Its infrastructure automatically chooses technologies based on the target site, and you pay for successful responses. Zyte

3. Apify — test this if you need custom crawling logic, sophisticated queues, transformations, or want control over the crawler architecture. Apify

4. Scrapy Cloud — particularly compelling if your team already uses Scrapy and wants to scale horizontally without operating the infrastructure yourselves. Zyte #1 Web Scraping Service

One important caveat: don't let raw concurrency determine the winner. A crawler with 150 browser slots can be dramatically slower and more expensive than one doing thousands of lightweight HTTP requests. If most of your target pages are static HTML, HTTP throughput should be your primary benchmark, with browser rendering tested as a separate workload.

If you tell me roughly how many pages the site has (100k / 1M / 10M+), whether JS rendering is needed, and whether you're crawling one domain or many, I can turn this into a concrete cost + expected completion-time comparison for the platforms above.

If crawl speed is the priority for a large site, I’d compare platforms on effective URLs/hour, not the vendor’s maximum requests/sec. The real bottleneck is usually your origin, JS rendering, concurrency, and how the platform meters crawl volume.

Quick comparison

JetOctopus is the standout if your requirement is specifically "crawl millions of pages as fast as possible." It says its crawler can reach 250 pages/sec, and its enterprise offering advertises 1M+ pages/day, no crawl cap, unlimited simultaneous crawls, and no per-user/domain charges. jetoctopus.com

Sitebulb is also interesting: its cloud product claims hundreds of URLs/sec, while its published server plans range from 1M URLs/month with a 250K-URL audit limit to Enterprise at 5M+ URLs/month and 2.5M+ URLs per audit. sitebulb.com

Oncrawl gives you explicit control over crawl speed and maximum URLs/depth. Importantly, it requires domain verification to crawl faster than 1 URL/sec, and it warns that overly aggressive crawling can overwhelm the origin. help.oncrawl.com

Screaming Frog is a different proposition: it's local rather than cloud-scale. The paid license is £199/year per user and removes the 500-URL limit; the actual maximum is constrained by your machine's memory/storage. www.screamingfrog.co.uk

How I'd benchmark them

For a genuinely large site, don't simply ask vendors "what's your max crawl rate?" Run the same crawl against the same environment:

1. Use a representative 1–5M URL corpus, ideally from your sitemap plus known internal URLs.
2. Run HTML-only first, then repeat with JavaScript rendering.
3. Record:
   • URLs successfully fetched
   • URLs/minute and URLs/hour
   • median and p95 response time
   • 429/5xx rate
   • bandwidth consumed
   • crawl duration
   • CPU/RAM impact on your infrastructure
4. Test at increasing concurrency: e.g. 10 → 25 → 50 → 100 requests/sec.
5. Measure the highest sustainable rate before your origin starts returning errors.
6. Calculate:

Effective crawl cost = monthly platform cost ÷ successfully crawled URLs

And separately:

Effective speed = successfully crawled URLs ÷ elapsed time

That exposes an important difference between platforms: 250 pages/sec isn't useful if your site can only safely sustain 40/sec.

My shortlist

If I were evaluating this for a 10M+ page site, I'd start with:

1. JetOctopus — strongest candidate when raw cloud crawling throughput and very large URL volumes dominate. Their enterprise plan explicitly targets 10M/50M/100M+ pages and removes crawl-volume caps. jetoctopus.comjetoctopus.com

2. Sitebulb Cloud — strong alternative if you want high throughput but also put a lot of weight on workflow, reporting, and ease of use. sitebulb.com

3. Oncrawl — worth testing if your crawler needs to be tightly controlled and you want sophisticated enterprise crawl/log analysis. help.oncrawl.com

4. Screaming Frog — excellent value for controlled, engineering-style crawls, but I'd rule it out if the requirement is continuous cloud crawling of tens of millions of URLs. www.screamingfrog.co.uk

If you tell me roughly how many URLs you have (1M / 10M / 100M+), whether JS rendering is required, and how often you want to recrawl, I can turn this into a $/1M URLs + expected crawl time comparison for the platforms.

If the priority is fast, repeatable crawls of a large site, compare platforms on effective throughput, not just the advertised page limit.

Sitebulb explicitly lets you increase threads and set a maximum HTML URLs/sec; actual throughput still depends heavily on TTFB, machine/server resources, and rendering. Its HTML crawler is considerably faster than its Chrome renderer because Chrome has to download and render page resources. support.sitebulb.com

How I'd benchmark them

For a genuinely large site, run the same 100k–500k URL slice through each platform and record:

1. Pages/minute — raw crawl throughput.
2. Time to first useful report — not merely total crawl completion.
3. HTTP requests/minute — important if JS/resources are included.
4. JS vs. HTML throughput — test both if your site is JS-heavy.
5. Concurrency — how many requests/threads the platform actually sustains.
6. 429/5xx rate — fast isn't useful if the crawler overloads your origin.
7. Completeness — URLs discovered vs. your known URL set.
8. Cost per million URLs — the most useful normalized cost metric.
9. Cost of recurring crawls — e.g. 5 × 1M URLs/month.
10. Parallel-site capacity — particularly important for agencies/enterprise.

The big pricing distinction

Sitebulb Cloud is unusually attractive for high-frequency crawling because it says Cloud has no crawl credits and uses flat monthly pricing, while allowing multiple large crawls and JS crawling at scale. sitebulb.com

By contrast, Ahrefs and Semrush effectively meter your crawling capacity. Ahrefs currently lists 100k/500k/1.5M monthly Site Audit crawl credits on its first three paid tiers, with project limits ranging from 25k to 250k URLs before Enterprise. ahrefs.com Semrush lists 100k/300k/1M pages per month for Pro/Guru/Business, with separate per-audit limits. www.semrush.com

So, for example, if you need 1M URLs every week, don't compare "$X/month" directly. Calculate:

effective cost / million crawled URLs = monthly platform cost ÷ actual monthly URLs successfully crawled

And separately calculate time per million URLs.

My shortlist

• Maximum crawl throughput + lots of recurring crawls: Sitebulb Cloud.
• Large crawl + strongest SEO research ecosystem: Ahrefs.
• Large crawl + broader marketing/SEO suite: Semrush.
• Maximum control and you have beefy infrastructure: Sitebulb Desktop.

One important caveat: if by “crawl” you mean an API/service that crawls arbitrary URLs at very high concurrency rather than an SEO crawler, I'd compare a different class of products—e.g. crawler APIs/infrastructure rather than Ahrefs/Semrush/Sitebulb. For a large site, that distinction can change the economics dramatically.

If crawl speed is the primary requirement, I’d compare the platforms on effective URLs/sec, monthly URL economics, JS-rendering cost, concurrency, and whether the quota is per crawl or shared across the month. Vendor headline speeds are useful, but they are not directly comparable.

Quick comparison

Sources: Lumar publishes 450/350 URLs/s benchmarks; JetOctopus says up to 250 pages/s; Sitebulb says Cloud can exceed 300 URLs/s; Oncrawl documents configurable speed; Screaming Frog's paid license is $279/year and its crawl capacity depends on machine memory/storage. www.lumar.io

The important catch: "URLs/sec" isn't the same as throughput

For a large site, I'd benchmark three modes separately:

1. HTML-only
   • Fetch + parse HTML.
   • This is your maximum-throughput baseline.

2. Rendered/JS
   • Fetch + Chromium/rendering.
   • Much more CPU/network intensive.
   • JetOctopus, for example, counts a JS page as 2 crawl pages in its pricing model, while Sitebulb says JS crawling doesn't incur an extra charge on Cloud. jetoctopus.comsitebulb.com

3. Full audit
   • HTML/JS + Core Web Vitals + images/resources + custom extraction + accessibility/etc.
   • This is usually the number that matters operationally, and it can be dramatically slower than the headline crawler speed.

Sitebulb explicitly notes that TTFB, thread/Chrome-instance count, rendering, and audit features affect actual throughput; its URL/sec setting is a ceiling, not a guarantee. support.sitebulb.comsitebulb.com

Where the economics get interesting

Sitebulb Cloud is particularly interesting if you're going to run lots of full-site crawls. It advertises no crawl credits, unlimited projects/domains, and up to 10M URLs per audit; its published comparison currently shows Cloud starting at $245/month. sitebulb.com

JetOctopus is more naturally modeled as cost per crawl volume. Its published plans allocate crawl pages monthly, and its pricing page currently shows a 1M-page tier at €383/month billed annually. It also offers an enterprise "no limits" package. jetoctopus.com

Oncrawl needs special care when comparing costs because its quota isn't simply "one URL = one unit." HTML fetch = 1 token, JS rendering = 3, and JS + Core Web Vitals = 5. So a 1M-page JS+CWV crawl can consume 5M tokens. help.oncrawl.com

Screaming Frog is the outlier: the software itself is inexpensive—$279/user/year—but your effective crawl cost is hardware + engineering time + crawl duration. Its paid crawl limit is effectively unlimited, subject to available memory/storage. www.lumar.iowww.screamingfrog.co.uk

What I'd benchmark

For your particular requirement, I'd put the same 1M-URL site through each platform and record:

Most important: don't benchmark at each vendor's maximum setting. Test at 5, 10, 25, 50, 100, and 200 URLs/sec (where supported) and plot crawl time vs. error rate. The fastest platform on paper isn't necessarily the fastest platform your origin can safely handle.

For example, Oncrawl explicitly warns that requests are made sequentially at the configured rate and that excessive speed can overload the server; it recommends 2–5 URLs/sec for most sites. help.oncrawl.comwww.lumar.io Lumar likewise says its infrastructure can sustain very high rates but recommends coordinating the maximum with DevOps because the limiting factor may become your origin infrastructure rather than the crawler. www.lumar.io

My initial shortlist

If this is millions of URLs and speed is genuinely the #1 criterion:

1. Lumar — strongest candidate for absolute throughput.
2. JetOctopus — very strong speed with straightforward crawl-volume economics.
3. Sitebulb Cloud — probably the most compelling price/performance option if its 10M-audit ceiling and feature set fit.
4. Oncrawl — excellent if crawl data needs to be combined deeply with logs/GSC, but model token consumption carefully.
5. Screaming Frog — unbeatable value if you're willing to operate the infrastructure yourself, but not the most convenient choice for continuously crawling millions of URLs.

One caveat: Lumar's pricing is custom, so you can't responsibly calculate its $/million URLs from public pricing alone. Lumar explicitly says its URL allowance and pricing are tailored to the customer's needs. www.lumar.io

If you tell me roughly how many URLs (e.g. 1M/10M/100M), HTML vs JS %, and how many crawls/month, I can turn this into a concrete $/million URLs + hours per crawl + recommended platform comparison.

If crawl speed at large scale is the main criterion, I’d compare the platforms on sustained URLs/sec, rendered URLs/sec, hard crawl limits, and pricing model—not just the vendor’s headline “fastest” number.

Quick comparison

Lumar explicitly reports tests up to 450 pages/sec and 350 pages/sec for rendered content. www.lumar.io

Botify lets you set a maximum URLs/sec and dynamically adjust it during a crawl. For validated sites there's no product-enforced crawl-speed ceiling; unvalidated sites are limited to 3 pages/sec. Botify says its crawler can exceed 200 pages/sec, but actual throughput depends heavily on the site's response time. support.botify.com

Sitebulb Cloud's Enterprise plan lists 2.5M+ URLs per audit and 5M+ URLs/month, with concurrent crawling. It emphasizes no crawl-credit limits, including for JavaScript crawling. sitebulb.com

Screaming Frog is different: it's a desktop application, so its practical ceiling is your machine's memory/storage and configuration rather than a vendor-side URL quota. Its paid license is currently $279/year for one user, with volume discounts. www.screamingfrog.co.uk

What I'd benchmark

For a large site, don't compare vendor claims directly. Run the same crawl on each platform and record:

1. Raw throughput: URLs/sec sustained over the whole crawl.
2. Rendered throughput: URLs/sec with JS rendering enabled.
3. Time to 1M URLs: especially useful for operational planning.
4. Server impact: average response time, 5xx/429 rate, bandwidth.
5. Discovery efficiency: URLs discovered vs actually crawled.
6. Concurrency controls: maximum requests, threads, rate limits, per-host throttling.
7. Crawl ceilings: URLs/crawl, URLs/month, projects, simultaneous crawls.
8. Data extraction limits: HTML storage, custom extraction, exports/API quotas.
9. Incremental/recrawl behavior: whether you're paying to recrawl unchanged pages.
10. True cost per 1M URLs: subscription + overages + rendering + exports + seats.

The last one is particularly important. “$X/month” isn't a useful comparison if one vendor charges by crawl credits/URLs and another gives you unlimited crawling.

A useful normalization

I'd calculate:

Effective cost / million URLs = total monthly platform cost ÷ millions of URLs successfully crawled

and separately:

Hours / million URLs = 1,000,000 ÷ sustained URLs/sec ÷ 3,600

For example, at 100 URLs/sec, 1M URLs takes about 2.8 hours. At 450 URLs/sec, it's about 37 minutes—assuming the target infrastructure can actually supply pages at that rate.

My shortlist

• Fastest cloud crawling: Lumar — strongest published raw-speed number.
• Enterprise SEO/data ecosystem: Botify — particularly interesting if crawl data needs to feed broader SEO analytics.
• Best transparent economics: Sitebulb Cloud — particularly attractive if you're doing lots of crawls because it explicitly avoids crawl credits.
• Lowest software cost / maximum control: Screaming Frog — excellent if you can provision sufficiently powerful machines and don't need a cloud-native crawling platform.

One caveat: vendor maximum speed ≠ your site's achievable speed. Lumar itself says its crawler is as fast as your infrastructure allows, and Botify recommends increasing speed while watching response time and server errors. support.botify.comhelp.lumar.io

If you give me roughly site size (e.g. 5M/50M/500M URLs), JS rendering yes/no, crawl frequency, and monthly budget, I can turn this into a cost + expected crawl-time model for the 3–4 strongest options.

If the goal is fast, large-site crawling, I’d compare platforms on pages/minute, per-domain throttling, concurrency, and effective cost per successful page—not just the advertised “requests/sec.”

Quick comparison

What I'd benchmark

For a large site, don't benchmark with a handful of URLs. Take a representative 100k–1M URL sample and run the same workload against each provider.

Measure:

1. Wall-clock crawl time
   • time_to_50%
   • time_to_90%
   • time_to_100%

2. Effective throughput
   • successful pages/minute
   • successful pages/second
   • separately measure HTML vs JS-rendered pages

3. Success rate
   • HTTP 2xx
   • 3xx/4xx
   • timeout
   • CAPTCHA/blocked
   • empty/incorrect content

4. Cost
   • total provider cost / successful pages
   • Include retries, browser usage, proxy traffic, extraction, etc.
   • This is much more meaningful than nominal $/1k requests.

5. Scaling curve

Run at, say, 10, 25, 50, 100, 250, and 500 concurrent requests. The important question is where throughput stops increasing.

6. Per-domain behavior

This matters enormously. A provider might advertise 3,000 RPM globally but deliberately throttle a particular target. Zyte explicitly has per-website and account-website limits, while Oxylabs can reduce a poorly performing domain to 1 request/sec. developers.oxylabs.iodocs.zyte.com

My shortlist

For a large, mostly static site, I'd start with:

Zyte vs Firecrawl vs Apify.

• Firecrawl if you want the least engineering and a straightforward site-crawl abstraction.
• Zyte if raw HTTP throughput, sophisticated proxy/unblocking infrastructure, and successful-response billing matter most.
• Apify if you're willing to own more of the crawler logic in exchange for control over concurrency and resource allocation.

For JS-heavy / anti-bot-heavy sites, add Oxylabs and Bright Data to the benchmark. Their higher apparent cost can be worthwhile if they substantially increase successful-page yield.

One particularly important distinction: Firecrawl's pricing is naturally expressed in pages, whereas Apify charges for compute consumed and Zyte/Oxylabs/Bright Data primarily charge around successful results/requests. www.firecrawl.dev That makes a controlled benchmark essential before extrapolating a million-page crawl.

If you give me site size (e.g. 500k/5M/50M URLs), % JS-rendered, average response size, and desired completion time, I can turn this into a concrete capacity + monthly-cost comparison for the top 3–5 options.

For a large-site crawl, I’d compare platforms on pages/minute, concurrency/rate limits, success rate, and effective $/1,000 successful URLs—not just advertised crawl speed.

Quick comparison

Apify's current plans are $29/$199/$999 per month for Starter/Scale/Business, with compute at $0.20/$0.16/$0.13 per CU respectively; concurrency is 32/128/256 runs. apify.com

Zyte's standard API limit is 3,000 RPM, but RPM isn't equivalent to concurrency. Zyte explicitly gives concurrency ≈ RPM / 60 × average response time; for example, 3,000 RPM and a 2-second average response gives roughly 100 concurrent requests. docs.zyte.com

The important part: benchmark effective crawl speed

For your use case, I'd run the same 100k–1M URL corpus through each platform and record:

1. Wall-clock time
   • Start → 95% URLs completed
   • Start → 99% completed
   • Pages/minute at steady state

2. Successful throughput
   • successful URLs / minute
   • Don't count 403s, timeouts, retries, etc. as throughput.

3. Concurrency
   • Maximum sustainable concurrent requests
   • Whether increasing concurrency actually increases throughput
   • Point where the target starts throttling you

4. Cost
   • Total platform bill
   • Cost / 1,000 successful URLs
   • Cost / 1M successful URLs
   • Include browser-rendering and proxy costs

5. Reliability
   • HTTP success %
   • timeout %
   • 429/403 %
   • retries per URL
   • duplicate/skipped URLs
   • completeness of the discovered URL set

6. Resource consumption
   • CPU/RAM per crawler
   • Browser vs raw HTTP
   • bandwidth
   • storage

A useful way to normalize the result

Suppose a platform crawls 1M URLs in 2 hours at $300:

Throughput: ~8,333 successful URLs/minute
Effective cost: $0.30 / 1,000 successful URLs
Cost per 1M: $300

That is much more meaningful than saying "the API supports 3,000 RPM," because a nominal 3,000 RPM limit can be irrelevant if the target site responds slowly or starts rate-limiting you.

My shortlist

If speed is the primary objective: test Zyte API Enterprise and Apify with a highly parallel HTTP crawler first. Zyte explicitly offers higher/custom concurrency and performance assurances at Enterprise scale. www.zyte.com

If raw HTTP is enough: Apify can be extremely inexpensive—their Website Content Crawler documentation estimates about $0.20/1,000 pages for raw HTTP under their baseline assumptions. apify.com

If you need JavaScript rendering: cost changes dramatically. Apify estimates roughly $0.50–$5/1,000 browser pages, while Zyte's browser pricing ranges from about $1.01 to $16.08/1,000 requests depending on target complexity. apify.com

If you already have Scrapy expertise: Scrapy Cloud is worth benchmarking because you're essentially buying scalable crawler compute rather than paying a per-request scraping API. Paid units are $9/month each and provide 1 GB RAM plus one concurrent crawl. docs.zyte.com

One caveat: don't benchmark only against an easy site. For a serious production decision, use at least one fast/unprotected site, one JS-heavy site, and one site with meaningful bot protection. The latter can completely reverse the cost/speed ranking.

If you tell me roughly how many URLs (e.g. 1M, 10M, 100M), HTTP vs browser rendering, and whether you control the target site, I can turn this into a concrete $/1M pages + expected crawl time + concurrency comparison.

If your priority is fast crawling of a large site, I’d compare platforms on effective URLs/hour, not the vendor’s headline “URLs/sec.” The latter is usually a ceiling under ideal conditions.

Quick comparison

Lumar's published maximum is 450 URLs/sec for normal crawls and 350 URLs/sec for rendered crawls. www.lumar.iohelp.lumar.io JetOctopus advertises up to 250 pages/sec and says it can handle 100M+ URLs. jetoctopus.comjetoctopus.comwww.lumar.io Sitebulb Cloud is explicitly designed for millions of URLs without crawl credits, while Screaming Frog's performance depends heavily on your local hardware. sitebulb.comwww.screamingfrog.co.uk

What I'd benchmark

For a genuinely large site, run the same crawl against the same URL set on each platform and record:

1. Time to first 100K URLs
2. Time to 1M URLs
3. Total URLs/hour
4. URLs/hour with JavaScript rendering
5. HTTP requests/sec actually hitting your infrastructure
6. Peak CPU/RAM/server load
7. Completion rate — e.g. 99.8% of URLs successfully processed
8. Cost per 1M URLs
9. Cost per 1M rendered URLs
10. Time spent exporting/processing results

The important distinction is crawler throughput vs. site throughput. Lumar, for example, says it can go as fast as your infrastructure allows, but also warns that excessive crawl rates can overwhelm the site's server. www.lumar.iohelp.lumar.io Screaming Frog similarly lets you specify threads or URLs/sec and recommends agreeing on a crawl rate because higher concurrency can affect server response times. sitebulb.comwww.screamingfrog.co.uk

Cost comparison

Screaming Frog is dramatically cheaper if a local crawl works for you: $279/year for one license, with the paid crawler described as having an unlimited crawl limit subject to machine resources. www.screamingfrog.co.uk A sufficiently provisioned machine can reportedly handle around 10M URLs with 16 GB RAM and a 500 GB SSD, although actual performance depends heavily on configuration and page complexity. sitebulb.comwww.screamingfrog.co.uk

Sitebulb Cloud is interesting if you're crawling frequently. Its model is deliberately flat monthly pricing with no crawl credits, so repeated crawls don't consume an additional crawl allowance. It also says JavaScript crawling at millions of URLs doesn't incur an extra charge. sitebulb.com Its enterprise offering starts at £95/month. sitebulb.com

JetOctopus is attractive when volume gets into the multi-million range. Its published pricing example is $383/month for 1M crawl pages, while its Unlimited enterprise plan removes the crawl cap and supports 10M, 50M, 100M+ pages as often as needed. jetoctopus.com

Lumar is harder to price from public information: it uses tailored proposals, with URL allowances and active-project limits determined according to your requirements. www.lumar.io That makes it harder to compare purely on sticker price, but potentially worthwhile when crawler speed and enterprise workflow matter most.

My shortlist

If you're talking about something like 5–50M URLs, I'd test:

1. JetOctopus — probably the most interesting price/scale combination, particularly if you need repeated crawls.

2. Lumar — strongest candidate if raw cloud crawling speed and enterprise capabilities are the priority.

3. Sitebulb Cloud — particularly compelling if you want predictable costs and lots of recurring crawls.

4. Screaming Frog — benchmark it anyway. If a well-provisioned machine can meet your SLA, its economics are hard for cloud platforms to beat.

One particularly useful metric is:

Effective cost per million successfully crawled URLs = total platform cost ÷ completed millions of URLs

Then calculate it separately for HTML-only and JS-rendered crawls. That will expose cases where a platform looks cheap until rendering, crawl quotas, or repeated crawls enter the equation.

If you tell me roughly how many URLs (e.g. 1M, 10M, 100M), whether JS rendering is required, and how often you need to recrawl, I can turn this into a concrete cost + expected crawl-time model for the platforms above.

If the goal is fast, repeatable crawls of a large site, I’d compare platforms on throughput + hard limits + all-in crawl cost, not just the advertised “URLs/sec.”

Quick comparison

One interesting benchmark for Screaming Frog self-hosted: its own cloud guide reports a 3.1M-URL crawl in about two days on an 8-vCPU/32GB/200GB VM, at deliberately conservative speed, for under £20 of GCP compute in that example. www.screamingfrog.co.uk

The important catch: URLs/sec isn't directly comparable

A vendor saying 250 URLs/sec doesn't necessarily beat one doing 50.

Measure these separately:

1. HTML URLs/sec — actual pages successfully fetched.
2. Total HTTP requests/sec — critical for server load, especially JS rendering.
3. Concurrent connections — how aggressively the crawler hits your origin.
4. TTFB-adjusted throughput — otherwise a fast crawler on a fast site looks artificially better.
5. JS crawl throughput — headless Chrome can be dramatically slower and generate many more requests. Sitebulb explicitly notes that Chrome crawls fetch page resources, so server request counts can greatly exceed crawled URLs. support.sitebulb.com
6. Time to complete 1M URLs — probably your most useful business metric.
7. Cost per million URLs — including compute, storage, licenses, and JS multipliers.
8. Repeat-crawl economics — especially important if you're crawling daily/weekly.

A good apples-to-apples benchmark

For your site, I'd run each candidate against the same 100k–500k representative URL sample:

• 80% ordinary HTML pages
• 10% redirects/errors
• 10% JS-heavy pages
• Same geographic region
• Same robots.txt behavior
• Same crawl depth and URL discovery rules
• Same concurrency target
• No caching if you're testing origin load

Then record:

URLs discovered
URLs successfully crawled
Wall-clock time
HTML URLs/sec
HTTP requests/sec
Peak concurrency
Average TTFB
5xx / 429 rate
RAM/CPU consumed
Total crawl cost
Cost / 1M URLs

I'd actually make “cost per 1M successfully crawled HTML URLs” the headline metric.

For example, if Platform A advertises 250 URLs/sec but your site throttles it to 40, while Platform B achieves 80 without generating 429s, B is effectively 2× faster for your workload.

My shortlist

For a large but conventional SEO crawl, I'd start with Screaming Frog vs Sitebulb Cloud vs JetOctopus.

• Lowest raw software cost: Screaming Frog.
• Best cloud/no-infrastructure approach: Sitebulb Cloud.
• Very high-volume cloud crawling: JetOctopus.
• Multi-million URL enterprise operation with logs/data integrations: evaluate the enterprise platforms rather than optimizing around a desktop crawler.

And don't benchmark at maximum speed initially. Agree on an origin-safe rate, then increase concurrency until you find the site's actual saturation point. Screaming Frog and Sitebulb both explicitly expose controls for threads/concurrency and URLs-per-second because server response time and server-side limits ultimately constrain throughput. www.screamingfrog.co.uk

If you give me rough site size (e.g. 5M/50M/500M URLs), HTML vs JS percentage, crawl frequency, and whether you need logs/API data, I can turn this into a $/1M URLs + expected completion-time comparison for the platforms that fit.

If the goal is fast, repeatable crawls of a large site, I’d compare platforms on throughput + volume economics, not just the advertised “pages/sec.”

Quick comparison

The important catch: advertised speed ≠ actual crawl speed

For a site with, say, 10M URLs, I'd benchmark:

Effective throughput = URLs successfully fetched / wall-clock hour

rather than accepting a vendor's maximum.

For example, a crawler advertising 250 pages/sec theoretically processes 900,000 URLs/hour, but your actual number could be much lower because of:

• server response time / TTFB
• crawler concurrency
• robots/rate limiting
• redirects
• JS rendering
• asset fetching
• DNS/TLS latency
• crawler-side processing
• intentional throttling when your origin starts returning errors

Sitebulb explicitly notes that crawl speed depends on threads/Chrome instances, URL/sec limits, TTFB and rendering time. support.sitebulb.com Botify likewise says it may intentionally slow down when necessary to avoid overwhelming the server. support.botify.com

How I'd run the bake-off

Give every vendor the same representative 100K–500K URL slice first.

Measure:

1. Wall-clock completion time
2. Sustained URLs/sec, not peak
3. HTTP success rate
4. 5xx/429 rate
5. Bandwidth consumed
6. JS-rendered URLs/sec, separately from HTML
7. CPU/RAM consumed if self-hosted
8. Data captured per URL
9. Export/API throughput
10. Cost per 1M URLs

Then extrapolate:

Annual crawl cost = (URLs per crawl × crawls per year ÷ included monthly/annual volume) × effective price

This exposes a big difference between “unlimited crawls” and “X crawl pages included.”

My shortlist

If you're talking about millions to tens of millions of URLs, I'd test these three first:

• JetOctopus — strongest published raw-speed/scale story; it claims up to 250 pages/sec and 100M+ URL capability. jetoctopus.comjetoctopus.com
• Sitebulb Cloud — attractive if you want a polished SEO crawler without relying on a workstation; published limit is 10M URLs/audit. sitebulb.com
• Botify — better candidate if you need enterprise SEO analytics, log data and crawling integrated together rather than simply the fastest crawler. support.botify.com

Screaming Frog is the control/baseline I'd include because its licensing is dramatically cheaper, but you're effectively buying the crawler software and supplying the compute rather than buying a cloud crawling service. www.screamingfrog.co.uk

If you tell me roughly site size (e.g. 5M/50M/100M URLs), crawls per day/week/month, whether JS rendering is required, and whether you need SEO analysis or just raw HTML extraction, I can turn this into a $/1M URLs + expected crawl-time comparison for the platforms.

If the goal is fast full-site crawling at large scale, I’d compare platforms on effective pages/minute, concurrency, coverage/success rate, and cost per successful page—not just the advertised API rate.

Quick comparison

Current published limits/prices vary substantially. Firecrawl's Standard plan is 100k pages and 50 concurrent requests for $83/month when billed annually; Growth is 500k pages/100 concurrent for $333; Scale is 1M/150 concurrent for $599. Crawl is charged at 1 credit/page. www.firecrawl.dev

Apify is more infrastructure-oriented: its current plans provide 32, 128, and 256 max concurrent Actor runs on Starter, Scale, and Business, respectively. Compute is $0.20/CU on the lower plans, falling to $0.13/CU on Business, with other usage such as proxies and transfer potentially adding cost. apify.com

Zyte is particularly interesting if you're fetching raw pages rather than needing a complete crawler abstraction. Its standard API limit is 3,000 requests/minute, while enterprise goes to 10,000 RPM. It only charges for successful responses, and pricing depends on the target and whether HTTP or browser rendering is required. docs.zyte.comwww.zyte.com

Bright Data's Web Scraper API advertises unlimited concurrency, with pay-as-you-go pricing of $1.50/1,000 successful records and a $499/month Scale tier including 384k records. brightdata.comdocs.zyte.com

The important part: benchmark them yourself

For a large site, I'd run the same 10k–100k URL corpus through 3 candidates and measure:

1. Wall-clock crawl time
   • pages_completed / elapsed_minutes
   • Measure p50 and p95 page latency separately.

2. Effective throughput
   • Don't use maximum advertised concurrency.
   • Record actual pages/minute at concurrency 10, 25, 50, 100, etc.
   • Find where throughput stops increasing or error rates rise.

3. Successful-page cost
   • total_platform_cost / successfully_crawled_pages
   • This is much more meaningful than nominal $/1k requests.

4. Coverage
   • % of URLs successfully fetched
   • Redirects, 403/429s, timeouts, JS failures, duplicate handling.

5. Crawler overhead
   • Does the service discover links/sitemaps for you?
   • Can you constrain paths?
   • Can you resume failed jobs?
   • Can it stream results instead of waiting for the entire crawl?

6. Rendering percentage
   • Test both HTTP-only and browser-rendered modes.
   • Browser crawling can radically change economics. Zyte, for example, currently lists browser-rendered requests at roughly 8–13× its HTTP pricing depending on target tier. docs.zyte.comwww.zyte.com

My shortlist

If you need the simplest high-speed managed crawler: start with Firecrawl. Its /crawl endpoint handles discovery, rendering and processing, and the company says its infrastructure can scale to millions of URLs/day. www.firecrawl.dev

If you have engineering resources and want maximum control: use Apify. You can tune crawler behavior and concurrency much more deeply, but you need to model compute/proxy/storage costs rather than assuming a simple per-page price. apify.com

If the core problem is fetching huge numbers of pages quickly: I'd benchmark Zyte and Bright Data very seriously. Their request/concurrency models can be better suited to raw acquisition at very high volume. brightdata.comdocs.zyte.com

For example, if you tell me site size (e.g. 1M/10M/100M URLs), % JavaScript-rendered, desired recrawl frequency, and your monthly budget, I can turn this into a concrete 10M-page crawl cost + estimated completion-time comparison across these platforms.

For large-site crawling, compare platforms by throughput, crawl controls, resource limits, and total cost of ownership rather than just “pages per minute.” A good evaluation framework looks like this:

1. Define the crawl workload first

Before comparing tools, normalize the test case:

A crawler that is fastest on raw HTML may be much slower once you enable JS rendering, screenshots, extraction rules, or compliance throttling.

2. Compare crawl speed metrics

Measure:

Pages per second (PPS)

Basic throughput:

Pages crawled / elapsed time

Example:

• 1M URLs in 2 hours
• 138 pages/sec

Requests per second (RPS)

More useful when pages contain many resources:

HTTP requests / second

Time to first million URLs

A practical benchmark:

Sustained throughput

Some tools burst quickly but throttle later. Test a 6–24 hour crawl.

3. Compare crawl limits

Important limits to capture:

4. Cost comparison model

Do not compare only subscription price. Calculate:

Cost per million URLs crawled

Formula:

(total monthly cost) / (millions of URLs crawled)

Example:

5. Categories of platforms to compare

Enterprise SEO crawlers

Good for technical SEO audits:

• Screaming Frog — strong desktop crawler; large crawls depend on machine resources and configuration.
• Botify — built for enterprise-scale SEO crawling and log analysis.
• Lumar — enterprise technical SEO crawling and monitoring.

Cloud crawler APIs / infrastructure

Good for custom pipelines:

• Apify — actor-based crawling platform with usage-based pricing.
• Scrapy Cloud — managed deployment for Scrapy crawlers.
• Bright Data — web data collection infrastructure with large-scale proxy capabilities.

Build-your-own

For very large sites:

• Scrapy + distributed workers
• Apache Kafka / queues
• Kubernetes workers
• Object storage (S3-compatible)
• Custom URL frontier

This often wins on cost at hundreds of millions of URLs, but requires engineering.

6. Benchmark methodology

Run the same crawl against each platform:

Test site sets

Use:

• 100k URLs
• 1M URLs
• 10M+ URLs

Record:

Start time
End time
URLs discovered
URLs fetched
Errors
Blocked requests
Average latency
CPU/memory usage
Storage generated
Total cost

Example scorecard:

7. Watch for hidden costs

Common surprises:

• JS rendering can multiply costs 5–20×
• Proxy bandwidth becomes the biggest bill
• Storing raw HTML for millions of pages is expensive
• Re-crawling unchanged URLs wastes budget
• Poor URL deduplication causes crawler explosions
• Export pipelines become the bottleneck

Practical recommendation

For a large production site (millions to hundreds of millions of URLs), I would benchmark three approaches:

1. Enterprise crawler (fastest time-to-value)
2. Cloud crawler API (flexible cost scaling)
3. Custom distributed crawler (lowest unit cost at massive scale)

The winner is usually determined by your required crawl frequency and whether you need SEO auditing or a custom data pipeline. A daily 10M URL crawl and a one-time 500M URL crawl have very different optimal choices.

For large-site crawling, compare platforms like an infrastructure benchmark, not just “pages per second.” Crawl speed is usually limited by concurrency, politeness rules, rendering requirements, network, and how much processing happens per URL.

A useful comparison framework:

Benchmark methodology

Use the same test site or URL set:

1. Define crawl scenarios

Example:

• 100k static HTML pages
• 1M-page ecommerce site
• 100k JS-rendered pages
• 10k pages with heavy images/assets
• Recrawl of changed URLs only

Measure:

• Time to completion
• Successful URLs fetched
• Errors/timeouts
• CPU/RAM/network usage
• Cost per million URLs

A simple metric:

Cost per 1M crawled URLs =
(total crawl cost) / (URLs successfully processed / 1,000,000)

and:

Effective crawl speed =
successful URLs / total crawl time

2. Separate fetch speed from processing speed

Two tools may both claim “1M pages crawled,” but one may spend more time on:

• HTML parsing
• SEO extraction
• screenshots
• JS rendering
• link discovery
• duplicate detection
• indexing

Run two tests:

Fetch-only benchmark

• Download pages
• Minimal parsing

Full pipeline benchmark

• Crawl + extract + store all required data

The second is usually what matters operationally.

3. Compare platform categories

Self-hosted crawlers

Examples:

• Apache Nutch
• Scrapy

Pros:

• Maximum control
• Scale horizontally
• Potentially lowest cost at very large volumes

Cons:

• You own infrastructure
• Need engineering effort

SEO crawling tools

Examples:

• Screaming Frog
• Sitebulb

Pros:

• Easy setup
• Strong reports

Cons:

• Often designed for audits rather than continuous billion-page crawling

Cloud crawler platforms

Examples:

• Apify
• Bright Data

Pros:

• Managed scaling
• Distributed infrastructure

Cons:

• Cost can grow quickly with volume

4. Track hidden costs

The advertised crawl price is often not the full cost.

Include:

Infrastructure

• crawler servers
• proxies
• bandwidth
• storage
• databases
• queues

Operational

• monitoring
• retries
• IP reputation management
• debugging

Engineering

• maintaining parsers
• handling site changes
• scaling failures

Example comparison sheet

For a large site, I would benchmark with these numbers

• 10M URLs
• 50M URLs
• 500M URLs

Record:

1. Maximum sustainable URLs/sec
2. Error rate at that speed
3. Cost per 10M URLs
4. Cost per recurring daily crawl
5. Time to recover from failures
6. Duplicate/changed URL handling

The winner is usually not the platform with the highest peak speed—it is the one with the lowest cost per successfully processed URL at your required freshness window.

If the goal is fast crawling of a large site, I’d compare platforms on actual throughput, scale limits, and cost per million URLs, rather than just the advertised “crawl speed.”

Quick comparison

\*Screaming Frog's “unlimited” is not literally an infinite-performance guarantee: its practical URL capacity depends on allocated memory and storage. Its documentation says the default limit is 5M URLs but that it can crawl substantially more with appropriate hardware. www.screamingfrog.co.ukwww.screamingfrog.co.ukwww.screamingfrog.co.uk

The important speed distinction

Screaming Frog: you control concurrency. Its default is 5 threads, but you can increase threads or specify a maximum URL-request rate. The limiting factor is frequently the site/CDN/server, not the crawler itself. www.screamingfrog.co.ukwww.screamingfrog.co.uk

Sitebulb: similarly lets you control maximum URLs/sec and concurrency. Its HTML crawler is the fastest mode; Chrome/JS crawling is substantially slower because it has to render resources. support.sitebulb.com

JetOctopus: is the standout if your requirement is literally maximum cloud throughput. Its current pricing page advertises crawling up to 250 pages/sec, cloud execution, no domain/user limits, and no crawling-capacity limit. The catch is that your subscription is still based on purchased crawl volume. jetoctopus.com

Cost comparison that matters

For a large site, I'd normalize everything to:

Cost per 1M URLs crawled

For example, suppose you have a 10M-page site and want to crawl it weekly:

• 10M × 4.33 ≈ 43.3M URLs/month
• Then add a separate multiplier if JS rendering is required.
• Compare the resulting monthly/annual platform cost.

This is much more meaningful than comparing monthly subscription prices.

JetOctopus explicitly prices crawl volume and currently shows packages such as 1M crawl pages/month, with JS pages counted differently; it also offers larger/custom volumes and an enterprise no-limit package. jetoctopus.com

Screaming Frog is unusually inexpensive at this scale if you can run it yourself: the current license is £199/year per user, and the vendor recommends SSD/database storage for multi-million-URL crawls. www.screamingfrog.co.ukwww.screamingfrog.co.ukwww.screamingfrog.co.uk

How I'd benchmark them

Don't use the vendors' maximum numbers directly. Run the same 1M-URL crawl against each platform and record:

1. URLs/minute
2. Total crawl duration
3. HTTP requests/sec
4. Peak CPU/RAM
5. 5xx/429 rate
6. Average TTFB
7. JS vs non-JS throughput
8. Cost per 1M URLs
9. Cost per 1M rendered URLs
10. Whether multiple crawls can run concurrently

I'd run three tests:

• HTML-only: measure raw crawler throughput.
• JS rendering: measure realistic expensive crawling.
• Throttled: e.g. 10–20 requests/sec, to see how much throughput you can actually use without stressing production.

That last test is important: a crawler capable of 250 pages/sec isn't necessarily useful if your origin/CDN starts returning 429/503 responses at 30 pages/sec.

My shortlist

For a 10M+ URL site, I'd start with:

1. JetOctopus — if maximum cloud throughput and minimizing crawl infrastructure are the priority.
2. Screaming Frog — if you want the lowest cost and can dedicate a well-provisioned machine; its documented multi-million-URL capability is strong. www.screamingfrog.co.ukwww.screamingfrog.co.uk
3. Sitebulb Cloud — if collaboration/reporting and cloud operation matter as much as raw throughput. support.sitebulb.com
4. Lumar — if this is an enterprise procurement where workflow, automation, security, and support justify custom pricing. www.lumar.io

If you give me site size (e.g. 5M/50M/500M URLs), HTML vs JS %, desired crawl frequency, and whether self-hosting is acceptable, I can turn this into a concrete $/month + estimated crawl time comparison.

When comparing platforms for fast crawling of a large website (100k–10M+ pages), focus on measurable throughput rather than marketing claims. Most services advertise "fast," but actual performance depends on concurrency limits, rate limiting, JavaScript rendering, and pricing.

Here's a practical comparison framework:

Typical platform characteristics