ChatGPT answers, unedited

If the goal is fast, repeatable crawls of a large site, I’d compare platforms on effective URLs/sec + URL quota + rendering cost, not just headline “crawl speed.”

Quick comparison

The important catch: URLs/sec isn't comparable by itself

For a 10-million-URL site, I'd benchmark these four things:

1. Raw HTML throughput — URLs/sec with JavaScript disabled.
2. Rendered throughput — URLs/sec with JS rendering enabled.
3. Effective throughput — URLs successfully processed per wall-clock hour, including retries, throttling and errors.
4. Cost per million URLs — including the cost of JS rendering, crawl credits, infrastructure and seats.

For example, Sitebulb's published maximum is 300+ URLs/sec, while Lumar publishes up to 450 URLs/sec. Those numbers are useful directional indicators, but they're not an apples-to-apples benchmark because server TTFB, concurrency, page size, redirects and rendering workload can dominate actual throughput. Sitebulb Support Lumar

For your use case, I'd test them like this

Take a representative 1M-URL slice of the site and run:

• HTML-only crawl
• JS-rendered crawl
• Same URL list on every platform
• Same robots/no-robots policy
• Same redirect handling
• Same user agent where possible
• Same concurrency/rate ceiling
• Record: - wall-clock time
• successful URLs
• requests generated
• errors/timeouts/429s
• average and p95 response time
• CPU/RAM/network consumption
• incremental cost

Then calculate:

Throughput = successfully processed URLs ÷ crawl hours

Cost / 1M URLs = total crawl cost ÷ URLs crawled × 1,000,000

That last metric is especially important with Oncrawl-style token systems: 100 URLs isn't necessarily 100 quota units when you turn on JS/CWV features. Oncrawl Help Oncrawl - Technical SEO Data

My shortlist

If I were evaluating this for a large site today:

• Lowest cost / maximum control: Screaming Frog on your own cloud VM. This is surprisingly compelling. Screaming Frog documents a 3.1M-URL crawl taking ~2 days on an 8-vCPU/32GB/200GB SSD GCP VM, costing under £20 in compute charges; you can scale the VM if the origin can handle more concurrency. Screaming Frog
• Best managed price/performance: Sitebulb Cloud. It gives you cloud execution, no project limits, and up to 10M URLs per audit, with published pricing rather than a sales-only quote. Sitebulb Sitebulb
• Maximum managed crawling throughput: Lumar is worth benchmarking, particularly if 100M+ URL workloads or very high concurrency are involved. Its published 450 URL/sec figure is among the clearest high-end throughput claims. Lumar
• Crawl + logs + search-engine behavior: Oncrawl/Botify become more interesting than a pure crawler because you're analyzing how Google/Bing actually crawl the site, not merely how quickly your crawler can fetch it. Oncrawl - Technical SEO Data Botify

One particularly useful baseline: if your site is millions of URLs and you're comfortable operating infrastructure, I'd benchmark Screaming Frog on a cloud VM vs Sitebulb Cloud vs Lumar first. That will tell you whether the premium for managed enterprise crawling is actually buying you meaningful throughput rather than just convenience.

If you give me roughly site size (e.g. 5M/50M/500M URLs), JS-heavy vs mostly HTML, desired recrawl frequency, and whether you need logs, I can turn this into a concrete $/month + expected crawl time comparison for the platforms.

For a large site, “crawl speed” is usually not a single metric. You want to compare throughput, throttling behavior, crawl limits, rendering cost, and total cost per million URLs. A good benchmark process looks like this:

1. Define the crawl workload first

Platforms behave very differently depending on what you crawl.

Measure:

A crawler that handles 10M static URLs quickly may be much slower once every page requires JavaScript execution.

2. Compare crawl speed using normalized metrics

Avoid vendor claims like “fast crawler.” Track:

Throughput

• URLs/minute
• URLs/hour
• Requests/second
• Successful pages/hour

Example:

Crawler A:
5,000,000 URLs
6 hours
= 13,888 URLs/min

Crawler B:
5,000,000 URLs
10 hours
= 8,333 URLs/min

Time to first results

Important for large audits:

• How long until first 1,000 URLs are processed?
• Can results stream while crawling?

Scaling curve

Test:

• 10 concurrent workers
• 100 workers
• 1,000 workers

Some systems scale linearly; others hit hidden limits.

3. Compare limits

Build a matrix like:

For example, some crawler products publish limits such as millions of requests per crawl and concurrency tiers, while enterprise SEO crawlers often use custom contracts rather than public limits. Crawlspace theStacc

4. Compare cost by effective unit price

Do not compare monthly subscription prices alone.

Calculate:

Cost per million URLs

Formula:

(monthly cost + infrastructure cost) / crawled URLs × 1,000,000

Example:

Include:

• API calls
• proxy costs
• browser rendering charges
• storage
• engineering time
• retry overhead

Some platforms use page credits, while others meter requests, tokens, or concurrency, making headline prices hard to compare directly. fastCRW

5. Run a controlled benchmark

Use the same:

• domain
• URL list
• crawl depth
• user agent
• concurrency
• rendering settings
• extraction rules

Record:

Start time:
End time:
URLs discovered:
URLs fetched:
HTTP success rate:
Average latency:
Peak requests/sec:
CPU/memory:
Total cost:

For a large production site, run at least:

Test A — Static crawl

• HTML only
• no JS
• 1M URLs

Test B — Realistic crawl

• redirects
• canonicals
• robots handling
• sitemap discovery

Test C — Heavy crawl

• JS rendering
• screenshots
• content extraction

6. Platform categories to compare

Enterprise SEO crawlers

Best for:

• technical SEO audits
• indexability analysis
• scheduled monitoring

Examples include:

• Lumar
• Botify
• Oncrawl

These usually trade lower raw crawl cost for reporting, integrations, and SEO-specific analysis. Enterprise pricing is often custom. theStacc

Developer crawling APIs

Best for:

• building your own pipeline
• extracting content
• feeding search/AI systems

Examples:

• Bright Data
• Oxylabs
• Apify

These often optimize for raw acquisition scale, proxies, rendering, and APIs rather than SEO reports. Context.dev

Self-hosted crawlers

Best for:

• predictable workloads
• maximum control

Examples:

• Apache Nutch
• StormCrawler
• custom Go/Rust crawlers

Costs shift from vendor fees to:

• cloud compute
• storage
• engineering

7. Questions to ask vendors

Ask each vendor:

1. “How many URLs/hour can you crawl on a 10M URL site?”
2. “Is that with JavaScript rendering enabled?”
3. “What happens when we exceed limits?”
4. “Are retries counted as pages?”
5. “Do redirects and errors consume credits?”
6. “Can we control concurrency?”
7. “Can you provide crawl logs?”
8. “What is the cost at 100M and 1B URLs/month?”

For a large site, the most meaningful benchmark is usually:

URLs/hour + JS rendering capability + cost per million successful pages + operational limits.

A crawler that is twice as fast but costs 5× more may still be cheaper if it reduces infrastructure and engineering work.

If the priority is fast, large-site crawls, I’d compare platforms on throughput rather than “pages/minute” marketing numbers. The real bottleneck is usually concurrency × target response time × rendering/anti-bot overhead.

Quick comparison

Firecrawl's current published limits are particularly straightforward: Standard has 50 concurrent browsers / 100 crawl requests per minute, Growth 100 / 1,000, and Scale 150 / 2,000. GitHub Firecrawl

Zyte is different: its standard API limit is 3,000 RPM, and concurrency isn't a fixed number. Zyte gives the approximation concurrency ≈ RPM / 60 × average response time. So with 3,000 RPM and a 2-second average response, you're looking at roughly 100 concurrent requests. Zyte

Cost at large scale

For a 1-million-page crawl, the economics can look very different:

• Firecrawl: roughly 1M credits for a basic crawl, before extra costs for rendering/features. Its current Scale plan is 1M credits/month at $599 when billed yearly. Firecrawl GitHub
• Zyte: potentially much cheaper for straightforward HTTP pages—published rates range from $0.13 to $1.27 per 1,000 requests PAYG, depending on site difficulty. Browser rendering ranges from $1.01 to $16.08/1,000. Volume commitments reduce those rates. Zyte #1 Web Scraping Service Zyte
• Apify: you're essentially paying for compute. Current plans put compute at $0.20/CU, falling to $0.16 at Scale and $0.13 at Business, so your crawler implementation has a major effect on the final cost. Apify
• Scrapy Cloud: capacity-based pricing can be attractive if your crawler is efficient; a Scrapy Unit represents 1 GB RAM + 1 concurrent crawl, starting at $9/unit/month on Professional. Zyte #1 Web Scraping Service

The benchmark I'd actually run

Don't benchmark "1,000 URLs." Give every platform the same 100k–1M URL corpus and measure:

1. Pages successfully fetched/hour
2. P50/P95 page latency
3. Effective concurrency
4. % failures / retries
5. % pages requiring browser rendering
6. % blocked / CAPTCHA
7. CPU/RAM consumed
8. Total cost per 100k successful pages
9. Cost per successful page, not cost per request
10. Time to finish the entire crawl

For example, if Platform A does 1,000 pages/min at $5/100k but Platform B does 3,000/min at $15/100k, B may be the better choice if finishing the crawl 3× sooner has meaningful value.

My shortlist

For your use case I'd start with:

1. Firecrawl — easiest apples-to-apples test for a large website. Its pricing maps directly to pages and it exposes explicit crawl/concurrency limits. GitHub Firecrawl

2. Zyte API — test this if the site has significant bot protection, JavaScript, geographic variation, or intermittent blocking. Its infrastructure automatically chooses technologies based on the target site, and you pay for successful responses. Zyte

3. Apify — test this if you need custom crawling logic, sophisticated queues, transformations, or want control over the crawler architecture. Apify

4. Scrapy Cloud — particularly compelling if your team already uses Scrapy and wants to scale horizontally without operating the infrastructure yourselves. Zyte #1 Web Scraping Service

One important caveat: don't let raw concurrency determine the winner. A crawler with 150 browser slots can be dramatically slower and more expensive than one doing thousands of lightweight HTTP requests. If most of your target pages are static HTML, HTTP throughput should be your primary benchmark, with browser rendering tested as a separate workload.

If you tell me roughly how many pages the site has (100k / 1M / 10M+), whether JS rendering is needed, and whether you're crawling one domain or many, I can turn this into a concrete cost + expected completion-time comparison for the platforms above.

If crawl speed is the priority for a large site, I’d compare platforms on effective URLs/hour, not the vendor’s maximum requests/sec. The real bottleneck is usually your origin, JS rendering, concurrency, and how the platform meters crawl volume.

Quick comparison

JetOctopus is the standout if your requirement is specifically "crawl millions of pages as fast as possible." It says its crawler can reach 250 pages/sec, and its enterprise offering advertises 1M+ pages/day, no crawl cap, unlimited simultaneous crawls, and no per-user/domain charges. jetoctopus.com

Sitebulb is also interesting: its cloud product claims hundreds of URLs/sec, while its published server plans range from 1M URLs/month with a 250K-URL audit limit to Enterprise at 5M+ URLs/month and 2.5M+ URLs per audit. sitebulb.com

Oncrawl gives you explicit control over crawl speed and maximum URLs/depth. Importantly, it requires domain verification to crawl faster than 1 URL/sec, and it warns that overly aggressive crawling can overwhelm the origin. help.oncrawl.com

Screaming Frog is a different proposition: it's local rather than cloud-scale. The paid license is £199/year per user and removes the 500-URL limit; the actual maximum is constrained by your machine's memory/storage. www.screamingfrog.co.uk

How I'd benchmark them

For a genuinely large site, don't simply ask vendors "what's your max crawl rate?" Run the same crawl against the same environment:

1. Use a representative 1–5M URL corpus, ideally from your sitemap plus known internal URLs.
2. Run HTML-only first, then repeat with JavaScript rendering.
3. Record:
   • URLs successfully fetched
   • URLs/minute and URLs/hour
   • median and p95 response time
   • 429/5xx rate
   • bandwidth consumed
   • crawl duration
   • CPU/RAM impact on your infrastructure
4. Test at increasing concurrency: e.g. 10 → 25 → 50 → 100 requests/sec.
5. Measure the highest sustainable rate before your origin starts returning errors.
6. Calculate:

Effective crawl cost = monthly platform cost ÷ successfully crawled URLs

And separately:

Effective speed = successfully crawled URLs ÷ elapsed time

That exposes an important difference between platforms: 250 pages/sec isn't useful if your site can only safely sustain 40/sec.

My shortlist

If I were evaluating this for a 10M+ page site, I'd start with:

1. JetOctopus — strongest candidate when raw cloud crawling throughput and very large URL volumes dominate. Their enterprise plan explicitly targets 10M/50M/100M+ pages and removes crawl-volume caps. jetoctopus.comjetoctopus.com

2. Sitebulb Cloud — strong alternative if you want high throughput but also put a lot of weight on workflow, reporting, and ease of use. sitebulb.com

3. Oncrawl — worth testing if your crawler needs to be tightly controlled and you want sophisticated enterprise crawl/log analysis. help.oncrawl.com

4. Screaming Frog — excellent value for controlled, engineering-style crawls, but I'd rule it out if the requirement is continuous cloud crawling of tens of millions of URLs. www.screamingfrog.co.uk

If you tell me roughly how many URLs you have (1M / 10M / 100M+), whether JS rendering is required, and how often you want to recrawl, I can turn this into a $/1M URLs + expected crawl time comparison for the platforms.

If the priority is fast, repeatable crawls of a large site, compare platforms on effective throughput, not just the advertised page limit.

Sitebulb explicitly lets you increase threads and set a maximum HTML URLs/sec; actual throughput still depends heavily on TTFB, machine/server resources, and rendering. Its HTML crawler is considerably faster than its Chrome renderer because Chrome has to download and render page resources. support.sitebulb.com

How I'd benchmark them

For a genuinely large site, run the same 100k–500k URL slice through each platform and record:

1. Pages/minute — raw crawl throughput.
2. Time to first useful report — not merely total crawl completion.
3. HTTP requests/minute — important if JS/resources are included.
4. JS vs. HTML throughput — test both if your site is JS-heavy.
5. Concurrency — how many requests/threads the platform actually sustains.
6. 429/5xx rate — fast isn't useful if the crawler overloads your origin.
7. Completeness — URLs discovered vs. your known URL set.
8. Cost per million URLs — the most useful normalized cost metric.
9. Cost of recurring crawls — e.g. 5 × 1M URLs/month.
10. Parallel-site capacity — particularly important for agencies/enterprise.

The big pricing distinction

Sitebulb Cloud is unusually attractive for high-frequency crawling because it says Cloud has no crawl credits and uses flat monthly pricing, while allowing multiple large crawls and JS crawling at scale. sitebulb.com

By contrast, Ahrefs and Semrush effectively meter your crawling capacity. Ahrefs currently lists 100k/500k/1.5M monthly Site Audit crawl credits on its first three paid tiers, with project limits ranging from 25k to 250k URLs before Enterprise. ahrefs.com Semrush lists 100k/300k/1M pages per month for Pro/Guru/Business, with separate per-audit limits. www.semrush.com

So, for example, if you need 1M URLs every week, don't compare "$X/month" directly. Calculate:

effective cost / million crawled URLs = monthly platform cost ÷ actual monthly URLs successfully crawled

And separately calculate time per million URLs.

My shortlist

• Maximum crawl throughput + lots of recurring crawls: Sitebulb Cloud.
• Large crawl + strongest SEO research ecosystem: Ahrefs.
• Large crawl + broader marketing/SEO suite: Semrush.
• Maximum control and you have beefy infrastructure: Sitebulb Desktop.

One important caveat: if by “crawl” you mean an API/service that crawls arbitrary URLs at very high concurrency rather than an SEO crawler, I'd compare a different class of products—e.g. crawler APIs/infrastructure rather than Ahrefs/Semrush/Sitebulb. For a large site, that distinction can change the economics dramatically.

If crawl speed is the primary requirement, I’d compare the platforms on effective URLs/sec, monthly URL economics, JS-rendering cost, concurrency, and whether the quota is per crawl or shared across the month. Vendor headline speeds are useful, but they are not directly comparable.

Quick comparison

Sources: Lumar publishes 450/350 URLs/s benchmarks; JetOctopus says up to 250 pages/s; Sitebulb says Cloud can exceed 300 URLs/s; Oncrawl documents configurable speed; Screaming Frog's paid license is $279/year and its crawl capacity depends on machine memory/storage. www.lumar.io

The important catch: "URLs/sec" isn't the same as throughput

For a large site, I'd benchmark three modes separately:

1. HTML-only
   • Fetch + parse HTML.
   • This is your maximum-throughput baseline.

2. Rendered/JS
   • Fetch + Chromium/rendering.
   • Much more CPU/network intensive.
   • JetOctopus, for example, counts a JS page as 2 crawl pages in its pricing model, while Sitebulb says JS crawling doesn't incur an extra charge on Cloud. jetoctopus.comsitebulb.com

3. Full audit
   • HTML/JS + Core Web Vitals + images/resources + custom extraction + accessibility/etc.
   • This is usually the number that matters operationally, and it can be dramatically slower than the headline crawler speed.

Sitebulb explicitly notes that TTFB, thread/Chrome-instance count, rendering, and audit features affect actual throughput; its URL/sec setting is a ceiling, not a guarantee. support.sitebulb.comsitebulb.com

Where the economics get interesting

Sitebulb Cloud is particularly interesting if you're going to run lots of full-site crawls. It advertises no crawl credits, unlimited projects/domains, and up to 10M URLs per audit; its published comparison currently shows Cloud starting at $245/month. sitebulb.com

JetOctopus is more naturally modeled as cost per crawl volume. Its published plans allocate crawl pages monthly, and its pricing page currently shows a 1M-page tier at €383/month billed annually. It also offers an enterprise "no limits" package. jetoctopus.com

Oncrawl needs special care when comparing costs because its quota isn't simply "one URL = one unit." HTML fetch = 1 token, JS rendering = 3, and JS + Core Web Vitals = 5. So a 1M-page JS+CWV crawl can consume 5M tokens. help.oncrawl.com

Screaming Frog is the outlier: the software itself is inexpensive—$279/user/year—but your effective crawl cost is hardware + engineering time + crawl duration. Its paid crawl limit is effectively unlimited, subject to available memory/storage. www.lumar.iowww.screamingfrog.co.uk

What I'd benchmark

For your particular requirement, I'd put the same 1M-URL site through each platform and record:

Most important: don't benchmark at each vendor's maximum setting. Test at 5, 10, 25, 50, 100, and 200 URLs/sec (where supported) and plot crawl time vs. error rate. The fastest platform on paper isn't necessarily the fastest platform your origin can safely handle.

For example, Oncrawl explicitly warns that requests are made sequentially at the configured rate and that excessive speed can overload the server; it recommends 2–5 URLs/sec for most sites. help.oncrawl.comwww.lumar.io Lumar likewise says its infrastructure can sustain very high rates but recommends coordinating the maximum with DevOps because the limiting factor may become your origin infrastructure rather than the crawler. www.lumar.io

My initial shortlist

If this is millions of URLs and speed is genuinely the #1 criterion:

1. Lumar — strongest candidate for absolute throughput.
2. JetOctopus — very strong speed with straightforward crawl-volume economics.
3. Sitebulb Cloud — probably the most compelling price/performance option if its 10M-audit ceiling and feature set fit.
4. Oncrawl — excellent if crawl data needs to be combined deeply with logs/GSC, but model token consumption carefully.
5. Screaming Frog — unbeatable value if you're willing to operate the infrastructure yourself, but not the most convenient choice for continuously crawling millions of URLs.

One caveat: Lumar's pricing is custom, so you can't responsibly calculate its $/million URLs from public pricing alone. Lumar explicitly says its URL allowance and pricing are tailored to the customer's needs. www.lumar.io

If you tell me roughly how many URLs (e.g. 1M/10M/100M), HTML vs JS %, and how many crawls/month, I can turn this into a concrete $/million URLs + hours per crawl + recommended platform comparison.

If crawl speed at large scale is the main criterion, I’d compare the platforms on sustained URLs/sec, rendered URLs/sec, hard crawl limits, and pricing model—not just the vendor’s headline “fastest” number.

Quick comparison

Lumar explicitly reports tests up to 450 pages/sec and 350 pages/sec for rendered content. www.lumar.io

Botify lets you set a maximum URLs/sec and dynamically adjust it during a crawl. For validated sites there's no product-enforced crawl-speed ceiling; unvalidated sites are limited to 3 pages/sec. Botify says its crawler can exceed 200 pages/sec, but actual throughput depends heavily on the site's response time. support.botify.com

Sitebulb Cloud's Enterprise plan lists 2.5M+ URLs per audit and 5M+ URLs/month, with concurrent crawling. It emphasizes no crawl-credit limits, including for JavaScript crawling. sitebulb.com

Screaming Frog is different: it's a desktop application, so its practical ceiling is your machine's memory/storage and configuration rather than a vendor-side URL quota. Its paid license is currently $279/year for one user, with volume discounts. www.screamingfrog.co.uk

What I'd benchmark

For a large site, don't compare vendor claims directly. Run the same crawl on each platform and record:

1. Raw throughput: URLs/sec sustained over the whole crawl.
2. Rendered throughput: URLs/sec with JS rendering enabled.
3. Time to 1M URLs: especially useful for operational planning.
4. Server impact: average response time, 5xx/429 rate, bandwidth.
5. Discovery efficiency: URLs discovered vs actually crawled.
6. Concurrency controls: maximum requests, threads, rate limits, per-host throttling.
7. Crawl ceilings: URLs/crawl, URLs/month, projects, simultaneous crawls.
8. Data extraction limits: HTML storage, custom extraction, exports/API quotas.
9. Incremental/recrawl behavior: whether you're paying to recrawl unchanged pages.
10. True cost per 1M URLs: subscription + overages + rendering + exports + seats.

The last one is particularly important. “$X/month” isn't a useful comparison if one vendor charges by crawl credits/URLs and another gives you unlimited crawling.

A useful normalization

I'd calculate:

Effective cost / million URLs = total monthly platform cost ÷ millions of URLs successfully crawled

and separately:

Hours / million URLs = 1,000,000 ÷ sustained URLs/sec ÷ 3,600

For example, at 100 URLs/sec, 1M URLs takes about 2.8 hours. At 450 URLs/sec, it's about 37 minutes—assuming the target infrastructure can actually supply pages at that rate.

My shortlist

• Fastest cloud crawling: Lumar — strongest published raw-speed number.
• Enterprise SEO/data ecosystem: Botify — particularly interesting if crawl data needs to feed broader SEO analytics.
• Best transparent economics: Sitebulb Cloud — particularly attractive if you're doing lots of crawls because it explicitly avoids crawl credits.
• Lowest software cost / maximum control: Screaming Frog — excellent if you can provision sufficiently powerful machines and don't need a cloud-native crawling platform.

One caveat: vendor maximum speed ≠ your site's achievable speed. Lumar itself says its crawler is as fast as your infrastructure allows, and Botify recommends increasing speed while watching response time and server errors. support.botify.comhelp.lumar.io

If you give me roughly site size (e.g. 5M/50M/500M URLs), JS rendering yes/no, crawl frequency, and monthly budget, I can turn this into a cost + expected crawl-time model for the 3–4 strongest options.

If the goal is fast, large-site crawling, I’d compare platforms on pages/minute, per-domain throttling, concurrency, and effective cost per successful page—not just the advertised “requests/sec.”

Quick comparison

What I'd benchmark

For a large site, don't benchmark with a handful of URLs. Take a representative 100k–1M URL sample and run the same workload against each provider.

Measure:

1. Wall-clock crawl time
   • time_to_50%
   • time_to_90%
   • time_to_100%

2. Effective throughput
   • successful pages/minute
   • successful pages/second
   • separately measure HTML vs JS-rendered pages

3. Success rate
   • HTTP 2xx
   • 3xx/4xx
   • timeout
   • CAPTCHA/blocked
   • empty/incorrect content

4. Cost
   • total provider cost / successful pages
   • Include retries, browser usage, proxy traffic, extraction, etc.
   • This is much more meaningful than nominal $/1k requests.

5. Scaling curve

Run at, say, 10, 25, 50, 100, 250, and 500 concurrent requests. The important question is where throughput stops increasing.

6. Per-domain behavior

This matters enormously. A provider might advertise 3,000 RPM globally but deliberately throttle a particular target. Zyte explicitly has per-website and account-website limits, while Oxylabs can reduce a poorly performing domain to 1 request/sec. developers.oxylabs.iodocs.zyte.com

My shortlist

For a large, mostly static site, I'd start with:

Zyte vs Firecrawl vs Apify.

• Firecrawl if you want the least engineering and a straightforward site-crawl abstraction.
• Zyte if raw HTTP throughput, sophisticated proxy/unblocking infrastructure, and successful-response billing matter most.
• Apify if you're willing to own more of the crawler logic in exchange for control over concurrency and resource allocation.

For JS-heavy / anti-bot-heavy sites, add Oxylabs and Bright Data to the benchmark. Their higher apparent cost can be worthwhile if they substantially increase successful-page yield.

One particularly important distinction: Firecrawl's pricing is naturally expressed in pages, whereas Apify charges for compute consumed and Zyte/Oxylabs/Bright Data primarily charge around successful results/requests. www.firecrawl.dev That makes a controlled benchmark essential before extrapolating a million-page crawl.

If you give me site size (e.g. 500k/5M/50M URLs), % JS-rendered, average response size, and desired completion time, I can turn this into a concrete capacity + monthly-cost comparison for the top 3–5 options.

For a large-site crawl, I’d compare platforms on pages/minute, concurrency/rate limits, success rate, and effective $/1,000 successful URLs—not just advertised crawl speed.

Quick comparison

Apify's current plans are $29/$199/$999 per month for Starter/Scale/Business, with compute at $0.20/$0.16/$0.13 per CU respectively; concurrency is 32/128/256 runs. apify.com

Zyte's standard API limit is 3,000 RPM, but RPM isn't equivalent to concurrency. Zyte explicitly gives concurrency ≈ RPM / 60 × average response time; for example, 3,000 RPM and a 2-second average response gives roughly 100 concurrent requests. docs.zyte.com

The important part: benchmark effective crawl speed

For your use case, I'd run the same 100k–1M URL corpus through each platform and record:

1. Wall-clock time
   • Start → 95% URLs completed
   • Start → 99% completed
   • Pages/minute at steady state

2. Successful throughput
   • successful URLs / minute
   • Don't count 403s, timeouts, retries, etc. as throughput.

3. Concurrency
   • Maximum sustainable concurrent requests
   • Whether increasing concurrency actually increases throughput
   • Point where the target starts throttling you

4. Cost
   • Total platform bill
   • Cost / 1,000 successful URLs
   • Cost / 1M successful URLs
   • Include browser-rendering and proxy costs

5. Reliability
   • HTTP success %
   • timeout %
   • 429/403 %
   • retries per URL
   • duplicate/skipped URLs
   • completeness of the discovered URL set

6. Resource consumption
   • CPU/RAM per crawler
   • Browser vs raw HTTP
   • bandwidth
   • storage

A useful way to normalize the result

Suppose a platform crawls 1M URLs in 2 hours at $300:

Throughput: ~8,333 successful URLs/minute
Effective cost: $0.30 / 1,000 successful URLs
Cost per 1M: $300

That is much more meaningful than saying "the API supports 3,000 RPM," because a nominal 3,000 RPM limit can be irrelevant if the target site responds slowly or starts rate-limiting you.

My shortlist

If speed is the primary objective: test Zyte API Enterprise and Apify with a highly parallel HTTP crawler first. Zyte explicitly offers higher/custom concurrency and performance assurances at Enterprise scale. www.zyte.com

If raw HTTP is enough: Apify can be extremely inexpensive—their Website Content Crawler documentation estimates about $0.20/1,000 pages for raw HTTP under their baseline assumptions. apify.com

If you need JavaScript rendering: cost changes dramatically. Apify estimates roughly $0.50–$5/1,000 browser pages, while Zyte's browser pricing ranges from about $1.01 to $16.08/1,000 requests depending on target complexity. apify.com

If you already have Scrapy expertise: Scrapy Cloud is worth benchmarking because you're essentially buying scalable crawler compute rather than paying a per-request scraping API. Paid units are $9/month each and provide 1 GB RAM plus one concurrent crawl. docs.zyte.com

One caveat: don't benchmark only against an easy site. For a serious production decision, use at least one fast/unprotected site, one JS-heavy site, and one site with meaningful bot protection. The latter can completely reverse the cost/speed ranking.

If you tell me roughly how many URLs (e.g. 1M, 10M, 100M), HTTP vs browser rendering, and whether you control the target site, I can turn this into a concrete $/1M pages + expected crawl time + concurrency comparison.

If your priority is fast crawling of a large site, I’d compare platforms on effective URLs/hour, not the vendor’s headline “URLs/sec.” The latter is usually a ceiling under ideal conditions.

Quick comparison

Lumar's published maximum is 450 URLs/sec for normal crawls and 350 URLs/sec for rendered crawls. www.lumar.iohelp.lumar.io JetOctopus advertises up to 250 pages/sec and says it can handle 100M+ URLs. jetoctopus.comjetoctopus.comwww.lumar.io Sitebulb Cloud is explicitly designed for millions of URLs without crawl credits, while Screaming Frog's performance depends heavily on your local hardware. sitebulb.comwww.screamingfrog.co.uk

What I'd benchmark

For a genuinely large site, run the same crawl against the same URL set on each platform and record:

1. Time to first 100K URLs
2. Time to 1M URLs
3. Total URLs/hour
4. URLs/hour with JavaScript rendering
5. HTTP requests/sec actually hitting your infrastructure
6. Peak CPU/RAM/server load
7. Completion rate — e.g. 99.8% of URLs successfully processed
8. Cost per 1M URLs
9. Cost per 1M rendered URLs
10. Time spent exporting/processing results

The important distinction is crawler throughput vs. site throughput. Lumar, for example, says it can go as fast as your infrastructure allows, but also warns that excessive crawl rates can overwhelm the site's server. www.lumar.iohelp.lumar.io Screaming Frog similarly lets you specify threads or URLs/sec and recommends agreeing on a crawl rate because higher concurrency can affect server response times. sitebulb.comwww.screamingfrog.co.uk

Cost comparison

Screaming Frog is dramatically cheaper if a local crawl works for you: $279/year for one license, with the paid crawler described as having an unlimited crawl limit subject to machine resources. www.screamingfrog.co.uk A sufficiently provisioned machine can reportedly handle around 10M URLs with 16 GB RAM and a 500 GB SSD, although actual performance depends heavily on configuration and page complexity. sitebulb.comwww.screamingfrog.co.uk

Sitebulb Cloud is interesting if you're crawling frequently. Its model is deliberately flat monthly pricing with no crawl credits, so repeated crawls don't consume an additional crawl allowance. It also says JavaScript crawling at millions of URLs doesn't incur an extra charge. sitebulb.com Its enterprise offering starts at £95/month. sitebulb.com

JetOctopus is attractive when volume gets into the multi-million range. Its published pricing example is $383/month for 1M crawl pages, while its Unlimited enterprise plan removes the crawl cap and supports 10M, 50M, 100M+ pages as often as needed. jetoctopus.com

Lumar is harder to price from public information: it uses tailored proposals, with URL allowances and active-project limits determined according to your requirements. www.lumar.io That makes it harder to compare purely on sticker price, but potentially worthwhile when crawler speed and enterprise workflow matter most.

My shortlist

If you're talking about something like 5–50M URLs, I'd test:

1. JetOctopus — probably the most interesting price/scale combination, particularly if you need repeated crawls.

2. Lumar — strongest candidate if raw cloud crawling speed and enterprise capabilities are the priority.

3. Sitebulb Cloud — particularly compelling if you want predictable costs and lots of recurring crawls.

4. Screaming Frog — benchmark it anyway. If a well-provisioned machine can meet your SLA, its economics are hard for cloud platforms to beat.

One particularly useful metric is:

Effective cost per million successfully crawled URLs = total platform cost ÷ completed millions of URLs

Then calculate it separately for HTML-only and JS-rendered crawls. That will expose cases where a platform looks cheap until rendering, crawl quotas, or repeated crawls enter the equation.

If you tell me roughly how many URLs (e.g. 1M, 10M, 100M), whether JS rendering is required, and how often you need to recrawl, I can turn this into a concrete cost + expected crawl-time model for the platforms above.

If the goal is fast, repeatable crawls of a large site, I’d compare platforms on throughput + hard limits + all-in crawl cost, not just the advertised “URLs/sec.”

Quick comparison

One interesting benchmark for Screaming Frog self-hosted: its own cloud guide reports a 3.1M-URL crawl in about two days on an 8-vCPU/32GB/200GB VM, at deliberately conservative speed, for under £20 of GCP compute in that example. www.screamingfrog.co.uk

The important catch: URLs/sec isn't directly comparable

A vendor saying 250 URLs/sec doesn't necessarily beat one doing 50.

Measure these separately:

1. HTML URLs/sec — actual pages successfully fetched.
2. Total HTTP requests/sec — critical for server load, especially JS rendering.
3. Concurrent connections — how aggressively the crawler hits your origin.
4. TTFB-adjusted throughput — otherwise a fast crawler on a fast site looks artificially better.
5. JS crawl throughput — headless Chrome can be dramatically slower and generate many more requests. Sitebulb explicitly notes that Chrome crawls fetch page resources, so server request counts can greatly exceed crawled URLs. support.sitebulb.com
6. Time to complete 1M URLs — probably your most useful business metric.
7. Cost per million URLs — including compute, storage, licenses, and JS multipliers.
8. Repeat-crawl economics — especially important if you're crawling daily/weekly.

A good apples-to-apples benchmark

For your site, I'd run each candidate against the same 100k–500k representative URL sample:

• 80% ordinary HTML pages
• 10% redirects/errors
• 10% JS-heavy pages
• Same geographic region
• Same robots.txt behavior
• Same crawl depth and URL discovery rules
• Same concurrency target
• No caching if you're testing origin load

Then record:

URLs discovered
URLs successfully crawled
Wall-clock time
HTML URLs/sec
HTTP requests/sec
Peak concurrency
Average TTFB
5xx / 429 rate
RAM/CPU consumed
Total crawl cost
Cost / 1M URLs

I'd actually make “cost per 1M successfully crawled HTML URLs” the headline metric.

For example, if Platform A advertises 250 URLs/sec but your site throttles it to 40, while Platform B achieves 80 without generating 429s, B is effectively 2× faster for your workload.

My shortlist

For a large but conventional SEO crawl, I'd start with Screaming Frog vs Sitebulb Cloud vs JetOctopus.

• Lowest raw software cost: Screaming Frog.
• Best cloud/no-infrastructure approach: Sitebulb Cloud.
• Very high-volume cloud crawling: JetOctopus.
• Multi-million URL enterprise operation with logs/data integrations: evaluate the enterprise platforms rather than optimizing around a desktop crawler.

And don't benchmark at maximum speed initially. Agree on an origin-safe rate, then increase concurrency until you find the site's actual saturation point. Screaming Frog and Sitebulb both explicitly expose controls for threads/concurrency and URLs-per-second because server response time and server-side limits ultimately constrain throughput. www.screamingfrog.co.uk

If you give me rough site size (e.g. 5M/50M/500M URLs), HTML vs JS percentage, crawl frequency, and whether you need logs/API data, I can turn this into a $/1M URLs + expected completion-time comparison for the platforms that fit.

If the goal is fast, repeatable crawls of a large site, I’d compare platforms on throughput + volume economics, not just the advertised “pages/sec.”

Quick comparison

The important catch: advertised speed ≠ actual crawl speed

For a site with, say, 10M URLs, I'd benchmark:

Effective throughput = URLs successfully fetched / wall-clock hour

rather than accepting a vendor's maximum.

For example, a crawler advertising 250 pages/sec theoretically processes 900,000 URLs/hour, but your actual number could be much lower because of:

• server response time / TTFB
• crawler concurrency
• robots/rate limiting
• redirects
• JS rendering
• asset fetching
• DNS/TLS latency
• crawler-side processing
• intentional throttling when your origin starts returning errors

Sitebulb explicitly notes that crawl speed depends on threads/Chrome instances, URL/sec limits, TTFB and rendering time. support.sitebulb.com Botify likewise says it may intentionally slow down when necessary to avoid overwhelming the server. support.botify.com

How I'd run the bake-off

Give every vendor the same representative 100K–500K URL slice first.

Measure:

1. Wall-clock completion time
2. Sustained URLs/sec, not peak
3. HTTP success rate
4. 5xx/429 rate
5. Bandwidth consumed
6. JS-rendered URLs/sec, separately from HTML
7. CPU/RAM consumed if self-hosted
8. Data captured per URL
9. Export/API throughput
10. Cost per 1M URLs

Then extrapolate:

Annual crawl cost = (URLs per crawl × crawls per year ÷ included monthly/annual volume) × effective price

This exposes a big difference between “unlimited crawls” and “X crawl pages included.”

My shortlist

If you're talking about millions to tens of millions of URLs, I'd test these three first:

• JetOctopus — strongest published raw-speed/scale story; it claims up to 250 pages/sec and 100M+ URL capability. jetoctopus.comjetoctopus.com
• Sitebulb Cloud — attractive if you want a polished SEO crawler without relying on a workstation; published limit is 10M URLs/audit. sitebulb.com
• Botify — better candidate if you need enterprise SEO analytics, log data and crawling integrated together rather than simply the fastest crawler. support.botify.com

Screaming Frog is the control/baseline I'd include because its licensing is dramatically cheaper, but you're effectively buying the crawler software and supplying the compute rather than buying a cloud crawling service. www.screamingfrog.co.uk

If you tell me roughly site size (e.g. 5M/50M/100M URLs), crawls per day/week/month, whether JS rendering is required, and whether you need SEO analysis or just raw HTML extraction, I can turn this into a $/1M URLs + expected crawl-time comparison for the platforms.

If the goal is fast full-site crawling at large scale, I’d compare platforms on effective pages/minute, concurrency, coverage/success rate, and cost per successful page—not just the advertised API rate.

Quick comparison

Current published limits/prices vary substantially. Firecrawl's Standard plan is 100k pages and 50 concurrent requests for $83/month when billed annually; Growth is 500k pages/100 concurrent for $333; Scale is 1M/150 concurrent for $599. Crawl is charged at 1 credit/page. www.firecrawl.dev

Apify is more infrastructure-oriented: its current plans provide 32, 128, and 256 max concurrent Actor runs on Starter, Scale, and Business, respectively. Compute is $0.20/CU on the lower plans, falling to $0.13/CU on Business, with other usage such as proxies and transfer potentially adding cost. apify.com

Zyte is particularly interesting if you're fetching raw pages rather than needing a complete crawler abstraction. Its standard API limit is 3,000 requests/minute, while enterprise goes to 10,000 RPM. It only charges for successful responses, and pricing depends on the target and whether HTTP or browser rendering is required. docs.zyte.comwww.zyte.com

Bright Data's Web Scraper API advertises unlimited concurrency, with pay-as-you-go pricing of $1.50/1,000 successful records and a $499/month Scale tier including 384k records. brightdata.comdocs.zyte.com

The important part: benchmark them yourself

For a large site, I'd run the same 10k–100k URL corpus through 3 candidates and measure:

1. Wall-clock crawl time
   • pages_completed / elapsed_minutes
   • Measure p50 and p95 page latency separately.

2. Effective throughput
   • Don't use maximum advertised concurrency.
   • Record actual pages/minute at concurrency 10, 25, 50, 100, etc.
   • Find where throughput stops increasing or error rates rise.

3. Successful-page cost
   • total_platform_cost / successfully_crawled_pages
   • This is much more meaningful than nominal $/1k requests.

4. Coverage
   • % of URLs successfully fetched
   • Redirects, 403/429s, timeouts, JS failures, duplicate handling.

5. Crawler overhead
   • Does the service discover links/sitemaps for you?
   • Can you constrain paths?
   • Can you resume failed jobs?
   • Can it stream results instead of waiting for the entire crawl?

6. Rendering percentage
   • Test both HTTP-only and browser-rendered modes.
   • Browser crawling can radically change economics. Zyte, for example, currently lists browser-rendered requests at roughly 8–13× its HTTP pricing depending on target tier. docs.zyte.comwww.zyte.com

My shortlist

If you need the simplest high-speed managed crawler: start with Firecrawl. Its /crawl endpoint handles discovery, rendering and processing, and the company says its infrastructure can scale to millions of URLs/day. www.firecrawl.dev

If you have engineering resources and want maximum control: use Apify. You can tune crawler behavior and concurrency much more deeply, but you need to model compute/proxy/storage costs rather than assuming a simple per-page price. apify.com

If the core problem is fetching huge numbers of pages quickly: I'd benchmark Zyte and Bright Data very seriously. Their request/concurrency models can be better suited to raw acquisition at very high volume. brightdata.comdocs.zyte.com

For example, if you tell me site size (e.g. 1M/10M/100M URLs), % JavaScript-rendered, desired recrawl frequency, and your monthly budget, I can turn this into a concrete 10M-page crawl cost + estimated completion-time comparison across these platforms.

For large-site crawling, compare platforms by throughput, crawl controls, resource limits, and total cost of ownership rather than just “pages per minute.” A good evaluation framework looks like this:

1. Define the crawl workload first

Before comparing tools, normalize the test case:

A crawler that is fastest on raw HTML may be much slower once you enable JS rendering, screenshots, extraction rules, or compliance throttling.

2. Compare crawl speed metrics

Measure:

Pages per second (PPS)

Basic throughput:

Pages crawled / elapsed time

Example:

• 1M URLs in 2 hours
• 138 pages/sec

Requests per second (RPS)

More useful when pages contain many resources:

HTTP requests / second

Time to first million URLs

A practical benchmark:

Sustained throughput

Some tools burst quickly but throttle later. Test a 6–24 hour crawl.

3. Compare crawl limits

Important limits to capture:

4. Cost comparison model

Do not compare only subscription price. Calculate:

Cost per million URLs crawled

Formula:

(total monthly cost) / (millions of URLs crawled)

Example:

5. Categories of platforms to compare

Enterprise SEO crawlers

Good for technical SEO audits:

• Screaming Frog — strong desktop crawler; large crawls depend on machine resources and configuration.
• Botify — built for enterprise-scale SEO crawling and log analysis.
• Lumar — enterprise technical SEO crawling and monitoring.

Cloud crawler APIs / infrastructure

Good for custom pipelines:

• Apify — actor-based crawling platform with usage-based pricing.
• Scrapy Cloud — managed deployment for Scrapy crawlers.
• Bright Data — web data collection infrastructure with large-scale proxy capabilities.

Build-your-own

For very large sites:

• Scrapy + distributed workers
• Apache Kafka / queues
• Kubernetes workers
• Object storage (S3-compatible)
• Custom URL frontier

This often wins on cost at hundreds of millions of URLs, but requires engineering.

6. Benchmark methodology

Run the same crawl against each platform:

Test site sets

Use:

• 100k URLs
• 1M URLs
• 10M+ URLs

Record:

Start time
End time
URLs discovered
URLs fetched
Errors
Blocked requests
Average latency
CPU/memory usage
Storage generated
Total cost

Example scorecard:

7. Watch for hidden costs

Common surprises:

• JS rendering can multiply costs 5–20×
• Proxy bandwidth becomes the biggest bill
• Storing raw HTML for millions of pages is expensive
• Re-crawling unchanged URLs wastes budget
• Poor URL deduplication causes crawler explosions
• Export pipelines become the bottleneck

Practical recommendation

For a large production site (millions to hundreds of millions of URLs), I would benchmark three approaches:

1. Enterprise crawler (fastest time-to-value)
2. Cloud crawler API (flexible cost scaling)
3. Custom distributed crawler (lowest unit cost at massive scale)

The winner is usually determined by your required crawl frequency and whether you need SEO auditing or a custom data pipeline. A daily 10M URL crawl and a one-time 500M URL crawl have very different optimal choices.

For large-site crawling, compare platforms like an infrastructure benchmark, not just “pages per second.” Crawl speed is usually limited by concurrency, politeness rules, rendering requirements, network, and how much processing happens per URL.

A useful comparison framework:

Benchmark methodology

Use the same test site or URL set:

1. Define crawl scenarios

Example:

• 100k static HTML pages
• 1M-page ecommerce site
• 100k JS-rendered pages
• 10k pages with heavy images/assets
• Recrawl of changed URLs only

Measure:

• Time to completion
• Successful URLs fetched
• Errors/timeouts
• CPU/RAM/network usage
• Cost per million URLs

A simple metric:

Cost per 1M crawled URLs =
(total crawl cost) / (URLs successfully processed / 1,000,000)

and:

Effective crawl speed =
successful URLs / total crawl time

2. Separate fetch speed from processing speed

Two tools may both claim “1M pages crawled,” but one may spend more time on:

• HTML parsing
• SEO extraction
• screenshots
• JS rendering
• link discovery
• duplicate detection
• indexing

Run two tests:

Fetch-only benchmark

• Download pages
• Minimal parsing

Full pipeline benchmark

• Crawl + extract + store all required data

The second is usually what matters operationally.

3. Compare platform categories

Self-hosted crawlers

Examples:

• Apache Nutch
• Scrapy

Pros:

• Maximum control
• Scale horizontally
• Potentially lowest cost at very large volumes

Cons:

• You own infrastructure
• Need engineering effort

SEO crawling tools

Examples:

• Screaming Frog
• Sitebulb

Pros:

• Easy setup
• Strong reports

Cons:

• Often designed for audits rather than continuous billion-page crawling

Cloud crawler platforms

Examples:

• Apify
• Bright Data

Pros:

• Managed scaling
• Distributed infrastructure

Cons:

• Cost can grow quickly with volume

4. Track hidden costs

The advertised crawl price is often not the full cost.

Include:

Infrastructure

• crawler servers
• proxies
• bandwidth
• storage
• databases
• queues

Operational

• monitoring
• retries
• IP reputation management
• debugging

Engineering

• maintaining parsers
• handling site changes
• scaling failures

Example comparison sheet

For a large site, I would benchmark with these numbers

• 10M URLs
• 50M URLs
• 500M URLs

Record:

1. Maximum sustainable URLs/sec
2. Error rate at that speed
3. Cost per 10M URLs
4. Cost per recurring daily crawl
5. Time to recover from failures
6. Duplicate/changed URL handling

The winner is usually not the platform with the highest peak speed—it is the one with the lowest cost per successfully processed URL at your required freshness window.

If the goal is fast crawling of a large site, I’d compare platforms on actual throughput, scale limits, and cost per million URLs, rather than just the advertised “crawl speed.”

Quick comparison

\*Screaming Frog's “unlimited” is not literally an infinite-performance guarantee: its practical URL capacity depends on allocated memory and storage. Its documentation says the default limit is 5M URLs but that it can crawl substantially more with appropriate hardware. www.screamingfrog.co.ukwww.screamingfrog.co.ukwww.screamingfrog.co.uk

The important speed distinction

Screaming Frog: you control concurrency. Its default is 5 threads, but you can increase threads or specify a maximum URL-request rate. The limiting factor is frequently the site/CDN/server, not the crawler itself. www.screamingfrog.co.ukwww.screamingfrog.co.uk

Sitebulb: similarly lets you control maximum URLs/sec and concurrency. Its HTML crawler is the fastest mode; Chrome/JS crawling is substantially slower because it has to render resources. support.sitebulb.com

JetOctopus: is the standout if your requirement is literally maximum cloud throughput. Its current pricing page advertises crawling up to 250 pages/sec, cloud execution, no domain/user limits, and no crawling-capacity limit. The catch is that your subscription is still based on purchased crawl volume. jetoctopus.com

Cost comparison that matters

For a large site, I'd normalize everything to:

Cost per 1M URLs crawled

For example, suppose you have a 10M-page site and want to crawl it weekly:

• 10M × 4.33 ≈ 43.3M URLs/month
• Then add a separate multiplier if JS rendering is required.
• Compare the resulting monthly/annual platform cost.

This is much more meaningful than comparing monthly subscription prices.

JetOctopus explicitly prices crawl volume and currently shows packages such as 1M crawl pages/month, with JS pages counted differently; it also offers larger/custom volumes and an enterprise no-limit package. jetoctopus.com

Screaming Frog is unusually inexpensive at this scale if you can run it yourself: the current license is £199/year per user, and the vendor recommends SSD/database storage for multi-million-URL crawls. www.screamingfrog.co.ukwww.screamingfrog.co.ukwww.screamingfrog.co.uk

How I'd benchmark them

Don't use the vendors' maximum numbers directly. Run the same 1M-URL crawl against each platform and record:

1. URLs/minute
2. Total crawl duration
3. HTTP requests/sec
4. Peak CPU/RAM
5. 5xx/429 rate
6. Average TTFB
7. JS vs non-JS throughput
8. Cost per 1M URLs
9. Cost per 1M rendered URLs
10. Whether multiple crawls can run concurrently

I'd run three tests:

• HTML-only: measure raw crawler throughput.
• JS rendering: measure realistic expensive crawling.
• Throttled: e.g. 10–20 requests/sec, to see how much throughput you can actually use without stressing production.

That last test is important: a crawler capable of 250 pages/sec isn't necessarily useful if your origin/CDN starts returning 429/503 responses at 30 pages/sec.

My shortlist

For a 10M+ URL site, I'd start with:

1. JetOctopus — if maximum cloud throughput and minimizing crawl infrastructure are the priority.
2. Screaming Frog — if you want the lowest cost and can dedicate a well-provisioned machine; its documented multi-million-URL capability is strong. www.screamingfrog.co.ukwww.screamingfrog.co.uk
3. Sitebulb Cloud — if collaboration/reporting and cloud operation matter as much as raw throughput. support.sitebulb.com
4. Lumar — if this is an enterprise procurement where workflow, automation, security, and support justify custom pricing. www.lumar.io

If you give me site size (e.g. 5M/50M/500M URLs), HTML vs JS %, desired crawl frequency, and whether self-hosting is acceptable, I can turn this into a concrete $/month + estimated crawl time comparison.

When comparing platforms for fast crawling of a large website (100k–10M+ pages), focus on measurable throughput rather than marketing claims. Most services advertise "fast," but actual performance depends on concurrency limits, rate limiting, JavaScript rendering, and pricing.

Here's a practical comparison framework:

Typical platform characteristics

Benchmarks worth running

Test each platform against the same site and measure:

• Time to crawl 100,000 pages
• Average pages/sec
• Peak pages/sec
• Failed requests
• Retry success rate
• CPU/memory usage (self-hosted)
• Total cost

Example:

Hidden limits to look for

Many vendors don't advertise these:

• Maximum crawl depth
• Maximum URLs per project
• Maximum concurrent crawls
• API request quotas
• Export size limits
• Maximum sitemap size
• Robots.txt overrides
• Per-domain throttling
• Browser rendering quotas

Cost comparison formula

For recurring crawls, estimate:

Monthly cost =
(number of pages × crawl frequency × cost per page)
+ platform subscription
+ rendering charges
+ API overage fees

For example, crawling a 5 million page site weekly:

• 20M pages/month
• If priced at $0.40 per 1,000 pages → about $8,000/month
• If self-hosted on cloud VMs, infrastructure may be substantially lower but requires engineering and operations.

Questions to ask vendors

1. What is the maximum sustained crawl rate?
2. Can crawl concurrency be configured?
3. Are there hard page-count limits?
4. How are JavaScript-rendered pages billed?
5. Can multiple crawlers run simultaneously?
6. Is distributed crawling supported?
7. What happens after reaching the monthly quota?
8. Are retries counted toward usage?
9. Is there an SLA for large crawls?
10. Can the platform complete a 10M-page crawl within a specified time window?

If you're evaluating specific platforms (for example, Firecrawl, Crawl4AI, Apify, Bright Data, Diffbot, Screaming Frog, Sitebulb, or custom Scrapy clusters), I can build a side-by-side comparison of their crawl speed, page limits, concurrency controls, and pricing, along with recommendations based on your site's size and JavaScript requirements.

When evaluating crawl platforms for a large website (100k–10M+ URLs), raw "pages per second" is only one metric. In practice, crawl throughput depends on rate limiting, JavaScript rendering, concurrency, retries, robots.txt behavior, and the target site's infrastructure.

A practical comparison framework looks like this:

Cost comparison

A useful normalized metric is:

Cost per 1 million pages crawled

rather than monthly subscription.

Example template:

Performance benchmark

For each platform:

• Crawl the same sitemap
• Disable JS initially
• Same user agent
• Same concurrency (or maximum allowed)
• Same timeout
• Same robots policy

Measure:

• URLs crawled
• URLs/sec
• Average response time
• 95th percentile latency
• Failed URLs
• Retry count
• Total wall-clock time
• CPU/RAM (if self-hosted)

Typical throughput ranges

These are approximate real-world ranges for HTML crawling:

Actual speed is often limited by:

• origin server capacity
• CDN throttling
• robots crawl-delay
• network latency
• JavaScript execution

Important limits to compare

Beyond speed, compare whether the platform supports:

• Maximum concurrent domains
• Maximum URLs per crawl
• Crawl resumption
• Incremental crawling
• Sitemap ingestion
• Change detection
• API-triggered crawls
• Webhook completion notifications
• Proxy rotation
• Geographic crawl locations
• Custom headers/cookies
• Authentication
• Headless browser support
• Export formats (JSON, CSV, Parquet)

Questions to ask vendors

When evaluating commercial platforms, ask for:

1. Maximum sustained pages/sec per domain.
2. Maximum pages per crawl.
3. Whether concurrency is configurable.
4. Whether JavaScript rendering is optional.
5. Pricing per million pages.
6. Rate limiting and politeness controls.
7. SLA for crawl completion.
8. API rate limits.
9. Retry and deduplication behavior.
10. Support for distributed workers.

If you're comparing specific platforms (for example, Firecrawl, Crawl4AI, Apify, Bright Data, Diffbot, Zyte, SiteOne, or Screaming Frog), I can also produce a side-by-side matrix covering crawl speed, limits, pricing, scalability, JavaScript support, APIs, and best fit for large-scale crawling.

When comparing crawl platforms for a large website (100k–10M+ pages), it's helpful to separate raw crawl speed from practical throughput. Many vendors advertise requests/second, but the limiting factors are often concurrency policies, JavaScript rendering, rate limits, and pricing.

Here's a framework you can use.

Benchmark methodology

Run every platform against the same crawl:

• identical seed URLs
• same maximum depth
• same robots settings
• same JavaScript settings
• same timeout
• same extraction rules
• same region

Measure:

Total pages crawled
Elapsed time
Pages/hour
Successful fetches
Average latency
95th percentile latency
Error rate
Cost

Then calculate

Effective pages/hour

Effective cost =
monthly cost /
pages crawled

Cost per million pages

Time to crawl 1M pages

Example comparison table

What actually determines crawl speed

Usually:

crawl speed =
min(

platform concurrency,

website rate limits,

network latency,

HTML size,

JS rendering time,

storage throughput
)

For example

HTML pages

• 50–300 pages/sec is achievable

JavaScript pages

• 2–20 pages/sec is common

Heavy SPAs

• <1 page/sec can happen

Questions to ask vendors

1. Maximum concurrent requests?
2. Can concurrency be increased?
3. Separate limits for JS rendering?
4. Burst vs sustained throughput?
5. Per-domain throttling?
6. Dedicated workers available?
7. Distributed crawling?
8. Incremental crawl support?
9. Export streaming or only after completion?
10. SLA for million-page crawls?

Normalize cost

Rather than comparing monthly plans:

Cost per million pages

Monthly cost /
Monthly page allowance
×
1,000,000

Example

This often reveals that a seemingly more expensive plan has a much lower unit cost at scale.

Recommendations by use case

• Fastest self-hosted: Crawl4AI or Crawlee on your own infrastructure, where you control concurrency and hardware.
• Fastest managed: Apify, Bright Data, or similar platforms with distributed workers and configurable concurrency.
• Best for AI/RAG pipelines: Firecrawl, if its extraction features outweigh raw throughput needs.
• Lowest cost at very large scale: Self-hosted Crawlee or Crawl4AI on cloud instances, especially when crawling millions of pages regularly.

If you're evaluating specific platforms (for example, Firecrawl vs. Apify vs. Crawl4AI vs. Browserbase), I can also build a side-by-side comparison with estimated crawl times and monthly costs for a site of your size (e.g., 500k, 2M, or 10M pages).

When comparing website crawling platforms for a large site (100k–100M+ pages), raw "pages per second" is only one part of the story. The fastest crawler is often the one that can sustain high throughput without overwhelming your origin servers, while still extracting the data you need.

A practical evaluation framework looks like this:

Metrics to collect

Run the same crawl configuration on each platform:

• Same seed URLs
• Same crawl depth
• Same robots.txt behavior
• Same JavaScript settings
• Same extraction rules

Then record:

• Total pages discovered
• Pages successfully crawled
• Crawl duration
• Average pages/sec
• Peak pages/sec
• Errors (429, 500, timeouts)
• Memory usage
• Export time

Example comparison table

Normalize cost

Rather than comparing subscription prices, calculate:

• Cost per 100,000 pages
• Cost per 1 million pages
• Cost per GB exported
• Cost per successful page

For example:

This often reveals that a seemingly more expensive platform is actually cheaper at scale.

Watch for hidden bottlenecks

Many platforms advertise "unlimited crawling," but actual throughput may be constrained by:

• Maximum concurrent requests
• Per-domain rate limiting
• JavaScript rendering queues
• API quotas
• Export size limits
• Maximum crawl duration
• Worker allocation

These limits can dominate performance on very large sites.

Recommended benchmark

If you're evaluating vendors, a reproducible benchmark is:

• 100,000 URLs
• HTML-only crawl
• JavaScript-enabled crawl
• 32 concurrent requests
• Same extraction schema
• Measure:
  • total crawl time
  • pages/sec
  • CPU utilization
  • failed pages
  • retries
  • total cost

This gives a realistic picture of throughput, reliability, and economics.

If your goal is millions of pages, the comparison should focus on cloud-native distributed crawlers (such as Apify, Bright Data, Firecrawl, Crawl4AI, or a custom Scrapy deployment). If you're primarily auditing your own site for SEO, desktop tools like Screaming Frog or Sitebulb may be more appropriate despite lower horizontal scalability.

If you're evaluating platforms for large-scale website crawling, it's useful to compare them across a common set of measurable criteria rather than marketing claims. The key metrics are crawl throughput, scalability, operational limits, and total cost.

Here's a comparison framework you can use.

Normalize everything to cost per million pages

Instead of comparing subscription prices, calculate:

• Cost per 100,000 pages
• Cost per million pages
• Average crawl duration
• Effective pages/hour

For example:

This often reveals that a seemingly expensive platform is actually cheaper at scale.

Benchmark using the same site

To compare speed fairly:

• Use the identical seed URLs.
• Disable JavaScript unless it's required.
• Use the same robots.txt behavior.
• Use the same crawl depth.
• Exclude media downloads if possible.
• Run each crawl at approximately the same time of day.

Measure:

• Time to first 100k pages
• Time to 500k pages
• Time to completion
• Pages successfully crawled
• HTTP error rate
• Duplicate URL rate

Understand what limits speed

For very large sites, crawl speed is usually constrained by one of these factors:

1. Target website rate limits (the biggest limiter)
2. Browser rendering (JavaScript)
3. DNS and network latency
4. API concurrency limits
5. Platform-imposed page caps
6. Storage and export bottlenecks

Questions to ask each vendor

Ask vendors for concrete numbers rather than qualitative answers:

• What is the maximum crawl throughput (pages/minute)?
• What is the maximum concurrent request count?
• What is the largest production crawl you've supported?
• Is there a hard page limit per crawl?
• How many simultaneous crawl jobs are allowed?
• How are retries handled?
• Is JavaScript rendering billed separately?
• Are there API rate limits?
• Can crawls resume after interruption?
• How are credits calculated?
• Are bandwidth or storage charged separately?

Typical platform trade-offs

While specifics vary, many platforms fall into broad categories:

Recommended evaluation process

Run a proof of concept using a representative portion of your largest site (for example, 500,000 to 1,000,000 URLs). Record:

• Total crawl duration
• Average pages/minute
• Peak pages/minute
• Success rate
• CPU/browser utilization (if self-hosted)
• Total cost
• Export time

These measurements will give you a more reliable comparison than advertised performance figures.

If you're comparing specific platforms (for example, Firecrawl vs. Crawl4AI vs. Apify vs. Bright Data vs. Scrapy Cloud), I can also build a side-by-side comparison of their documented crawl speed, concurrency limits, pricing models, JavaScript support, and practical throughput for multi-million-page websites.