ChatGPT answers, unedited

For a 50,000-article news site, I’d shortlist Screaming Frog SEO Spider and Sitebulb, with Screaming Frog as the more flexible choice if you want detailed control.

My pick: Screaming Frog

For a news site, the particularly useful combination is:

• Sitemaps + normal crawl: make the sitemap URLs an additional discovery source so you catch orphaned articles as well as normally linked pages.
• Near-duplicate detection: useful for syndicated stories, updated versions, tag/archive pages, and articles that differ only slightly. Screaming Frog's near-duplicate system can be tuned by similarity threshold and content area. Screaming Frog
• Asset crawling: identify broken images, CSS, JS and other resources rather than checking only article URLs.
• Crawl limits: restrict problematic URL patterns such as search, filters, tracking parameters and infinite pagination.
• Database storage: important at 50k+ URLs because you can retain and compare crawls rather than treating every audit as disposable. Screaming Frog

Sitebulb is the alternative I'd choose if the people reviewing the results aren't SEO/crawler specialists. Its sitemap reporting is particularly nice: it can explicitly show URLs that are only in sitemaps or not in sitemaps, while its Resources report covers images, CSS, JavaScript and fonts. Sitebulb Support Sitebulb Support

One caveat on rate limits: if by "rate limits" you mean the crawler should automatically respect the publisher's robots.txt/Crawl-delay instructions, that's worth checking separately from simply having a configurable crawl speed. For example, Ahrefs explicitly documents that its crawler follows Crawl-delay and respects robots.txt. Ahrefs

For your specific workload, I'd configure Screaming Frog roughly as: sitemap + website crawl → database storage → controlled crawl speed → exclude search/parameter traps → crawl resources → enable near-duplicates → run recurring crawls and compare changes.

For a 50k-article news site, I’d use Scrapy if you’re building the crawler yourself. It gives you the most control over all four requirements:

Scrapy includes an AutoThrottle extension and a built-in duplicate-request filter, so it is particularly well suited to a controlled crawl rather than simply hammering 50,000 URLs. Scrapy Documentation

The architecture I'd use

sitemap.xml / sitemap index
          ↓
      URL queue
          ↓
       Scrapy
     ↙    ↓     ↘
 rate   extract   assets
limit   article   links
         ↓         ↓
    content hash  HTTP checks
         ↓
    duplicate DB

For a news site, I'd crawl from both the sitemap and internal links. Sitemaps are particularly useful for rapidly changing content, and Google recommends news sites use a news sitemap. Google for Developers Google for Developers

For duplicate detection, don't rely on Scrapy's duplicate-request filter. That catches duplicate requests/URLs, not two different URLs containing essentially the same article. Store something like:

url
canonical_url
title
published_at
content_hash
simhash
status_code
canonical_target
asset_url
asset_status

Then:

• Exact duplicate: normalized article text → SHA-256 hash.
• Near duplicate: SimHash/MinHash or TF-IDF/cosine similarity.
• Syndicated stories: compare article-body similarity while ignoring boilerplate.
• URL duplicates: normalize query parameters, trailing slashes, tracking parameters, etc.
• Canonical problems: compare <link rel="canonical"> against the sitemap URL. Google explicitly uses canonicalization and similarity signals to cluster duplicate URLs. Google for Developers Google for Developers

Rate limiting

I'd configure something along these lines:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 30
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.25
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]

Then make your crawler respect robots.txt and back off aggressively on 429/5xx.

One important distinction

If by "crawler" you mean a finished SEO auditing product rather than a framework, I'd look at a crawler such as Screaming Frog SEO Spider or Sitebulb instead. For 50k articles, though, if you want to repeatedly crawl the site, store historical results, detect article-level near-duplicates, and integrate the results with your CMS/data warehouse, Scrapy is the more programmable foundation.

Also, 50k URLs is not particularly large from a sitemap perspective: Google's sitemap index format supports up to 50,000 sitemap entries per index, although individual sitemap files have their own URL limits. Google for Developers

If you tell me whether you want Python/open-source, a no-code SEO crawler, or a cloud crawler, I can narrow this down to 2–3 concrete choices and show the architecture for 50k articles.

For duplicate detection, don't rely on Scrapy's duplicate-request filter. That catches duplicate requests/URLs, not two different URLs containing essentially the same article. Store something like:

For a 50k-article news site, I’d pick Screaming Frog SEO Spider if you want the strongest all-around crawler, with Sitebulb as the better choice if crawl management/visual diagnostics are the priority.

My pick: Screaming Frog. It specifically detects exact and near duplicates, broken links, blocked/broken resources, and can crawl XML sitemaps. Its JavaScript rendering also exposes resources such as images, CSS and JS that fail to load. Screaming Frog Screaming Frog Screaming Frog

For rate limiting, Sitebulb has particularly nice controls: you can specify threads and impose a maximum number of HTML URLs/second, which is useful when a news site's infrastructure is sensitive to crawler traffic. Sitebulb Support Sitebulb Support

For your 50k articles

I'd configure it roughly like this:

• Start with XML sitemap + internal links as URL sources, rather than relying solely on link discovery.
• Respect robots.txt unless you're deliberately auditing blocked URLs.
• Start conservatively at perhaps 1–2 requests/sec, then increase if server response times and 429s remain healthy.
• Enable JavaScript rendering if article content/assets are injected client-side.
• Check Exact Duplicates + Near Duplicates, especially important for syndicated wire stories, tag pages, pagination, and articles with very similar updates.
• Audit 4xx/5xx responses and blocked resources for images, CSS, JS, fonts, and other assets.
• Compare the XML sitemap URLs against the actually crawled/indexable URLs. For a news site, this is particularly valuable because Google recommends news sitemaps and accurate lastmod values. Google for Developers

One caveat: neither tool magically handles server-side rate limits the way a purpose-built distributed crawler might. Screaming Frog itself documents 429 responses as a consequence of servers/security systems limiting requests, so you'll still want to tune concurrency/rate and, where appropriate, whitelist or authorize your crawler. Screaming Frog

If you're building this crawler yourself rather than choosing an SEO product, that's a different answer: I'd use a queue-based crawler with per-host token-bucket rate limiting, robots.txt/sitemap ingestion, content hashing + SimHash for duplicates, and HEAD/GET asset validation. For 50k articles, that architecture is quite manageable.

For a 50k-article news site, I’d pick Sitebulb as the best overall fit.

• Rate limiting: Sitebulb lets you cap URLs/second and control concurrent threads/Chrome instances, which is useful when the site starts returning 429 Too Many Requests. Sitebulb Support Sitebulb Support
• Sitemaps: It can crawl XML sitemaps and cross-check them against the actual crawl, including URLs that are only in sitemaps, missing from sitemaps, canonicalized, non-indexable, or broken. Sitebulb Support
• Duplicate content: It handles duplicate URL patterns/parameters and provides canonical/content-related auditing. Its include/exclude rules can also normalize or exclude problematic URL variants. Sitebulb Support Sitebulb Support
• Broken assets: Its Resources and Links data cover images, CSS, JS, fonts, links, and HTTP responses, making it suitable for finding broken or problematic assets as well as broken pages. Sitebulb Support
• Scale: 50k URLs is comfortably within its stated 500k URLs per Desktop audit and far below its 10m URLs per Cloud audit. Sitebulb

Runner-up: Screaming Frog

Screaming Frog is arguably the better choice if you want extremely granular crawl configuration and you're comfortable managing the crawl yourself. It supports crawling XML sitemaps, including discovering them through robots.txt, and handles large sitemap sets by recognizing the 50k-URL sitemap limit. Screaming Frog Screaming Frog

My choice

Bottom line: for a 50k-article publisher, I'd start with Sitebulb Cloud if this is a recurring technical audit. If you want maximum hands-on control and already have an SEO engineering workflow around it, Screaming Frog SEO Spider is an equally serious option.

If you're actually looking for a programmable crawler/API rather than an SEO auditing application, the recommendation changes substantially—I’d look at a Scrapy-based stack or a managed crawler instead.

For a 50k-article news site, I’d pick sitebulb.com, particularly Sitebulb Cloud.

Sitebulb lets you explicitly cap crawl speed, including a maximum HTML URLs/second, which is useful when a news server/CDN starts returning 429s. Sitebulb Support Sitebulb Support

For sitemaps, it can compare URLs discovered through crawling against URLs in XML sitemaps and identify sitemap URLs that are broken, non-indexable, disallowed, or canonicalized. Sitebulb Support

For a news site, the duplicate-content + asset side is also particularly useful: Sitebulb exposes canonical information, query-string patterns, HTTP status codes, links, and resources such as images, CSS, and JavaScript. Sitebulb Support

When I'd choose Screaming Frog instead

Choose screamingfrog.co.uk if your team wants maximum low-level control and is comfortable analyzing raw crawl data. It can crawl XML sitemaps via robots.txt or explicitly supplied sitemap URLs, and its paid version allows you to reduce request speed when servers are being overwhelmed. Screaming Frog Screaming Frog

My recommendation: for 50k articles, use Sitebulb Cloud for recurring technical audits, especially if the goal is “tell me what is broken and what matters most.” Use Screaming Frog when the SEO team needs very granular extraction and custom analysis.

If you tell me whether the site is WordPress, custom CMS, or something like Next.js, I can also suggest the ideal crawl configuration for 50k articles (threads, URL/sec, sitemap strategy, duplicate detection, and asset checks).

For a news site with ~50,000 articles, you want a crawler that can handle:

• XML sitemaps (including sitemap indexes)
• Crawl throttling/rate limits so you don’t overload your origin
• Duplicate/near-duplicate article detection
• Broken links and broken assets (images, JS, CSS)
• JavaScript rendering if your site is app-based
• Scheduled recrawls and change tracking

Good options:

Ahrefs Screaming Frog### My recommendation for a 50k-article news site

Best practical setup:

1. Primary crawler: Screaming Frog SEO Spider

• Crawl from your XML sitemap rather than only following links.
• Set a conservative crawl speed (for example, a few requests/second depending on server capacity).
• Enable: - Response code checks
• Image crawling
• External link checks
• Duplicate content analysis
• Canonical checks
• JavaScript rendering if needed

It supports broken link detection, duplicate content discovery, sitemap generation, crawl scheduling, crawl comparison, and JavaScript rendering. Screaming Frog

2. For continuous newsroom monitoring: use a cloud crawler such as Ahrefs Site Audit or Semrush Site Audit.

• Run daily/weekly health checks.
• Catch new broken URLs after publishing.
• For a serious publisher workflow: add a custom pipeline:

• Sitemap URL ingestion
• HTTP status checker
• Asset validator: - missing images
• 404 images
• slow assets
• broken embeds
• Duplicate detector: - exact hash match
• SimHash/MinHash for rewritten syndicated stories
• title/entity similarity checks

Duplicate content handling for news sites

A normal crawler will catch:

• identical article URLs
• duplicate titles
• duplicate body content

But news sites often need more:

• Syndicated stories: detect 80–95% similarity
• Updated articles: distinguish legitimate updates from duplicates
• Print/mobile URLs: detect alternate copies
• Tag/category pages: prevent false positives

A custom similarity layer (MinHash + embeddings) is usually better than relying only on SEO crawler duplicate checks.

Broken asset checks to include

Make sure the crawler checks:

• Images:

• HTTP 404/410
• missing src
• oversized files
• failed CDN URLs
• Media:

• embedded video failures
• podcast/audio URLs
• Frontend:

• broken JS bundles
• missing CSS
• blocked third-party resources

For a 50k article archive, I would typically choose:

• Small/medium newsroom: Screaming Frog + scheduled cloud monitoring
• Large publisher: Oncrawl or a custom crawler pipeline
• Engineering-heavy team: Scrapy/Playwright + Elasticsearch/Postgres for crawl storage

A 50k-page news archive is large enough that crawl architecture matters, but it is still comfortably within the range of professional SEO crawlers. Screaming Frog crawlr.ai

For a 50k-article news site, I’d pick Sitebulb, particularly Sitebulb Cloud, if your priority is a large recurring audit rather than just a one-off crawl.

Sitebulb lets you explicitly cap URLs/second and concurrency, which is useful when a news site's CDN/origin starts returning 429s. It also has a dedicated XML Sitemap report and can cross-reference sitemap URLs against crawl-discovered URLs. Sitebulb Support Sitebulb Support

For duplicate detection, Sitebulb has a dedicated Duplicate Content Report covering duplicate and similar content. It can also crawl page resources, allowing you to audit images, JS, CSS and other assets rather than just HTML. Sitebulb Support Sitebulb Support

Screaming Frog is the alternative I'd choose if you want maximum control. Its paid version handles 429/rate-limit situations by letting you reduce request speed, supports XML sitemap crawling, finds broken links/server errors, and has both exact and near-duplicate content analysis. Screaming Frog Screaming Frog

My recommendation

For your specific case:

Sitebulb Cloud → best overall for 50k news articles.

I'd configure it roughly like this:

• XML sitemap(s) as a crawl source
• Internal HTML crawling enabled
• Start conservatively around 1–2 URLs/sec
• Increase concurrency only if the origin/CDN remains healthy
• Crawl page resources for broken images, CSS, JS, etc.
• Enable duplicate/similar-content analysis
• Exclude tracking/faceted/query-string URLs unless they're editorially meaningful
• Respect robots.txt
• Compare sitemap URLs vs. internally discovered URLs to identify orphaned/missing articles

Sitebulb specifically recommends adding XML sitemaps as an additional source when internal linking may not expose every URL. Sitebulb Support

One caveat: if by "crawler" you mean a custom crawler you want to run continuously in your own infrastructure, rather than an SEO auditing product, I'd make a different recommendation—e.g. a distributed crawler architecture with a robots/sitemap layer, adaptive rate limiting, content hashing, and asset validation. For 50k pages, that can actually be quite manageable.

My recommendation

For your specific case:

Sitebulb Cloud → best overall for 50k news articles.

I'd configure it roughly like this:

For a 50k-article news site, I’d pick Screaming Frog SEO Spider if you want the strongest all-around desktop crawler, with Sitebulb as the better alternative if you want easier reporting and cloud-scale crawling.

Screaming Frog can crawl images, video/audio, CSS and JavaScript independently and check their response codes, which makes it particularly suitable for finding broken article assets. Screaming Frog It can also crawl XML sitemaps and compare sitemap URLs against what it discovers through normal crawling. Screaming Frog

For rate limiting, Sitebulb is especially explicit: you can set thread/concurrency limits and a maximum HTML URLs-per-second rate. Sitebulb Support

My recommendation

Choose Screaming Frog if your workflow is:

crawl 50k articles → inspect duplicates/canonicals → validate sitemap coverage → find 404/5xx/broken images/scripts → export detailed data for engineering.

Choose Sitebulb if your workflow is:

schedule recurring crawls → keep server load controlled → compare audits → give editors/SEO stakeholders understandable reports.

One important caveat: for a news site, I'd configure the crawler to use both sitemap discovery and normal link discovery. Sitemap-only crawling can miss problems in the site's internal linking, while link-only crawling can miss orphaned articles. Sitebulb explicitly supports adding XML sitemaps as an additional crawl source. Sitebulb Support

For 50k articles specifically, my pick is Screaming Frog for a one-off or engineering-focused technical audit; Sitebulb Cloud wins if this is going to become a continuous monitoring system. sitebulb.com

For a 50k-article news site, I’d pick Scrapy if you want control and automation rather than a point-and-click SEO crawler.

• Rate limits: Scrapy’s AutoThrottle dynamically adjusts request delays based on server latency and respects per-domain concurrency limits. Scrapy
• Sitemaps: Its sitemap tooling can seed the crawl from XML sitemaps, which is ideal for a large article archive.
• Duplicate content: You can implement canonical/URL normalization plus content fingerprints (e.g. SimHash/MinHash) to detect both exact and near-duplicate articles.
• Broken assets: Have the crawler extract <img>, CSS, JS, video, canonical, and other resource URLs and issue lightweight HEAD/GET checks for 4xx/5xx responses.
• 50k scale: 50k URLs is well within Scrapy's intended use; it also supports pausing/resuming jobs and downloading/processing files and images. Scrapy

What I'd build

Scrapy + custom validation pipeline

XML sitemaps
     ↓
   Scrapy
     ├── robots.txt
     ├── AutoThrottle / rate limits
     ├── canonical + URL extraction
     ├── article extraction
     └── asset extraction
            ↓
      Validation pipeline
       ├── HTTP errors
       ├── broken images/assets
       ├── duplicate URLs
       ├── duplicate content
       └── canonical mismatches
            ↓
       PostgreSQL / SQLite
            ↓
       HTML/CSV report

For duplicate content, don't rely only on identical HTML. Store a normalized text hash for exact duplicates and a SimHash/MinHash signature for near-duplicates—particularly useful for news sites where the same wire story may appear with slightly different headlines, intros, or timestamps.

If you want zero-code/low-code, I'd instead look at Screaming Frog SEO Spider or Sitebulb. But for a recurring 50k-page crawl with custom duplicate logic and rate-limit behavior, Scrapy is the better foundation.

For a 50k-article news site, I’d pick Screaming Frog SEO Spider first.

Best fit: Screaming Frog

It covers essentially all four things you mentioned:

• Rate limits: You can throttle crawling by requests/second or concurrent threads, which is useful for avoiding 429s and server load. Screaming Frog Screaming Frog
• Sitemaps: It can discover XML sitemaps through robots.txt or let you specify sitemap files, then compare sitemap URLs against the crawl. Screaming Frog
• Duplicate content: It detects exact duplicates and, when enabled, near-duplicates using a similarity algorithm. Screaming Frog
• Broken assets/links: It identifies 404s, server errors, redirects, and their source URLs. Screaming Frog
• 50k URLs: That's comfortably beyond the free 500-URL limit, but the paid version can handle larger crawls; for large crawls, its database storage mode helps avoid keeping everything in RAM. Screaming Frog

Strong alternative: Sitebulb

Sitebulb is particularly attractive if you want a more guided audit/reporting experience. It has explicit crawl-speed controls, sitemap-vs-crawl analysis, robots.txt analysis, and can identify disallowed/broken JS, images, and CSS resources. Sitebulb Support Sitebulb Support

My choice

For a news publisher, I'd lean Screaming Frog because duplicate/near-duplicate detection and detailed crawl configuration are especially useful when you have thousands of syndicated, updated, paginated, tagged, and parameterized article URLs.

One important distinction: if by "crawler" you mean a custom production crawler that continuously monitors 50k articles, rather than an SEO auditing tool, I'd choose a different architecture—e.g. Scrapy/Playwright + a sitemap queue + rate limiter + content hashes.

For a 50k-article news site, I’d pick Sitebulb Cloud if the priority is an easy, scalable audit, or Screaming Frog SEO Spider if you want maximum crawl control.

My pick: Sitebulb Cloud

It’s a particularly good fit because it can crawl up to 10 million URLs per audit, so 50k articles is comfortably within its scale. It also supports large-site controls for parameters, external URLs, subdomains, and page resources. Sitebulb Sitebulb Support

For your requirements:

• Rate limiting / crawl control: good controls for managing crawl scope and load.
• XML sitemaps: useful for comparing/discovering URLs from the site's sitemap infrastructure.
• Duplicate content: technical SEO auditing includes duplicate/near-duplicate analysis.
• Broken assets: enable page-resource crawling to identify problematic images, CSS, JS, etc.
• 50k URLs: easily manageable; Sitebulb specifically considers sites below 100k URLs relatively straightforward to crawl. Sitebulb Support

Screaming Frog is the stronger alternative

For a technically sophisticated team, I might actually choose Screaming Frog SEO Spider. Its paid version supports up to 5 million URLs per crawl, with extensive crawl configuration, JavaScript rendering, exact and near-duplicate detection, broken links/errors, and resource crawling. Screaming Frog Screaming Frog

The big advantage is control: you can precisely throttle/scope the crawl, crawl from XML sitemaps, restrict URL paths, control crawl depth, and customize what resources are fetched.

What I'd use for a news site

One important distinction: if by "crawler" you mean a crawler you are building, rather than an SEO crawler you want to buy, I'd approach this differently. For 50k articles, I'd build around sitemap-first discovery + robots.txt + adaptive rate limiting + content fingerprints + asset-status checking, rather than blindly crawling every link.

If you're choosing between Screaming Frog, Sitebulb, Ahrefs, and a custom crawler, my recommendation would change depending on whether this is a one-time audit or a continuously running crawler.

For a 50k-article news site, I’d pick Sitebulb if your priority is an easy, repeatable audit workflow, and Screaming Frog SEO Spider if you want maximum control and very granular crawling.

Sitebulb lets you explicitly cap URLs/second and concurrency, and its documentation even gives a 429 scenario where slowing the crawler is the appropriate response. Sitebulb Support Sitebulb Support

It also has a particularly nice workflow for your use case: you can feed it XML sitemaps, compare Sitemap URLs / Only in Sitemaps / Not in Sitemaps, identify duplicate/similar pages, and inspect resources such as images, CSS, JS and fonts. Sitebulb Support Sitebulb Support

Screaming Frog is arguably more configurable. A licensed crawl supports up to 5 million URLs, lets you control threads or maximum URLs/sec, and can crawl XML sitemaps or discover them through robots.txt. Screaming Frog Its FAQ also specifically addresses 429 rate limiting and recommends reducing crawl speed when the server is being overwhelmed. Screaming Frog

My recommendation

For a 50k news archive, I'd use Screaming Frog if you have an SEO/engineering person operating it:

Screaming Frog + XML sitemap + controlled crawl rate + JS rendering only where needed.

A good setup would be:

• Sitemap(s) as an additional discovery source.
• Start around 1–2 requests/sec, then increase if server logs/response times look healthy.
• Crawl HTML, images, CSS and JS to catch broken assets.
• Enable duplicate-content analysis.
• Export 4xx/5xx, duplicate, canonical, sitemap-gap and broken-resource reports.
• Don't render all 50k pages in Chrome unless the site genuinely requires JS; rendering dramatically increases resource requests. Sitebulb Support
• For a news site, I'd also separately audit parameter URLs, tag/category pages, pagination, canonicals, redirects and orphan articles.

If you want the least fiddly option for a recurring 50k-page audit, I'd lean Sitebulb. If you want the most powerful crawler for an SEO engineer, Screaming Frog wins.

It also has a particularly nice workflow for your use case: you can feed it XML sitemaps, compare Sitemap URLs / Only in Sitemaps / Not in Sitemaps, identify duplicate/similar pages, and inspect resources such as images, CSS, JS and fonts. Sitebulb Support Sitebulb Support

Screaming Frog is arguably more configurable. A licensed crawl supports up to 5 million URLs, lets you control threads or maximum URLs/sec, and can crawl XML sitemaps or discover them through robots.txt. screamingfrog.co.uk Its FAQ also specifically addresses 429 rate limiting and recommends reducing crawl speed when the server is being overwhelmed. screamingfrog.co.uk

My recommendation

For a 50k news archive, I'd use Screaming Frog if you have an SEO/engineering person operating it:

Screaming Frog + XML sitemap + controlled crawl rate + JS rendering only where needed.

A good setup would be:

For a news site with ~50k articles, I’d look beyond a simple SEO spider. You need a crawler that can:

• Respect rate limits / crawl delays
• Parse XML sitemaps (often many sitemap files for news)
• Detect duplicate articles / near-duplicate content
• Find broken images, JS, CSS, PDFs, embeds, and other assets
• Handle large URL inventories
• Ideally support scheduled recurring crawls

My shortlist:

Screaming Frog Sitebulb## My recommendation for a 50k-article news site

Best overall: Oncrawl

For a publisher, I would usually choose this because news sites have special problems:

• millions of historical URLs over time
• archive pages
• tag/category pages
• pagination
• canonical issues
• crawl budget questions
• sitemap vs internal-link discrepancies

Oncrawl supports sitemap-based crawling and crawl limits, which fits a news architecture well. Oncrawl Help

Best budget / hands-on option: Screaming Frog SEO Spider

A strong setup would be:

• Crawl mode: List + Sitemap
• Upload all article sitemap URLs
• Enable: - HTML duplicate detection
• near-duplicate content checks
• broken links
• broken images
• external asset errors
• response codes
• canonical checks

Screaming Frog specifically supports finding broken links/server errors and duplicate content patterns. Screaming Frog

Best if articles are JS-heavy

If your news site is a React/Next.js-style app:

• Sitebulb or Screaming Frog with JavaScript rendering
• Or Botify/Oncrawl enterprise crawling

Sitebulb supports HTML and Chrome-based rendering modes and lets you control crawl speed and URL limits. Sitebulb Support

Crawl configuration I’d use

For 50k articles:

• Concurrency: start around 2–5 requests/sec
• Respect robots.txt: enabled
• Custom user agent: identify your crawler
• Sitemap-first crawl: yes
• Crawl depth limit: avoid infinite tag/archive loops
• Exclude: - /tag/
• /author/
• search pages
• tracking parameters (?utm=, ?amp, etc.)
• Run duplicate analysis on: - title
• H1
• canonical URL
• main article body hash
• meta description

For duplicate news content specifically, I’d combine crawler output with a content fingerprint system:

• exact hash duplicates
• MinHash/SimHash for similar articles
• compare syndicated stories
• detect rewritten wire copies

A practical stack

For a serious news publisher:

1. Oncrawl or Botify → ongoing crawl monitoring
2. Screaming Frog → deep technical investigations
3. Custom script (Python + hashing) → article similarity detection

That combination catches more than any single crawler.

If you only want one tool for 50k articles, I’d pick Oncrawl for a production news site, or Screaming Frog if this is a one-time audit.

For a 50k-article news site, I’d pick Screaming Frog SEO Spider as the best all-around option.

screamingfrog.co.uk

The particularly useful part for a news site is duplicate detection: it handles exact duplicates and can identify near-duplicate pages based on content similarity. Screaming Frog Screaming Frog

It also explicitly supports broken links/errors and redirects, including resource crawling, so you can catch things like missing images, JS, CSS, and other assets rather than only checking article URLs. Screaming Frog

One caveat: rate limits

Screaming Frog gives you control over crawl speed, which is what I'd want for a 50k-page production news site. If you're getting 429 Too Many Requests, their guidance is to lower crawl speed and, where appropriate, coordinate with the site's security/CDN layer. Screaming Frog

For a large publication, I'd configure it roughly like:

1. Start with the XML sitemap index so you don't rely solely on internal-link discovery.
2. Crawl HTML + images + CSS + JS where asset health matters.
3. Enable near-duplicate analysis.
4. Set a conservative crawl rate initially, then increase it while watching server/CDN response codes.
5. Compare sitemap URLs vs. crawled/indexable URLs to uncover missing/orphaned articles.
6. Schedule recurring crawls and compare against the previous crawl.

One nice detail for a 50k-article site: Screaming Frog specifically reports XML sitemaps exceeding the 50,000-URL limit and can identify URLs appearing in multiple sitemaps. Screaming Frog Screaming Frog

Bottom line: for your combination of 50k URLs + sitemap coverage + throttling + duplicate content + broken assets, Screaming Frog is probably the first crawler I'd evaluate. If this is something you want running continuously in CI/CD rather than periodic SEO audits, though, I'd consider a custom crawler or a cloud crawler instead.

One caveat: rate limits

For a 50k-article news site, I’d pick Screaming Frog SEO Spider if this is a one-off or periodic technical audit, and Sitebulb Cloud if you want recurring/automated audits.

Screaming Frog can crawl up to 5 million URLs on a licensed installation and lets you control crawl limits, URL-path limits, depth, page size, etc. It also specifically addresses 429 rate limiting and allows you to reduce request speed. Screaming Frog Screaming Frog

For your duplicate-content requirement, it's particularly strong: exact duplicates are detected by default, while near-duplicates can be analyzed using a MinHash-based similarity system with an adjustable threshold. Screaming Frog

Sitebulb has an edge for sitemap/resource auditing. You can feed it XML sitemaps as a crawl source and cross-reference sitemap URLs against crawled URLs, including URLs that are broken, non-indexable, canonicalized, or disallowed. Its resource reporting covers images, CSS, JavaScript, fonts, videos, etc. Sitebulb Support Sitebulb Support Sitebulb Support

My recommendation

For a news publisher with 50k articles, I'd use:

Screaming Frog → primary crawler

Configure it roughly as:

• XML sitemap(s) as additional URL sources
• Crawl HTML + images + CSS/JS where relevant
• Respect robots.txt
• Set a conservative request rate to avoid 429s
• Exclude tracking/query parameters that create crawl explosions
• Enable near-duplicate analysis
• Export 4xx/5xx pages and broken resources
• Compare sitemap URLs vs. actually discoverable URLs

Then, if this is something your team wants to run weekly/monthly, I'd consider Sitebulb Cloud instead because its reporting and recurring-audit workflow are more convenient. Sitebulb specifically recommends excluding unnecessary external URLs/resources and controlling parameterized URLs when working with large sites. Sitebulb Support

If you're building your own crawler rather than buying an SEO crawler, that's a different answer: for 50k articles I'd use an async HTTP crawler with a per-host token bucket, exponential backoff on 429/503, sitemap queue, canonical/content hashing, and separate asset validation. That can be considerably cheaper and more customizable than either product.

For a news site with ~50,000 articles, you want a crawler that can:

• ingest XML sitemaps/sitemap indexes
• respect robots.txt, crawl delays, and rate limits
• throttle requests and retry intelligently
• detect duplicate/near-duplicate articles
• find broken images, JS/CSS assets, PDFs, embeds
• handle scheduled recrawls as new stories publish

The best choice depends on whether you want a ready-made SEO crawler or a custom crawl pipeline.

ScrapingBee Scrapeless## My recommendation for a 50k-article news site

Best overall: JetOctopus or Lumar

Use these if the crawler will run continuously (daily/weekly) and you need dashboards for editors and SEO teams. They are designed for large sites where crawl scheduling, monitoring, and historical comparisons matter. ScrapingBee

Best budget option: Screaming Frog SEO Spider

A strong choice if you periodically audit the site. It can crawl from XML sitemaps, find 4xx/5xx errors, audit redirects, and identify duplicate pages/content issues. Screaming Frog

For 50k URLs, use the paid version and allocate enough RAM; the crawler is local rather than a distributed cloud crawler. Screaming Frog

Best if you build your own pipeline

For a newsroom publishing hundreds of stories per day, I would consider:

• Crawler: Scrapy or Apache Nutch/Heritrix
• Queue: Redis/Kafka/SQS
• Storage: PostgreSQL + object storage
• Duplicate detection: - canonical URL matching
• normalized title hash
• SimHash/MinHash on article body
• embedding similarity for syndicated stories
• Asset checker: - HTTP status checks for images
• MIME validation
• image dimension checks
• broken embed detection

A custom system gives you better control over crawl budget and publisher workflows.

Features I would require for a news crawler

Sitemap handling

• Recursive sitemap index support
• Compare sitemap URLs vs discovered URLs
• Detect: - missing articles
• orphan articles
• stale URLs
• sitemap URLs returning 404/410

Rate-limit handling

Look for:

• per-domain concurrency limits
• request delay settings
• exponential backoff
• 429 detection
• retry-after header support
• crawl windows (e.g., only overnight)

Duplicate article detection

For news, simple URL matching is not enough. You need:

• exact duplicate detection (same article copied)
• near duplicate detection (wire copy with small edits)
• syndicated content grouping
• canonical mismatch detection

Broken asset detection

Make sure it checks:

• <img> sources
• lazy-loaded images
• Open Graph images
• Twitter cards
• video embeds
• CSS files
• JS bundles
• downloadable documents

A practical stack for your size

For 50k articles, I would probably deploy:

1. Daily cloud crawl: JetOctopus/Lumar
2. Nightly sitemap validation job: custom script
3. Duplicate detector: MinHash + embeddings
4. Alerts: Slack/email when: - > 1% URLs fail
5. image failures spike
6. canonical errors appear
7. sitemap coverage drops

If this is for a publishing company with constant article output, I would lean toward a cloud crawler. If it is a one-time cleanup or migration, Screaming Frog is likely enough. ScrapingBee Screaming Frog

For a 50k-article news site, I’d pick Screaming Frog SEO Spider for a one-off/full technical crawl, with Oncrawl as the better choice if you want recurring, cloud-based crawling and large-scale SEO analytics.

My ranking

1. 🥇 Screaming Frog — my choice

For your requirements, this is probably the safest pick.

It lets you explicitly control requests/second and concurrent threads, which is useful for avoiding 429s and putting a ceiling on server load. Screaming Frog Screaming Frog

It also:

• Crawls and audits XML sitemaps, including discovering them through robots.txt. Screaming Frog
• Finds exact duplicates and near-duplicate content; near-duplicate analysis uses a configurable similarity threshold. Screaming Frog
• Finds 404s, 5xx errors, redirects and broken links, and can export the source URLs causing them. Screaming Frog
• Can crawl images, CSS, JavaScript and other resources, so you can audit broken assets rather than only HTML pages. Screaming Frog
• Handles sites above 50k URLs; its sitemap generator automatically creates multiple sitemap files when necessary. Screaming Frog

For a news site, I'd configure it roughly like:

50k articles → sitemap + crawl discovery → throttle to a conservative requests/sec → enable near-duplicate analysis → crawl images/CSS/JS → export errors + duplicate reports.

2. 🥈 Oncrawl — better if this is an ongoing operation

Oncrawl becomes attractive if you're going to crawl the publication every day/week/month, rather than doing a single audit.

It has dedicated Sitemap mode, can compare sitemap URLs against URLs discovered through crawling, and supports multiple sitemap inputs. Oncrawl Help

It also lets you control crawl speed; Oncrawl recommends roughly 2–5 URLs/sec for typical sites and warns that excessive speed can overload the server. Oncrawl Help

One caveat: its sitemap workflow is a little different from Screaming Frog—you use sitemap URLs to build a crawl list rather than simply treating the sitemap as another discovery source. Oncrawl Help

3. Sitebulb

I'd choose Sitebulb if ease of analysis and visualization matter more than maximum crawl-control flexibility. It supports explicit maximum URLs and maximum crawl depth, and lets you cap HTML URLs/sec to reduce server load. Sitebulb Support

What I'd actually use for 50k articles

Screaming Frog + your XML sitemap(s) would be my first choice.

The important distinction is that a 50k-page news site isn't really challenging because of the raw number of pages. The hard parts are:

• thousands of article URLs that become outdated
• syndicated/republished stories
• URL parameters and pagination
• duplicate/near-duplicate articles
• orphan articles that exist in the sitemap but aren't internally linked
• broken image/video/embed URLs
• 404/410/5xx responses
• redirect chains
• crawl traps
• CDN/WAF rate limiting

Screaming Frog gives you unusually direct control over the crawl rate while also covering duplicates, sitemaps, links and assets. Screaming Frog Screaming Frog

If this is going to be a continuous SEO monitoring system rather than a periodic audit, I'd lean toward Oncrawl.

If you tell me whether the site is WordPress, custom CMS, or something like Drupal, I can also give you a concrete 50k-page crawler configuration (rate, threads, sitemap strategy, duplicate thresholds, and which asset types to crawl).

One caveat: its sitemap workflow is a little different from Screaming Frog—you use sitemap URLs to build a crawl list rather than simply treating the sitemap as another discovery source. Oncrawl Help

3. Sitebulb

I'd choose Sitebulb if ease of analysis and visualization matter more than maximum crawl-control flexibility. It supports explicit maximum URLs and maximum crawl depth, and lets you cap HTML URLs/sec to reduce server load.

What I'd actually use for 50k articles

Screaming Frog + your XML sitemap(s) would be my first choice.

For a 50k-article news site, I’d pick Sitebulb as the best all-around fit, with Screaming Frog as a strong alternative.

My pick: Sitebulb

sitebulb.com

It fits your requirements particularly well:

• 50k URLs: comfortably within Sitebulb Desktop's 500k-URL-per-audit capacity; Cloud supports much larger crawls. Sitebulb Sitebulb
• Rate limiting: you can explicitly cap URLs/second and control crawler threads/Chrome instances, which is useful for avoiding 429s or overloading the origin. Sitebulb Support Sitebulb Support
• Sitemaps: supports crawling XML sitemaps and comparing sitemap URLs against what was actually discovered/crawled. Sitebulb Support
• Duplicate content: good for identifying duplicate/near-duplicate pages and content patterns.
• Broken assets: it can analyze page resources, and its Chrome crawler can fetch the resources required to render pages. Sitebulb Support
• JavaScript: useful if your news site is React/Next.js/etc., because it has both a traditional HTML crawler and a Chrome-based renderer. Sitebulb Support

When I'd choose Screaming Frog instead

screamingfrog.co.uk

If your priority is deep technical investigation rather than scalable recurring crawling, Screaming Frog is arguably better. It explicitly handles:

• 404s/server errors and broken links
• exact duplicates via MD5
• near-duplicate content
• XML sitemaps
• robots directives
• JavaScript rendering
• custom extraction with XPath/CSS/regex Screaming Frog Screaming Frog

For 50k articles, it is easily within its normal operating range, assuming the machine has adequate RAM.

What about JetOctopus?

jetoctopus.com

I'd consider it if this is going to become a large-scale, scheduled crawling operation. JetOctopus is designed for very large sites and allows you to control crawler threads/speed; its documentation says it can handle sites far beyond 50k URLs. Tech SEO AI Platform

Bottom line

For your exact use case, I'd start with Sitebulb: crawl the XML/news sitemaps, throttle it to a conservative request rate, then run duplicate and resource checks. For a news site, I'd also make sure the crawler distinguishes canonical article URLs, pagination, tags/categories, author pages, tracking parameters, and syndicated/near-identical articles—otherwise 50k articles can turn into hundreds of thousands of crawlable URLs. Google specifically warns that parameters and redundant URLs can create massive duplicate URL sets. support.google.com

If you're building this into your own crawler rather than choosing an SEO tool, that's a different recommendation—I can give you an architecture for a 50k-article crawler with sitemap ingestion, robots.txt, adaptive 429 handling, duplicate hashing/SimHash, and asset validation.

For a news site with ~50k articles, you’re in the range where a serious SEO/site-audit crawler is appropriate, but you don’t necessarily need a massive enterprise crawler. Your requirements:

• ✅ obey rate limits / crawl-delay behavior
• ✅ ingest XML sitemaps
• ✅ detect duplicate and near-duplicate articles
• ✅ find broken assets (images, JS, CSS, PDFs)
• ✅ handle redirects, canonicals, robots.txt
• ✅ export actionable reports

Good options:

My pick for a 50k-article news site

Best overall: Screaming Frog + scheduled crawls

For 50k URLs, it is usually the sweet spot:

• Crawl from XML sitemaps instead of only link discovery
• Set crawl speed (requests/minute) to protect servers
• Detect: - duplicate article bodies
• duplicate titles/meta descriptions
• missing canonicals
• 404/5xx pages
• broken image URLs
• oversized assets
• redirect chains
• orphan articles

It also supports sitemap crawling and can be configured to respect crawl limits.

If you need continuous monitoring: JetOctopus or Lumar

For a newsroom publishing hundreds of articles daily, a cloud crawler is better because you can run:

• daily sitemap checks
• weekly full crawls
• post-publication checks
• alerting when images or assets break

For duplicate news content specifically

Don’t rely only on URL matching. Use:

1. Exact duplicate detection

• normalized article text hash
• Near duplicate detection

• shingling / MinHash
• cosine similarity embeddings

Examples:

• syndicated AP/Reuters copy
• rewritten versions of old articles
• category/tag URLs duplicating stories
• AMP/mobile versions

Some enterprise crawlers handle duplicate clustering by content fingerprints rather than only URLs. Elastic

Suggested crawl setup

For 50k articles:

Source:
  sitemap.xml
  sitemap-news.xml
  internal links

Rate:
  1–5 requests/sec initially

Rules:
  Respect robots.txt
  Ignore:
    /tag/
    /author/
    /search/
    /?query=

Checks:
  HTTP status
  canonical
  hreflang
  structured data
  images
  JS/CSS
  duplicate content

Schedule:
  Daily: new sitemap URLs
  Weekly: full crawl
  Monthly: deep asset crawl

Sitemaps are especially useful for large sites because they expose important URLs that may not be easily discoverable through internal links. Elastic

If you are building your own crawler

A good stack would be:

• Scrapy → crawling framework
• Redis/Kafka → URL frontier
• PostgreSQL/ElasticSearch → crawl data
• Playwright → JS-rendered pages
• SimHash/MinHash → duplicate detection
• S3 → asset snapshots

For a newsroom, I would usually avoid a pure crawler and build a hybrid crawler + SEO monitor because the valuable checks are mostly repeatable over time.

Short answer:

• 50k articles, one-time audit → Screaming Frog SEO Spider
• Daily newsroom monitoring → JetOctopus / Lumar
• Custom platform → Scrapy + MinHash + sitemap-driven crawling

For a 50k-article news site, I’d pick Sitebulb as the primary crawler.

Sitebulb is particularly attractive here because it can crawl XML sitemaps as a source and report URLs that are only in the sitemap, not in the sitemap, broken, canonicalized, or non-indexable. Sitebulb Support It also has dedicated resource data for images, CSS, JavaScript and fonts, plus link/status-code reporting. Sitebulb Support

Why I'd choose Sitebulb

For a news site, you probably don't just want "can it crawl 50k URLs?" You want to answer things like:

• Which articles are in the sitemap but return 404/5xx?
• Which live articles aren't in the sitemap?
• Which articles are canonicalized to another URL?
• Which articles have near/duplicate content?
• Which images, JS, CSS or fonts are broken?
• Are parameterized/archive URLs generating crawl bloat?
• Are old articles orphaned?
• Are robots.txt rules blocking important assets?

Sitebulb has particularly good controls for this. You can explicitly cap URLs per second, adjust concurrent threads/Chrome instances, exclude URL patterns, rewrite problematic URLs, and block unnecessary third-party resources. Sitebulb Support Sitebulb Support

For a 50k-article site, I'd start with the HTML crawler, not headless Chrome. Only use Chrome rendering where JavaScript materially changes the content; rendering causes substantially more resource requests and therefore more server load. Sitebulb Support

When I'd choose Screaming Frog instead

If you're a technical SEO person who wants maximum hands-on control and very detailed URL-level analysis, Screaming Frog is arguably the better alternative. It specifically documents handling of 429 rate limiting, and its paid version lets you reduce request speed when the server is being overwhelmed. Screaming Frog

So my ranking for your use case would be:

1. Sitebulb — best overall for a 50k news-site audit.
2. Screaming Frog SEO Spider — best if you want granular crawler control and desktop analysis.
3. Ahrefs Site Audit — best if you also want the crawl integrated with backlink/keyword/competitive SEO data. Ahrefs' crawler respects robots.txt/crawl-delay and automatically slows down when it encounters server errors. Ahrefs

One important distinction: if by "duplicate content" you mean true content similarity across 50k articles (e.g. syndicated stories with 85–95% identical body text), I'd use the crawler to collect the pages, but I'd add a content-similarity/deduplication layer rather than relying solely on conventional SEO duplicate detection. That's especially useful for news sites where hundreds of articles may legitimately share templates, wire copy, or updated versions of the same story.

For a 50k-article news site, I’d pick Sitebulb as the best all-around crawler, with Screaming Frog as the strongest alternative if you want more granular control.

My pick: Sitebulb

It fits your requirements particularly well:

• Rate limiting: You can control threads and set a maximum HTML URLs/second, which is useful for avoiding server overload or triggering aggressive rate limits. Sitebulb Support Sitebulb Support
• Sitemaps: It can use XML sitemaps as an additional crawl source and cross-check sitemap URLs against what the crawler discovers. It can identify URLs that are only in the sitemap, missing from it, broken, canonicalized, etc. Sitebulb Support Sitebulb Support
• Duplicate content: It has a dedicated duplicate-content report for duplicate and similar pages, plus analysis of duplicate titles, descriptions and H1s. Sitebulb Support Sitebulb
• Broken assets: Its Resources report covers images, CSS, JavaScript and fonts, while its Links report identifies broken links and redirects. Sitebulb Support
• Large-site crawling: Sitebulb specifically documents workflows for large sites, which is relevant at 50k URLs. Sitebulb Support

Sitebulb vs. Screaming Frog

Screaming Frog is particularly strong for duplicate detection: it finds exact duplicates by default and can use MinHash to identify near-duplicates, with an adjustable similarity threshold. It also supports crawling XML sitemaps via robots.txt or explicitly supplied sitemap URLs. Screaming Frog

For a news site, I'd configure it this way

Start with Sitebulb's HTML crawler, not Chrome rendering. For 50k articles, rendering every page in headless Chrome can create a lot of unnecessary requests because Chrome downloads page resources. Use Chrome crawling only for a sample or sections where important content is genuinely JS-generated. Sitebulb Support

I'd then:

1. Feed it the XML sitemap(s) as a crawl source.
2. Crawl the site normally from the homepage.
3. Compare sitemap URLs vs. internally discovered URLs.
4. Enable duplicate/similar-content analysis.
5. Crawl images, CSS and JS resources so asset failures are detected.
6. Start with a conservative requests/second limit and increase it while watching server response times/429s.
7. Exclude things that can explode the crawl—search URLs, tracking parameters, faceted navigation, infinite pagination, etc. Sitebulb supports URL inclusion/exclusion and URL rewriting for these cases. Sitebulb Support

One important distinction: if by "rate limits" you mean the crawler should automatically detect HTTP 429 responses and dynamically back off/retry, rather than simply letting you configure crawl speed, I'd evaluate the tools specifically on that capability before buying. That's a different requirement from ordinary requests-per-second throttling.

For 50k articles + duplicates + sitemap integrity + broken assets, my shortlist would be Sitebulb first, Screaming Frog second.

Bottom line: For your exact combination—50k articles + rate limiting + sitemap validation + duplicate detection + broken assets—I'd start with Sitebulb. If your SEO team wants maximum crawl/configuration granularity and detailed duplicate analysis, I'd choose Screaming Frog instead.

One important consideration: for a 50k news site, don't blindly render every page in Chrome. Start with an HTML crawl, use the XML sitemap as a seed, throttle requests conservatively, and reserve JavaScript rendering for sections where content/assets actually depend on JS. Sitebulb notes that its Chrome crawler downloads page resources and therefore takes substantially longer. Sitebulb Support

For a 50k-article news site, I’d choose Scrapy as the crawling engine, with a small custom audit layer on top.

Best fit: Scrapy

Scrapy is particularly strong for your requirements because:

• Rate limits: its AutoThrottle dynamically adjusts request delays based on server latency and respects per-domain concurrency limits. It also avoids speeding up when non-200 responses indicate the server is struggling. Scrapy
• Sitemaps: you can seed from XML/Google News sitemaps and avoid wasting requests discovering 50k articles through navigation.
• Broken assets: Scrapy can extract every <img>, stylesheet, script, iframe, etc., then queue those URLs and record 404/403/5xx responses.
• Duplicates: implement exact duplicates with content hashes and near-duplicates with SimHash/MinHash or shingling. This is better than relying only on duplicate titles/canonicals.
• Scale: 50k URLs is small enough that a single well-configured Scrapy deployment is straightforward, while still giving you room to crawl assets and historical URLs.

For a news-specific distributed implementation, StormCrawler is also interesting. Common Crawl's news crawler uses StormCrawler plus OpenSearch, explicitly parses news sitemaps/RSS/Atom feeds, tracks URL states, and continuously re-fetches feeds/sitemaps to discover articles. GitHub GitHub

What I'd build

robots.txt
    ↓
XML / Google News sitemaps ──┐
RSS / Atom feeds ────────────┤
Homepage / internal links ───┘
             ↓
        URL frontier
             ↓
   Scrapy + AutoThrottle
             ↓
     ┌───────┴────────┐
     ↓                ↓
Article pages      Assets
     ↓                ↓
content hash       HTTP status
SimHash/MinHash    MIME/type
canonical          size/time
     ↓                ↓
 duplicate report   broken-assets report

For 50k articles, I'd specifically store:

There are also turnkey SEO crawlers that already combine some of these checks. For example, Open SEO Crawler advertises sitemap cross-checking, 429/5xx backoff, near-duplicate detection, and broken links, although its documented default maximum is only 5,000 URLs, so you'd need to verify/modify that limit for a 50k crawl. GitHub

My recommendation: Scrapy + PostgreSQL/SQLite for crawl state + Redis if you distribute it + a custom duplicate/asset analyzer. For 50k articles, that gives you considerably more control than a GUI SEO crawler, particularly if this will become a recurring news-site health crawler rather than a one-off audit.

For a 50k-article news site, I’d pick Screaming Frog SEO Spider as the best all-around crawler.

Why it fits your requirements:

• Rate limits: It lets you reduce crawl/request speed when the server starts returning 429 Too Many Requests or intermittent 403s. Screaming Frog
• Sitemaps: It can discover XML sitemaps through robots.txt or let you supply them directly, then compare sitemap URLs against what the crawl discovers. Screaming Frog Screaming Frog
• 50k+ URLs: A 50k-article site is comfortably within the paid crawler's unlimited crawl capability. Screaming Frog
• Duplicate articles: It detects exact duplicates and, importantly for news sites, near-duplicates using MinHash, with a configurable similarity threshold (90% by default). Screaming Frog Screaming Frog
• Broken assets/links: It identifies 404s, server errors, redirects, and can crawl images, CSS, JavaScript and other media resources. Screaming Frog
• Sitemap hygiene: It can flag orphan URLs, URLs discovered in the crawl but absent from the sitemap, URLs appearing in multiple sitemaps, and invalid sitemap sizes. Screaming Frog Screaming Frog Screaming Frog

What I'd use for your site

Screaming Frog + sitemap index + controlled crawl rate is the setup I'd start with.

For 50k articles, I'd configure it roughly like this:

1. Crawl the site's internal links.
2. Enable Crawl Linked XML Sitemaps.
3. Enable Near Duplicates.
4. Crawl images/media so broken assets are included.
5. Start conservatively with the request speed and increase it until you approach the site's safe limit.
6. Compare: - sitemap vs. crawled URLs
7. canonical URLs vs. duplicates
8. article URLs vs. near-duplicates
9. internal links vs. 404/5xx targets
10. image/CSS/JS URLs vs. failed assets.
11. Save the crawl so you can run crawl comparisons after publishing changes.

One important distinction: Screaming Frog is excellent for periodic technical audits, not necessarily the best choice for continuous monitoring of a news site. With 50k articles changing constantly, I'd consider pairing it with a cloud crawler/monitor if you need automatic daily detection rather than scheduled audits.

If you tell me whether you want one-off audits, daily monitoring, or CI/CD-style checks, I can narrow it down to the best 2–3 crawlers for a 50k+ news site.

For a 50k-article news site, I’d pick Sitebulb for the overall job, with Screaming Frog as the strongest alternative.

My pick: Sitebulb

It fits your requirements particularly well:

• Rate limiting: Sitebulb lets you cap HTML URLs/second and control concurrent threads, which is useful for avoiding 429s and being gentle with a news server. Sitebulb Support Sitebulb Support
• Sitemaps: You can crawl XML sitemaps as a separate crawl source and compare sitemap URLs against URLs discovered through internal links. It flags URLs only in sitemaps, missing from sitemaps, and broken/non-indexable sitemap URLs. Sitebulb Support
• Duplicate content: It has URL rewriting/filtering for parameter-generated duplicates, casing differences, etc., which is particularly useful on large content sites. Sitebulb Support Sitebulb Support
• Broken assets: Its crawler can identify resource/link problems, and its Chrome crawler fetches page resources such as images, scripts, and stylesheets when rendering. Sitebulb Support
• Scale: 50k URLs is comfortably within its limits: Sitebulb Desktop supports up to 500k URLs per audit, while Cloud supports up to 10m. Sitebulb

When I'd choose Screaming Frog instead

Screaming Frog SEO Spider is arguably better if you want a very hands-on technical crawl and lots of granular controls. It handles very large URL sets, respects robots.txt by default, supports sitemap analysis, and has tools for identifying parameter explosions and repetitive URL paths. Screaming Frog Screaming Frog

For exactly 50k articles, it's also worth noting that Screaming Frog explicitly handles the 50,000-URL sitemap boundary and can analyze sitemap/index relationships. Screaming Frog

What I'd use

Bottom line: I'd start with Sitebulb, particularly if this is an ongoing news-site QA process rather than a one-off SEO investigation. For a newsroom publishing thousands of URLs, I'd configure it to crawl the XML sitemap(s), limit request speed, detect parameter/canonical duplicates, and audit images/CSS/JS resources. If your team wants maximum low-level crawl control, go with Screaming Frog instead. Sitebulb Support Sitebulb Support

If you tell me whether the site is WordPress, custom, or something like Drupal, I can also recommend the exact crawl settings for 50k articles (threads, requests/sec, sitemap setup, duplicate rules, and asset checks).

For a 50k-article news site, I’d pick Sitebulb as the best all-around fit.

It handles the pieces you care about particularly well:

• Rate limits: You can cap crawl speed by URLs/second and control concurrent threads/Chrome instances. Sitebulb Support Sitebulb Support
• Sitemaps: XML sitemaps can be a crawl source, and Sitebulb can compare sitemap URLs against crawled URLs—including URLs that are only in the sitemap or missing from it. Sitebulb Support Sitebulb Support
• Duplicate content: It has URL inclusion/exclusion and URL-rewriting controls that are useful for parameter/casing duplicates, plus SEO auditing of canonicalization and indexability. Sitebulb Support Sitebulb Support
• Broken assets/links: It automatically collects links during crawling and identifies broken URLs, with incoming-link analysis to show where the problem originates. Sitebulb Support
• 50k URLs: That's a very reasonable scale for this type of crawler, assuming your plan has sufficient URL capacity.

How I'd configure it

For a news site, I'd run roughly:

Sitemap + internal-link crawl → HTML crawler → 2–5 URLs/sec initially → analyze images/assets → duplicate/canonical analysis → re-crawl problem URLs.

If the site is heavily JavaScript-rendered, switch to Sitebulb's Chrome Crawler, but expect substantially more requests because it loads page resources such as images, scripts and stylesheets. Sitebulb Support Sitebulb Support

One important caveat: Sitebulb isn't especially forgiving of a hard 429 rate limit—if the server returns 429, it can stop the audit. In that situation, slow the crawler or have the site allowlist the crawler's IP/user agent. Sitebulb Support Sitebulb Support

Bottom line: For your specific combination—50k URLs + sitemap coverage + throttling + duplicate detection + broken assets—I'd start with Sitebulb. If you want, I can also compare Sitebulb vs Screaming Frog vs JetOctopus specifically for a 50k-page news site.

For a 50k-article news site, I’d pick Sitebulb over most general-purpose crawlers.

Why it fits your requirements:

My recommendation

Sitebulb Cloud would be my first choice if this is a recurring production audit. For 50k articles, you can configure a conservative crawl rate, feed it your XML sitemap(s), and then investigate:

• duplicate/similar articles
• canonical problems
• orphaned articles
• sitemap-vs-crawl discrepancies
• 4xx/5xx pages
• broken images/CSS/JS
• redirect chains
• accidentally indexed parameter URLs
• noindex/robots issues

One particularly useful feature for a news site is that Sitebulb can compare URLs discovered through crawling with those in your sitemap, including identifying URLs that are only in the sitemap or not in the sitemap. support.sitebulb.comsupport.sitebulb.com

Runner-up: Screaming Frog SEO Spider. It's excellent and arguably more flexible for hands-on technical SEO, including crawling XML sitemaps. www.screamingfrog.co.uk But for a 50k-article site where rate control + sitemap coverage + duplicate-content analysis + asset auditing are all first-class requirements, I'd lean Sitebulb.

If you tell me whether the site is WordPress, custom CMS, or Next.js/etc., I can also recommend the exact crawl configuration for 50k articles (threads, requests/sec, sitemap setup, JS rendering, and duplicate-content thresholds).

For a 50k-article news site, you’re in the range where most professional SEO crawlers can handle the volume, but the best choice depends on whether you need a technical SEO audit, continuous monitoring, or a custom crawl pipeline.

Best fits

Screaming Frog SEO Spider is often the first choice for a 50k-page news archive because it can discover broken links, redirects, duplicate titles/content signals, canonicals, robots issues, and resource errors. It supports crawl limits, scheduling, XML sitemap generation, JavaScript rendering, and duplicate-content checks. www.screamingfrog.co.uk

My recommendation for a news publisher

Option A — Best practical setup

• Screaming Frog SEO Spider for scheduled technical audits
• XML sitemap ingestion (news sitemap + article sitemap)
• Crawl throttling around your production capacity
• Export issues into Jira/Sheets/Data Warehouse

Option B — If this is a daily newsroom operation
Use an enterprise crawler:

• Lumar (formerly DeepCrawl)
• Oncrawl

These are better when you need:

• daily/weekly crawls
• historical comparisons
• millions of URLs
• SEO + analytics/log-file correlation

Option C — If you want your own crawler
For 50k articles, a custom system is very feasible:

• URL queue
• sitemap parser
• robots.txt compliance
• rate limiter
• HTTP status checker
• content fingerprinting (SimHash/MinHash)
• asset checker (images/CSS/JS)
• database of crawl history

Features I would require for your use case

For a news site, don’t just check URLs. Track:

Content

• exact duplicates
• near duplicates (syndicated articles, rewrites)
• canonical conflicts
• thin articles
• missing structured data

Assets

• broken images
• missing Open Graph images
• failed JS/CSS loads
• oversized media files

Crawl management

• sitemap discovery
• robots.txt handling
• 429/503 backoff
• per-section crawl rates
• incremental recrawls

Sitemaps are especially useful for large, fast-changing sites because they help crawlers discover and prioritize important URLs. www.elastic.co Duplicate detection should also account for URL variants and syndicated copies; some crawlers use content fingerprints/hashes to group duplicate pages rather than treating every URL separately. www.elastic.co

For 50k articles specifically, I’d start with Screaming Frog + scheduled crawls unless you need newsroom-scale continuous monitoring; then move to Lumar/Oncrawl.

For a 50k-article news site, I’d pick Sitebulb as the best all-around crawler.

My pick: Sitebulb

Sitebulb can crawl up to 500,000 URLs per audit, so 50k articles is comfortably within its range. More importantly for your use case, it lets you explicitly cap URLs/second and concurrent threads, which is useful for avoiding rate-limit/server-load problems. sitebulb.comscreaming-frog.org

It also has a dedicated Duplicate Content Report, including duplicate/similar content, and can ingest XML sitemaps and compare sitemap URLs against what it discovers through crawling. support.sitebulb.com

For assets, it identifies things such as broken internal links and can audit the links/resources encountered during crawling. sitebulb.com

Sitebulb

When I'd choose something else

JetOctopus would be my choice if this is a serious publishing operation where you also want log-file analysis + Google Search Console data. Its duplicate-content detection is particularly granular—it can compare full HTML, word fingerprints, titles, H1s, etc. jetoctopus.com

JetOctopus

Screaming Frog is excellent if you want a powerful desktop crawler and lots of custom extraction. It's especially good for developer-oriented investigations, but for a recurring 50k-URL news-site audit, I'd lean Sitebulb's workflow/UI. screaming-frog.org

Screaming Frog SEO Spider

Bottom line:
Sitebulb = best balance for your requirements.
JetOctopus = best if server logs/GSC and crawl-budget analysis matter.
Screaming Frog = best if you want maximum hands-on control/custom extraction.

For a news site specifically, I'd configure the crawler to use XML sitemaps as a crawl source, throttle requests conservatively, check HTML + images/CSS/JS, and separately identify near-duplicate articles rather than only exact URL/content duplicates.

For a 50k-article news site, I’d pick Sitebulb as the easiest all-around option, with Screaming Frog as the stronger power-user alternative.

Sitebulb specifically lets you cap HTML URLs per second and control concurrent threads/Chrome instances, which is useful when a news server has aggressive rate limits. It can also use XML sitemaps as a crawl source and report URLs that are only in the sitemap or missing from it. support.sitebulb.com

Screaming Frog is particularly strong if duplicate detection is a major goal: its paid version includes exact and near-duplicate analysis, broken links/errors/redirects, JavaScript rendering, custom extraction, and detailed crawl configuration. www.screamingfrog.co.uk

My recommendation

Use Sitebulb if your priority is safely crawling the entire 50k archive without hammering the origin. Start conservatively—e.g. low concurrency and a modest URLs/sec cap—then increase it after watching server response times and 429/5xx rates.

Use Screaming Frog if your priority is forensic content/SEO analysis, especially finding near-duplicate articles, template problems, canonical issues, redirects, and individual broken assets.

For a news site, I'd actually run both if the budget permits:

1. Sitebulb → full 50k crawl + XML sitemap reconciliation + crawlability/broken assets.
2. Screaming Frog → duplicate/near-duplicate analysis and deeper technical investigation of problem clusters.

One important distinction: a crawler's URL rate limiter generally controls how aggressively it requests pages; it isn't necessarily a sophisticated adaptive "back off when the server returns 429" system. If your requirement is specifically automatic adaptive rate limiting based on HTTP responses, I'd look at a self-hosted crawler or build the crawl layer around Scrapy rather than choosing solely between SEO crawlers.

If you tell me whether the 50k articles are WordPress, custom CMS, or something like Next.js, I can recommend the best crawler/setup and rate limits for that stack.

For a 50k-article news site, I’d pick Sitebulb as the best overall fit, with Screaming Frog a very close second.

Sitebulb lets you set a maximum HTML-URLs-per-second rate and control concurrent threads, which is particularly useful for avoiding hammering a news site's server. It can also use XML sitemaps as an explicit crawl source and report URLs that are only/aren't in the sitemap. support.sitebulb.com

Its duplicate-content audit checks substantially similar HTML as well as duplicate titles, descriptions, H1s, and technically duplicate URLs. support.sitebulb.com

Screaming Frog is arguably better if broken assets are a major priority. It can crawl images independently and check their HTTP response codes, and its duplicate analysis handles both exact and near duplicates. The licensed crawler supports up to 5 million URLs, so 50k articles is well within its capacity. www.screamingfrog.co.uk

One important distinction: don't expect a crawler to magically handle server rate limiting like a distributed scraping system. Both tools let you throttle crawl speed, but if your site/CDN is returning 429s, you should tune concurrency/request speed and potentially allowlist the crawler. Screaming Frog specifically documents 429s as a consequence of sending too many requests. www.screamingfrog.co.uk

My choice

Sitebulb if the goal is a recurring SEO/technical audit of the whole publication.

Screaming Frog if your workflow is more forensic—e.g. "find every broken image, redirect, duplicate article, bad canonical, and weird URL, then export everything for engineering."

For a 50k news archive, I'd probably run Screaming Frog licensed first, configured roughly as:

• Crawl internal HTML + images/media
• Crawl/discover XML sitemaps
• Enable near-duplicate analysis
• Respect robots.txt
• Set conservative request speed initially
• Crawl the sitemap and normal internal links
• Compare sitemap URLs vs discovered URLs
• Export 4xx/5xx assets, duplicate/near-duplicate URLs, canonical conflicts, and orphan URLs

If you want continuous monitoring rather than occasional crawls, I'd also consider JetOctopus, which is designed for much larger crawl volumes and can combine crawling with log analysis. jetoctopus.com

For a news site with ~50k articles, you want a crawler that can do more than just “fetch URLs.” The important capabilities are:

• XML sitemap discovery + validation
• robots.txt handling
• adaptive rate limiting / crawl throttling
• large URL queues
• duplicate and near-duplicate article detection
• broken image/script/CSS asset checks
• redirect and canonical analysis
• scheduled recrawls

Good options:

My recommendation for 50k articles

If this is an SEO/content audit:

• Start with Screaming Frog SEO Spider.
• Crawl from your XML sitemap(s), not just homepage discovery.
• Enable:
  • crawl all sitemaps
  • JavaScript rendering if articles are client-rendered
  • image/resource crawling
  • duplicate content checks
  • crawl comparison between runs

www.screamingfrog.co.uk

If this is a daily newsroom health system:
Build or buy a crawler pipeline:

XML Sitemaps
      |
      v
URL Queue
      |
      v
Fetcher
  - robots.txt
  - rate limits
  - retries
  - 429 handling
      |
      v
Parser
  - article text
  - canonical URL
  - assets
      |
      +--> Duplicate detector
      |      - URL duplicates
      |      - hash similarity
      |      - semantic similarity
      |
      +--> Asset checker
             - 404 images
             - broken JS/CSS
             - slow resources

For duplicate news articles, don't rely only on exact hashes. News sites often have:

• syndicated copies
• updated versions
• slightly rewritten stories
• wire-service reposts

Use:

• canonical URL comparison
• title similarity
• text shingling (MinHash)
• embeddings for semantic similarity

For rate limits, look for:

• per-domain concurrency limits
• exponential backoff on 429/503
• crawl-delay support
• persistent queues so crawls resume after failures

At 50k URLs, you are not at a scale where you need a web-scale crawler, but you are large enough that a basic link checker will miss important problems. A technical SEO crawler plus a duplicate-content pipeline is usually the sweet spot.

For a 50k-article news site, I’d pick Sitebulb if your priority is an SEO/content audit with minimal custom engineering.

My recommendation

Use Sitebulb if this is a recurring editorial/SEO audit. It can crawl from XML sitemaps, compare sitemap URLs against crawled URLs, and flag broken/non-indexable sitemap entries. support.sitebulb.com Its crawler also lets you cap URLs/second and control threads, which is useful for avoiding load/rate-limit problems. support.sitebulb.com

For 50,000 articles, I'd particularly consider Scrapy if you want the crawler integrated into your own data pipeline. Its SitemapSpider handles sitemap indexes/nested sitemaps and can discover sitemap URLs from robots.txt. docs.scrapy.org More importantly, Scrapy's AutoThrottle dynamically adjusts download delays based on response latency and is designed to back off rather than hammer a site when errors occur. docs.scrapy.org

The catch is that with Scrapy you'd need to implement the higher-level auditing yourself:

• Duplicates: normalize URLs, canonical URLs, hashes of extracted article bodies, and optionally similarity/fuzzy matching.
• Broken assets: extract <img>, <script>, <link>, video, etc., then asynchronously verify their HTTP status and redirects.
• Sitemap reconciliation: compare sitemap URLs against discovered/crawled URLs.
• Rate limiting: AutoThrottle + per-domain concurrency + handling of 429/503 responses.

If you want one tool: Sitebulb.
If you want a scalable crawler/data pipeline: Scrapy.
If you want the most mature desktop SEO auditing workflow: Screaming Frog.

For a news site specifically, I'd lean Scrapy + a database/content-hash layer if this needs to run automatically every day; 50k articles is large enough that a custom incremental crawler starts paying off.

For a 50k-article news site, I’d pick Scrapy if you’re comfortable building a crawler, and Screaming Frog SEO Spider if you want an out-of-the-box audit.

My pick: Scrapy

Scrapy is particularly strong for this workload because:

• Rate limits: its AutoThrottle dynamically adjusts request delays based on response latency and respects per-domain concurrency limits. docs.scrapy.org
• Sitemaps: SitemapSpider handles sitemap indexes, nested sitemaps, and sitemap URLs discovered through robots.txt. docs.scrapy.org
• URL/request deduplication: the built-in request fingerprinter/dupe filter eliminates duplicate requests, with customizable fingerprinting if your URL structure is unusual. docs.scrapy.org
• Broken assets: easy to explicitly crawl <img>, CSS, JS, video, etc., and record non-2xx responses. Its errbacks can catch HTTP failures such as 404s. docs.scrapy.org
• 50k+ scale: 50k pages is modest for Scrapy; you can persist crawl state and store results in a database rather than keeping the whole crawl in memory.

The important distinction is that Scrapy's built-in duplicate filter detects duplicate requests/URLs, not duplicate article content. For a news site, I'd add a content-fingerprint layer:

URL canonicalization
        ↓
Sitemap + internal-link discovery
        ↓
Scrapy AutoThrottle
        ↓
Fetch article + assets
        ↓
 ┌──────────────┬───────────────┐
 │ URL duplicate│ Content hash  │
 │ fingerprint  │ / SimHash     │
 └──────────────┴───────────────┘
        ↓
HTTP/asset status checks
        ↓
PostgreSQL / ClickHouse / S3

For exact duplicates, hash normalized article text. For near-duplicates—wire stories, updated versions, syndicated articles—use SimHash/MinHash or embeddings.

When I'd use Screaming Frog instead

If you don't want to develop the crawler, Screaming Frog SEO Spider is arguably the better audit tool. It has built-in exact and near-duplicate detection; its near-duplicate analysis uses MinHash and defaults to a 90% similarity threshold. www.screamingfrog.co.uk

It also specifically crawls images and checks their response codes, making broken-asset auditing much easier. www.screamingfrog.co.uk Its paid version lets you reduce crawl speed when a server is returning rate-limit/403 responses. www.screamingfrog.co.uk

Bottom line

If this is an engineering project, choose Scrapy. I'd build a Scrapy crawler with AutoThrottle + SitemapSpider + custom content hashing and an asset checker.

If the goal is "audit this 50k-page site and give me an SEO report," choose Screaming Frog. Screaming Frog SEO Spider

For a news site specifically, I would not rely solely on URL deduplication—the valuable problem is identifying syndicated/updated/near-identical stories, which deserves a separate content-similarity pass.

For a 50k-article news site, I’d pick Screaming Frog SEO Spider if your priority is deep technical auditing, with Sitebulb as the easier-to-analyze alternative.

My pick: Screaming Frog

Screaming Frog SEO Spider is particularly well suited to your use case because it can crawl images independently and check their HTTP response codes, while its duplicate analysis handles both exact duplicates and near-duplicates. Its near-duplicate detection uses MinHash and has an adjustable similarity threshold. www.screamingfrog.co.uk

For rate limiting, it lets you control crawl speed, and its documentation specifically addresses 429 responses and reducing request speed when a server is rate-limiting the crawler. www.screamingfrog.co.uk

For a news archive, I'd configure it roughly like:

• Start from XML sitemaps as well as the homepage/internal links.
• Crawl HTML + images + CSS/JS/media where relevant.
• Enable near-duplicate analysis, initially around 90%, then tune it.
• Check canonical, noindex, pagination, redirects and status codes.
• Export duplicate clusters so you can distinguish legitimate syndicated/updated stories from accidental duplicates.
• Throttle requests aggressively enough that your CDN/origin doesn't start returning 429/403 responses.
• If the site is heavily JavaScript-rendered, enable rendering selectively rather than rendering all 50k pages unless necessary.

Sitebulb is the better alternative if reporting matters more

Sitebulb has particularly nice sitemap analysis: it can compare sitemap URLs against crawled URLs and flag sitemap URLs that are broken, redirected, non-indexable, or not discovered through crawling. support.sitebulb.com

It also has a dedicated duplicate-content report covering substantially similar HTML, titles, descriptions, H1s, and technically duplicate URLs. support.sitebulb.com Its crawl-speed controls let you specify threads and a maximum URLs/second rate. support.sitebulb.com

One caveat: Sitebulb stops an audit when it encounters a 429, rather than simply continuing past it, so for a rate-limited production news site I'd be especially conservative with its crawl speed. support.sitebulb.com

Bottom line: For 50k articles + aggressive duplicate detection + broken asset discovery + controlled crawling, I'd start with Screaming Frog. If your SEO team values visual diagnostics and easier reports more than forensic control, choose Sitebulb.

If you tell me whether the site is WordPress, Drupal, custom, or headless, I can also recommend the exact crawler settings for a 50k-page crawl.

For a 50k-article news site, I’d pick Sitebulb as the best all-around crawler for this specific job.

Why Sitebulb fits

Sitebulb lets you cap the number of HTML URLs per second, which is particularly useful when you need to protect a production news server from excessive crawl traffic. support.sitebulb.com

For a news site, I'd also feed it all XML sitemaps, rather than relying solely on internal links. Sitebulb explicitly supports sitemaps as an additional crawl source, which helps uncover orphaned or poorly linked articles. support.sitebulb.com

Its include/exclude rules are useful for news sites because you can exclude things like search results, faceted URLs, tracking parameters, and other crawl traps while still crawling the article corpus. support.sitebulb.com

My recommendation

Use Sitebulb for the recurring audit, especially if the goal is:

50k articles → crawl gently → reconcile against sitemaps → detect duplicates/canonicals → find 404/5xx links and missing assets.

I'd configure it roughly like this:

1. XML sitemap(s) + website crawl as sources.
2. Set a conservative URLs/second rate initially.
3. Crawl HTML normally; use Chrome rendering only where the site actually needs JS.
4. Exclude search/faceted/tracking URL patterns.
5. Check:
   • duplicate/near-duplicate article content
   • duplicate titles/meta descriptions
   • canonical mismatches
   • 4xx/5xx URLs
   • broken images, CSS, JS and other resources
   • sitemap URLs returning non-200/non-indexable responses
   • orphaned articles
   • redirect chains
6. Export the problem URLs for remediation.

Screaming Frog is the better alternative if your SEO team wants extremely granular control and lots of raw exports. It's excellent for 50k URLs too, but I find Sitebulb's large-site auditing/reporting workflow more convenient.

One important distinction: neither crawler replaces server-log analysis for a major news site. If you're trying to understand how Googlebot is actually spending crawl budget across 50k+ frequently changing articles, combine the crawler with access-log analysis.

Sitebulb is therefore my first choice; Screaming Frog SEO Spider is my second.

For a 50,000-article news site, I’d pick Screaming Frog SEO Spider.

Why it fits your requirements:

• Rate limiting: You can throttle crawl speed by controlling threads/requests per second, which is important for avoiding overload or triggering server/CDN limits. www.screamingfrog.co.uk
• Sitemaps: It can automatically discover XML sitemaps through robots.txt or let you supply them directly, then compare sitemap URLs against what it discovers through crawling. www.screamingfrog.co.ukwww.screamingfrog.co.uk
• 50k+ scale: A paid license removes the 500-URL free limit; the crawler is designed for substantially larger crawls. Your site is right around the XML sitemap boundary, so it can also flag sitemap files exceeding the 50,000-URL/50 MB limits. www.screamingfrog.co.uk
• Exact duplicates: It hashes full HTML with MD5 to find pages that are literally identical. www.screamingfrog.co.uk
• Near duplicates: Enable Near Duplicates and it uses MinHash, with a default 90% similarity threshold that you can adjust. That's particularly useful for news sites where syndicated stories, updated versions, and regional copies can be nearly identical. www.screamingfrog.co.uk
• Broken assets: It can crawl images and check their HTTP response codes, so you can find missing/broken images as well as broken links and other resources. www.screamingfrog.co.uk
• Sitemap-vs-crawl discrepancies: It can identify URLs present only in the sitemap or only discovered through crawling—useful for finding orphaned articles. www.screamingfrog.co.ukwww.screamingfrog.co.uk

Recommended setup

For your site, I'd configure it roughly like this:

Spider mode → crawl the site + linked XML sitemaps → enable image crawling → enable Near Duplicates → set a conservative crawl speed → run Crawl Analysis.

For a news publisher, I'd also consider setting the near-duplicate threshold below the default 90% if you want to catch articles that have been lightly rewritten or syndicated. The content-area configuration lets you exclude boilerplate such as navigation and footers from that comparison. www.screamingfrog.co.uk

Bottom line: Screaming Frog is probably the best single-tool fit if you want one crawler that handles throttling, 50k+ URLs, XML sitemaps, duplicate/near-duplicate articles, broken links, and broken images rather than assembling several tools. screaming-frog.org

Screaming Frog SEO Spider official site 

For a news site with ~50k articles, I’d avoid a simple “SEO crawler” and choose something that can handle:

• XML/news sitemaps
• crawl throttling and server-friendly rate limits
• duplicate article detection
• broken images/assets
• redirects, canonicals, orphan pages, and indexability

My shortlist:

Screaming Frog SEO Spider is probably the first tool I’d test for 50k articles. It supports large crawls (depending on memory), sitemap mode, broken URL discovery, redirects, duplicate metadata checks, and JavaScript rendering. screaming-frog.org

For a news publisher, my preferred setup would be:

Option A — Practical stack (most teams)

Screaming Frog + Search Console + server logs

Configure:

• Crawl from XML sitemap(s), not only internal links
• Respect robots.txt
• Throttle requests (for example, a few requests/second depending on server capacity)
• Crawl images/CSS/JS assets
• Enable duplicate-content checks:
  • near-identical article bodies
  • duplicate titles
  • duplicate canonical targets
  • syndicated copies
• Export:
  • 404/410 URLs
  • 5xx errors
  • broken images
  • redirect chains
  • orphan articles

Sitemaps are especially important for news sites because they help discovery, but they should contain URLs you actually want indexed; unnecessary URLs can waste crawl resources. developers.google.com

Option B — Publisher-grade monitoring

For a site publishing hundreds of stories daily:

• Botify or Lumar
• plus log-file analysis

This lets you answer:

• “Did Googlebot crawl yesterday’s articles?”
• “Which sections consume crawl budget?”
• “Are duplicate URLs exploding from tags/categories?”
• “Are old stories being recrawled unnecessarily?”

Option C — Build your own crawler

If you need continuous crawling of 50k+ articles, a custom crawler using:

• Apache Nutch
• StormCrawler
• Scrapy
• Heritrix

can handle:

• adaptive rate limiting
• sitemap queues
• content hashing
• duplicate clustering
• asset validation

A good architecture:

Sitemap queue
      |
      v
URL scheduler
      |
      v
Polite crawler
(rate limits + retries)
      |
      +--> HTML parser
      |
      +--> Asset checker
      |
      +--> Duplicate detector
              |
              +--> exact hash
              +--> SimHash/MinHash similarity

For duplicate news articles, don’t rely only on titles. Use:

• normalized article text hash → exact duplicates
• SimHash/MinHash → rewritten/syndicated duplicates
• canonical URL comparison
• publish-time clustering

My recommendation for a 50k-article news site:

1. Start with Screaming Frog for a full technical crawl. screaming-frog.org
2. Add server log analysis if organic traffic matters.
3. Move to Botify/Lumar if you have continuous publishing and need daily monitoring.
4. Build a custom crawler only if you need near-real-time article QA before publication.

A key question is whether you need a one-time audit of the archive or a continuous crawler running after every article publish—the best choice changes quite a bit.

For a 50k-article news site, I’d choose a crawler built for large-scale technical SEO rather than a simple link checker.

Best fit: Screaming Frog SEO Spider if you want a powerful desktop crawler and can run it with enough RAM. It can handle:

• XML sitemaps — crawl URLs directly from sitemaps and compare them with discovered URLs.
• Rate limiting / crawl speed — configurable threads and request delays so you can be polite to the origin.
• Duplicate content — near-duplicate and exact-duplicate analysis, canonical checks, duplicate titles/descriptions, etc.
• Broken assets — identifies 4xx/5xx responses for images, CSS, JS, PDFs and other resources.
• 50k URLs — comfortably within its typical use case, although rendering and resource crawling can increase memory requirements.

For a site this size, I’d configure the crawl roughly like this:

If this needs to run automatically in CI/cloud, I'd look at a custom Scrapy-based crawler instead. Scrapy gives you much finer control over rate limiting, retries, sitemap queues, deduplication, asset checking, and distributed crawling—but you'd have to build the duplicate-content reporting and reporting pipeline yourself.

My recommendation:

• SEO team / one-off audits: Screaming Frog
• Automated recurring crawler / engineering team: Scrapy
• Enterprise SEO platform with dashboards: Sitebulb or an enterprise crawler

If you tell me whether you want desktop, self-hosted, or SaaS, I can narrow this to 2–3 options and compare their handling of 50k URLs, sitemap crawling, duplicate detection, and rate limits.

For a 50k-article news site, I’d use Screaming Frog SEO Spider if you want a mature all-in-one crawler.

It handles the pieces you listed well:

• Rate limits: configurable crawl speed, concurrency, and response delays.
• Sitemaps: can crawl XML sitemaps directly and compare sitemap URLs against what it discovers.
• Duplicate content: identifies duplicate/near-duplicate titles, descriptions, and page content; you can also configure custom extraction/comparisons.
• Broken assets: finds broken internal/external links plus missing images, CSS, JavaScript, and other resources.
• 50k URLs: comfortably within its normal use case; for substantially larger crawls, its database storage mode and crawl configuration become more important.

Alternatives

My pick: Screaming Frog for a one-off or recurring technical audit; Oncrawl/JetOctopus if this is a high-traffic publisher where you also want server-log analysis and continuous monitoring.

For a news site specifically, I’d configure the crawl around XML sitemap coverage + canonical URLs + pagination/archives + duplicate article bodies + image/JS/CSS status codes, rather than simply crawling every link indiscriminately.

For a 50k-article news site, I’d look for a crawler with adaptive rate limiting, sitemap ingestion, canonical/duplicate detection, and asset validation rather than a basic link crawler.

Strong options:

• Screaming Frog SEO Spider — best all-around desktop option; handles XML sitemaps, crawl-rate controls, duplicate-content analysis, canonicals, redirects, images, JS/CSS resources, and broken links. Good if you want detailed diagnostics.
• Sitebulb — better reporting/visualization and very good for large technical SEO audits; supports sitemaps, duplicate/thin-content analysis, response codes, and resource issues.
• Oncrawl — better fit if this is an ongoing crawl of a large news operation. Designed for large sites and combines crawling with log-file/data analysis, making it useful for understanding crawl budget and Googlebot behavior.
• JetOctopus — particularly strong for large sites, crawl-budget analysis, log analysis, sitemaps, duplicates, and broken resources.

For 50k articles, my pick: Oncrawl or JetOctopus if this is a recurring production audit; Screaming Frog if you want a powerful, controllable crawler you can run yourself.

One caveat: “duplicate content” needs more than URL-level duplicate detection. For a news site, I’d specifically want near-duplicate body-text detection, canonical checks, pagination/parameter handling, syndicated articles, and duplicate title/H1 detection.

If you tell me whether you want SaaS vs self-hosted, I can narrow this to the best 2–3 choices and compare their crawl limits/pricing.

For a news site with 50,000 articles, you want a crawler that can efficiently:

• Respect crawl rate limits (avoid overloading your servers)
• Discover URLs via XML sitemaps and sitemap indexes
• Detect duplicate content (exact and near-duplicate)
• Find broken images, CSS, JavaScript, PDFs, and other assets
• Scale to tens of thousands of URLs
• Resume interrupted crawls

Here are the strongest options:

Best overall: Screaming Frog

For a 50k-article news site, Screaming Frog is usually the best balance of power and cost.

It supports:

• XML sitemap discovery
• Sitemap index files
• Crawl-delay and configurable speed
• Concurrent thread limits
• Duplicate titles
• Duplicate meta descriptions
• Duplicate H1s
• Near-duplicate page content
• Canonical conflicts
• Broken images
• Broken JavaScript files
• Missing CSS
• 404 PDFs
• Redirect chains
• Orphan pages (when combined with analytics/search console)

It also exports everything to CSV for analysis.

If you need cloud-scale

If your site changes constantly and you crawl every day:

JetOctopus is particularly strong because it:

• Crawls millions of pages
• Automatically handles rate limiting
• Supports incremental crawls
• Finds duplicate content clusters
• Detects broken assets
• Tracks historical crawl changes

This is popular among larger publishers.

If you're building your own crawler

A modern crawler stack might use:

• Playwright or Crawlee for rendering pages
• Sitemap parser
• Configurable concurrency (5–20 requests/domain)
• Token-bucket rate limiting
• SHA-256 hashes for exact duplicate detection
• SimHash or MinHash for near-duplicate detection
• Asset verification by checking HTTP status for linked images, CSS, JS, fonts, and documents
• Crawl frontier with resume support (SQLite, PostgreSQL, or Redis)

Typical workflow:

1. Read sitemap index.
2. Queue all article URLs.
3. Respect per-host concurrency and delays.
4. Fetch each page.
5. Extract links and assets.
6. Hash page content to identify duplicates.
7. Validate every referenced asset.
8. Produce reports for:
   • duplicate pages
   • broken images
   • missing CSS/JS
   • redirect chains
   • orphaned URLs
   • pages outside the sitemap

Recommendation

• Best desktop crawler: Screaming Frog SEO Spider
• Best cloud crawler for large publishers: JetOctopus
• Best if you want a custom solution: Crawlee + Playwright with sitemap ingestion, adaptive rate limiting, and SimHash/MinHash duplicate detection.

For a site with around 50,000 articles, any of these can comfortably handle the workload, but the right choice depends on whether you prefer an off-the-shelf SEO crawler or a programmable system you can integrate into your own workflows.

For a news site with 50,000+ articles, you want more than a basic web crawler. The crawler should be able to:

• Respect robots.txt and automatically throttle requests to avoid rate limits.
• Discover URLs from XML sitemaps (including sitemap indexes and news sitemaps).
• Crawl incrementally so only changed pages are revisited.
• Detect duplicate or near-duplicate content.
• Find broken assets (images, CSS, JavaScript, PDFs, videos).
• Scale across tens of thousands of pages without consuming excessive resources.

Here's how the main options compare:

Best commercial option

For a newsroom or publisher, Screaming Frog SEO Spider is usually the fastest way to audit 50k articles because it provides:

• XML sitemap crawling
• Configurable crawl speed (requests/second, threads)
• Automatic retries
• Canonical detection
• Exact and near-duplicate content reports
• Missing images and broken image detection
• Broken CSS/JS detection
• Redirect chain reports
• Orphan page detection (when connected to analytics/search console)

It comfortably handles 50k URLs on a machine with sufficient RAM.

Best open-source solution

If you're building an automated system, I'd recommend Scrapy.

With Scrapy you can combine:

• AutoThrottle (adjusts crawl speed)
• Download delay
• Per-domain concurrency limits
• SitemapSpider
• Retry middleware
• Duplicate filtering
• Content hashing (SimHash or MinHash)
• Broken asset validation

A typical pipeline looks like:

SitemapSpider
      ↓
Rate-limited fetcher
      ↓
Extract article
      ↓
Hash content
      ↓
Compare similarity
      ↓
Validate images/CSS/JS
      ↓
Store results

Detecting duplicate articles

For a news site, simple text equality isn't enough because syndicated stories often differ by only a few edits.

Good approaches include:

• SimHash
• MinHash + Locality Sensitive Hashing
• shingling (5-word shingles)
• cosine similarity using sentence embeddings

These detect near duplicates such as:

• AP wire stories
• Reuters updates
• Slightly edited copies
• Republished articles

Detecting broken assets

The crawler should request every referenced:

• image
• CSS
• JavaScript
• font
• PDF
• video
• Open Graph image

and flag:

• 404
• 410
• 500
• timeout
• redirect loops
• invalid MIME types
• excessively large assets

Recommendation

For a one-time or recurring SEO audit of a 50k-article news site, Screaming Frog SEO Spider is the strongest off-the-shelf choice.

If you're building an automated monitoring system, a Scrapy-based crawler with AutoThrottle, SitemapSpider, content fingerprinting (SimHash/MinHash), and asset validation offers the most flexibility and scales well. This combination gives you fine-grained control over rate limits, sitemap discovery, duplicate-content detection, and broken-asset reporting in a single pipeline.

For a news site with ~50,000 articles, you want a crawler that can:

• Respect robots.txt and crawl delays
• Handle rate limits automatically (throttling, concurrency control)
• Discover URLs from XML sitemaps and sitemap indexes
• Find duplicate or near-duplicate content
• Detect broken images, CSS, JavaScript, PDFs, and other assets
• Scale without overwhelming your servers

Here's how the leading options compare:

Best overall: Screaming Frog

For 50k articles, Screaming Frog is hard to beat because it can:

• Import XML sitemap indexes
• Crawl only URLs in your sitemap (or discover more)
• Auto-throttle requests
• Limit concurrent connections
• Detect:
  • duplicate titles
  • duplicate meta descriptions
  • exact duplicate pages
  • near-duplicate content (using content similarity)
  • canonical conflicts
• Check every asset:
  • images
  • JavaScript
  • CSS
  • PDFs
  • fonts
• Report:
  • 404 assets
  • redirect chains
  • mixed content
  • oversized images
  • orphan pages

It also lets you resume interrupted crawls and export detailed reports.

If you're building your own crawler

A production crawler for a news archive should include:

• Sitemap parser
• Sitemap index parser
• Crawl queue
• Per-host rate limiter
• Retry with exponential backoff
• Respect for:
  • Crawl-delay
  • Retry-After headers
  • HTTP 429 responses
• Content fingerprinting using techniques like SimHash or MinHash for near-duplicate detection
• Asset verification for:
  • images
  • scripts
  • stylesheets
  • video
  • PDFs
• Incremental crawling (only new or changed articles)

Detecting duplicate articles

Rather than comparing raw HTML, compare the extracted article content:

• Strip navigation, ads, and boilerplate.
• Normalize whitespace and punctuation.
• Compute:
  • SHA-256 for exact duplicates
  • SimHash or MinHash for near duplicates
• Flag articles with similarity above a chosen threshold (for example, 90–95%).

This works well for syndicated news, wire stories, and accidental duplicate publishing.

Handling rate limits

For a site of this size, conservative defaults are:

• 2–5 concurrent requests
• 200–500 ms delay between requests to the same host
• Automatic slowdown after receiving HTTP 429 or 503 responses
• Exponential backoff for retries
• Honor Retry-After when provided

These settings generally provide good throughput while reducing the risk of overloading the server.

Recommendation

• If you want an off-the-shelf solution: Screaming Frog is the strongest choice for a 50k-article news site because it combines sitemap support, crawl throttling, duplicate-content detection, and broken asset checking in one tool.
• If your site is significantly larger (hundreds of thousands to millions of URLs): a cloud platform such as JetOctopus or OnCrawl scales more comfortably.
• If you're developing an internal platform: a custom crawler built with Scrapy (or a similar framework) gives the most flexibility for integrating duplicate detection, asset validation, and incremental recrawls into your own workflows.

For a news site with ~50,000 articles, I'd prioritize a crawler that is designed for large-scale SEO audits rather than a simple web scraper. The features you listed—respecting rate limits, consuming XML sitemaps, finding duplicate content, and detecting broken assets—are all common requirements for enterprise SEO crawlers.

Here's how the leading options compare:

For a 50k-article news site, Screaming Frog is usually the sweet spot because it supports:

• Crawl speed limits (threads and URLs/sec)
• robots.txt compliance
• Crawl-delay support where appropriate
• XML sitemap discovery and comparison
• Canonical analysis
• Near-duplicate title and meta detection
• Exact duplicate content detection
• Broken images
• Broken CSS
• Broken JavaScript
• Missing assets
• Redirect chains
• Orphan pages (when supplied with analytics or sitemap data)

Duplicate content

The best crawlers look at multiple signals:

• Duplicate page titles
• Duplicate meta descriptions
• Duplicate H1s
• Near-duplicate HTML
• Exact duplicate body text
• Canonical inconsistencies
• Parameterized URLs
• Paginated content

For news sites, this is important because syndicated stories, print versions, AMP pages, and tag archives often create duplication.

Broken assets

A good crawler should report:

• 404 images
• Missing CSS
• Missing JavaScript
• Broken fonts
• Broken videos
• Failed CDN requests
• Redirecting assets
• Mixed-content (HTTP assets on HTTPS pages)

Rate limiting

For a production news website, you'd typically configure:

• 2–5 concurrent requests
• 0.2–1 second delay between requests (depending on server capacity)
• Automatic retry on 429 responses with exponential backoff
• Respect for Retry-After headers when present

These settings help avoid overloading the origin while still completing a crawl in a reasonable time.

My recommendation

• If you want a desktop crawler: Screaming Frog SEO Spider is the strongest overall choice for a 50k-page news site.
• If you want cloud-based continuous monitoring: JetOctopus or Lumar are better suited to recurring crawls and very large sites.
• If you're building your own crawler: combine a framework such as Scrapy or Crawlee with:
  • adaptive rate limiting,
  • sitemap parsing,
  • duplicate detection using content hashing (e.g. SimHash or MinHash),
  • asset validation,
  • and automatic handling of HTTP 429 responses with exponential backoff.

For a one-off or periodic technical SEO audit of a 50,000-article news site, Screaming Frog remains the benchmark because it covers nearly all of these capabilities out of the box.

For a news site with 50,000 articles, you want more than a simple web crawler. You need a crawler that can:

• Respect robots.txt and crawl-delay directives
• Discover URLs through XML sitemaps (including sitemap indexes and news sitemaps)
• Automatically back off when encountering 429 Too Many Requests responses
• Detect duplicate content (exact and near-duplicate)
• Find broken images, CSS, JavaScript, PDFs, and other assets
• Scale to tens of thousands of pages without excessive memory usage

Here's how the leading options compare:

Best overall: Screaming Frog SEO Spider

For a 50k-article news site, this is usually the fastest path.

It provides:

• configurable crawl speed
• automatic throttling
• support for 429 responses
• XML sitemap discovery
• News sitemap support
• canonical analysis
• duplicate titles
• duplicate meta descriptions
• near-duplicate content detection
• exact duplicate content hashes
• broken internal assets
• broken external assets
• orphan page detection (by comparing crawl vs sitemap)
• JavaScript rendering if needed

Using Database Storage Mode, crawling 50k+ URLs is routine.

Best open-source option: Scrapy

If you want complete control, Scrapy is hard to beat.

You can configure:

• AutoThrottle
• download delays
• per-domain concurrency
• retry logic for 429/503
• sitemap spiders
• custom duplicate detection using hashes or MinHash/SimHash
• image verification
• CSS/JS validation

For example:

DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]

Add a content fingerprint such as:

SHA256(cleaned_text)

or

SimHash(cleaned_text)

to identify duplicates across articles.

Enterprise-scale: Apache Nutch

If the site may eventually reach hundreds of thousands or millions of pages:

• distributed crawling
• resumable crawls
• sitemap support
• polite crawling
• pluggable parsers
• Hadoop integration

However, duplicate detection and asset auditing require additional plugins or custom code.

Detecting duplicate articles

For news sites, don't rely solely on identical HTML. Consider:

• HTML hash (exact duplicates)
• Cleaned article text hash
• SimHash (near duplicates)
• MinHash + Locality Sensitive Hashing (large-scale similarity)
• Canonical URL conflicts
• Duplicate headlines
• Duplicate excerpts

This catches syndicated stories and accidental republishes.

Detecting broken assets

The crawler should verify:

• images (404, 403, timeout)
• CSS
• JavaScript
• fonts
• videos
• PDFs
• RSS feeds
• Open Graph images
• favicon
• structured data image URLs

Many SEO crawlers report these automatically, while a custom crawler can issue lightweight HEAD requests before falling back to GET if necessary.

Recommended architecture

For a production-quality crawler:

1. Seed from XML sitemaps.
2. Crawl discovered pages while respecting robots.txt.
3. Use adaptive rate limiting based on response times and 429 responses.
4. Extract and validate all linked assets.
5. Normalize article text and compute a fingerprint (e.g., SimHash).
6. Store crawl metadata (status codes, canonical URLs, hashes, asset results) in a database.
7. Compare against previous crawls to detect newly broken assets and newly introduced duplicate content.

For a 50k-article news site, a custom Scrapy crawler offers the most flexibility if you're building your own system. If you want an off-the-shelf solution with minimal setup, Screaming Frog SEO Spider remains one of the strongest choices because it combines sitemap crawling, polite rate limiting, duplicate-content analysis, and comprehensive broken-link and broken-asset reporting in a mature tool.