ChatGPT answers, unedited

Yes. If your workflow is “I have a list of target domains/pages → crawl them → extract outbound links → identify dead resources → evaluate which ones are worth replacing”, a few tools stand out.

Best options

1. Screaming Frog is probably the closest match.

Its List Mode lets you upload a list of URLs, and it can be configured to crawl the external links found on those pages. Screaming Frog specifically documents this as a broken-link-building workflow. Screaming Frog Screaming Frog

The workflow is essentially:

target URLs/domains
        ↓
Screaming Frog List Mode
        ↓
crawl target pages
        ↓
extract external links
        ↓
check external URLs
        ↓
filter 404 / 410 / other errors
        ↓
export:
  referring page
  broken URL
  anchor text
  status code

You can also control external-link crawling independently in the configuration. Screaming Frog

Important distinction: if you give it a list of homepage URLs, you don't necessarily want it to stop at those pages. For broken-link building, you generally want it configured to follow the target site's internal pages while checking the external links encountered along the way. Screaming Frog's own broken-link-building tutorial describes removing the List Mode crawl-depth limit for this purpose. Screaming Frog

2. Sitebulb

Sitebulb is another strong option. It can use a URL list as a crawl source and reports links, including external links and broken links. Sitebulb Support Sitebulb Support

One limitation is worth knowing: its URL-list mode is primarily intended to analyze the URLs supplied rather than indiscriminately crawl a huge collection of unrelated domains. Its documentation says a URL-list crawl is restricted to the subdomain of the starting URL. Sitebulb Support

For normal one-domain-at-a-time prospect analysis, though, it's very capable.

3. Ahrefs

Ahrefs approaches the problem differently.

Its Outgoing Links → Broken Links report can show pages on a domain that contain broken links to other websites, including the broken destination and referring page. Ahrefs Help Center

This is particularly useful because you don't necessarily have to crawl the prospect yourself. Ahrefs has already crawled many of these pages.

Even better for the other half of broken-link building, Ahrefs' Broken Backlinks report finds external pages linking to 404 pages on a target domain. That lets you identify dead resources that already have backlinks. Ahrefs Help Center

So there's a useful distinction:

• Screaming Frog: “Here are my 500 prospect pages; find their broken outbound links.”
• Ahrefs: “Here are domains/resources with broken links/backlinks; show me the SEO opportunity.”
• Sitebulb: Similar to Screaming Frog, with strong link auditing/reporting.

What I'd use for a scaled workflow

If you're doing this seriously, I'd combine Screaming Frog + Ahrefs:

1. Build prospect list
       ↓
2. Screaming Frog
   crawl prospect pages
       ↓
3. Extract broken external URLs
       ↓
4. Deduplicate broken URLs
       ↓
5. Ahrefs
   check each dead resource for:
      - referring domains
      - backlinks
      - authority
      - organic traffic
      - historical relevance
       ↓
6. Wayback Machine / historical research
   determine what the dead resource actually contained
       ↓
7. Find prospects whose page currently
   links to the dead resource
       ↓
8. Create/recreate genuinely relevant replacement
       ↓
9. Outreach

The Ahrefs qualification step is important. A 404 isn't automatically a good broken-link-building opportunity; you want dead resources that are topically relevant and have worthwhile referring links. Ahrefs explicitly supports finding competitors' broken pages with backlinks for this purpose. Ahrefs Help Center Ahrefs

If your actual use case is “I have 10,000–100,000 target domains and want to automatically discover their broken outbound resources”, that's a slightly different problem. In that case, I'd look at a crawler/API-based pipeline rather than manually configuring Screaming Frog, and I can lay out the most scalable architecture and tools for that.

Important distinction: if you give it a list of homepage URLs, you don't necessarily want it to stop at those pages. For broken-link building, you generally want it configured to follow the target site's internal pages while checking the external links encountered along the way. Screaming Frog's own broken-link-building tutorial describes removing the List Mode crawl-depth limit for this purpose. Screaming Frog

2. Sitebulb

Sitebulb is another strong option. It can use a URL list as a crawl source and reports links, including external links and broken links. Sitebulb Support Sitebulb Support

One limitation is worth knowing: its URL-list mode is primarily intended to analyze the URLs supplied rather than indiscriminately crawl a huge collection of unrelated domains. Its documentation says a URL-list crawl is restricted to the subdomain of the starting URL. Sitebulb Support

For normal one-domain-at-a-time prospect analysis, though, it's very capable.

3. Ahrefs

Ahrefs approaches the problem differently.

Its Outgoing Links → Broken Links report can show pages on a domain that contain broken links to other websites, including the broken destination and referring page. help.ahrefs.com

This is particularly useful because you don't necessarily have to crawl the prospect yourself. Ahrefs has already crawled many of these pages.

Even better for the other half of broken-link building, Ahrefs' Broken Backlinks report finds external pages linking to 404 pages on a target domain. That lets you identify dead resources that already have backlinks.

So there's a useful distinction:

For broken link building, you usually want a tool that can do one of two things:

1. Crawl a list of target domains/pages and find outbound broken links (the classic “find dead resources and suggest your replacement” workflow).
2. Find broken pages that already have backlinks (often higher-value because you know there is link equity).

Here are the main options:

The workflow I’d use

For 100–10,000 target domains:

1. Build your prospect list (resource pages, directories, “useful links” pages).
2. Feed URLs into Screaming Frog in List Mode.
3. Crawl external links.
4. Filter:

• Status code: 404, 410, 5xx
• External links only
• Relevant anchor text
• Export:

• Target page
• Broken URL
• Anchor text
• Context/snippet
• Check the dead URL in the Internet Archive Wayback Machine to recreate a better replacement.
• Outreach.

For finding the highest-value opportunities:

Use Ahrefs:

• Find competitor domains.
• Go to broken backlinks / broken pages.
• Sort by referring domains.
• Recreate the missing resource.
• Contact sites linking to the dead page. Ahrefs

My practical stack

• Small campaign (<500 prospects): Screaming Frog + Ahrefs + Wayback Machine
• Agency workflow: Screaming Frog + Ahrefs/Semrush + Pitchbox
• Large-scale automation: custom crawler + Ahrefs API/data export + outreach CRM

One important distinction: a normal crawler finds dead outbound links on pages; Ahrefs-style tools find dead pages that already have backlinks. The second category often produces fewer but more valuable opportunities. Ahrefs BestPage

Yes. If your workflow is “I have a list of target domains → crawl each site → identify broken outbound resources → evaluate the dead URLs as link-building opportunities,” there are a few tools that fit particularly well.

Best options

• Screaming Frog SEO Spider — probably the most straightforward for exactly this workflow. It has List Mode, so you can upload a list of URLs/domains, crawl the supplied URLs, and increase crawl depth to follow their outlinks. It can export the source page, broken destination, anchor text, and response code. The free version is limited to 500 URLs; paid removes that limit. Screaming Frog

screamingfrog.co.uk

• Ahrefs — better if you want to go beyond simply finding the 404 and immediately determine how many sites link to the dead resource, its backlink profile, historical importance, etc. Ahrefs specifically documents using competitors' broken outbound links for broken-link building, and its Content Explorer can find broken pages by topic that have backlinks. Ahrefs Ahrefs

ahrefs.com

• Sitebulb — another strong crawler if you want a more visual/SEO-oriented workflow. It exposes broken URLs together with incoming/referring links and allows bulk export. Sitebulb

sitebulb.com

What I'd use for scaled broken-link building

For a campaign with, say, 500–10,000 prospect domains, I'd split the job into two stages:

1. Crawl prospects:
Screaming Frog → List Mode → crawl depth 1+ → external links → filter 4xx/5xx

You end up with something like:

2. Qualify the dead resources:
Run the broken URLs through Ahrefs to see referring domains/backlinks, then use Wayback Machine to determine what the resource actually contained. Ahrefs specifically recommends sorting competitors' dead pages by referring domains when doing broken-link building. Ahrefs

That distinction is important: the crawler finds the broken link; the backlink index tells you whether the dead resource is worth pursuing.

One important caveat

If by "crawl a list of target domains" you mean give a tool 5,000 domains and have it autonomously crawl every page of every domain looking for external broken resources, Screaming Frog can technically be configured toward that workflow, but it isn't really designed as a high-scale cloud prospecting platform. You can run multiple crawls/automation around it, but bandwidth, crawl time, robots restrictions, rate limiting, and 403/429 responses become significant.

For serious scale, I'd consider building a pipeline around a crawler/API:

domain list → crawl → extract external URLs → HTTP validation → deduplicate dead URLs → Ahrefs backlink enrichment → Wayback lookup → opportunity scoring

If that's the workflow you're after, I can also give you a specific tool stack for 10k–100k domains, including which parts can be automated and which tool is cheapest/fastest for each stage.

For broken link building at scale, you generally want tools that can:

1. Take a list of target domains or URLs
2. Crawl their pages/resources
3. Find outbound links returning 404/410/5xx
4. Export the broken URLs + source pages
5. Let you evaluate whether the dead resource has enough authority/backlinks to justify outreach

The best options:

My recommended workflow

Option 1: Best practical setup (most SEOs)

Screaming Frog + Ahrefs

1. Build a prospect list:

• industry resource pages
• competitor resource pages
• curated link pages
• .edu/.gov resource pages
• Upload URLs into Screaming Frog:

• Mode → List
• Upload your target URLs
• Crawl external links
• Filter Response Codes → Client Error (4XX)

Screaming Frog specifically supports this broken link building workflow. Screaming Frog

3. Export:

• broken URL
• linking page
• anchor text
• context
• Put broken URLs into Ahrefs:

• check referring domains
• check historical traffic
• see who linked to the dead resource

Ahrefs' broken-link reports and Content Explorer workflows are commonly used to find dead pages worth replacing. Ahrefs

5. Create replacement content and outreach.

If you have thousands of domains

A custom pipeline is often better:

Crawler layer

• Scrapy
• Playwright (for JS-heavy pages)
• aiohttp/requests

Checks

• HTTP status
• redirect chains
• canonical status
• content similarity

Enrichment

• Ahrefs API
• Semrush API
• Moz API
• Majestic API

Example output:

Source page:
example.com/resources/tools

Broken link:
oldtool.com/report.pdf

Status:
404

Anchor:
"SEO checklist"

Referring domains:
142

Replacement candidate:
yourdomain.com/seo-checklist

Tools I would choose by campaign size

• <1,000 prospects: Screaming Frog
• 1,000–50,000 prospects: Screaming Frog + automation/scripts
• 50,000+ prospects: custom crawler + backlink API
• Finding the highest-value dead pages: Ahrefs

For pure "give me 10,000 target pages and find broken resources I can replace", Screaming Frog List Mode is probably the closest off-the-shelf solution. Screaming Frog

Yes. For broken link building at scale, there are two different jobs:

1. Crawl a list of target pages/domains and discover outbound broken links — best handled by a crawler.
2. Find dead pages on competitors that already have backlinks — best handled by a backlink index.

Best tools

1. Screaming Frog — probably exactly what you're describing

screamingfrog.co.uk

You can upload a list of URLs, switch to List Mode, and configure it to crawl the external links found on those pages. Screaming Frog specifically documents a broken-link-building workflow where you upload target URLs, crawl their external links, and then filter the results for 4XX responses. Screaming Frog Screaming Frog

So your workflow could be:

targets.csv
   ↓
Screaming Frog List Mode
   ↓
crawl target pages
   ↓
extract external links
   ↓
check HTTP status
   ↓
filter 404 / 410 / other dead resources
   ↓
export:
source page → broken URL → anchor text

This is probably the best answer if you already have a curated list of target domains/pages.

2. Ahrefs — best for finding the opportunity

ahrefs.com

Ahrefs has a Broken links report that identifies outbound links on a target site pointing to dead pages. It can also export the results. Ahrefs Help Center Ahrefs Help Center

More importantly for classic broken-link building, Ahrefs lets you find competitors' 404 pages that have backlinks. You can then identify the sites linking to those dead resources and pitch your replacement. Ahrefs Help Center Ahrefs

That's a different—and often more valuable—workflow:

Competitor domain
   ↓
find highly linked 404 pages
   ↓
identify referring pages
   ↓
find relevant replacement resource
   ↓
outreach

Ahrefs also exposes its broken-backlinks functionality through an API, which makes it interesting if you're building an automated pipeline. Ahrefs for Developers

3. Sitebulb

sitebulb.com

Sitebulb is another strong crawler. It automatically collects links during a crawl and lets you drill into broken URLs and their incoming links. Sitebulb Support

I'd choose it over Screaming Frog mainly if you prefer its reporting/visualization workflow.

4. Semrush

semrush.com

Semrush's Site Audit crawls a domain and reports URL-level issues, while its backlink tools can identify broken/lost target pages. Semrush Semrush

It's useful if you're already paying for Semrush, but for dedicated broken-link-building prospecting I'd generally pick Ahrefs + Screaming Frog.

What I'd use for a serious BLB campaign

If you have, say, 500–50,000 target domains, I'd use:

Screaming Frog → Ahrefs → custom spreadsheet/database

Screaming Frog handles the actual crawling:

1. Import target URLs
2. Crawl external links
3. Extract 404/410/5xx resources
4. Export source URL + broken URL + anchor

Then Ahrefs enriches the opportunities:

5. Check whether the dead resource has backlinks
6. Number/quality of referring domains
7. Organic traffic / relevance
8. Historical context
9. Identify potential replacement content

The key distinction is that Screaming Frog answers "what links are currently broken on these sites?", while Ahrefs answers "which dead resources are actually worth replacing because people link to them?"

For pure broken-link building, I'd prioritize the second question. A random 404 isn't necessarily a link-building opportunity; a dead resource with 30 relevant referring domains can be extremely valuable. Ahrefs Help Center

If you tell me roughly how many domains/pages you want to crawl (e.g. 100, 10k, 1M), I can recommend the most efficient setup and explain how I'd automate the whole pipeline.

Yes. If your workflow is “give me 50–500 target domains → crawl them → identify broken outbound resources → prioritize the best replacement opportunities,” there are a few good options, but they differ substantially.

Best options

My pick: Ahrefs + Screaming Frog

For broken link building specifically, I'd lean toward Ahrefs because you don't just want to know “this URL is 404.” You want to know:

1. Which target page contains the broken link
2. What the dead resource was
3. How authoritative the target page/domain is
4. How many other sites link to the dead resource
5. Whether the opportunity is actually worth outreach

Ahrefs' Outgoing Links → Broken Links report can show pages containing broken links to external sites and lets you export the results. Ahrefs Help Center

Even more useful for classic broken-link building, Ahrefs' Broken Backlinks data lets you identify dead pages on a competitor's site and the sites linking to those dead pages—the latter are your outreach prospects. Ahrefs Help Center Ahrefs for Developers

If you specifically have a list of target domains

For example:

competitor1.com
competitor2.com
industryassociation.org
university.edu
resource-site.com
...

I'd structure the process like this:

Domain list → crawler → broken outbound URLs → deduplicate → classify resource → check historical content → find/create replacement → score referring page → outreach

Screaming Frog is particularly useful for the crawler portion. It can crawl a site, identify 404s/server errors, and export the broken URLs along with their source URLs. Screaming Frog

The important distinction is that Screaming Frog is a crawler, whereas Ahrefs gives you a huge link graph on top of its crawling/indexing. That's why Ahrefs is generally more powerful for discovering link-building opportunities, while Screaming Frog is excellent for actually crawling the sites you specify.

One potentially better approach

If your goal is scale, rather than manually opening each domain, I'd consider building a small pipeline around the Ahrefs API. Their API exposes broken-backlink data with fields including the referring URL, target URL, URL Rating, traffic, link type, last-seen dates, etc. Ahrefs for Developers

That allows something like:

500 target domains
       ↓
broken URLs
       ↓
HTTP status verification
       ↓
deduplicate URLs
       ↓
Wayback/content identification
       ↓
referring-domain metrics
       ↓
relevance scoring
       ↓
outreach list

If you tell me roughly how many domains you want to process (e.g. 100, 1,000, or 100,000), I can recommend the most practical setup and tools/API for doing it at that scale.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

If your goal is broken link building at scale (upload hundreds or thousands of domains, crawl them, identify broken outbound links or 404 resources, then pitch your replacement content), the tools fall into a few categories.

Best overall

1. Screaming Frog (my top recommendation)

Best if you're serious about broken link building.

Features:

• Import a CSV of domains or URLs
• Crawl thousands of pages
• Detect - 404 links
• 410 links
• broken redirects
• external broken resources
• images
• PDFs
• JS/CSS
• Export all broken links with source page
• Supports automation
• API integrations (Ahrefs, GA, GSC, etc.)

Typical workflow:

Prospect domains
      ↓
Import into List Mode
      ↓
Crawl entire sites
      ↓
Export external 404s
      ↓
Filter by niche
      ↓
Outreach

For agencies, this is probably the standard tool.

2. Sitebulb

Very similar to Screaming Frog but with a friendlier interface.

Advantages:

• Better visual reports
• Easier prioritization
• Good site health reports
• Excellent crawl diagnostics

Less flexible than Screaming Frog for very large campaigns.

3. Ahrefs

Excellent for finding broken pages that already have backlinks.

Examples:

• Best by links → Broken
• Broken backlinks
• Outgoing broken links

This helps you find:

Dead resource that 75 sites still link to

Perfect replacement opportunity.

Strengths:

• Shows link equity
• Finds link opportunities quickly

Weakness:

Doesn't crawl arbitrary domains the way Screaming Frog does.

4. Semrush

Offers similar capabilities:

• Broken pages
• Broken backlinks
• Site Audit
• Backlink Audit

Good if you're already using Semrush.

5. Check My Links

Chrome extension.

Great for:

• One page
• Resource pages
• University pages
• Government pages

Not suitable for scale.

6. Xenu Link Sleuth

Old but still works.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• Outdated UI
• Limited reporting

Enterprise options

For agencies crawling tens of thousands of sites:

• ContentKing
• Deepcrawl (Lumar)
• Botify
• Oncrawl

These are more focused on technical SEO than link building.

If you want automation

Many SEOs build a pipeline like:

Google search operators
        ↓
Scrape SERPs
        ↓
Deduplicate domains
        ↓
Screaming Frog List Mode
        ↓
Export broken external links
        ↓
Wayback Machine API
        ↓
LLM generates replacement angle
        ↓
Hunter.io / Apollo
        ↓
Outreach

If you want to scale to tens of thousands of domains

Many teams combine a crawler with custom scripts:

• Scrapy (Python) for distributed crawling
• Playwright or Puppeteer for JavaScript-heavy sites
• requests + BeautifulSoup for lightweight HTML crawling
• aiohttp or httpx with async Python for very high throughput

This approach gives full control over crawl rules, concurrency, and filtering, but requires development effort.

Recommended stack by budget

• Free: Xenu + Check My Links
• Best value: Screaming Frog + Ahrefs
• Best for agencies: Screaming Frog + Ahrefs + Pitchbox/BuzzStream for outreach
• Maximum scale: Custom Python crawler + Ahrefs API + outreach platform

For most SEO professionals running broken link building campaigns, Screaming Frog in List Mode paired with Ahrefs is the most widely used and effective combination: Screaming Frog discovers broken opportunities across your prospect list, while Ahrefs helps prioritize those with the strongest backlink value.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract:
  • External 404s
  • Internal 404s
  • Redirect chains
  • Broken images
  • PDFs
  • JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract:
  • External 404s
  • Internal 404s
  • Redirect chains
  • Broken images
  • PDFs
  • JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

Yes. If your workflow is:

list of target domains → crawl their pages → find outbound links/resources that are dead → identify the dead resource → find a replacement opportunity, there are a few good options.

\* Usually means setting up separate crawls/projects or feeding URLs/sitemaps rather than simply dropping thousands of domains into one click.

1. Ahrefs — probably your best fit

Ahrefs has a particularly useful Outgoing Links → Broken Links report. Give it a target domain/URL and it can show pages on that site containing broken external links, including the destination URL and the reason it's broken (e.g. 404 or DNS failure). You can export the results. help.ahrefs.com

This is almost exactly the data you want for broken-link building:

Target site → page containing broken link → dead resource → replacement opportunity

It also has the complementary Broken Backlinks report, which is extremely useful after finding a dead resource: you can see other sites linking to that dead URL and prospect those sites for replacement links. help.ahrefs.com

My pick if you're doing this manually or semi-automatically.

2. Screaming Frog SEO Spider — best if you want control

Screaming Frog is better if you want to actually crawl the target websites yourself rather than relying primarily on an SEO link index.

It can crawl a site's internal and external links, return HTTP status codes, and export the results. That makes it particularly useful when you want to inspect every link/resource discovered during a crawl. www.techradar.com

For example, you could build a workflow like:

targets.csv
   ↓
Screaming Frog
   ↓
crawl each domain
   ↓
extract external URLs
   ↓
HTTP status = 404/410/5xx/etc.
   ↓
deduplicate dead resources
   ↓
Ahrefs → backlinks to those resources
   ↓
prospect sites for outreach

This is probably the better choice if you have hundreds/thousands of target domains and want to automate the crawling side.

3. Semrush

Semrush's Site Audit can check external links when "Check HTTP status of external links" is enabled, and it reports pages containing links to broken destinations. help.ahrefs.com

I'd rank it behind Ahrefs for this particular use case, mainly because Ahrefs' link-index data is exceptionally useful for the second half of broken-link building—finding backlinks to the dead resource.

4. Sitebulb

Sitebulb is another strong crawler if you want a more visual technical-SEO interface. It's worth considering if you're doing large audits and want to analyze the relationships between source pages, links, redirects, status codes, etc.

But there's an important distinction

If you're specifically doing broken-link building, I wouldn't just look for:

"Which links on these websites return 404?"

I'd optimize for:

Which broken external resources have the highest-quality replacement potential?

So I'd combine a crawler + backlink database.

The stack I'd use

For ~10–500 target domains:

Ahrefs → Broken Outgoing Links → filter → export → Ahrefs backlinks

For each dead URL:

1. Find the pages linking to it.
2. Check whether the dead resource was genuinely useful/relevant.
3. Use the Wayback Machine to understand what the original resource contained.
4. Create/improve your replacement resource.
5. Contact the sites linking to the dead resource.

Ahrefs itself recommends essentially this approach: find broken outbound links, then identify all backlinks pointing at the broken resource and offer a relevant replacement. help.ahrefs.com

For thousands+ of domains:

I'd use Screaming Frog or a custom crawler for discovery, then Ahrefs API/data for backlink enrichment. That gives you much more control over crawling, concurrency, deduplication, status-code validation, and prioritization.

If you're planning to do this at scale (e.g. 1,000–100,000 domains), I can also lay out the exact cheap → mid-range → enterprise stack, including which APIs/crawlers can accept a CSV of domains and how I'd structure the pipeline.

If you're doing broken link building at scale, the tools fall into two categories:

1. Site crawlers that find broken outbound/internal links on a domain.
2. Backlink databases that find broken pages with existing backlinks (often more valuable).

Here's how the main options compare.

Best options by workflow

1. Crawl hundreds or thousands of target sites

Use Screaming Frog.

You can:

• Import a list of domains
• Crawl each automatically
• Export:
  • 404s
  • External broken links
  • Broken images
  • Redirect chains
• Schedule crawls

This is the closest to an industry standard for broken link building.

2. Find the highest-value opportunities

Instead of looking for broken outbound links, look for dead pages that already have backlinks.

Ahrefs excels here:

• Best by Links report
• Broken pages report
• Broken backlinks report

This finds pages with dozens or hundreds of referring domains that you can recreate and pitch as replacements.

This approach generally produces better outreach opportunities than simply finding random broken outbound links.