ChatGPT answers, unedited

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract:
  • External 404s
  • Internal 404s
  • Redirect chains
  • Broken images
  • PDFs
  • JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract: - External 404s
• Internal 404s
• Redirect chains
• Broken images
• PDFs
• JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

For broken link building at scale, you generally want a crawler that can:

1. Crawl a list of domains in bulk.
2. Identify 404s, 410s, and sometimes soft 404s.
3. Show which pages link to those broken URLs.
4. Export the data for outreach.

Here's how the major tools compare:

Best options

1. Ahrefs (best overall)

If you already have Ahrefs, it's hard to beat.

Workflow:

• Upload target domains.
• Site Explorer → Outgoing Links → Broken Links.
• Export every broken outbound link.
• See referring pages.
• Create replacement content.

This is the classic broken link building workflow.

Pros:

• Very fast
• Huge index
• Doesn't require crawling from your computer
• Shows link metrics

2. Screaming Frog (best desktop crawler)

Use List Mode.

You can:

• Import thousands of domains or URLs.
• Crawl each site.
• Extract:
  • External 404s
  • Internal 404s
  • Redirect chains
  • Broken images
  • PDFs
  • JS/CSS errors

It's extremely configurable.

Useful reports:

• Response Codes
• Client Error (4xx)
• External Links
• Bulk Export → All Outlinks

3. Sitebulb

Essentially a friendlier version of Screaming Frog.

Advantages:

• Better visual reports
• Easier exports
• Automatic issue prioritization

If you're not already a Screaming Frog power user, many people find Sitebulb more intuitive.

For large-scale broken link building

If you're targeting 10,000+ domains, combine tools:

1. Build prospect list.
2. Crawl with Screaming Frog in List Mode.
3. Import results into Excel or Google Sheets.
4. De-duplicate broken URLs.
5. Use Ahrefs to check whether those broken resources have backlinks.
6. Prioritize broken pages with lots of referring domains.

Enterprise-scale options

For agencies or very large campaigns:

• Netpeak Spider — very fast desktop crawler with batch processing.
• JetOctopus — cloud crawler that can handle millions of URLs.
• OnCrawl — enterprise cloud crawler.
• Lumar (formerly Deepcrawl) — enterprise technical SEO.

These are designed for massive crawls that exceed what a desktop crawler comfortably handles.

If your goal is outreach at scale

A workflow that many SEO agencies use looks like this:

1. Export a list of relevant domains.
2. Crawl them with Screaming Frog (List Mode).
3. Export all external 404 links.
4. Group by broken destination URL.
5. Use Ahrefs to see how many sites link to each broken resource.
6. Create a better replacement page.
7. Contact every site linking to the dead resource.

This approach lets you identify "link opportunities" where many websites reference the same dead page.

If you're planning to crawl tens or hundreds of thousands of domains, cloud-based solutions like JetOctopus or a custom crawler built with frameworks such as Scrapy are generally more efficient than desktop tools.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If your workflow is "I have a list of 100–10,000 target domains and I want to crawl them for broken outbound resources I can replace", the tools differ quite a bit. Here's what actually works at scale.

My top recommendations

1. Screaming Frog (best overall)

This is what most professional link builders use.

You can:

• Import thousands of URLs
• Crawl only resource pages
• Check external links
• Export every broken outbound link
• Export the source page, anchor text, status code, redirect chain, etc. www.screamingfrog.co.uk

Typical workflow:

1. Import a CSV of target URLs or domains.
2. Crawl.
3. Filter for External → Client Error (4xx).
4. Export.
5. Pitch your replacement.

2. Scrapebox (best for very large campaigns)

If you're crawling:

• 50,000 domains
• millions of pages
• huge outreach campaigns

Scrapebox is still hard to beat.

You can combine it with:

• custom footprints
• proxies
• multithreaded crawling
• broken link checker plugin

This is common among agencies running large broken-link-building campaigns.

3. Ahrefs (best for finding valuable opportunities)

Instead of crawling everything yourself, Ahrefs can show:

• broken pages with backlinks
• broken outbound links
• competitors' dead resources
• pages linking to dead content ahrefs.com

This saves a lot of crawling.

4. Sitebulb

Very similar to Screaming Frog but easier to use.

It supports:

• batch crawling
• broken external links
• exporting source URLs
• crawl scheduling

Many agencies prefer it because the reports are easier to interpret.

If you're building your own pipeline

Many advanced SEOs now use:

• Playwright or Puppeteer
• Python (requests, httpx, or aiohttp)
• BeautifulSoup or lxml
• Async crawling

The pipeline typically:

1. Reads a list of domains.
2. Crawls only likely resource pages (e.g. /resources, /links, /tools, /useful-links).
3. Extracts outbound links.
4. Checks HTTP status codes.
5. Records 404/410/5xx responses.
6. Exports:
   • source page
   • broken URL
   • anchor text
   • surrounding text
   • domain authority/traffic (via APIs)

This approach scales well into the hundreds of thousands of pages.

For really high-volume prospecting

If you're processing 10k–100k+ domains, a common stack is:

• Crawler: Screaming Frog or a custom Python crawler
• Enrichment: Ahrefs API or Majestic API (to prioritize authoritative sites)
• Outreach: BuzzStream, Pitchbox, or Instantly

That gives you both the technical discovery of broken links and the ability to prioritize outreach based on link equity rather than chasing every 404.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

If you're doing broken link building at scale (i.e. you have a list of hundreds or thousands of prospect domains and want to find dead outbound resources you can replace), the tools fall into a few categories.

Ahrefs can report broken outbound links for a domain and export them, while Screaming Frog and Sitebulb discover them by crawling the live site. help.ahrefs.com

My recommendations by scale

100–500 domains

• Screaming Frog in List Mode
• Import your domain list
• Crawl each site
• Filter for External → Client Error (404/410)
• Export CSV

This is the workflow most SEO agencies use.

500–10,000 domains

Use:

• Screaming Frog CLI (command line)
• Multiple crawl instances
• Export results automatically
• Process with Python or Sheets

This scales surprisingly well if you're comfortable scripting.

Finding higher-value opportunities

Instead of crawling random sites, use Ahrefs first:

1. Find pages ranking for your topic.
2. Export referring domains.
3. Crawl only those domains with Screaming Frog.
4. Find outbound 404s.
5. Replace with your resource.

This dramatically improves outreach quality.

If you want backlinks with existing authority

Ahrefs has another powerful workflow:

• Find broken pages on competitors ("Best by Links" → filter for 404).
• See every site linking to those dead pages.
• Contact those sites with your replacement.

This is often more efficient than crawling for broken outbound links because you're targeting pages that already have proven backlink profiles. ahrefs.com

Enterprise options

If you're crawling tens of thousands of domains:

• Screaming Frog + CLI
• Sitebulb Cloud
• JetOctopus
• Lumar (formerly Deepcrawl)
• Custom crawler (Python + Playwright/Requests)

These let you automate large-scale crawling and exports.

The stack I'd build today for an agency

1. Ahrefs → discover good prospect sites and broken competitor pages.
2. Screaming Frog → crawl prospect domains for live broken outbound links.
3. Wayback Machine → inspect what the dead page used to contain.
4. AI + your CMS → create a better replacement resource.
5. Outreach platform (BuzzStream, Pitchbox, or Respona) → contact site owners.

This combination tends to produce higher-quality broken link opportunities than relying on any single tool.

Yes. If your workflow is:

list of target domains → crawl their pages → find outbound links/resources that are dead → identify the dead resource → find a replacement opportunity, there are a few good options.

\* Usually means setting up separate crawls/projects or feeding URLs/sitemaps rather than simply dropping thousands of domains into one click.

1. Ahrefs — probably your best fit

Ahrefs has a particularly useful Outgoing Links → Broken Links report. Give it a target domain/URL and it can show pages on that site containing broken external links, including the destination URL and the reason it's broken (e.g. 404 or DNS failure). You can export the results. help.ahrefs.com

This is almost exactly the data you want for broken-link building:

Target site → page containing broken link → dead resource → replacement opportunity

It also has the complementary Broken Backlinks report, which is extremely useful after finding a dead resource: you can see other sites linking to that dead URL and prospect those sites for replacement links. help.ahrefs.com

My pick if you're doing this manually or semi-automatically.

2. Screaming Frog SEO Spider — best if you want control

Screaming Frog is better if you want to actually crawl the target websites yourself rather than relying primarily on an SEO link index.

It can crawl a site's internal and external links, return HTTP status codes, and export the results. That makes it particularly useful when you want to inspect every link/resource discovered during a crawl. www.techradar.com

For example, you could build a workflow like:

targets.csv
   ↓
Screaming Frog
   ↓
crawl each domain
   ↓
extract external URLs
   ↓
HTTP status = 404/410/5xx/etc.
   ↓
deduplicate dead resources
   ↓
Ahrefs → backlinks to those resources
   ↓
prospect sites for outreach

This is probably the better choice if you have hundreds/thousands of target domains and want to automate the crawling side.

3. Semrush

Semrush's Site Audit can check external links when "Check HTTP status of external links" is enabled, and it reports pages containing links to broken destinations. help.ahrefs.com

I'd rank it behind Ahrefs for this particular use case, mainly because Ahrefs' link-index data is exceptionally useful for the second half of broken-link building—finding backlinks to the dead resource.

4. Sitebulb

Sitebulb is another strong crawler if you want a more visual technical-SEO interface. It's worth considering if you're doing large audits and want to analyze the relationships between source pages, links, redirects, status codes, etc.

But there's an important distinction

If you're specifically doing broken-link building, I wouldn't just look for:

"Which links on these websites return 404?"

I'd optimize for:

Which broken external resources have the highest-quality replacement potential?

So I'd combine a crawler + backlink database.

The stack I'd use

For ~10–500 target domains:

Ahrefs → Broken Outgoing Links → filter → export → Ahrefs backlinks

For each dead URL:

1. Find the pages linking to it.
2. Check whether the dead resource was genuinely useful/relevant.
3. Use the Wayback Machine to understand what the original resource contained.
4. Create/improve your replacement resource.
5. Contact the sites linking to the dead resource.

Ahrefs itself recommends essentially this approach: find broken outbound links, then identify all backlinks pointing at the broken resource and offer a relevant replacement. help.ahrefs.com

For thousands+ of domains:

I'd use Screaming Frog or a custom crawler for discovery, then Ahrefs API/data for backlink enrichment. That gives you much more control over crawling, concurrency, deduplication, status-code validation, and prioritization.

If you're planning to do this at scale (e.g. 1,000–100,000 domains), I can also lay out the exact cheap → mid-range → enterprise stack, including which APIs/crawlers can accept a CSV of domains and how I'd structure the pipeline.

If you're doing broken link building at scale, the best tools depend on whether you want an all-in-one SEO platform or a dedicated crawler. Here's how they compare.

Best options

1. Screaming Frog (my top recommendation)

This is the standard tool for broken link building.

You can:

• Import a CSV of hundreds or thousands of domains.
• Crawl each site.
• Export every external link returning 404, 410, 5xx, or timeout.
• Filter by HTTP status code.
• Automate via CLI and scheduled crawls.

Typical workflow:

1. Import target domains.
2. Crawl.
3. Export "External Links."
4. Filter for broken URLs.
5. Find which pages link to those URLs.
6. Create or identify replacement content.
7. Conduct outreach.

2. Sitebulb

Very similar crawling capability, with:

• Better visualizations
• Easier reporting
• Excellent filtering

Many agencies prefer it if they don't need Screaming Frog's advanced customization.

3. Ahrefs

Instead of crawling everything yourself, Ahrefs already has a massive web index.

Useful reports include:

• Broken outbound links
• Broken backlinks
• Best by Links → 404 pages
• Outgoing links returning 404

It's excellent for prioritizing opportunities with stronger authority.

4. Semrush

Includes:

• Site Audit
• Broken external links
• Crawl management
• API (on higher plans)

Useful if your agency already uses Semrush.

For very large campaigns (10k–100k+ domains)

Many agencies build custom pipelines using:

• Scrapy
• Crawl4AI
• Playwright (for JavaScript-heavy sites)
• HTTPX (ProjectDiscovery) to verify broken URLs quickly
• Python Requests for lightweight HTTP checks

A common workflow is:

1. Crawl pages.
2. Extract all external links.
3. Deduplicate URLs.
4. Check HTTP status.
5. Identify 404/410/5xx resources.
6. Map referring pages.
7. Match broken resources to your content inventory.
8. Generate outreach lists.

If you're using AI for outreach

A productive stack is:

• Screaming Frog for crawling
• Ahrefs for authority and backlink data
• Crawl4AI or Scrapy for automation
• An LLM to analyze the dead resource (using archived versions where available), identify the closest replacement on your site, and draft personalized outreach emails.

That combination scales well from a few hundred prospects to tens of thousands while keeping manual work to a minimum.

If you're doing broken link building at scale, the tools fall into two categories:

1. Site crawlers that find broken outbound/internal links on a domain.
2. Backlink databases that find broken pages with existing backlinks (often more valuable).

Here's how the main options compare.

Best options by workflow

1. Crawl hundreds or thousands of target sites

Use Screaming Frog.

You can:

• Import a list of domains
• Crawl each automatically
• Export:
  • 404s
  • External broken links
  • Broken images
  • Redirect chains
• Schedule crawls

This is the closest to an industry standard for broken link building.

2. Find the highest-value opportunities

Instead of looking for broken outbound links, look for dead pages that already have backlinks.

Ahrefs excels here:

• Best by Links report
• Broken pages report
• Broken backlinks report

This finds pages with dozens or hundreds of referring domains that you can recreate and pitch as replacements.

This approach generally produces better outreach opportunities than simply finding random broken outbound links.

3. Fully automated prospecting

Many agencies combine:

• Screaming Frog
• Ahrefs API
• Custom Python
• Google Sheets or Airtable
• AI for outreach personalization

Typical workflow:

1. Import 5,000 domains.
2. Crawl with Screaming Frog.
3. Export all external 404 links.
4. Deduplicate broken URLs.
5. Use Ahrefs API to check backlink value.
6. Score opportunities.
7. Generate outreach lists.

Open-source options

If you're comfortable coding:

• Scrapy (Python)
• Playwright (for JavaScript-heavy sites)
• requests + BeautifulSoup
• crawl4ai
• Apache Nutch (large-scale)

A simple crawler can:

• Visit each domain
• Collect every external link
• Check HTTP status
• Record 404/410/5xx responses
• Export CSV for outreach

This scales well if you have tens of thousands of targets.

Recommended stack

For most SEO teams:

• Screaming Frog for crawling domains.
• Ahrefs for prioritizing broken pages that have valuable backlinks.
• A custom Python script (or no-code automation) to merge exports, remove duplicates, and prepare outreach.

That combination gives a strong balance of scale, accuracy, and efficiency for broken link building campaigns.

If you're doing broken link building at scale, you generally want tools that can:

1. Crawl a list of domains.
2. Find broken outbound links (404s, 410s, DNS failures, etc.).
3. Export the pages containing those broken links.
4. Ideally identify the anchor text and the dead URL so you can pitch a replacement.

Here are the strongest options, depending on your workflow.

1. Screaming Frog (my top recommendation)

This is probably the gold standard.

Workflow:

• Import a CSV of target domains.
• Crawl each site.
• Filter External → Client Error (4xx).
• Export:
  • Source page
  • Broken URL
  • Anchor text
  • Status code

Pros:

• Extremely fast
• Millions of URLs possible
• Supports scheduling
• API integrations (Ahrefs, GA, GSC)
• JavaScript rendering if needed

Best for agencies and large campaigns.

2. Ahrefs

Instead of crawling yourself, you can use Ahrefs' existing index.

Useful reports:

• Broken Outgoing Links
• Best by Links → Broken
• Broken Backlinks
• Outgoing Links

This is excellent when you're looking for pages already linking to dead resources.

3. Sitebulb

Very similar to Screaming Frog but with more visual reporting.

Useful if you're auditing dozens of sites and want easier interpretation.

4. Netpeak Spider

A favorite for very large crawls.

Advantages:

• Faster on Windows
• Memory efficient
• Great bulk export
• Custom extraction
• Batch crawling

Many link-building agencies use it for large prospect lists.

5. Custom crawling

If you have hundreds or thousands of domains, a custom crawler can outperform desktop tools.

Typical stack:

• Python
• requests/httpx
• BeautifulSoup or lxml
• Playwright (for JavaScript-heavy sites)
• Async crawling
• Check every outbound link
• Store:
  • referring page
  • dead URL
  • anchor text
  • HTTP status
  • surrounding paragraph

This gives you exactly the dataset needed for outreach.

6. Scrapebox

Still popular among link builders.

Features:

• Bulk URL harvesting
• Broken link checking
• Page scraping
• Proxy support
• Very fast

It's more of a power-user tool than an SEO platform.

7. LinkMiner (broken links)

Useful for finding broken links on individual sites.

Less suitable for crawling hundreds of domains.

Ideal workflow for broken link building

Many agencies combine tools rather than relying on just one:

1. Build a prospect list (relevant websites).
2. Crawl those sites with Screaming Frog or Netpeak Spider.
3. Export all broken outbound links.
4. Use Ahrefs to check whether the dead resource had backlinks.
5. Recreate or improve the dead content.
6. Reach out to the page owners suggesting your replacement.

If you're crawling 1,000+ domains

For that scale, I'd recommend:

• Discovery & metrics: Ahrefs
• Bulk crawling: Screaming Frog SEO Spider or Netpeak Spider
• Automation: a custom Python crawler with asynchronous requests and a database for results

That combination gives you the flexibility of commercial SEO data along with the efficiency of your own crawls.

If you're doing broken link building at scale, the ideal setup depends on whether you want to crawl your prospects' sites for broken outbound links or find pages across the web that link to dead resources.

Here's a breakdown of the strongest options.

For large prospect lists (500–100,000 domains)

If you have a CSV of target domains:

1. Import into Screaming Frog.
2. Enable crawling of external links.
3. Crawl each site.
4. Export:
   • External 404s
   • Broken outbound links
   • Anchor text
   • Source URL

This workflow is common among agencies doing broken link outreach.

If you want to find "dead resources" across many sites

Instead of crawling every prospect, use backlink databases:

• Ahrefs
• Majestic

Typical workflow:

1. Find a dead page or expired resource.
2. Look up all backlinks pointing to it.
3. Contact those sites with your replacement content.

This is often faster than crawling thousands of sites manually.

For automation

If you're building your own pipeline:

• Scrapy — Python framework for custom crawlers.
• Crawl4AI — Open-source crawler designed for AI and web extraction workflows.
• Apify — Cloud platform for running scalable crawlers and scraping jobs.
• Firecrawl — API-first crawler that extracts structured website content.

With a custom crawler, you can:

• Crawl a list of target domains.
• Extract every external link.
• Check HTTP status codes (404, 410, 500, timeouts).
• Store the referring page, anchor text, destination URL, and domain authority metrics.
• Feed opportunities directly into your outreach CRM.

A scalable stack

For most SEO teams doing broken link building at scale:

• Prospecting: Ahrefs
• Crawling: Screaming Frog SEO Spider
• Automation: Apify or Scrapy
• Outreach: BuzzStream or Pitchbox

If you're planning to crawl tens of thousands of domains, a cloud-based crawler (Apify or a custom Scrapy deployment) generally scales better than desktop tools, while Screaming Frog remains an excellent choice for smaller batches and detailed audits.

If you're doing broken link building at scale, you generally want tools that can:

1. Crawl a list of websites.
2. Detect 404s or broken outbound links.
3. Export the URLs, anchor text, and linking pages.
4. (Ideally) integrate with backlink data so you can prioritize opportunities.

Here's how the main tools compare:

1. Screaming Frog (most flexible)

Ideal if you already have a list of prospects.

Workflow:

• Import hundreds or thousands of domains.
• Crawl each site.
• Filter for:
  • External links
  • Response Code = 404
  • 410
  • Timeout
• Export:
  • Source page
  • Broken URL
  • Anchor text

Pros:

• Extremely configurable
• JavaScript rendering
• API integrations
• Automation mode
• Can crawl millions of URLs (with enough RAM)

Cons:

• Doesn't tell you backlink value.

2. Ahrefs (best overall for broken link building)

This is arguably the strongest dedicated tool.

You can:

• Enter a domain.
• Go to:
  • Broken Links
  • Best by Links → 404
  • Outgoing Links → Broken
• Find pages that:
  • Used to exist
  • Had backlinks
  • Now return 404

You can then recreate similar content and reach out to linking sites.

This is one of the classic broken link building workflows.

3. Sitebulb

Very similar to Screaming Frog but easier to interpret.

Finds:

• Broken internal links
• Broken external links
• Redirect chains
• 404 resources

Good for agencies.

4. Xenu Link Sleuth

Old but still effective.

Advantages:

• Free
• Very fast
• Finds broken links

Disadvantages:

• Windows only
• No JavaScript
• Dated interface

5. Custom crawler

If you have thousands of target domains, many SEOs build their own pipeline using:

• Scrapy
• Crawl4AI
• Playwright
• BeautifulSoup
• Async HTTP libraries

Typical workflow:

target domains
      ↓
crawl homepage
      ↓
crawl important pages
      ↓
extract external links
      ↓
HEAD/GET requests
      ↓
identify 404/410
      ↓
export:
- source page
- anchor
- dead URL
- status

This is much cheaper than paying for enterprise SEO tools if you're comfortable with Python.

Large-scale SaaS options

If you're crawling tens of thousands of prospects:

• Ahrefs
• Screaming Frog (CLI + scheduling)
• Sitebulb Cloud
• Botify (enterprise)
• Oncrawl (enterprise)

Recommended stack for outreach

Many link builders combine tools rather than relying on one:

• Screaming Frog → crawl prospect sites and discover broken outbound links.
• Ahrefs → evaluate the authority of the linking pages and investigate whether the dead resource previously attracted backlinks.
• Outreach platform (such as Pitchbox, BuzzStream, or Respona) → manage contact discovery and email campaigns.

That combination gives you both technical discovery (what's broken) and prioritization (which opportunities are most valuable).