ChatGPT answers, unedited

For large-scale broken-link prospecting on authority sites, I’d use a stack rather than one tool. The key is separating discovery, validation, prioritization, and outreach.

My preferred stack

1. Ahrefs should probably be your primary engine

This is the strongest fit for what you're describing.

In Ahrefs Site Explorer, you can take a competitor or authoritative site and use Best by Links → HTTP code: 404 to identify dead pages that previously accumulated backlinks. You can then inspect the referring domains and prioritize the pages worth pursuing. Ahrefs specifically documents this as a broken-link-building workflow. ahrefs.comwww.semrush.comwww.semrush.comhelp.ahrefs.comhelp.ahrefs.com

Even better, Ahrefs has an Outgoing Links → Broken Links report that identifies pages on a target site containing broken external links. That means you can approach an authority site's webmaster with a relevant replacement resource. help.ahrefs.com

For scale, I'd run two prospecting modes:

A. Dead competitor resources

competitor → 404 pages → referring domains → filter for authority/relevance

B. Broken links on authority sites

authority domain → broken outbound links → identify relevant dead resource → create/identify replacement → outreach

The second workflow is particularly useful if your niche has lots of universities, associations, publications, government sites, resource hubs, etc.

2. Semrush is a useful second database

I'd use Semrush primarily to expand the universe of prospects and cross-check Ahrefs rather than replacing it.

Its backlink tooling lets you filter backlink data by authority and link status, and its Backlink Audit includes a Target URL Error filter for backlinks pointing to broken pages. www.semrush.com

Its backlink database also supports competitor-oriented prospecting, including finding domains that link to competitors but not you. www.semrush.com

So a good large-scale workflow is:

Ahrefs discovery → Semrush enrichment/cross-check → crawler validation → outreach

3. Screaming Frog is excellent when you already have target sites

This is where I'd change the approach if you have, say, 5,000 authority domains you've already identified.

Instead of asking a backlink database to discover everything, feed those domains into a crawler and look for:

• 404/410 external links
• broken resource links
• redirected links
• broken internal links
• pages containing multiple broken references
• relevant resource/library pages

Then use Ahrefs/Semrush to determine whether the dead destination had meaningful backlinks and whether the referring domain is actually valuable.

4. Use Wayback Machine for qualification

Don't automatically pursue every 404.

Once you find something promising, check the historical version of the dead URL. You're trying to establish:

Was this actually a useful resource that an authority site intentionally cited?

If the old page was a substantive guide, study, dataset, tool, report, etc., that's much more attractive than a random dead commercial page.

It also tells you what your replacement needs to cover.

5. The biggest scaling trick: prioritize pages, not domains

Don't make your prospecting unit:

"Find authoritative websites with broken links."

Make it:

"Find dead resources with lots of relevant authoritative referring pages."

For example, imagine you discover a dead page that formerly covered 2023 renewable-energy statistics.

Instead of contacting one website, you might find:

• 73 referring domains
• 18 highly authoritative domains
• 11 genuinely relevant resource pages
• 6 university/association/government pages

That's one dead-resource discovery potentially producing multiple outreach prospects.

Ahrefs' broken-backlink reporting lets you sort/filter referring pages using metrics such as DR and inspect anchor/surrounding text, which makes this prioritization practical. ahrefs.comwww.semrush.comwww.semrush.comhelp.ahrefs.comhelp.ahrefs.com

A scalable pipeline

I'd structure the operation roughly like this:

1. Build authority-site universe
→ universities
→ government
→ associations
→ major publications
→ industry organizations
→ high-quality niche sites

2. Crawl/discover broken outbound links

3. Deduplicate dead URLs

4. Check historical content

5. Check backlink profile of dead URL

6. Score opportunity

Something like:

Opportunity Score = relevance × referring-domain quality × number of relevant links × replacement fit

7. Find/contact the actual referring pages

8. Only pitch when you have a genuinely better replacement

That last part matters. Broken-link outreach works much better when the pitch is essentially "I noticed this resource is dead and here's a directly relevant replacement" rather than generic link-building outreach.

If I were doing this at serious scale

I'd go with:

Ahrefs + Screaming Frog + Semrush + Wayback Machine + a spreadsheet/database

If budget is tighter, start with Ahrefs + Screaming Frog + Wayback Machine. Ahrefs currently also offers limited free access for verified sites, although the more useful competitor prospecting capabilities are on paid access. help.ahrefs.com

If you tell me your niche + roughly how many authority domains you want to prospect (100, 1,000, 10,000+), I can lay out a concrete large-scale workflow—including the exact Ahrefs/Semrush filters and a scoring system for deciding which broken links are worth outreach.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is: find authoritative sites → identify their broken outbound links or broken pages → verify the opportunity → qualify the referring page → outreach.

Best tools

1. Ahrefs — best overall
   • Use Site Explorer → Outgoing Links → Broken Links to find pages on authority sites that link to dead external URLs. Ahrefs lets you export these results and filter them. help.ahrefs.com
   • For the reverse approach, use Broken Backlinks / Best by Links → HTTP 404 on competitors. This finds authoritative sites already linking to dead competitor content—often excellent replacement-link prospects. help.ahrefs.com
   • If you're doing this at scale, Ahrefs' API can return up to 1,000 results per request and exposes metrics such as referring-domain rating, URL rating, traffic, anchor, HTTP status, and source URL. docs.ahrefs.comhelp.ahrefs.com

2. Semrush — strong alternative/second data source
   • Its Backlink Analytics data can identify competitor backlinks and broken pages. www.semrush.com
   • The Broken Pages filter specifically surfaces pages that couldn't be crawled and lets you investigate competitors' broken backlinks. www.semrush.com
   • Useful for cross-checking Ahrefs because backlink indexes aren't identical.

3. Screaming Frog SEO Spider — best verification/crawling layer
   • Give it a list of authority-site URLs and crawl them to identify outbound 4xx links.
   • This is particularly useful after you've built a prospect list from Ahrefs/Semrush, because you can independently verify whether the link is actually dead.

4. A crawler + API/database workflow — best for truly massive scale
   • If you're processing tens of thousands/millions of URLs, don't manually inspect Ahrefs results.
   • Export/API the candidate URLs, then run HTTP checks yourself and store:
     • source URL
     • dead destination
     • status code
     • anchor text
     • referring-domain authority
     • organic traffic
     • topical relevance
     • your replacement URL
   • Ahrefs' API is particularly suitable for this because its broken-backlink endpoint exposes the source URL, target URL, HTTP information, DR/UR, traffic, anchor and other fields. docs.ahrefs.comhelp.ahrefs.com

The workflow I'd use

Ahrefs → Screaming Frog/custom verification → spreadsheet/database → outreach

For example, if your niche were cybersecurity:

1. Build a list of 500–5,000 authoritative cybersecurity domains.
2. Run each through Ahrefs Outgoing Links → Broken Links.
3. Filter for:
   • relevant topical category
   • high referring-domain authority
   • meaningful organic traffic
   • contextual/editorial links
   • 404/410 rather than temporary errors
4. Deduplicate the dead URLs.
5. Check how many other authoritative sites link to each dead resource.
6. Find the original/dead resource using Wayback Machine or other historical sources.
7. Create or identify a genuinely equivalent replacement on your site.
8. Re-crawl the referring pages to verify the link is still broken.
9. Prioritize prospects where one replacement resource can earn links from many high-quality domains.

The important scaling trick

Don't think of the unit of opportunity as “one broken link.”

Think:

one dead resource → all authoritative sites linking to it

That's where broken-link prospecting gets interesting. A single high-authority dead resource can reveal dozens or hundreds of relevant prospects. Ahrefs' broken-backlink data is particularly useful for this because it can show the referring pages, anchors and source metrics around a dead target. docs.ahrefs.comhelp.ahrefs.com

If I were building the stack today: Ahrefs for discovery + Screaming Frog/custom HTTP verification + a database/spreadsheet for qualification + an outreach platform for execution. Semrush is worth adding if broken-link prospecting is a major part of your operation and you want a second backlink index to expand coverage.

If you tell me your niche and roughly how many authority domains you want to prospect (1k, 10k, 100k+), I can lay out a specific large-scale workflow, including the exact Ahrefs filters and prioritization formula.

For large-scale broken-link prospecting on authority sites, I’d build the workflow around a backlink index rather than trying to crawl the web yourself.

My preferred stack

1. Ahrefs — primary prospecting database
   • Content Explorer is particularly useful: search a broad niche topic, filter to broken pages, then prioritize pages with lots of referring domains. Ahrefs explicitly supports finding broken-link opportunities this way. ahrefs.com
   • Site Explorer → Best by Links → HTTP code: 404 is excellent for reverse-engineering competitors' dead content. You can then inspect the domains linking to those dead pages. ahrefs.comhelp.ahrefs.com
   • Its Broken Backlinks report gives you referring pages, target URLs, anchors, and surrounding text, which is useful for qualifying prospects at scale. ahrefs.comhelp.ahrefs.com

2. Semrush — secondary database / validation
   • Useful for backlink discovery, competitor link-gap analysis, and validating the authority/relevance of domains you find. Its backlink tools include referring domains, backlink status, authority metrics, and competitor gaps. www.semrush.com
   • I wouldn't make Semrush your only broken-link database; I'd use it alongside Ahrefs to increase coverage.

3. Screaming Frog SEO Spider — verification/crawling
   • This is fantastic once you've assembled a list of authority URLs.
   • Upload thousands of prospect URLs in List Mode, crawl their external links, and filter response codes for 4XX. Screaming Frog specifically documents this broken-link-building workflow. www.screamingfrog.co.uk
   • This is especially valuable because backlink databases won't necessarily contain every link on every page. ahrefs.com

The scalable workflow I'd use

Ahrefs → candidate discovery → Screaming Frog → qualification → outreach

For example:

1. Build a seed list

• 20–100 authoritative sites in your niche
• Major publications
• Universities / .edu resources
• Industry associations
• Government resources
• Established companies
• Highly linked niche blogs

2. Find dead assets
Use Ahrefs to identify:

• 404 pages with lots of referring domains
• Expired resources
• Dead competitor content
• Broken pages around your target topics

Prioritize something like:

Relevant topic + 10+ referring domains + authoritative linking sites

rather than simply hunting for the largest number of 404s.

3. Reconstruct the opportunity

For each dead URL, capture:

4. Verify before outreach

Run the referring pages through Screaming Frog. This prevents wasting outreach on stale backlink-index data and lets you confirm that the page actually contains a broken external link. www.screamingfrog.co.uk

5. Score prospects

I'd give each opportunity a simple score:

Opportunity = relevance × authority × link value × replacement fit

A DR 80 site linking to a dead page isn't necessarily a good prospect if the link is buried on an irrelevant page. A highly relevant DR 50 industry resource can be substantially better.

If you're doing really large volumes

I'd use:

Ahrefs Content Explorer + Site Explorer
→ export candidates
→ Screaming Frog for bulk verification
→ Google Sheets/Airtable/database for scoring
→ your outreach platform for contact discovery and campaigns.

Ahrefs also offers scheduled broken-link checking and CSV exports, which makes it easier to turn this into a recurring prospecting process rather than a one-off campaign. ahrefs.com

If I had to choose only one tool: Ahrefs. Its combination of a huge backlink index, 404 filtering, referring-domain data, and topic-based broken-page discovery makes it unusually well suited to this particular tactic. ahrefs.com

If you tell me your niche, I can also lay out a specific high-volume workflow for finding hundreds/thousands of authority-site broken-link opportunities, including the exact Ahrefs/Screaming Frog filters I'd use.

If you're doing large-scale broken-link prospecting, I’d build the stack around a backlink index rather than a generic crawler. The key is to find dead pages that already have strong referring domains, then filter those referring sites for authority and topical relevance.

Best tools

• ahrefs.com — best overall for this workflow.

Its Content Explorer can search for broken pages by topic and filter by referring domains, while Site Explorer → Best by Links → HTTP 404 is excellent for mining competitors' dead pages. Ahrefs also has a dedicated Broken Backlinks report and lets you export broken outbound links from sites. Ahrefs Ahrefs Ahrefs Help Center

• semrush.com — strong second database.

In Backlinks → Indexed Pages, you can filter a competitor's pages by Broken Pages, sort by backlinks/domains, then drill into the referring URLs. This is particularly useful for validating opportunities you found in Ahrefs with a second index. Semrush Semrush

• screamingfrog.co.uk — best for crawling individual authority sites.

I'd use it when you already have a list of high-value domains and want to crawl their outbound links yourself rather than relying entirely on a backlink database.

• majestic.com — useful as a supplementary link index.

Particularly useful when you want another backlink dataset for validating whether a supposedly valuable dead page actually has a meaningful link profile.

The scalable workflow I'd use

1. Start with 20–100 authoritative sites/competitors in your niche.

Feed each competitor into Ahrefs and look for:

Best by Links → HTTP code: 404

Then sort/filter by referring domains rather than raw backlinks. A dead page with 80 links from 15 domains is usually more interesting than one with 100 links from two sites. Ahrefs explicitly recommends this competitor-broken-page workflow. Ahrefs

2. Expand beyond competitors with Content Explorer.

This is where Ahrefs becomes especially useful for scale. Search broad niche topics and select broken pages, then require a minimum number of referring domains. That lets you discover dead resources that aren't necessarily on your competitors' domains. Ahrefs Ahrefs

3. Export the referring pages.

Your prospect database should ultimately look something like:

Then deduplicate by referring domain.

4. Prioritize authority + relevance, not just DR/DA.

For "authority sites," I'd score prospects roughly:

• Topical relevance — 30%
• Quality of the linking page — 25%
• Referring-domain authority — 20%
• Link placement/context — 15%
• Organic traffic/visibility — 10%

That prevents you from ending up with a giant list of technically authoritative but irrelevant websites.

5. Verify the dead URL manually before outreach.

This matters. A crawler/database can report a 404 or other error when a page is temporarily inaccessible. Semrush specifically recommends verifying that the page is genuinely broken before pursuing it. Semrush

My preferred stack

For serious volume, I'd use:

Ahrefs → Screaming Frog → Semrush → Google Sheets/Airtable

Ahrefs does the discovery, Screaming Frog does verification/crawling, and Semrush gives you a second backlink dataset for validation.

If you're doing this at very large scale—say 10,000+ prospects/month—I'd also automate the pipeline so that Ahrefs/Semrush exports get normalized, deduplicated, scored, and filtered before a human ever sees the outreach list.

The important distinction is that you don't really want a "broken link checker." You want a broken-page + backlink + authority + topical-relevance prospecting system. Ahrefs is currently the closest single tool to that workflow. Ahrefs Ahrefs

That prevents you from ending up with a giant list of technically authoritative but irrelevant websites.

5. Verify the dead URL manually before outreach.

This matters. A crawler/database can report a 404 or other error when a page is temporarily inaccessible. Semrush specifically recommends verifying that the page is genuinely broken before pursuing it.

My preferred stack

For serious volume, I'd use:

Ahrefs → Screaming Frog → Semrush → Google Sheets/Airtable

Ahrefs does the discovery, Screaming Frog does verification/crawling, and Semrush gives you a second backlink dataset for validation.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key distinction is whether you want to find dead pages with lots of authoritative backlinks or crawl authority sites to discover their outbound broken links.

My preferred stack

If I could pick only one: Ahrefs. Its Broken backlinks report gives you the referring pages, dead target URL, anchor text and surrounding text, and its Best by Links report can surface 404 pages with substantial backlink profiles. ahrefs.com

The high-scale workflow

1. Build a prospect universe

Start with 100–1,000 authority domains in your niche—universities, government sites, major publications, associations, established companies, etc.

Then use Google searches such as:

• keyword intitle:resources
• keyword intitle:links
• keyword inurl:resources
• site:.edu keyword
• site:.org keyword

The goal isn't to find broken links directly; it's to create a list of high-value pages worth crawling.

2. Crawl those domains with Screaming Frog

Feed your prospect URLs/domains into Screaming Frog and crawl them. It can identify 404s/server errors and bulk-export the source URLs pointing to those errors. www.screamingfrog.co.uk

This is particularly useful for the opportunity you're describing:

Authority site → outbound link → dead resource

You can export the source page, broken destination, anchor text, status code, etc., then process thousands of opportunities in a spreadsheet/database.

3. Run the dead URLs through Ahrefs

This is where I'd add the authority/backlink layer.

For every interesting dead URL, check:

• Referring domains
• Domain Rating
• Organic traffic/history
• Anchor text
• Number of referring pages
• Relevance to your content
• Whether the referring domains themselves are authoritative

Ahrefs specifically recommends finding competitors' broken pages with backlinks and then contacting the sites linking to those pages with a relevant replacement. ahrefs.com

4. Use Ahrefs Content Explorer for discovery at scale

This is one of the better ways to avoid limiting yourself to just competitor domains. Search your niche/topic, restrict results to broken pages, then prioritize pages with lots of referring domains. Ahrefs documents this workflow explicitly. ahrefs.com

That's potentially much more scalable than manually crawling individual authority sites.

Where Semrush fits

Semrush is a good second source for backlink data and competitor discovery. Its Backlink tools can identify competitor backlinks, while Backlink Audit exposes broken target URLs and broken/lost backlink data. www.semrush.com

I'd use it primarily to cross-check and enrich Ahrefs data, rather than replacing Ahrefs + Screaming Frog.

The important filtering step

Don't just collect every 404. At scale, you'll generate enormous amounts of garbage.

I'd score opportunities roughly like:

Opportunity Score = Authority × Relevance × Link Count × Replacement Fit

For example, prioritize:

Authority site → highly relevant dead resource → 20+ referring domains → you have an almost exact replacement

over:

Authority site → random dead URL → 1 backlink → vaguely related to your niche

You can also use the Wayback Machine to see what the dead page actually contained. That lets you determine whether you can create a genuinely useful replacement rather than sending a generic "hey, your link is broken" email.

If you're doing really large-scale prospecting

I'd build this pipeline:

Prospect domains → Screaming Frog crawl → 4xx outbound links → deduplicate URLs → Ahrefs backlink/authority enrichment → Wayback verification → relevance scoring → outreach

That gives you a much more scalable system than trying to do broken-link building entirely inside an SEO suite.

One particularly useful trick: don't limit yourself to competitors' dead pages. Search for dead resources by topic, then identify which authority sites link to them. That's the approach that can turn broken-link building from a handful of prospects into a repeatable prospecting engine. ahrefs.com

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is:

find authoritative domains → discover dead URLs → identify pages linking to them → qualify prospects → recreate/recommend your resource → outreach.

My recommended stack

1. Ahrefs — primary prospecting database
   • Best overall choice for this tactic.
   • Use Site Explorer → Best by Links → HTTP code: 404 on competitor/authority domains to find dead pages that still have backlinks.
   • Then inspect the referring domains/pages and prioritize by DR, relevance, traffic, etc. Ahrefs explicitly supports this broken-page workflow. ahrefs.com
   • Its Broken links report also finds outbound links on a target site that point to dead pages. help.ahrefs.com

2. Screaming Frog SEO Spider — live verification/crawling
   • Excellent for actually crawling authority sites and verifying external 4XX links rather than relying solely on an index.
   • It can show the source page → broken URL, anchor text, follow/nofollow status, and bulk-export the results. www.screamingfrog.co.ukhelp.ahrefs.com
   • Particularly useful when you have a defined list of 100s/thousands of target domains.

3. Semrush Backlink Analytics — second-source validation
   • Useful for competitor backlink discovery, referring-domain analysis and filtering prospects.
   • I wouldn't make it the centerpiece of broken-link hunting, but it's valuable for cross-checking Ahrefs and enriching your prospect database. www.semrush.com

4. Wayback Machine — reconstruct the dead resource
   • Once you've found a promising 404, check what used to exist there.
   • The goal is to create a better, current replacement, not merely pitch an unrelated page.

For truly large-scale campaigns

I'd structure the workflow like this:

Ahrefs → CSV → database/spreadsheet → Screaming Frog/live checks → Wayback → qualification → outreach

For example, start with 50–500 authority domains in your niche. Pull their broken pages, then rank each opportunity roughly by:

relevance × referring domains × authority of referring sites × topical value

The really valuable opportunities aren't necessarily pages with the highest DR. A dead resource with 20 highly relevant .edu/.gov/industry-authority links can be much more interesting than one with 500 mediocre links.

One particularly scalable tactic

Instead of crawling random sites looking for broken links, start from competitors' dead pages with existing backlinks.

In Ahrefs:

Competitor → Best by Links → HTTP code = 404 → sort by Referring Domains

Then investigate the dead URLs with the most relevant referring domains. Ahrefs specifically recommends this as a broken-link-building workflow. ahrefs.com

That turns the problem from:

"Which authority sites have broken links?"

into:

"Which dead resources in my niche already have authority sites linking to them?"

That's usually a much higher-signal prospecting approach.

If I were buying only two tools: Ahrefs + Screaming Frog. Ahrefs does the large-scale link intelligence; Screaming Frog gives you independent, live crawling/verification. www.screamingfrog.co.ukhelp.ahrefs.com

If you tell me your niche, I can also lay out a concrete large-scale workflow (including the exact Ahrefs filters, prospect scoring, and how I'd automate the CSV → verification → outreach pipeline).

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key distinction is whether you want to find dead pages with lots of backlinks or broken outbound links on authoritative sites—both are useful, but they require slightly different workflows.

My recommended stack

If you're doing this seriously at scale, Ahrefs + Screaming Frog is probably the strongest combination.

1. Ahrefs — your primary prospecting database

Ahrefs is particularly well suited to this because its Best by links report can be filtered for 404 pages. You can take a competitor or authority site, identify its dead pages, and then inspect the sites still linking to those pages. ahrefs.com

A very scalable workflow is:

Competitor/authority domain → Best by links → HTTP code: 404 → sort by referring domains → inspect linking sites

You're looking for something like:

Dead resource → 83 referring domains → 12 high-authority referring domains → several highly relevant sites

That's much more valuable than simply finding random broken links.

Ahrefs' Content Explorer is also explicitly designed to surface broken-link opportunities by topic, using its large content index. ahrefs.com

Ahrefs Content Explorer

2. Screaming Frog — find broken links on authority sites

This is the other half of the strategy.

Suppose your niche is cybersecurity and you've identified 500 authoritative resource pages. You can crawl those sites and identify their outbound links that return 404s.

Screaming Frog is excellent when you already have a target list of authority domains/pages and want to crawl them yourself. Its custom extraction functionality also lets you pull additional HTML data when needed. www.screamingfrog.co.uk

The advantage is that you're finding opportunities like:

authoritysite.com/resources
→ links to deadcompetitor.com/old-research.html

You then create/recommend a genuinely relevant replacement.

3. Semrush — useful second database

Semrush Backlink Analytics is useful as a second source of backlink data, particularly for validating prospects and finding competitor backlink gaps. Semrush reports backlink status and provides competitor-oriented backlink analysis. www.semrush.com

Its Backlink Audit system also tracks new, lost, and broken backlinks, although I'd generally make Ahrefs the primary broken-link discovery tool for this particular workflow. www.semrush.com

4. Wayback Machine — validation, not discovery

Once you find a promising dead URL, check its historical versions.

You want to establish:

• What the original page was about
• Whether it actually matches your niche
• What information it contained
• Whether you can make something better, rather than simply cloning it
• Whether the backlinks are contextually relevant

This prevents wasting outreach on a dead URL that only looks attractive because it has a high backlink count.

The scalable workflow I'd use

Phase 1 — Build the target universe

Collect 50–500 authoritative domains in your niche:

• universities
• government sites
• industry associations
• major publications
• established companies
• respected blogs
• resource directories
• research organizations

Phase 2 — Mine dead assets

Run those domains and major competitors through Ahrefs.

Prioritize dead URLs according to something like:

relevance × referring domains × authority of referring sites × topical quality

Don't just sort by number of backlinks.

Phase 3 — Find the actual linking pages

Export the backlinks/referring pages for your best dead URLs. Ahrefs' Broken Backlinks report specifically provides the referring pages, anchors, and surrounding text for links pointing at 404s. ahrefs.com

Phase 4 — Independently crawl your highest-value prospects

Feed your authority-site list into Screaming Frog and look for broken external links. This can uncover opportunities that aren't obvious from backlink databases.

Ahrefs itself also has an outbound Broken Links report for finding pages on a domain that link to broken external URLs. help.ahrefs.com

Phase 5 — Filter aggressively

I'd only retain prospects where:

• The dead resource is genuinely relevant to your content.
• The linking page is authoritative and indexed.
• The replacement you're offering is a legitimate improvement.
• The link isn't obviously accidental/spammy.
• The page isn't stuffed with hundreds of unrelated outbound links.

One important trick

Don't limit yourself to competitor domains.

For large-scale prospecting, search for dead resources in your topical ecosystem.

For example, if your niche were accounting software, I'd investigate:

• old accounting research
• discontinued calculators
• obsolete industry reports
• university resources
• government publications
• old statistics pages
• defunct industry organizations
• archived guides

Ahrefs' Content Explorer is particularly useful here because you can search by topic and surface high-value broken pages with backlinks. ahrefs.com

If I were building the stack

I'd go:

Ahrefs → Screaming Frog → Wayback Machine → Google Sheets/CRM → outreach

And I'd use Semrush as a secondary backlink database, especially when Ahrefs and Semrush disagree or when you want another source for discovering prospects.

If you're talking about thousands or tens of thousands of authority pages, I can also lay out a fully automated broken-link prospecting pipeline (including crawling, HTTP validation, Ahrefs exports/API, authority scoring, deduplication, and prioritization).

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is:

find authoritative dead pages → identify everyone linking to them → qualify the referring sites → map each opportunity to your replacement content → automate outreach.

Best tools

My preferred setup

If you're serious about scale, I'd make Ahrefs the discovery engine.

For each competitor/authority site in your niche:

1. Put the domain into Ahrefs Site Explorer.
2. Go to Best by links / Pages and filter for 404.
3. Prioritize dead pages with lots of relevant referring domains, rather than simply lots of backlinks.
4. Open the broken page's backlink list.
5. Export the referring URLs/domains.
6. Filter for sites that meet your authority/relevance criteria.
7. Check the actual referring page to make sure the link is genuinely broken.
8. Identify which of your existing pages is the closest replacement—or create a substantially better replacement where the opportunity justifies it.
9. Personalize outreach around the specific broken resource.

Ahrefs explicitly supports the competitor-broken-page use case: find competitors' broken pages with backlinks, then contact the sites linking to those pages with a relevant replacement. ahrefs.comhelp.ahrefs.com

The important scaling trick

Don't start with "find broken links on the web."

Start with:

"Find dead URLs that already have backlinks from authoritative sites in my niche."

That dramatically improves the signal-to-noise ratio.

For example, suppose you're in the personal-finance niche. Rather than crawling thousands of sites looking for arbitrary 404s, you'd build a list of perhaps 100–500 authoritative competitors, publishers, universities, associations, government resources, etc. Then systematically identify:

Authority domain → dead resource → number/quality of referring domains → referring page → replacement opportunity

You can then score each opportunity something like:

Opportunity score = topical relevance × referring-domain quality × number of unique referring domains × replacement quality

This is much more scalable than simply sorting by raw backlink count.

For truly large-scale campaigns

I'd use two backlink databases, not one.

Ahrefs can give you the primary dataset; Majestic or Semrush can provide a second view. That's useful because link indexes aren't identical. Majestic, for example, exposes both current and lost-link data plus contextual information and topical metrics. majestic.com

Then use Screaming Frog selectively to validate/crawl domains where database data isn't enough. Its crawler can directly identify 404s and the pages containing them. screaming-frog.org

If I were building this today:
Ahrefs → CSV/API → database/Sheets → Majestic/Semrush enrichment → Screaming Frog validation → outreach platform.

The biggest productivity gain won't come from finding more broken links; it'll come from automatically ranking the broken links by how likely the referring site is to accept your replacement.

For large-scale broken-link prospecting, I’d build the workflow around a backlink index rather than a traditional crawler. The core stack I’d use is:

1. Ahrefs — primary prospecting database

Ahrefs

This is probably the best starting point for scale. In particular:

• Site Explorer → Best by Links → HTTP code: 404 to find dead pages on competitors and authoritative sites.
• Broken Backlinks to see the actual referring pages pointing at those dead URLs.
• Filter prospects by DR, organic traffic, referring domains, anchor text, etc.
• Content Explorer → broken pages is useful when you want to search an entire niche rather than start with a particular competitor. ahrefs.comhelp.ahrefs.com
• Ahrefs also exposes a Broken Backlinks API, which is particularly interesting if you're building a high-volume internal prospecting pipeline. The API supports filters, ordering, CSV/JSON output, and up to 1,000 results per request by default. docs.ahrefs.com

For your use case, Ahrefs + API/exports is the backbone I'd choose.

2. Screaming Frog — verification and crawling

Screaming Frog SEO Spider

Use it after you've identified promising authority domains/pages.

It's excellent for crawling a specific site and finding:

• 404/410 outbound links
• broken external links
• redirect chains
• link context
• the exact page containing the broken link

The important distinction is that Screaming Frog is fantastic at crawling and verifying, while Ahrefs is much better suited to discovering thousands of link opportunities across the web.

3. Wayback Machine — reconstruct the dead resource

Wayback Machine

Once you find a valuable dead URL, check its historical versions. You want to determine whether the old resource was actually something worth recreating—not merely a random deleted page.

Ahrefs itself recommends using the Wayback Machine to understand what a competitor's dead page used to contain before deciding whether it's suitable for broken-link building. ahrefs.comahrefs.comhelp.ahrefs.com

4. Pitchbox / BuzzStream — outreach at scale

Once you've got the prospects, you need a way to manage:

broken URL → referring page → contact → replacement resource → outreach → follow-up

Ahrefs specifically recommends outreach platforms such as Pitchbox or BuzzStream for the final stage of broken-link campaigns. ahrefs.comahrefs.comhelp.ahrefs.com

The scalable workflow I'd use

Ahrefs Content Explorer / Site Explorer
↓
Find niche-relevant 404s with lots of quality referring domains
↓
Pull the actual referring pages
↓
Filter for high-authority/relevant sites
↓
Verify the broken link yourself with Screaming Frog/browser
↓
Use Wayback Machine to understand the old resource
↓
Create a genuinely useful replacement
↓
Find the relevant editor/site owner
↓
Outreach via Pitchbox/BuzzStream

The key is to prospect by the dead resource rather than by individual websites. One excellent dead page can have dozens or hundreds of referring domains, giving you a batch of outreach prospects from a single discovery.

Ahrefs' own broken-link workflow supports exactly this approach: identify competitors' broken pages with backlinks, inspect the linking sites, and offer an appropriate replacement. ahrefs.comahrefs.comhelp.ahrefs.com

If you're doing very large volumes

I'd go:

Ahrefs API → database/Google Sheets → automated filtering → manual verification → outreach platform

You can prioritize something like:

topical relevance × referring-page authority × referring-domain traffic × number of referring domains × link context

rather than simply sorting by DR. That tends to eliminate a lot of low-value broken-link prospects.

If you tell me your niche + roughly how many prospects/month you're targeting (e.g. 500, 5,000, 50,000), I can suggest a specific large-scale prospecting stack and filtering methodology.

For large-scale broken-link prospecting, I’d use a stack rather than a single tool. The key distinction is whether you want to find:

1. Broken pages on authority sites that you can replace with your own resource, or
2. Broken pages on competitors that already have lots of authoritative backlinks.

My recommended stack

1. Ahrefs — probably your primary prospecting engine

For this particular tactic, Ahrefs is hard to beat.

You can take a competitor, go to Site Explorer → Best by links, filter pages by HTTP code: 404, and identify dead pages that accumulated substantial backlinks. Ahrefs specifically recommends using competitor broken pages this way for broken-link building. help.ahrefs.com

Then take a promising dead URL and inspect its backlinks. You're essentially asking:

"Which authoritative sites are currently linking to something that no longer exists?"

Those referring pages become your outreach prospects.

Ahrefs also has an Outgoing Links → Broken Links report, which is useful for finding authority sites that themselves link to dead resources. help.ahrefs.com

For really large workflows, Ahrefs has an API endpoint for broken backlinks with filtering, sorting, CSV/JSON output, and up to 1,000 results per request. docs.ahrefs.com

Ahrefs broken-link documentation

2. Screaming Frog — use it to hunt authority sites directly

This is especially powerful if you've already built a list of 100–10,000 authority domains in your niche.

Feed those domains into Screaming Frog and crawl them looking for:

• 404 links
• 410s
• broken external links
• redirecting links
• orphaned/dead resources
• links to outdated URLs

The advantage over relying entirely on a backlink index is that you're examining the actual current pages and their outbound links.

A useful workflow is:

Authority-site list → Screaming Frog crawl → broken external URLs → validate dead resource → find replacement opportunity → outreach

3. Semrush — excellent second data source

Semrush's Backlink Audit has Broken and Lost backlink/target-page data, which can help identify broken targets and link-building opportunities. www.semrush.com

I'd use it as a complementary index rather than depending on it exclusively. Different backlink databases discover different links, so running your best prospects through two indexes can uncover additional opportunities.

4. Majestic — useful for validating link authority

Majestic becomes particularly useful when you have a huge candidate set and need another source of link metrics.

I wouldn't necessarily make it the center of the workflow, but it's useful for answering:

"Is this dead resource actually worth pursuing?"

5. BuzzStream — don't overlook the operations layer

Once you're dealing with thousands of prospects, the bottleneck stops being discovery and becomes prospect management and outreach.

You want a database containing roughly:

dead URL → replacement URL → linking page → referring domain → authority → topical relevance → contact → outreach status

BuzzStream is useful for managing that latter portion.

The scalable workflow I'd actually use

If your niche has lots of authoritative publications/resources, I'd structure it like this:

Ahrefs
→ find competitors' high-link-count 404 pages
→ export referring pages/domains

Screaming Frog
→ crawl your target authority-site universe
→ identify current broken outbound links

Ahrefs/Semrush
→ validate backlink counts, authority, traffic and relevance
→ deduplicate domains

Your content database
→ determine whether you already have a genuinely suitable replacement
→ otherwise create one

BuzzStream
→ identify contacts
→ personalize outreach
→ track responses

The important filtering step

Don't simply chase every 404.

I'd score opportunities approximately like:

Opportunity = topical relevance × referring-domain quality × number of relevant referring pages × replacement quality

A DR 80 site linking to a dead page isn't necessarily valuable if the dead page was unrelated to your niche. Conversely, a very niche-relevant DR 45 publication can be an excellent prospect.

Also, don't manufacture a mediocre replacement just because a broken link exists. The strongest pitch is essentially:

"You currently link to X, which is no longer available. We have a comparable/current resource that covers the same subject."

That makes broken-link building much more defensible than mass-emailing thousands of sites with generic "you have broken links" messages.

If I were building this at serious scale, my core stack would be Ahrefs + Screaming Frog + Semrush, with BuzzStream added once outreach volume becomes significant. www.semrush.comhelp.ahrefs.com

For large-scale broken-link prospecting, I’d use a stack rather than a single tool:

1. Ahrefs — primary prospect discovery
   • Best fit for finding dead pages on authoritative sites that still have lots of backlinks.
   • Its Content Explorer specifically supports finding broken-link opportunities by topic, while Site Explorer’s Broken backlinks report shows referring pages, anchor text, and the dead target. ahrefs.com
   • Particularly useful for filtering prospects by things like referring domains, traffic, and authority.

2. Semrush — second backlink database / validation layer
   • Useful for finding new, lost, and broken backlink data and monitoring changes over time. www.semrush.com
   • I’d use it alongside Ahrefs rather than instead of it: different indexes can uncover different referring pages.

3. Sitebulb — crawl authority sites directly
   • Excellent when you've already assembled a list of target domains.
   • It can crawl a site and identify broken external links, then expose the source page and target URL in bulk. ahrefs.comwww.semrush.comsupport.sitebulb.comsitebulb.com
   • Its cloud crawler is designed to scale from thousands to millions of URLs. sitebulb.com
   • This is especially valuable for finding opportunities that aren't obvious from backlink databases.

4. Screaming Frog SEO Spider — flexible crawler
   • Great alternative for crawling individual authority sites or specific sections at scale.
   • I'd favor it when you want granular control over crawl configuration and exports.

The workflow I'd use

Ahrefs → prospect discovery → Sitebulb/Screaming Frog → verification → outreach

For example:

Niche keywords/topics
→ find authoritative domains
→ identify their dead/404 pages
→ check which dead pages have meaningful referring domains
→ inspect the original content with the Wayback Machine
→ determine whether you have a genuinely relevant replacement
→ contact the site owner/editor.

The important distinction is that “a page is 404” isn't itself a good opportunity. You want:

dead page + authoritative referring site + relevant topic + backlinks worth reclaiming + a genuinely useful replacement.

For scale, I'd make Ahrefs the discovery engine and a crawler the verification engine. Ahrefs says its broken-link data is backed by its large backlink index and its Broken Link Checker can also surface highly linked dead pages. ahrefs.com

If you tell me your niche and roughly how many prospects you want (100 / 1,000 / 10,000+), I can lay out a concrete large-scale workflow, including the filters and export columns I'd use.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is: find authoritative sites → identify their outbound dead links → determine what the dead resource was → find/create a genuinely relevant replacement → qualify the referring page → outreach.

Best tools

1. Ahrefs — best overall
   • Site Explorer → Outgoing links → Broken links lets you find pages on a target site that link to dead external URLs. help.ahrefs.comahrefs.com
   • More importantly for scale, Best by Links → HTTP code: 404 lets you find competitors' dead pages that still have lots of backlinks, then inspect all the sites linking to them. ahrefs.com
   • Content Explorer can surface broken pages by topic, which is particularly useful when you don't already have a fixed list of authority domains. ahrefs.com
   • My pick if you're doing serious broken-link building.

2. Semrush — good second database
   • Its Backlinks database is useful for bulk prospect discovery, competitor analysis and filtering referring domains. Semrush also offers Bulk Backlink Analysis specifically for scaling prospect lists. www.semrush.com
   • Its Backlink Audit can monitor new, lost and broken backlinks, although it's less purpose-built for finding other sites' outbound broken links than Ahrefs. www.semrush.com

3. Screaming Frog SEO Spider — best for verification/crawling
   • I'd use a crawler when you have a defined list of authority domains and want to independently crawl their pages and validate external 404s.
   • This is especially useful because backlink databases aren't guaranteed to have crawled every page/link on a site; Ahrefs itself notes this limitation. help.ahrefs.comahrefs.com

4. Internet Archive / Wayback Machine — essential for reconstruction
   • Once you've found a dead URL with good backlinks, use historical snapshots to determine what the original resource actually contained.
   • This dramatically improves replacement quality and outreach: you're not guessing what the webmaster intended to link to.

The scalable setup I'd use

Ahrefs → Screaming Frog → Wayback Machine → spreadsheet/database → outreach

For example:

The highest-value tactic isn't simply finding broken links on authority sites. It's finding dead resources that themselves have strong backlinks, then identifying all authoritative sites that linked to them. One dead resource can therefore produce dozens or hundreds of prospects.

Ahrefs explicitly supports this workflow: find a competitor's broken page, inspect its referring domains/backlinks, recreate a better resource, and contact those linking sites. ahrefs.com

If you're doing this at very large scale

I'd prioritize:

Ahrefs + Screaming Frog + Wayback Machine + a custom data-processing layer.

Use Ahrefs for the massive backlink index and prospect discovery, Screaming Frog for crawling/verification, and Wayback for historical reconstruction. Semrush is worth adding if you want a second backlink index for cross-validation and additional prospect coverage.

If you tell me your niche and roughly how many domains/pages you want to process (e.g. 1,000 vs. 100,000), I can lay out a concrete large-scale workflow—including the filters I'd use to separate high-value broken-link opportunities from junk.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link prospecting, you'll typically combine three capabilities:

1. Crawling websites to find broken outbound links.
2. Finding authoritative websites in your niche.
3. Managing and prioritizing prospects for outreach.

A stack that many SEO professionals use looks like this:

My favorite workflow (most scalable)

Method 1: Dead resource pages (high success rate)

Search for resource pages like:

• intitle:resources "your keyword"
• inurl:links your keyword
• intitle:recommended sites
• your keyword useful links

Crawl the resulting domains with Screaming Frog configured to:

• Check external links
• Report 404s
• Export broken external links

This quickly surfaces pages linking to dead resources.

Method 2: Find dead pages with backlinks (my preferred approach)

Instead of crawling millions of pages, start with backlink data.

Using Ahrefs:

1. Search your competitors.
2. Go to Best by links.
3. Filter:
   • HTTP code = 404
   • Referring domains > 20 (or another threshold)
4. Export.

Now you have dead pages that dozens or hundreds of websites still link to.

This is often far more efficient than crawling random authority sites.

Method 3: Crawl authority sites directly

If you have a list of authority sites:

• Crawl with Screaming Frog.
• Enable:
  • Check external links
  • Store HTML
  • Follow external links (if appropriate)
• Export:
  • Broken external links
  • Source page
  • Anchor text

You now know exactly which page contains each broken link.

Method 4: Build a custom crawler

For very large campaigns, many SEO teams build a pipeline like:

Google search →
URL collection →
Crawler →
Broken link detection →
Page authority scoring →
Email finder →
CRM

Typical tools:

• n8n
• Python (Scrapy or Playwright)
• PostgreSQL
• OpenAI (for content categorization and prioritization)

This approach scales to hundreds of thousands of pages.

Prioritize prospects

Don't chase every broken link. Score opportunities using factors like:

• Domain authority or equivalent authority metrics
• Topical relevance
• Number of organic visitors
• Whether the linking page is indexed
• Number of outbound links on the page
• Whether the dead resource closely matches content you can provide

A site with moderate authority but a highly relevant, curated resource page is often a better target than a very large site with low editorial attention.

At enterprise scale

If you're looking at 100,000+ pages, a common workflow is:

• Use Ahrefs or Semrush to identify authoritative sites and dead pages with backlinks.
• Feed target URLs into Screaming Frog (or a cloud crawler) in List Mode.
• Store results in a database.
• Use automation (such as n8n) to enrich each opportunity with authority metrics, contact information, and topical relevance.
• Push qualified prospects into an outreach platform like BuzzStream.

This avoids wasting time crawling the entire web and focuses effort on pages that are both authoritative and likely to yield valuable link opportunities.

If your goal is millions of prospects across an entire niche, the workflow changes significantly from desktop SEO tools to distributed crawling and automation. I can outline an enterprise-scale pipeline that agencies use for that level of prospecting.

If you're doing broken link building at scale, you'll typically need four capabilities:

1. Find authoritative sites in your niche.
2. Crawl those sites for broken outbound links.
3. Evaluate the SEO value of the opportunities.
4. Manage outreach.

The best stack depends on your budget and technical comfort.

Budget-friendly workflow

If you're a solo SEO:

• Ahrefs
• Screaming Frog
• Hunter
• BuzzStream

This combination covers nearly everything without paying for multiple enterprise platforms.

Agency-scale workflow

For hundreds or thousands of prospects each month:

• Ahrefs for prospect discovery
• Screaming Frog in List Mode
• Custom exports into Google Sheets or Airtable
• BuzzStream or Pitchbox for outreach
• Optional Python automation for filtering and prioritization

How to find opportunities

A common workflow looks like this:

1. Find resource pages in your niche.
   • intitle:resources
   • inurl:links
   • helpful links
   • recommended websites

2. Export candidate domains.

3. Crawl them with Screaming Frog.

4. Filter:
   • External links
   • 404 responses
   • High-authority domains
   • Relevant topic

5. Verify the dead page with the Internet Archive to understand what the original content covered.

6. Create or improve a replacement resource.

7. Reach out to the webmaster explaining the broken link and suggesting your relevant replacement.

At very large scale (10,000+ sites)

Many advanced SEOs automate the process:

• Crawl target domains with Screaming Frog CLI or custom crawlers.
• Use APIs from Ahrefs or Semrush to enrich results with authority metrics.
• Check HTTP status codes programmatically.
• Query the Wayback Machine API to identify the original content.
• Store opportunities in a database (PostgreSQL, BigQuery, or Airtable).
• Prioritize by domain authority, topical relevance, and estimated traffic.
• Feed prioritized prospects into an outreach platform.

This approach can uncover tens of thousands of broken-link opportunities while allowing you to focus outreach on the highest-value prospects.

My recommended stack by budget

• Under $100/month: Screaming Frog + Hunter + Google Search operators.
• $200–500/month: Ahrefs + Screaming Frog + BuzzStream.
• Agency/enterprise: Ahrefs + Screaming Frog + BuzzStream or Pitchbox, with custom automation for crawling, filtering, and outreach.

One additional tactic that often produces higher-quality opportunities than generic resource pages is competitor broken backlink analysis. Using Ahrefs, you can identify pages on competitors' sites that have gone offline but still attract backlinks, then create a superior replacement and reach out to the sites linking to the dead page. This tends to yield more relevant and authoritative prospects than searching for random broken external links.

For large-scale broken link building, the most effective workflow is usually a combination of a backlink database, a crawler, and automation. No single tool does everything well.

Here's a stack that scales from hundreds to hundreds of thousands of prospects.

For large-scale broken link prospecting, you'll usually want a stack rather than a single tool. The best workflow combines a crawler, a backlink database, and automation.

Here's a practical setup depending on your budget.

The highest-ROI workflow

This approach scales well into tens of thousands of prospects.

1. Build a list of authority domains in your niche.
   • Competitors
   • Industry associations
   • Universities
   • Government resources
   • Resource pages
   • Blogs

2. Crawl them with Screaming Frog.
   • Find:
     • 404 pages
     • Broken outbound links
     • Resource pages

3. Take every discovered 404 URL and check it in Ahrefs.
   • How many referring domains?
   • How many backlinks?
   • Is it still getting links?

If a dead page has 100+ referring domains, it's often a valuable opportunity.

4. Export all pages linking to that dead URL.

5. Qualify prospects.
   • DR/DA
   • Traffic
   • Relevance
   • Language
   • Spam score

6. Reach out suggesting your replacement content.

An even faster method (my favorite)

In Ahrefs:

Best by Links
→ filter

• HTTP Code = 404
• Referring Domains > 20
• Language = English
• One niche at a time

This immediately surfaces dead pages that still have lots of links.

Then:

Dead page
↓
Backlinks report
↓
Every linking page becomes an outreach prospect.

This often uncovers opportunities in minutes instead of crawling hundreds of sites yourself.

At enterprise scale

People doing broken link building across hundreds of thousands of pages often automate the process:

Seed domains
      ↓
Crawler
      ↓
Extract every 404
      ↓
Check backlinks via Ahrefs API
      ↓
Filter by referring domains
      ↓
Find contact emails
      ↓
Push to Airtable/Sheets
      ↓
Outreach sequence

This lets you continuously discover new opportunities instead of running one-off campaigns.

Useful Google search operators

To find pages likely to contain external links:

intitle:resources "keyword"

inurl:links keyword

"helpful links" keyword

"recommended resources" keyword

site:.edu keyword resources

site:.gov keyword resources

"useful websites" keyword

Resource pages tend to accumulate outbound links over time, making them good candidates for broken link opportunities.

If you have a larger budget

A stack like this covers most needs:

• Ahrefs for backlink discovery and prioritization
• Screaming Frog SEO Spider for crawling
• Hunter.io for contact discovery
• Airtable for managing prospects
• n8n for automation

If you're comfortable with Python, you can automate much of this using the Ahrefs API (or another backlink provider's API), a crawler like crawl4ai or Scrapy, and an orchestration tool such as n8n. That combination can scale to processing hundreds of thousands of URLs with relatively little manual work.