ChatGPT answers, unedited

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key distinction is whether you want to find dead pages with valuable backlinks or broken outbound links on authority sites.

Recommended stack

• ahrefs.com — primary prospecting database.

Probably the best centerpiece for this workflow. Its Broken backlinks report shows pages linking to 404s, with referring pages, anchors, surrounding text, and filters such as referring-domain strength. Its API can also return up to 1,000 results per request, which makes it useful for programmatic/bulk prospecting. Ahrefs Help Center Ahrefs for Developers

• semrush.com — useful second data source.

Its Backlinks database has a Broken Pages filter that can expose broken pages on competitors and the backlinks pointing to them. Running the same prospects through two indexes can uncover opportunities one crawler missed. Semrush Semrush

• screamingfrog.co.uk — authority-site validation/crawling.

This is especially valuable when you already have a list of high-authority domains or URLs. Give it a URL list, crawl external links, then filter for 4XX responses. It can bulk-export the source pages and broken destinations, making spreadsheet/database processing straightforward. Screaming Frog Screaming Frog

• Wayback Machine — historical-content reconstruction.

Once you find a genuinely valuable dead URL, check what used to live there before deciding whether you can create a legitimate replacement. Ahrefs specifically recommends this step in its broken-link-building workflow. Ahrefs

• Pitchbox / BuzzStream — outreach at scale.

These are useful after you've identified and qualified prospects. The important thing is not to automate outreach before you've established that your replacement is genuinely relevant; Ahrefs lists both as outreach options for broken-link campaigns. Ahrefs

The workflow I'd use

Ahrefs → Screaming Frog → Wayback → outreach

1. Build a list of 50–500 authoritative sites in your niche.
2. In Ahrefs, inspect their Outgoing Links → Broken Links reports. This directly finds pages containing broken external links and allows exports. Ahrefs Help Center
3. Separately, search Ahrefs for competitor broken pages with referring domains. This finds the inverse opportunity: a dead resource that already has backlinks from authoritative sites. Ahrefs
4. Crawl/validate promising source URLs with Screaming Frog so you're not acting on stale crawler data.
5. Put dead URLs through the Wayback Machine.
6. Prioritize opportunities where:

• the referring site is genuinely authoritative/relevant;
• the dead resource is topically close to your content;
• multiple worthwhile sites link to it;
• you can create something that actually deserves to replace it.
• Contact the site owner/editor with the replacement resource.

For large-scale prospecting

If you're talking tens of thousands+ of URLs, I'd lean toward:

Ahrefs API + Screaming Frog exports + your own spreadsheet/database + outreach platform.

Ahrefs' API is particularly useful here because you can programmatically pull broken-backlink data and filter/group it rather than manually working through Site Explorer. Ahrefs for Developers

One important distinction: don't use Domain Rating/Domain Authority alone as your opportunity metric. A highly authoritative domain linking to an irrelevant dead resource is usually less interesting than a moderately authoritative, extremely topical page whose broken link is a perfect match for your replacement.

If you tell me your niche + roughly how many prospects you want per month, I can lay out a concrete high-volume workflow (queries, Ahrefs filters, Screaming Frog settings, scoring columns, and outreach process).

For large-scale broken-link prospecting, I’d use this stack:

• Ahrefs — primary prospect discovery: find dead pages with backlinks, filter by referring domains/traffic, and analyze competitors. 12AM Agency Linkforce Blog
• Screaming Frog — crawl entire authority sites to discover broken outbound links and export them for bulk processing. Clickolytics
• Semrush — strong alternative/all-in-one option for backlink and competitor research. Link Building Journal
• BuzzStream or Pitchbox — manage large outreach campaigns, contacts, follow-ups, and personalization. theStacc BestPage
• Wayback Machine — inspect the original dead page so your replacement actually matches what was previously linked. Linkforce Blog
• Check My Links — free browser-based validation for quickly checking individual resource pages. BestPage

Recommended workflow: Ahrefs → Screaming Frog → Wayback Machine → qualification/spreadsheet → BuzzStream/Pitchbox outreach. This separates finding lots of dead links from finding dead links worth pitching.

For large-scale broken-link prospecting, I’d use a combination rather than one tool. The key is separating discovery (finding authoritative pages linking to dead resources) from verification (confirming the link is genuinely broken) and prospect qualification.

My recommended stack

• Ahrefs — primary discovery tool.

This is probably the most useful centerpiece for broken-link building. Its Broken backlinks report lets you find pages linking to 404s on a target domain, with referring pages, anchor text, surrounding text, and filters such as DR. You can also find competitors' broken pages with lots of backlinks and prospect the sites linking to them. Ahrefs Help Center Ahrefs
ahrefs.com

• Screaming Frog SEO Spider — best for crawling your target authority sites yourself.

Give it a list of high-value sites/URLs, crawl them, filter external links returning 4XX responses, then bulk-export the source page → broken destination → anchor text relationships. It supports list mode and bulk exports, which makes it particularly useful when you've assembled hundreds or thousands of target URLs. Screaming Frog Screaming Frog
screamingfrog.co.uk

• Semrush Site Audit — useful as a second verification layer.

It identifies broken links and 4XX pages, although crawler access issues can occasionally make a live URL appear broken, so I'd manually/independently verify high-value opportunities before outreach. Semrush

A scalable workflow

I'd build the process roughly like this:

1. Build an authority-site universe

Collect several hundred/thousand relevant sites in your niche, prioritizing things like:

• Universities and research organizations
• Government sites
• Major publications
• Industry associations
• Established companies
• Resource/links pages
• High-DR niche publications

2. Find their dead outbound links

With Screaming Frog, upload your URL list in List Mode and crawl the pages. Its broken-link-building workflow specifically supports this approach: identify external links returning 4XX responses and export the source and destination URLs. Screaming Frog

3. Use Ahrefs to determine whether the dead resource is actually valuable

For each dead URL, check:

• Referring domains
• Backlinks
• DR/UR
• Historical traffic
• Anchor text
• Number and quality of linking sites
• Topical relevance
• Whether the dead page had substantial content

This is where Ahrefs is particularly powerful: you're not merely finding broken links; you're finding broken links that are worth replacing. Ahrefs explicitly supports using competitors' heavily linked broken pages for broken-link-building prospecting. Ahrefs Help Center

4. Reconstruct the dead resource

Use the Wayback Machine or other historical sources to understand what the dead page contained. Then determine whether you already have a genuinely equivalent resource—or create one.

5. Match prospects to your replacement

Your outreach becomes substantially stronger when the replacement isn't just "my article is relevant," but something like:

Your page links to [dead resource], which is no longer available. We published an updated resource covering the same subject here: [URL].

6. Verify before outreach

Don't blindly trust a crawler's 404. Check the URL yourself. Crawlers can encounter blocking, transient server errors, redirects, etc.; even Semrush notes that crawler access problems can result in false broken-link reports. Semrush

If you're doing this at serious scale

I'd use:

Ahrefs → Screaming Frog → spreadsheet/database → automated HTTP verification → manual qualification → outreach

The particularly valuable dataset to build is:

That lets you deduplicate opportunities where 50 different authority sites all link to the same dead resource. Those clusters are especially interesting because creating one genuinely strong replacement can potentially give you dozens of outreach targets.

If I were starting from scratch, I'd prioritize Ahrefs + Screaming Frog. Ahrefs handles the backlink intelligence and opportunity sizing; Screaming Frog handles high-volume crawling of the actual authority-site pages. Ahrefs also has a free broken-link checker for smaller-scale checks and can export results. Ahrefs

If you tell me your niche + roughly how many sites you want to prospect (100, 1,000, 10,000+), I can lay out a concrete large-scale workflow, including the filters I'd use in Ahrefs and Screaming Frog.

For large-scale broken-link prospecting, I’d build the workflow around a backlink index rather than relying primarily on traditional crawlers.

Best tool stack

• Ahrefs — best overall for scale. Its Site Explorer can identify competitors’ broken pages, show the referring pages linking to those 404s, and let you filter by metrics such as DR. Its Content Explorer can also surface high-value broken pages by topic. Ahrefs Help Center Ahrefs

ahrefs.com

• Screaming Frog — best for crawling specific authority sites. Give it a list of target domains and crawl them to find outbound 4xx links, then bulk-export the source URLs/inlinks. This is particularly useful when you've already assembled a prospect list of authoritative sites. Screaming Frog Screaming Frog

screamingfrog.co.uk

• Ahrefs API — best if you're doing hundreds/thousands of prospects. The API exposes the broken-backlinks report with filtering, sorting, aggregation, and CSV/JSON output, making it much easier to automate prospect discovery and scoring. Ahrefs for Developers

The workflow I'd use

1. Build an authority-site universe

Start with 500–5,000 sites in your niche—industry publications, universities, associations, major blogs, resource pages, government sites, etc.

2. Find their broken outbound links

For each domain, use Ahrefs' Outgoing Links → Broken Links report. Ahrefs specifically supports exporting these results, and the report identifies pages containing broken external links. Ahrefs Help Center

For sites where you need a fresh crawl rather than relying on an existing index, use Screaming Frog.

3. Prioritize the dead URLs, not merely the referring domains

The goldmine is something like:

High-authority site → relevant resource page → dead external URL → your content can genuinely replace it

I'd score opportunities using:

• Authority/quality of referring domain
• Organic traffic to referring page
• Relevance to your niche
• Number of sites linking to the dead URL
• Number of referring domains to the dead URL
• Whether the dead URL had substantial historical backlinks
• Whether you already have a genuinely suitable replacement

4. Reverse-engineer the dead pages

This is where Ahrefs becomes especially useful. Find competitors' 404s with lots of backlinks, inspect the pages that link to them, and determine what the dead resource originally provided. Ahrefs explicitly recommends using competitors' broken pages as broken-link-building opportunities. Ahrefs Help Center

5. Crawl the resulting prospect list

Once you've identified, say, 10,000 potentially valuable referring pages, feed those URLs into Screaming Frog. Verify that the link is actually broken and extract:

• Source URL
• Source page title
• Broken destination
• Anchor text
• HTTP status
• Context around the link

That gives you a much cleaner outreach dataset.

If you're serious about large-scale prospecting

I'd use:

Ahrefs API → database/Sheets → automated relevance scoring → Screaming Frog verification → outreach

rather than trying to make Screaming Frog do the entire job.

The key distinction is that Ahrefs is excellent for discovering the backlink relationships at scale, while Screaming Frog is excellent for actually crawling and validating the pages you care about. Ahrefs' own API supports up to 1,000 results per request for the broken-backlinks endpoint, which makes it considerably more suitable for programmatic prospecting. Ahrefs for Developers

One particularly powerful variation is topic-first broken-link prospecting: use Ahrefs Content Explorer to find authoritative dead pages within your niche, then harvest all the sites linking to those pages. That can produce a much more relevant outreach list than blindly crawling thousands of domains. Ahrefs

If you tell me your niche + approximate number of prospects (e.g. 1k, 10k, 100k), I can lay out a concrete large-scale workflow, including the exact Ahrefs/Screaming Frog filters and a prospect-scoring schema.

For large-scale broken link prospecting, you’ll usually want a stack rather than a single tool. The best setup depends on whether you’re finding broken backlinks to competitors, broken outbound links on authority sites, or resource-page opportunities.

Best tools for enterprise-scale broken link prospecting

1. Ahrefs — Best overall for finding linkable broken pages

Best for:

• Finding competitor pages that used to have lots of backlinks
• Finding 404 pages with strong backlink profiles
• Prioritizing opportunities by authority and traffic

Workflow:

1. Enter competitor domains into Site Explorer.
2. Go to Best by Links.
3. Filter for 404 Not Found pages.
4. Export referring domains.
5. Create replacement content and outreach.

Ahrefs also has broken link reports that surface dead pages and broken links tied to backlink data. Ahrefs Ahrefs

Best for: SaaS, affiliate, content-heavy niches, agencies.

2. Screaming Frog SEO Spider — Best for crawling authority sites at scale

Best for:

• Finding broken outbound links on sites you target
• Crawling resource pages, libraries, associations, universities, etc.
• Exporting the exact page + broken URL + anchor text

A common large-scale workflow:

• Build a list of 100–1,000 authority domains.
• Crawl them with Screaming Frog.
• Export external links returning 404/410 errors.
• Filter by relevance and authority.
• Pitch your replacement resource.

Screaming Frog specifically supports broken link building workflows by crawling prospect lists and identifying broken external links. Screaming Frog

Best for: Finding "this site links to a dead resource" opportunities.

3. Semrush — Best if you already run broader SEO campaigns

Useful for:

• Backlink gap analysis
• Competitor link research
• Site audits
• Managing outreach workflows

It’s less specialized than Ahrefs for broken-link discovery, but works well if your team already uses the platform. Techcognate

4. Majestic — Best for link intelligence

Useful when you need:

• Large backlink datasets
• Historical link discovery
• Trust Flow / Citation Flow filtering

Good as a secondary data source when you need more link history.

5. BuzzStream or Pitchbox — Best for outreach at scale

Once you have prospects, these help with:

• Finding contacts
• Sending personalized campaigns
• Managing follow-ups
• Tracking replies

They don’t usually find broken links themselves; they operationalize the outreach stage.

A scalable broken-link prospecting stack

For an agency or serious campaign, I’d use:

The highest-volume workflow

1. Build your authority-site list

• Industry publications
• Universities
• Associations
• Government resources
• Resource pages
• Crawl for broken links

• Screaming Frog in list mode
• Export broken external URLs
• Score opportunities

Prioritize:

• Relevant topic match
• Linking page authority
• Number of referring domains
• Whether the broken page had historical value
• Recover the asset

• Use the Wayback Machine to see what the dead page contained.
• Create a better replacement.
• Outreach

Pitch it as:

"I noticed your resources page links to X, but that page appears to be gone. We created an updated resource covering the same topic that may be useful for your readers."

If you’re doing thousands of prospects/month

A strong enterprise workflow is:

Ahrefs → Screaming Frog → Airtable → enrichment API → Pitchbox/BuzzStream

The biggest bottleneck usually isn’t finding broken links—it’s filtering out low-value ones. A broken link on a relevant DR 70 industry resource page is worth far more than thousands of random 404s. SEOSyrup

For most niches, I’d start with Ahrefs + Screaming Frog; that combination covers the two main discovery angles: broken backlinks and broken outbound links. Ahrefs Screaming Frog

Ahrefs → Screaming Frog → Airtable → enrichment API → Pitchbox/BuzzStream

The biggest bottleneck usually isn’t finding broken links—it’s filtering out low-value ones. A broken link on a relevant DR 70 industry resource page is worth far more than thousands of random 404s.

For large-scale broken link prospecting on authority sites, you’ll usually want a stack that covers four stages:

1. Find broken opportunities
2. Validate link value
3. Find contacts
4. Manage outreach

Here are the tools I’d consider:

1. Backlink intelligence (finding high-value broken pages)

Ahrefs — best overall for broken link discovery

Best for:

• Finding competitor pages that are now 404 but still have backlinks
• Discovering “lost” content with strong referring domains
• Filtering by authority, traffic, referring domains, and relevance

Typical workflow:

• Enter competitor domains into Site Explorer
• Go to broken pages / best by links
• Export pages with strong backlink profiles
• Create replacement content
• Pitch the sites still linking to the dead resource

Ahrefs’ Content Explorer and Site Explorer workflows are specifically designed for finding link prospects and broken-link opportunities. Ahrefs

Best for: agencies, serious link builders, competitive niches.

Semrush — good all-in-one alternative

Best for:

• Backlink gap analysis
• Competitor research
• Combining prospecting with broader SEO workflows
• Teams already using Semrush for audits/rank tracking

It’s especially useful if broken link building is only one part of your SEO process. Listicler

Best for: in-house SEO teams.

2. Crawlers (finding broken outbound links on authority sites)

Screaming Frog SEO Spider — essential for scale

This is the tool I’d pair with Ahrefs.

Use it to:

• Crawl curated lists of authority sites
• Extract external links
• Find 404s, redirects, and dead resources
• Export: - linking page
• broken destination URL
• anchor text

Example process:

1. Build a list of 500 niche authority sites.
2. Crawl them in Screaming Frog.
3. Filter external links by 404/4xx.
4. Prioritize pages with strong authority metrics.
5. Outreach.

Screaming Frog documents this exact broken-link-building workflow using list mode and external link reports. Screaming Frog

Best for: finding “you have a broken link on your page” opportunities.

3. Prospect discovery tools

BuzzSumo

Useful for:

• Finding heavily linked content in your niche
• Identifying publishers and influencers
• Finding outdated resources worth replacing

Google search operators (free)

Still useful for building your initial authority-site list:

Examples:

keyword + "resources"
keyword + "useful links"
keyword + "recommended sites"
keyword + intitle:resources
keyword + inurl:links

Then run those pages through Screaming Frog.

4. Contact finding and outreach

Hunter

Use for:

• Finding editor/webmaster emails
• Verifying addresses

Pitchbox

Use for:

• Large-scale outreach campaigns
• Automated follow-ups
• Prospect workflows

BuzzStream

Use for:

• CRM-style relationship management
• Tracking conversations
• Managing outreach teams

My recommended stacks by budget

Solo operator / low budget

• Screaming Frog
• Google Sheets
• Check My Links browser extension
• Hunter free tier
• Ahrefs Webmaster Tools (where applicable)

Serious niche campaign

• Ahrefs
• Screaming Frog
• Hunter
• BuzzStream

Agency scale

• Ahrefs
• Screaming Frog
• Pitchbox
• Hunter / email verification tool
• Airtable or CRM layer

The workflow that tends to produce the best opportunities

1. Start with authority sites, not random broken links

• Universities
• Industry associations
• Government resources
• Major publications
• Established blogs
• Find broken resources

• Use Screaming Frog on curated lists
• Use Ahrefs for competitor dead pages
• Score opportunities

Prioritize:

• Relevant topic match
• Referring domains
• Organic traffic
• Real editorial page
• Recently broken resource
• Recreate or improve the missing resource

Don’t just replace a dead URL with any page; match the original intent.

5. Outreach with a service-first angle

“I noticed this resource is no longer available” works better than immediately asking for a link.

For a large-scale campaign, the core combination I’d pick is Ahrefs + Screaming Frog + Hunter + BuzzStream/Pitchbox. seosyrup.co.uk Screaming Frog

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key is separating finding high-authority prospects from crawling those prospects for broken outbound links.

Best tool stack

• Ahrefs — best for discovering opportunities at scale.

Its Site Explorer has a Broken backlinks report, and you can analyze competitors' dead pages and the sites linking to them. You can filter by things like referring-page DR and export the results. Ahrefs also exposes the broken-backlinks endpoint through its API, which is particularly useful if you're automating prospecting. Ahrefs Help Center Ahrefs for Developers
help.ahrefs.com

• Screaming Frog — best for crawling the actual authority sites.

Feed it hundreds/thousands of relevant URLs in List Mode, restrict crawling to external links, and identify 4xx destinations plus the pages containing them. Screaming Frog specifically documents this workflow for broken-link building. Screaming Frog Screaming Frog
screamingfrog.co.uk

• Semrush — useful as a complementary backlink database.

I'd mainly use it to discover competitors, authoritative domains and content/resource pages, then pass those URLs into your crawler. It becomes more valuable if Semrush is already part of your SEO workflow.

• Majestic — useful when link intelligence is the priority.

Its historical backlink data can be useful for finding previously authoritative/deleted resources, although I'd generally choose Ahrefs first for this particular workflow.

• Google/Bing + SERP scraping — useful for prospect discovery.

Search queries around your niche's resource pages, statistics pages, guides, associations, universities, etc. can produce a much better prospect universe than simply crawling random domains.

The workflow I'd use

1. Build an authority-site universe

Start with your niche's:

• universities and research organizations
• government/industry organizations
• major publications
• established SaaS/company blogs
• associations
• high-quality resource pages
• competitor sites
• "best resources" / "recommended tools" pages

Use Ahrefs to prioritize them by DR, referring domains, organic traffic, and topical relevance.

2. Find dead resources

This is where Ahrefs is especially powerful.

For each major competitor:

Site Explorer → Pages → Best by links → HTTP code: 404

That lets you find competitor pages that disappeared but still have backlinks. Ahrefs explicitly recommends this approach for broken-link building: identify the dead competitor page, find who links to it, and approach those sites with a relevant replacement. Ahrefs Help Center Ahrefs

3. Crawl the authority sites

Take your prioritized prospect URLs and feed them into Screaming Frog in List Mode.

You can configure it to crawl external links from those pages and then export the pages containing 4xx links. Screaming Frog

This is much more scalable than manually checking pages.

4. Qualify the broken links

Don't treat every 404 as an opportunity. I'd score each one roughly on:

Opportunity = Authority × Relevance × Replacement quality × Number of linking sites

A broken link on a DR 80 university resource page that perfectly matches something you have is vastly more interesting than a random 404 on a low-value blog.

5. Automate the boring parts

For serious volume, I'd build a pipeline like:

Ahrefs API → CSV/database → URL deduplication → Screaming Frog/API crawler → HTTP validation → Ahrefs metrics → opportunity scoring → outreach

Ahrefs' current API documentation supports retrieving broken backlinks with filters, sorting and up to 1,000 results per request, which makes it considerably more interesting for an automated prospecting pipeline. Ahrefs for Developers

My preferred stack

If I were doing this for an agency or an in-house SEO team:

The important distinction: Ahrefs is excellent at answering "Who links to this dead resource?", while Screaming Frog is excellent at answering "Which pages on these authority sites contain dead outbound links?". Combining those two workflows gives you much broader coverage than relying on either alone. Ahrefs Help Center Screaming Frog

If you're doing 10k–100k+ URLs, I can also lay out a concrete high-volume workflow—including URL discovery, crawling architecture, deduplication, API calls, scoring, and how to avoid wasting crawl/API credits.

For large-scale broken-link prospecting on authority sites, I’d use a stack rather than a single tool:

• ahrefs.com — best overall for prospect discovery. Its Site Explorer has a Broken backlinks report that shows referring pages linking to 404s, with filters for things like DR, URL, and anchor text. More importantly, you can analyze competitors and find their highly linked-to dead pages—the classic broken-link-building workflow. Ahrefs Help Center Ahrefs
• semrush.com — strong second database / validation layer. Backlink Audit can surface target-URL errors and broken/lost backlinks, and its Authority Score and referring-domain data are useful for prioritizing prospects. Semrush Semrush
• screamingfrog.co.uk — best for crawling specific authority sites. Give it a curated list of sites and have it crawl outbound links to identify dead destinations. This is particularly useful when you want opportunities that aren't already neatly surfaced by backlink indexes.
• ahrefs.com — useful for lightweight/batch checking. It can identify broken outbound links and export results, and Ahrefs explicitly supports using competitors' broken pages for link-building prospecting. Ahrefs Ahrefs Help Center
• Wayback Machine — use it to reconstruct what a dead page actually contained. This is important before pitching a replacement, because you want your replacement resource to genuinely match the original.

The workflow I'd use

1. Build a target list of authority domains.
Start with competitors, industry publications, universities, associations, government sites, major blogs, resource pages, and other sites with strong topical relevance.

2. Put competitor domains into Ahrefs.
Go to Best by Links → HTTP 404 and sort by referring domains/backlinks. You're looking for dead pages that have lots of relevant authority links, not merely lots of backlinks. Ahrefs specifically recommends this approach for finding competitors' broken pages. Ahrefs Help Center Ahrefs

3. Extract the referring pages.
For each promising dead URL, pull the sites actually linking to it. Then filter for things like:

• high DR/Authority Score
• topical relevance
• followed links
• genuine editorial pages
• meaningful organic traffic
• pages where the broken link is contextually useful

4. Verify the opportunity manually.
Check the dead URL in the Wayback Machine, determine what it originally covered, and decide whether you already have—or can create—a genuinely better replacement.

5. Crawl high-value authority sites separately.
This is where Screaming Frog becomes valuable. For a list of, say, 500 carefully selected authority domains, crawl their outbound links and identify dead destinations. That gives you a second prospecting mechanism beyond backlink-index data.

6. Prioritize by opportunity, not just authority.

A useful scoring model is:

Opportunity = topical relevance × referring-domain quality × link prominence × replacement fit

A DR 90 site linking to an irrelevant dead page isn't necessarily better than a DR 60 niche publication linking to a dead page that your resource perfectly replaces.

If you're doing this at serious scale

I'd go with:

Ahrefs API → export broken pages/referrers → database/Google Sheets → Screaming Frog validation → Wayback reconstruction → outreach.

Ahrefs exposes a Broken Backlinks API endpoint with filtering, sorting, aggregation, and CSV/JSON output, which makes it much more suitable for programmatic prospecting than manually clicking through reports. Ahrefs for Developers

If you tell me your niche + roughly how many domains you want to prospect (e.g. 1,000 / 10,000 / 100,000), I can lay out a concrete large-scale workflow, including the exact Ahrefs/Semrush filters and how I'd score and deduplicate the opportunities.

If you’re doing large-scale broken-link prospecting, I’d build the workflow around a backlink index + crawler rather than relying on browser extensions alone.

My recommended stack

1. Ahrefs — primary prospecting database

• Best overall choice for this tactic.
• Use Site Explorer → Broken backlinks to find authoritative sites linking to dead pages on competitors.
• Use Content Explorer to search for broken pages around your niche/topic, then filter for pages with substantial referring domains.
• You can also inspect competitors' outgoing broken links and export them. Ahrefs specifically supports filtering broken-link opportunities by things like referring-domain authority and exporting the results. Ahrefs Help Center Ahrefs Help Center Ahrefs
• ahrefs.com
• Screaming Frog — bulk verification

• Excellent when you've already assembled a big list of authority pages.
• Put hundreds/thousands of prospect URLs into List Mode, configure it to crawl external links, and identify 4xx destinations.
• It can export the source page → broken destination → status code relationship, which is exactly what you need for outreach. Screaming Frog Screaming Frog
• screamingfrog.co.uk
• Ahrefs + Google/Bing — authority-page discovery

• Search for resource pages, statistics pages, guides, libraries, associations, universities, industry organizations, etc.
• Queries such as: - "[keyword]" intitle:resources
• "[keyword]" inurl:resources
• "[keyword]" intitle:links
• Feed the resulting URLs into Screaming Frog for bulk checking. This is essentially the scalable version of the traditional broken-resource-page tactic. Ahrefs
• Semrush — useful secondary database

• I wouldn't buy it solely for broken-link building if you already have Ahrefs.
• But if you're already using Semrush for competitive research, its backlink data can complement Ahrefs when building a large prospect database.
• Wayback Machine — historical validation

• Once you find a valuable dead URL, check what the page actually contained historically.
• This is important because you want to create a genuinely relevant replacement, not merely something vaguely related to the anchor text.

The scalable workflow I'd use

Ahrefs → Screaming Frog → historical validation → qualification → outreach

For example:

Competitors / niche authorities
          ↓
     Ahrefs
          ↓
Find dead pages with 10–100+ referring domains
          ↓
Export referring pages
          ↓
     Screaming Frog
          ↓
Verify 404 / dead destinations
          ↓
Wayback Machine
          ↓
Determine what the dead resource covered
          ↓
Build a better replacement on your site
          ↓
Prioritize high-authority, relevant linking pages
          ↓
Personalized outreach

The key distinction is broken page discovery vs. broken-link verification. Ahrefs is much better for the former because you're searching its backlink graph; Screaming Frog is excellent for the latter because you can throw a large URL list at it and systematically identify broken external links. Screaming Frog Ahrefs

For genuinely large scale

I'd prioritize prospects using something like:

Relevance × referring-domain quality × number of linking domains × page authority × likelihood of replacement

Don't simply scrape every 404 you can find. A dead page with 500 junk backlinks is usually less interesting than a highly relevant dead industry resource with 15 legitimate authority-site links.

If you tell me your niche + roughly how many prospects you want per month (e.g. 1,000 / 10,000 / 100,000), I can lay out a specific large-scale prospecting pipeline, including the Ahrefs filters, Screaming Frog configuration, spreadsheet/database fields, and prioritization formula.

For large-scale broken-link prospecting, I’d build the workflow around a backlink index + crawler + historical URL data. The best stack depends on whether you’re hunting for broken pages on authority sites or broken pages that authority sites link to.

Best tools

• ahrefs.com — best overall

• Site Explorer → Best by links → HTTP 404 finds dead pages on competitors/authority domains with backlinks.
• Content Explorer → Broken pages lets you search for broken pages around a topic and filter by referring domains.
• Outgoing Links → Broken links finds authority sites that currently link to dead resources—often the most directly actionable prospecting angle.
• You can export the results and prioritize by referring domains, URL Rating, Domain Rating, traffic, etc. Ahrefs specifically supports filtering broken backlinks and broken outbound links. Ahrefs Help Center Ahrefs Help Center Ahrefs
• semrush.com — strong second database

• Particularly useful for competitor backlink discovery, backlink gaps, authority/quality filtering and monitoring lost/broken links.
• Its backlink database currently covers 43T+ backlinks, so it can be useful when you're trying to expand beyond a relatively small prospect list. Semrush
• Screaming Frog SEO Spider — best crawler for validating prospects

• Once you've identified authority domains, crawl them and extract their external links/status codes.
• This is useful because a backlink database can tell you potentially broken; a live crawl lets you verify what the page is returning now.
• I'd use it downstream rather than trying to discover thousands of opportunities from scratch with it.
• Wayback Machine — essential for reconstructing dead content

• When you find a valuable 404, check its historical versions to understand exactly what the old resource covered.
• This lets you create a genuinely relevant replacement rather than pitching something vaguely related.
• Pitchbox / BuzzStream — outreach layer

• These aren't discovery tools so much as campaign-management tools. Once you've identified broken-link prospects, they help scale personalized outreach.
• Ahrefs itself recommends tools such as Pitchbox or BuzzStream for contacting the sites linking to broken resources. Ahrefs

The workflow I'd use

For large-scale prospecting, I'd do:

Ahrefs → export → filter/score → Wayback → crawl/verify → outreach

For example:

1. Build a list of 50–500 authority domains in your niche.
2. Put them into Ahrefs and look for their broken outbound links.
3. Extract the dead URLs and consolidate duplicates.
4. Run those URLs through backlink analysis to determine which dead resources have meaningful referring domains.
5. Use Wayback to determine what each resource originally contained.
6. Keep only opportunities where you can produce a better/current equivalent.
7. Identify every relevant authority site linking to the dead resource.
8. Verify the link is actually broken.
9. Score prospects by relevance + authority + topical fit + number of referring domains.
10. Run personalized outreach.

The important distinction is that you don't necessarily want "sites with broken links." You want:

High-authority pages in your niche → linking to a genuinely dead resource → where you have (or can create) a highly relevant replacement.

That produces dramatically better prospects than simply scraping thousands of 404s.

If you're doing this at serious scale

I'd choose Ahrefs + Screaming Frog + Wayback + an outreach platform.

Ahrefs is particularly well suited because its broken-link workflow can go in both directions: find broken pages with valuable backlinks, and find pages containing broken outbound links. Ahrefs Help Center Ahrefs

If you tell me your niche + approximate scale (e.g. 10k, 100k, or 1M prospects), I can also lay out a concrete scraping/filtering workflow, including the exact Ahrefs/Screaming Frog filters I'd use and how I'd score opportunities.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key distinction is whether you want to find broken pages that have valuable backlinks or authority sites that themselves contain broken outbound links.

Best tools

• ahrefs.com — best overall for scalable prospecting.

Its Site Explorer has both Broken backlinks and Broken links reports. More importantly, you can take a competitor, filter their pages to HTTP 404, and then inspect all the sites linking to those dead pages. That is probably the highest-value workflow for your use case. Ahrefs Help Center Ahrefs

• semrush.com — strong second database.

Backlink Audit can identify backlinks pointing to broken target URLs, while its Site Audit can crawl sites and identify broken external links. Using it alongside Ahrefs can uncover opportunities the other index missed. Semrush Semrush

• screamingfrog.co.uk — best for actually crawling target authority sites.

Once you've assembled a list of high-authority domains, crawl them to independently verify their outbound 404s. This is particularly useful because backlink databases don't necessarily contain every link on every page.

• Wayback Machine — use it for reconstructing what the dead resource used to contain. This isn't your primary discovery tool; it's the validation/research layer after you've found a promising dead URL.

The workflow I'd use

For a niche with, say, 500–5,000 authority domains:

1. Build your authority-site universe.

• Industry publications
• Universities / research organizations
• Government resources
• Major associations
• Established blogs
• Resource pages
• Competitor sites
• Start with competitor dead pages in Ahrefs.

For each major competitor:
Site Explorer → Pages → Best by Links → HTTP code: 404

Then prioritize dead pages with:

• Lots of referring domains
• High-quality referring domains
• Strong URL Rating
• Clear topical relevance
• A topic you can realistically recreate better

Ahrefs explicitly recommends this approach for broken-link building. Ahrefs

3. Export the referring pages.

Now you've transformed:

competitor → dead URL

into:

authority website → page containing dead link → dead resource

That's your prospect list.

4. Run the prospects through a crawler.

For particularly valuable domains, crawl them with Screaming Frog and verify the link is actually broken. This avoids wasting outreach on stale third-party index data.

5. Use Wayback to understand the dead resource.

Determine what the original page covered, then create something genuinely comparable or better.

6. Score prospects before outreach.

I'd use something like:

Opportunity score = topical relevance × referring-domain quality × page authority × likelihood of link replacement

Don't simply chase DR/DA. A highly relevant niche publication with moderate authority can be much more valuable than a generic high-DR site.

If you're doing this at serious scale

My preferred stack would be:

Ahrefs → Screaming Frog → Wayback Machine → spreadsheet/database → outreach platform

And I'd keep Semrush as a second backlink index for discovering opportunities Ahrefs misses.

The big trick is not crawling thousands of sites blindly. Start from dead resources that already attracted links, because one good dead resource can give you dozens or hundreds of qualified prospects. Ahrefs' current documentation specifically supports finding competitors' broken pages with lots of backlinks and then contacting those linking sites. Ahrefs

If you tell me your niche + approximate number of prospects (e.g. 1,000 vs. 100,000), I can lay out a concrete large-scale workflow, including the exact Ahrefs/Screaming Frog filters and a prospect-scoring schema.

For large-scale broken link prospecting, you’ll want a stack that can do three things well:

1. Find authoritative pages with dead links
2. Prioritize opportunities by SEO value
3. Scale crawling + outreach workflows

Here are the tools I’d consider:

1. Ahrefs — best overall for broken link discovery

Best for: finding competitors’ dead pages, broken backlinks, and high-authority targets.

Useful workflows:

• Site Explorer → Best by Links → filter 404 - Finds competitor pages that have earned backlinks but disappeared.
• You can recreate the resource and pitch sites still linking to the dead page.
• Broken Backlinks report - Finds backlinks pointing to dead URLs.
• Content Explorer - Find popular content in your niche, then check for outdated/dead resources.

Ahrefs also provides broken link reports and backlink analysis features designed for finding these kinds of opportunities. Ahrefs Ahrefs

Good for scale: ⭐⭐⭐⭐⭐
Cost: $$$
Best use case: “Find 1,000 high-value broken link opportunities in my niche.”

2. Semrush — strong alternative with broader SEO workflows

Best for: agencies that already use Semrush for:

• competitor analysis
• backlink audits
• prospect management
• outreach tracking

Useful workflow:

• Analyze competitor backlinks
• Filter lost/broken links
• Export prospects
• Enrich with authority metrics

It’s especially useful if broken link building is one part of a larger SEO campaign.

Good for scale: ⭐⭐⭐⭐
Cost: $$$
Best use case: “I need broken links plus a full SEO platform.”

3. Screaming Frog SEO Spider — best crawler for finding outbound broken links

Best for: crawling authority sites and finding their dead external links.

Workflow:

1. Build a prospect list: - industry resource pages
2. university pages
3. associations
4. blogs
5. “best resources” pages
6. Crawl them with Screaming Frog.
7. Export: - source URL
8. broken destination URL
9. anchor text

Screaming Frog specifically supports broken link building workflows by crawling prospect pages and identifying broken external links. Screaming Frog

Good for scale: ⭐⭐⭐⭐⭐
Cost: $ (relative to enterprise tools)
Best use case: “I already have 10,000 authority pages and want to find dead links on them.”

4. Majestic — useful for link intelligence

Best for:

• historical backlink analysis
• finding old authoritative resources
• analyzing topical link neighborhoods

Less of a crawler, more of a backlink intelligence tool.

Good for scale: ⭐⭐⭐
Best use case: “Find forgotten pages that used to attract links.”

5. Moz Link Explorer — good lightweight option

Useful for:

• checking domain authority
• finding competitor link gaps
• validating prospects

Not usually my first choice for large-scale broken link campaigns, but useful for smaller teams.

6. Supporting tools

Check My Links

Browser extension for quickly checking individual resource pages.

Good for:

• manual prospecting
• validating opportunities before outreach

Bad for:

• thousands of pages

Internet Archive Wayback Machine

Essential for figuring out what a dead page used to contain.

Workflow:

• Find broken URL
• Check archived version
• Create a better replacement
• Pitch linking sites

A scalable broken link prospecting stack

If I were building a serious campaign:

Budget setup

• Screaming Frog
• Ahrefs free tools
• Wayback Machine
• Google Sheets/Airtable

Professional setup

• Ahrefs
• Screaming Frog
• BuzzStream or Pitchbox for outreach
• Airtable for opportunity management

Agency-scale setup

• Ahrefs API
• Screaming Frog exports
• Custom scripts for: - URL checking
• relevance scoring
• DR/traffic filtering
• contact enrichment

My recommended workflow

1. Find authority pages

• Search: - "keyword" + resources
• "keyword" + useful links
• site:.edu "keyword" + links
• intitle:resources "keyword"
• Crawl them

• Screaming Frog → find 404 external links
• Score opportunities

Filter for:

• DR/DA
• organic traffic
• topical relevance
• number of referring domains
• quality of the original resource
• Recreate the missing asset

• Better guide
• Updated statistics
• New research
• Better UX
• Outreach

• “I noticed your resources page links to X, but that page is no longer available. We created a replacement covering the same topic…”

For true large-scale broken link building, the strongest combination is usually:

Ahrefs + Screaming Frog + Wayback Machine + an outreach CRM. Ahrefs Screaming Frog

That combination lets you go from “find random broken links” to building a repeatable prospecting pipeline.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core job is:

find authoritative sites → identify their broken outbound links → determine whether the dead resource is relevant to your niche → find sites linking to dead competitor pages → prioritize outreach.

Best tools

• ahrefs.com — best overall

• Site Explorer → Outgoing Links → Broken links lets you find pages containing broken external links and export them. Ahrefs Help Center
• More importantly for broken-link building, Best by Links → HTTP 404 lets you find competitors' dead pages that still have lots of backlinks, then prospect the sites linking to them. Ahrefs Help Center Ahrefs
• Its API can return up to 1,000 broken-backlink records per request and supports filtering, sorting, DR, traffic, URL, etc., which makes it particularly useful for scaling the process. Ahrefs for Developers
• semrush.com — strong second database

• Backlink Analytics is useful for finding competitor link sources and analyzing referring domains. Semrush
• Its Backlink Audit product can also monitor new, broken and lost backlinks. Semrush
• I like using it as a second data source to catch opportunities that don't appear in Ahrefs.
• screamingfrog.co.uk — best for crawling specific authority sites

• Give it a list of 500–5,000 authority domains/pages and crawl them for broken external links.
• This is particularly useful when you've already assembled a curated prospect list (industry associations, universities, publications, government resources, major blogs, etc.).
• google.com — prospect discovery

• Use search operators to build your initial authority-site universe, e.g. site:.edu, site:.gov, industry associations, resource pages, statistics pages, guides, etc.
• Then feed those domains into your crawler/link database rather than trying to discover everything through Google alone.

The scalable workflow I'd use

1. Build an authority-site list

Start with 500–5,000 domains relevant to your niche. Don't just filter on DR; categorize them by things like:

• topical relevance
• organic traffic
• editorial quality
• number of referring domains
• likelihood of actually editing old content

2. Run the list through Ahrefs

For each domain, pull:

referring page → broken outbound URL → anchor text → surrounding text → HTTP status → referring-page traffic/authority

Ahrefs specifically exposes the referring page, broken target, anchor and surrounding text in its broken-link reports. Ahrefs

3. Run competitor analysis in parallel

This is often the higher-value source.

For each major competitor:

Competitor → Best by Links → 404 → sort by referring domains → inspect dead pages

Then find all authoritative domains linking to those dead pages. Ahrefs explicitly recommends this approach for broken-link building. Ahrefs Help Center

4. Validate the opportunity

Don't automatically pitch every 404.

I'd score opportunities roughly:

Opportunity = authority × topical relevance × number of linking domains × replacement quality

A dead resource with 40 relevant referring domains is dramatically more interesting than a random broken link on a DR 80 site.

5. Use Screaming Frog for verification

Before outreach, recrawl the prospective page to confirm the link is actually broken. This eliminates stale backlink-database data and false positives.

If you're doing serious volume

My preferred stack would be:

Ahrefs API → Python/Sheets database → Screaming Frog validation → outreach platform

The Ahrefs API is particularly attractive here because you can programmatically pull broken-backlink data and filter/aggregate it rather than manually inspecting thousands of reports. Ahrefs for Developers

One important distinction: broken outbound links and broken backlinks are two different prospecting strategies. For authority-site campaigns, I'd run both, but I'd generally prioritize competitor dead pages with many quality referring domains because they give you a ready-made list of sites that have already demonstrated willingness to link to that type of resource.

For large-scale broken link prospecting on authority sites, you’ll usually want a stack that covers four stages:

1. Find high-value broken pages
2. Identify who links to them
3. Verify relevance and authority
4. Manage outreach at scale

Here are the tools I’d consider:

1. Backlink databases (best for finding broken link opportunities)

ahrefs.com — Best overall for broken link building

Best for: competitor broken backlinks, lost content discovery, authority filtering.

Key workflows:

• Enter competitor domains → Site Explorer → Best by Links → filter 404
• Find dead pages that still have referring domains
• Export referring sites and prioritize by DR, traffic, relevance
• Use broken backlink reports to find pages where replacement content could earn links Ahrefs

Why it’s strong:

• Huge backlink index
• Good filtering by referring domains, DR, traffic, anchors
• Easy export workflow Ahrefs

semrush.com — Strong alternative

Best for: teams already using Semrush for SEO campaigns.

Useful features:

• Backlink Analytics
• Lost backlinks
• Competitor link gap analysis
• Authority/traffic filtering

A common workflow is:

• Find competitor pages that disappeared
• Check who linked to them
• Create a better replacement resource
• Pitch those linking sites

moz.com — Good for simpler workflows

Best for: smaller teams that want domain authority metrics and link analysis without a larger enterprise stack.

2. Crawlers (best for finding broken outbound links on authority sites)

screaming-frog.org — The workhorse for resource-page prospecting

Best for: crawling lists of niche authority sites and finding broken external links.

Workflow:

1. Build a list of 100–1,000 relevant authority sites
2. Crawl them
3. Export external links returning: - 404
4. 410
5. dead redirects
6. Filter for pages with topical relevance

Screaming Frog can identify broken links and export source URLs for analysis. Screaming Frog

Good companion tools:

• Ahrefs Batch Analysis → check authority metrics
• Wayback Machine → see what the dead page used to contain

3. Browser-based helpers (good for smaller batches)

chromewebstore.google.com

Best for: quickly checking resource pages manually.

Useful when you find pages like:

• “Best resources for [topic]”
• “Helpful links”
• “Industry tools”
• “Research libraries”

4. Content reconstruction tools

web.archive.org

Best for: understanding what the dead page used to be.

A strong broken link campaign usually works like this:

1. Find dead URL with many backlinks
2. Look at historical snapshots
3. Recreate something better: - updated statistics
4. better design
5. more comprehensive guide
6. original research
7. Contact linking sites

5. Outreach + pipeline management

buzzstream.com

Best for: managing hundreds/thousands of outreach prospects.

Useful features:

• Contact discovery
• Email tracking
• Follow-up sequences
• Prospect tagging

Alternatives:

• pitchbox.com — more agency/enterprise oriented
• hunter.io — email finding and verification

A scalable broken-link prospecting stack

For an agency-level workflow, I’d use:

My recommended process at scale

1. Create a niche authority list

• industry associations
• universities
• government resources
• respected publications
• major blogs
• Run Screaming Frog crawls

• extract dead external links
• Run dead URLs through Ahrefs

• referring domains
• DR
• traffic
• anchor text
• Prioritize

• DR 40+
• topical match
• multiple referring domains
• previously valuable content
• Recreate the missing asset

• don't just replace a broken page; make the best available version
• Outreach with a repair angle

• “I noticed your resource page links to X, but that page is no longer available. We created an updated resource covering the same topic…”

For very large campaigns, the biggest efficiency gain usually comes from Ahrefs + Screaming Frog + a CRM/outreach tool, rather than trying to find everything inside one platform. seosyrup.co.uk

For large-scale broken-link prospecting, I’d build the workflow around a backlink index rather than relying on a basic broken-link crawler.

Best tool stack

• ahrefs.com — best overall for prospect discovery.

Its Site Explorer can identify competitors’ 404 pages and show the referring pages linking to them. More importantly for scale, Content Explorer lets you search for broken pages by topic and filter for pages with referring domains. Ahrefs also has an API endpoint for broken backlinks, which is useful if you're building a large prospecting pipeline. Ahrefs Ahrefs for Developers

• semrush.com — strong alternative/second data source.

Its Backlinks tools have a Broken Pages filter, including competitor analysis, so you can find broken competitor URLs and the sites linking to them. Semrush

• screamingfrog.co.uk — best for crawling specific authority sites.

Once you've identified a target site, crawl it and inspect external links returning 4xx/5xx responses. It's particularly useful when you have a curated list of hundreds or thousands of authority domains and want to systematically check them.

• web.archive.org — essential validation layer.

Before creating replacement content, check what the dead URL actually contained historically. You don't want to build a replacement based solely on a URL/title.

• pitchbox.com / buzzstream.com — outreach layer.

These aren't prospecting engines, but they're useful once you've turned broken URLs into qualified prospects. Ahrefs specifically recommends outreach tools such as Pitchbox or BuzzStream for contacting the sites linking to dead pages. Ahrefs

For true scale, I'd use this workflow

1. Build an authority-site universe

Start with, say, 500–10,000 sites in your niche and categorize them by:

• DR/DA
• topical relevance
• organic traffic
• referring-domain count
• whether they are editorial/resource sites
• whether they have a history of linking externally

2. Find dead assets with backlinks

In Ahrefs:

Competitor → Best by Links → HTTP code: 404

Then prioritize dead pages with lots of relevant referring domains, rather than simply chasing the highest DR pages. Ahrefs' own documentation confirms this workflow. Ahrefs

For broader discovery, use:

Content Explorer → niche/topic → broken pages → referring domains ≥ X

That lets you discover dead resources beyond your immediate competitors. Ahrefs

3. Extract the actual linking prospects

You ultimately want a dataset like:

This is much more valuable than a list of 404 URLs.

4. Validate historically

Run the dead URLs through the Wayback Machine. Determine:

• what the original page was about
• whether it was genuinely useful
• whether its backlinks are editorial
• whether you can produce something substantially equivalent or better

5. Verify the link is actually broken

Don't blindly trust a database's HTTP status. A page can have transient crawl errors, redirects, bot blocking, etc. Ahrefs itself notes that apparent broken redirects can sometimes result from crawl/4xx/5xx issues. Ahrefs Help Center

If you're doing thousands of prospects

I'd go with:

Ahrefs API → database/Google Sheets → Wayback validation → Screaming Frog spot checks → automated qualification → Pitchbox/BuzzStream outreach

The important distinction is that Ahrefs/Semrush find opportunities from backlink data, whereas Screaming Frog verifies links by actually crawling pages. For large-scale campaigns, combining those approaches is substantially better than trying to crawl the entire web yourself.

If you tell me your niche + roughly how many authority domains you want to prospect (e.g. 1,000 vs. 100,000), I can lay out a specific large-scale workflow, including the filters/metrics I'd use to automatically score the opportunities.

This is much more valuable than a list of 404 URLs.

4. Validate historically

Run the dead URLs through the Wayback Machine. Determine:

• what the original page was about
• whether it was genuinely useful
• whether its backlinks are editorial
• whether you can produce something substantially equivalent or better

5. Verify the link is actually broken

Don't blindly trust a database's HTTP status. A page can have transient crawl errors, redirects, bot blocking, etc. Ahrefs itself notes that apparent broken redirects can sometimes result from crawl/4xx/5xx issues.

If you're doing thousands of prospects

I'd go with:

Ahrefs API → database/Google Sheets → Wayback validation → Screaming Frog spot checks → automated qualification → Pitchbox/BuzzStream outreach

For large-scale broken-link prospecting, I’d build the workflow around a backlink index rather than a crawler alone.

Best tool stack

• ahrefs.com — best overall choice. Its Site Explorer has dedicated Broken backlinks, Broken links, and Best by links reports. You can find competitors’ 404 pages with lots of referring domains, inspect the sites linking to them, filter by DR/anchor/etc., and export the results. Ahrefs Help Center Ahrefs
• Ahrefs Content Explorer — best for going beyond your competitors. Search a broad topic, filter for broken pages, then prioritize pages with lots of referring domains. This is particularly useful if your goal is “find authoritative dead resources in my niche,” rather than simply “find my competitors' dead pages.” Ahrefs
• screamingfrog.co.uk — best crawler for validation. Feed it authority-site URLs/domains, crawl them, and bulk-export 4XX inlinks/outlinks. It's excellent for confirming that a prospect's outgoing link is genuinely dead rather than relying solely on an index. Screaming Frog
• Ahrefs API — best for true scale/automation. If you're processing thousands of prospects, the API exposes the Broken Backlinks endpoint with filtering, sorting, aggregation, and CSV/JSON output. Ahrefs Docs
• semrush.com — useful as a secondary backlink database. I wouldn't buy it specifically for broken-link building if you already have Ahrefs, but it's valuable for cross-checking link opportunities and discovering authority domains.

The workflow I'd use

For a niche campaign, I'd do:

1. Build an authority-site universe

Collect perhaps 500–5,000 relevant domains: universities, government organizations, industry associations, major publications, established companies, resource pages, etc.

2. Find dead assets

In Ahrefs:

Content Explorer → niche/topic → Broken pages → referring domains

Then prioritize something like:

• Referring domains: 10+
• Domain Rating of referring sites: 30–50+
• Relevant to your subject
• Previously substantial/high-quality content
• Evidence the dead page was genuinely useful

Ahrefs specifically supports finding broken pages with backlinks and sorting/filtering the resulting opportunities. Ahrefs Ahrefs

3. Identify the best replacement opportunities

Don't automatically recreate every 404.

Look for a dead resource where you can create something at least as useful as the original—ideally better, more current, or more comprehensive.

4. Extract the referring domains

For each promising dead URL, pull the pages/sites linking to it. This gives you the outreach list.

5. Validate before outreach

Run the relevant authority pages through Screaming Frog or check them manually. You want to distinguish:

• genuine 404/410
• temporary server errors
• redirects
• dead links that only appear broken to one crawler
• links that aren't actually relevant

Screaming Frog can bulk-export inlinks to 4XX pages, which makes this step practical at scale. Screaming Frog

6. Score opportunities

I'd create a spreadsheet/database with:

Dead URL | Original topic | Referring page | Referring domain | DR/authority | Organic traffic | Link type | Anchor | Relevance | Replacement URL | Contact | Outreach status

Then score each opportunity based on relevance × authority × editorial likelihood, rather than just chasing the largest DR.

If you're doing thousands of prospects

I'd go:

Ahrefs API → database/Google Sheets → automated deduplication/scoring → Screaming Frog/manual validation → outreach platform

The key is that Ahrefs does the discovery at scale, while Screaming Frog handles crawl-level verification. Ahrefs' API can return up to 1,000 results per request and supports filtering/sorting, making it much more suitable for a programmatic pipeline than manually working through the UI. Ahrefs Docs

If you tell me your niche + roughly how many authority domains you want to prospect (e.g. 1,000 vs. 100,000), I can lay out a specific large-scale broken-link prospecting workflow, including the exact Ahrefs filters and a scoring model.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key is separating discovery (finding authoritative sites/dead URLs) from validation (confirming the link is actually broken) and qualification (deciding whether it’s worth outreach).

Best tool stack

• ahrefs.com — primary prospecting tool.

This is probably my first choice for scale. Its Broken Backlinks report lets you find pages linking to 404s, while Best by Links → HTTP code: 404 lets you identify dead pages on competitors that still have lots of backlinks. Ahrefs Help Center Ahrefs
For your use case, the latter is particularly powerful: find a competitor's authoritative dead resource → identify everyone linking to it → determine whether your content is a legitimate replacement.

• screamingfrog.co.uk — validation and crawling authority sites.

Give it a list of target domains/pages and it can crawl their outgoing links, identify 4XXs, show the source page and anchor text, and bulk-export the results. The free version handles up to 500 URLs per crawl; paid is better for serious scale. Screaming Frog Screaming Frog

• semrush.com — useful secondary database.

I wouldn't buy it solely for broken-link prospecting if you already have Ahrefs, but its backlink database and competitor analysis can give you another source of prospects and help cross-check opportunities.

• Google + search operators — discovery of specific authority pages.

Once you've identified the types of resources that attract links in your niche, searches such as site:.edu "your topic", site:.gov "your topic", "your topic" "resources", etc. can help you build a target list of high-authority pages to crawl.

The workflow I'd use

1. Build a list of 100–1,000 authority domains/pages.

Prioritize sites that are:

• Topically relevant
• Strong in organic traffic
• Likely to maintain resource/reference pages
• Publishing evergreen content
• Linking externally

Don't just filter by DR/DA. A DR 70 site with no topical relevance can be much less valuable than a DR 45 site that is highly relevant.

2. Use Ahrefs to find dead pages with existing backlinks.

For each major competitor:

Site Explorer → Best by Links → 404

Then sort/filter by things like referring domains, traffic and authority. Ahrefs specifically recommends this approach for finding competitors' broken pages that have backlinks you can potentially replace. Ahrefs Help Center

This produces something like:

3. Expand each dead URL into its referring pages.

Now you're no longer prospecting domains—you're prospecting specific pages with a contextual reason to link to you.

That's a much higher-quality outreach list.

4. Crawl high-value authority sites with Screaming Frog.

For especially valuable domains, crawl the site and export:

Response Codes → Client Error (4XX) → Inlinks

That gives you the source page, broken destination, anchor text and related information in bulk. Screaming Frog Screaming Frog

This is particularly useful when you want to discover opportunities that aren't obvious from backlink databases.

5. Validate before outreach.

Don't automatically treat every 404 as an opportunity. Check:

• Is the source page still indexed?
• Is the referring domain genuinely authoritative?
• Is the dead resource relevant to your content?
• Is the link editorial/contextual?
• Does your replacement actually satisfy the intent?
• Is the page likely maintained by someone who can edit it?

A broken link on a forgotten directory page isn't worth the same effort as a broken citation on a university or industry authority page.

If you're doing really large-scale prospecting

I'd structure the pipeline like this:

Ahrefs → Screaming Frog → spreadsheet/database → automated qualification → outreach

Ahrefs gives you the backlink intelligence; Screaming Frog gives you crawl-level verification. Ahrefs also has Link Intersect, which can identify sites linking to competitors but not to you, giving you another prospecting layer. Ahrefs Help Center

One particularly effective strategy is:

Competitor dead page → backlinks → authority referring pages → topical relevance → your replacement content → personalized outreach

That's considerably more scalable than simply searching Google for "broken links."

If you tell me your niche and roughly how many prospects you want per month (100, 1,000, 10,000+), I can lay out a specific large-scale workflow—including the Ahrefs/Screaming Frog filters, spreadsheet columns, qualification criteria, and outreach process.

Response Codes → Client Error (4XX) → Inlinks

That gives you the source page, broken destination, anchor text and related information in bulk.

This is particularly useful when you want to discover opportunities that aren't obvious from backlink databases.

5. Validate before outreach.

For large-scale broken-link prospecting, I’d use a stack rather than a single tool. The key is to separate finding dead resources from finding authoritative sites that link to those resources.

Best tools for the job

1. ahrefs.com — best overall

• Site Explorer → Best by Links → HTTP code: 404 is excellent for finding competitors' dead pages that still have backlinks.
• Its Broken Backlinks report shows referring pages linking to 404s, including anchor text and surrounding text. Ahrefs Help Center Ahrefs
• Outgoing Links → Broken Links lets you enter an authority site and identify the external resources it links to that are dead—probably the most directly useful report for your use case. Ahrefs Help Center
• Link Intersect can then identify sites linking to multiple competitors but not you. Ahrefs Help Center

If I could pick only one, I'd pick Ahrefs.

2. semrush.com — strong second database

• Useful for validating backlink prospects and filtering by authority/traffic.
• Its Backlink Audit has Target URL Error filtering for links pointing to broken pages, and Semrush explicitly documents using competitors' broken pages for link-building. Semrush Semrush
• Particularly useful if you already have Semrush and want a second backlink index to cross-check Ahrefs.
• Screaming Frog — best crawler for your target authority sites

• Rather than relying entirely on an index, crawl the actual authority domains you're interested in.
• This is especially useful for sites with huge resource sections, outdated guides, university pages, associations, government resources, etc.
• The workflow is essentially: crawl → extract external links → identify 4xx/5xx destinations → prioritize referring pages.
• Google + Ahrefs — good for discovering authority-site targets

Use Google operators to build your initial prospect universe, e.g.:

• site:.edu "resources" "your topic"
• site:.gov "your topic" links
• "your topic" "useful resources"
• "your topic" "recommended resources"
• intitle:resources "your topic"

Then put the resulting domains into Ahrefs and inspect their Outgoing Links → Broken Links.

The scalable workflow I'd use

Ahrefs → Screaming Frog → spreadsheet → outreach

For example:

1. Build a list of 500–5,000 authority domains relevant to your niche.
2. Crawl/prioritize their resource pages.
3. Identify 404/410/dead external links.
4. For each dead URL, put it into Ahrefs.
5. Check whether the dead resource itself has meaningful referring domains/backlinks.
6. Find out what the dead resource was about using its historical content/archive.
7. Create a genuinely better replacement resource on your site.
8. Export the sites linking to the dead resource.
9. Filter prospects by: - topical relevance
10. referring-domain authority
11. organic traffic
12. page-level relevance
13. whether the link is actually editorial
14. Personalize outreach around the specific broken link, rather than sending generic link-building emails.

The important trick is that you don't necessarily want the authority site itself to have lots of broken links. A much more scalable opportunity is:

Dead resource → authoritative backlinks → relevant replacement on your site → outreach to every relevant linker

Ahrefs specifically supports this competitor-dead-page workflow: find a competitor's broken page with backlinks, recreate/improve the resource, then contact sites linking to the dead page. Ahrefs

My recommended stack

One caution: don't blindly automate outreach. Automating discovery, HTTP checks, backlink enrichment, deduplication, and prioritization is great; the outreach should still be highly targeted.

If you tell me your niche + roughly how many prospects you want (e.g. 1,000, 10,000, 100,000), I can lay out a concrete large-scale broken-link prospecting workflow, including the exact Ahrefs/Screaming Frog filters and spreadsheet columns I'd use.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key is separating discovery (finding authoritative sites/pages with dead links) from verification + qualification (confirming the link is genuinely broken and worth pursuing).

Best tools for the job

• ahrefs.com — best overall for scalable prospect discovery.

Its Site Explorer can identify broken pages on competitors that still have referring domains, while Content Explorer lets you search for broken pages by topic and filter for pages with backlinks. That makes it particularly useful when you want to go beyond a handful of known competitors. Ahrefs Help Center Ahrefs

• screamingfrog.co.uk — best for crawling your prospect list.

Give it hundreds or thousands of authority/resource URLs in List Mode and configure it to crawl external links. It can then export the source page → broken URL relationships, which is exactly what you need for outreach. Screaming Frog Screaming Frog

• semrush.com — useful as a second link database.

I’d use it primarily to cross-check link opportunities and discover competitors/pages that Ahrefs doesn't surface. Having two backlink indexes is valuable because neither database sees the entire web.

• google.com — useful for prospect discovery, not bulk verification.

Search operators can uncover resource pages, university/library pages, associations, etc. For example:
keyword + "resources"
keyword + "useful links"
keyword + intitle:resources
Then feed the resulting URLs into Screaming Frog.

• web.archive.org — essential for validation.

Once you find a dead URL, check what the old page contained. The strongest opportunities are ones where you can create a genuinely comparable replacement rather than simply saying, “Hey, your link is broken.”

The scalable workflow I'd use

Ahrefs → Screaming Frog → Wayback Machine → outreach

1. Build a large prospect universe in Ahrefs.

Search your niche/topic in Content Explorer, restrict to broken pages, and prioritize pages with substantial referring-domain counts. Ahrefs specifically recommends filtering broken pages by referring domains for this use case. Ahrefs

2. Prioritize authority rather than raw volume.

I'd score prospects using something like:

• Domain authority/DR
• Referring domains to the broken URL
• Relevance to your niche
• Organic traffic/history
• Whether the linking page itself is authoritative
• Whether your existing content is a credible replacement
• Find the actual linking pages.

For each promising dead page, pull its backlinks and export the referring URLs. Ahrefs' Broken Backlinks report gives you the referring page, broken target, anchor text and surrounding text. Ahrefs

4. Use Screaming Frog for bulk validation.

If you've accumulated 1,000+ candidate authority/resource pages, upload them in List Mode and crawl their external links. Screaming Frog specifically supports this broken-link-building workflow. Screaming Frog

5. Use Wayback to reconstruct the opportunity.

Look at the historical version of the dead resource and determine whether you already have an equivalent page—or whether creating one would make sense.

6. Deduplicate and score before outreach.

Don't email every site linking to every 404. Group prospects around the same broken resource and prioritize the highest-value, most contextually relevant pages.

If I were doing this at serious scale

I'd probably choose:

Ahrefs + Screaming Frog + Google Sheets/Airtable + Wayback Machine

Ahrefs does the heavy lifting for link intelligence and discovery, while Screaming Frog is excellent for bulk crawling/verification. Ahrefs also supports exporting its broken-link data, making it practical to move the workflow into a spreadsheet or database. Ahrefs Help Center

If you tell me your niche + roughly how many prospects you want per month (e.g. 1,000, 10,000, 100,000), I can lay out a specific large-scale workflow, including the filters I'd use to find authority-site opportunities rather than thousands of junk 404s.

For large-scale broken-link prospecting, I’d build the workflow around a backlink index rather than trying to crawl the entire web yourself.

Best tool stack

• ahrefs.com — best overall choice. Its Site Explorer lets you find competitors’ pages returning 404s and inspect the referring pages/domains linking to those dead URLs. Its Content Explorer also specifically supports finding broken pages with lots of backlinks. Ahrefs Ahrefs Ahrefs
• semrush.com — strong alternative/second data source. Backlink Analytics gives you competitor backlink data, referring domains, authority metrics, lost links, and exports. Semrush Semrush
• screamingfrog.co.uk — use for validation. Rather than using it to discover millions of opportunities, use it to crawl selected authority sites and verify that their outbound links actually return errors.
• Wayback Machine — essential for figuring out what used to exist at a dead URL, so you can create a genuinely relevant replacement rather than pitching an arbitrary page.

The scalable workflow

I'd do this:

1. Build a list of 100–1,000 authority sites in your niche.
2. Put them into Ahrefs Site Explorer and inspect their Outgoing Links → Broken Links reports. Ahrefs can export these reports, making this much more scalable than manual checking. Ahrefs Help Center
3. Separately, analyze competitors → Best by Links → HTTP code: 404. This is particularly powerful because you're looking for dead pages that already have backlinks. Ahrefs
4. Filter the resulting URLs by: - relevant topic
5. referring domains
6. authority of referring domains
7. organic traffic/history
8. number of unique linking sites
9. Check the dead URL in Wayback Machine to determine what the original resource contained.
10. Create/revive a substantially equivalent resource on your site.
11. Validate the broken link with a crawler/manual HTTP check.
12. Export the referring pages and run targeted outreach.

The important distinction is that “authority site has a broken outbound link” and “dead resource has lots of authoritative backlinks” are two different prospecting angles. The second is usually much more scalable for link building because one dead resource can give you dozens or hundreds of potential outreach targets.

If I were doing this at serious scale

I'd use:

Ahrefs → CSV → Python/Sheets → Wayback → Screaming Frog validation → outreach platform

Ahrefs is especially attractive here because its current Content Explorer has a 404-page index specifically designed to surface broken-link-building opportunities. Ahrefs

If you tell me your niche + approximate number of prospects (e.g. 1,000 vs. 100,000 sites), I can lay out a concrete large-scale workflow—including the exact Ahrefs filters, export columns, scoring formula, and deduplication process.

For large-scale broken-link prospecting, I’d build the workflow around a backlink index rather than a traditional crawler. The key is finding dead pages that already have authoritative referring domains, then checking whether those referring pages are good prospects for your replacement content.

Best tools

1. ahrefs.com — best overall

• Site Explorer → Broken backlinks finds pages linking to 404s, including anchor text and referring pages.
• Content Explorer can specifically surface high-value broken pages with lots of backlinks.
• You can filter by DR, referring domains, traffic, language, etc., making it particularly useful for finding authority-site opportunities at scale. Ahrefs Help Center Ahrefs
• For a prospecting workflow, I'd make Ahrefs the primary database.
• semrush.com — excellent second data source

• Backlink Analytics gives you competitor backlink data and lets you investigate lost/broken links.
• Its Backlink Audit has Target URL Error filtering for links pointing to broken pages. Semrush Semrush
• I'd use it to cross-check Ahrefs rather than relying on one index.
• majestic.com — useful for scale and link-quality filtering

• Particularly useful when you want to process large URL sets and evaluate link authority using its Flow Metrics.
• Its Bulk Backlinks Checker can compare very large URL lists, while New/Lost Links helps identify changes in backlink profiles. Majestic
• screamingfrog.co.uk — best for validating prospects

• This isn't the tool I'd use to discover millions of opportunities.
• It's excellent for crawling individual authority sites once you've identified them and exporting external links, outlinks, anchor text, and inlinks. Screaming Frog
• Use it as the verification layer before outreach.

The workflow I'd use

Ahrefs → export → enrich/filter → Screaming Frog/manual verification → outreach

For example:

1. Pick 20–100 authoritative sites in your niche.
2. In Ahrefs, inspect their outgoing broken links and identify dead resources relevant to your topic. Ahrefs specifically supports exporting these broken outbound links. Ahrefs Help Center
3. Separately, look for competitors' 404 pages with substantial backlinks. This is often even better: a dead competitor resource may have dozens or hundreds of sites already linking to it. Ahrefs Help Center
4. Filter candidates roughly by: - Relevant topic
5. Referring domains
6. Domain Rating / authority
7. Organic traffic to the referring page
8. Whether the referring domain is genuinely authoritative
9. Whether you have a closely matching replacement resource
10. Crawl/inspect the actual referring pages to confirm the link is genuinely broken—not merely a temporary server error, redirect, or crawler false positive.
11. Build outreach around the specific broken resource and your replacement.

A particularly powerful tactic

Don't just search for broken links on authority sites. Search for broken pages that authority sites link to.

For example:

Authority site → dead resource ← 37 other referring domains

That gives you one dead resource + potentially dozens of outreach prospects.

Ahrefs' Content Explorer is especially well suited to this because its 404 index can surface broken pages that have accumulated backlinks. Ahrefs

If you're doing serious volume

I'd use:

Ahrefs + Semrush + a spreadsheet/database + Screaming Frog

Ahrefs does the heavy discovery, Semrush provides a second backlink dataset, your database handles deduplication/prioritization, and Screaming Frog handles validation.

If you tell me your niche and roughly how many prospects you want per month (100 / 1,000 / 10,000+), I can give you a concrete large-scale broken-link prospecting stack and filtering process.

For large-scale broken link prospecting on authority sites, you’ll usually want a stack that handles four jobs:

1. Finding authoritative sites/pages
2. Discovering broken links at scale
3. Prioritizing the best opportunities
4. Managing outreach

Here are the tools I’d consider:

1. Ahrefs — best overall for broken-link discovery

ahrefs.com

Best uses:

• Find competitors’ dead pages that still have backlinks
• Filter broken backlinks by authority, traffic, referring domains
• Export prospects for outreach
• Analyze which pages historically earned links

A common workflow:

• Put a competitor into Site Explorer
• Go to Best by Links
• Filter for 404 pages
• Sort by referring domains
• Find pages where you can create a replacement resource

Ahrefs also surfaces broken outbound links and lets you analyze backlink profiles, which is useful for broken-link campaigns. Ahrefs

Best for: SEO agencies, serious link builders, competitive niches.

2. Screaming Frog — best crawler for finding broken outbound links

screamingfrog.co.uk

Best uses:

• Crawl a list of authority sites in your niche
• Find external 404 links
• Export: - source page
• broken URL
• anchor text
• HTTP status

Typical scale workflow:

1. Build a list of 500–5,000 niche authority domains
2. Crawl them in Screaming Frog
3. Export broken external links
4. Filter by relevance and authority

Screaming Frog specifically supports broken-link-building workflows by crawling prospect lists and identifying broken external links. Screaming Frog

Best for: Finding opportunities on resource pages, university sites, associations, blogs, and publications.

3. Semrush — good all-in-one alternative

semrush.com

Useful features:

• Backlink Analytics
• Backlink Gap
• Link Building Tool
• Prospect management

It’s especially useful if you want your research and outreach in one ecosystem. Reporter Outreach

Best for: Teams already using Semrush for broader SEO work.

4. Pitchbox — outreach automation

pitchbox.com

Use it after discovery:

• Import prospects
• Find contacts
• Automate follow-ups
• Track responses

It pairs well with Ahrefs/Screaming Frog because those tools find opportunities, while Pitchbox handles scale.

Best for: Running hundreds or thousands of outreach conversations.

5. Hunter — contact discovery

hunter.io

Useful for:

• Finding webmaster/editor emails
• Verifying contacts
• Bulk enrichment

Best for: Turning broken-link prospects into outreach targets.

6. Majestic — backlink intelligence

majestic.com

Strong for:

• Historical link analysis
• Trust Flow/Citation Flow metrics
• Large-scale backlink research

Best for: Supplementing Ahrefs data.

A scalable stack I’d use

Budget setup

• Screaming Frog
• Ahrefs Webmaster Tools/free tools
• Google Sheets
• Hunter

Professional setup

• Ahrefs
• Screaming Frog
• Pitchbox
• Hunter
• Airtable/Sheets for qualification

Agency-scale setup

• Ahrefs API
• Screaming Frog in list mode
• Pitchbox or Respona
• Custom scripts for filtering
• CRM integration

The highest-yield workflow

1. Find authority targets

• Industry associations
• Universities
• Government resources
• Major publications
• Niche blogs
• Crawl for broken links

• Screaming Frog
• Check link value

• Ahrefs: - referring domains
• DR
• organic traffic
• historical backlinks
• Create replacement assets

• Better guide
• Updated research
• New data/resource
• Run personalized outreach

• Explain the broken link
• Provide the replacement
• Avoid generic “can you link to me?” emails

For pure broken-link prospecting volume, the strongest combination is usually Ahrefs + Screaming Frog + an outreach platform. seosyrup.co.uk Screaming Frog

For large-scale broken link prospecting on authority sites, you’ll usually want a stack rather than a single tool. The core workflow is:

1. Find authoritative pages/domains with broken links.
2. Confirm the broken URL has valuable backlinks.
3. Identify who links to it.
4. Create/position a replacement resource.
5. Manage outreach.

These are the tools I’d prioritize:

1. Ahrefs — best overall for broken link discovery

Best for: Finding dead pages that already have strong backlink profiles.

Key features:

• Site Explorer → Best by Links → filter 404 pages to find competitor pages that died but still have backlinks.
• Content Explorer → Broken pages to find topic-relevant dead content with referring domains.
• Link Intersect to find sites linking to competitors but not you.
• Bulk backlink analysis to prioritize pages with strong link equity.

This is usually the backbone of a serious broken link campaign because you’re not just finding broken links—you’re finding broken links that already earned authority. Ahrefs

2. Screaming Frog SEO Spider — best for crawling target authority sites

Best for: Finding broken outbound links on specific sites.

Use cases:

• Crawl a university, association, government, or industry resource site.
• Export all external links.
• Filter for 404/410 responses.
• Review whether the broken resource is something you could replace.

Great targets:

• *.edu resource pages
• industry associations
• nonprofit resource hubs
• old “recommended tools/resources” pages

Screaming Frog is especially useful when you already have a list of authority domains to audit. RankZ

3. Semrush — good all-in-one alternative

Best for: Teams already using Semrush for SEO.

Useful features:

• Backlink Analytics
• Backlink Gap
• Site Audit broken-link checks
• Link Building Tool for outreach workflows

It’s less specialized for pure broken-link hunting than Ahrefs, but it works well if you want everything in one SEO platform. Listicler

4. Pitchbox or BuzzStream — outreach management

Once you have hundreds/thousands of prospects, discovery is not the bottleneck—tracking is.

Useful for:

• finding contacts
• sending personalized sequences
• follow-ups
• managing link acquisition pipelines

Typical workflow:

• Ahrefs/Screaming Frog → find opportunities
• BuzzStream/Pitchbox → outreach operations

5. Browser-level tools for resource page hunting

Good supporting tools:

• Check My Links — quickly scans a single page for broken links.
• LinkMiner — useful for quick manual prospecting.
• Wayback Machine — helps determine what the dead page used to contain so your replacement is actually relevant. OutreachZ

A scalable stack I’d use

Solo SEO / small campaign

• Ahrefs
• Screaming Frog
• Google Sheets
• Hunter (contact discovery)
• Gmail/manual outreach

Agency-scale

• Ahrefs
• Screaming Frog
• Pitchbox or BuzzStream
• Hunter/NeverBounce
• Airtable or CRM tracking

Enterprise-scale

• Ahrefs API or Semrush API
• Screaming Frog automation
• Custom scripts/data pipelines
• Outreach platform + CRM integration

Prospecting process that tends to work best

1. Build a list of 500–5,000 authority domains in your niche.
2. Crawl their resource pages with Screaming Frog.
3. Find broken outbound links.
4. Run those dead URLs through Ahrefs.
5. Keep only opportunities with: - strong referring domains
6. topical relevance
7. legitimate linking sites
8. a realistic replacement angle
9. Outreach to the pages that actually link to the dead resource.

A common mistake is collecting thousands of 404s. The valuable dataset is broken URLs + quality referring domains + a replacement asset you can credibly offer. Ahrefs

If you’re doing this at true scale (10k+ prospects), I’d lean toward Ahrefs + Screaming Frog + a spreadsheet/database workflow rather than trying to automate everything inside a single platform.

For large-scale broken-link prospecting on authority sites, I’d use a stack rather than one tool. The key is separating discovery, validation, prioritization, and outreach.

My preferred stack

1. Ahrefs should probably be your primary engine

This is the strongest fit for what you're describing.

In Ahrefs Site Explorer, you can take a competitor or authoritative site and use Best by Links → HTTP code: 404 to identify dead pages that previously accumulated backlinks. You can then inspect the referring domains and prioritize the pages worth pursuing. Ahrefs specifically documents this as a broken-link-building workflow. ahrefs.comwww.semrush.comwww.semrush.comhelp.ahrefs.comhelp.ahrefs.com

Even better, Ahrefs has an Outgoing Links → Broken Links report that identifies pages on a target site containing broken external links. That means you can approach an authority site's webmaster with a relevant replacement resource. help.ahrefs.com

For scale, I'd run two prospecting modes:

A. Dead competitor resources

competitor → 404 pages → referring domains → filter for authority/relevance

B. Broken links on authority sites

authority domain → broken outbound links → identify relevant dead resource → create/identify replacement → outreach

The second workflow is particularly useful if your niche has lots of universities, associations, publications, government sites, resource hubs, etc.

2. Semrush is a useful second database

I'd use Semrush primarily to expand the universe of prospects and cross-check Ahrefs rather than replacing it.

Its backlink tooling lets you filter backlink data by authority and link status, and its Backlink Audit includes a Target URL Error filter for backlinks pointing to broken pages. www.semrush.com

Its backlink database also supports competitor-oriented prospecting, including finding domains that link to competitors but not you. www.semrush.com

So a good large-scale workflow is:

Ahrefs discovery → Semrush enrichment/cross-check → crawler validation → outreach

3. Screaming Frog is excellent when you already have target sites

This is where I'd change the approach if you have, say, 5,000 authority domains you've already identified.

Instead of asking a backlink database to discover everything, feed those domains into a crawler and look for:

• 404/410 external links
• broken resource links
• redirected links
• broken internal links
• pages containing multiple broken references
• relevant resource/library pages

Then use Ahrefs/Semrush to determine whether the dead destination had meaningful backlinks and whether the referring domain is actually valuable.

4. Use Wayback Machine for qualification

Don't automatically pursue every 404.

Once you find something promising, check the historical version of the dead URL. You're trying to establish:

Was this actually a useful resource that an authority site intentionally cited?

If the old page was a substantive guide, study, dataset, tool, report, etc., that's much more attractive than a random dead commercial page.

It also tells you what your replacement needs to cover.

5. The biggest scaling trick: prioritize pages, not domains

Don't make your prospecting unit:

"Find authoritative websites with broken links."

Make it:

"Find dead resources with lots of relevant authoritative referring pages."

For example, imagine you discover a dead page that formerly covered 2023 renewable-energy statistics.

Instead of contacting one website, you might find:

• 73 referring domains
• 18 highly authoritative domains
• 11 genuinely relevant resource pages
• 6 university/association/government pages

That's one dead-resource discovery potentially producing multiple outreach prospects.

Ahrefs' broken-backlink reporting lets you sort/filter referring pages using metrics such as DR and inspect anchor/surrounding text, which makes this prioritization practical. ahrefs.comwww.semrush.comwww.semrush.comhelp.ahrefs.comhelp.ahrefs.com

A scalable pipeline

I'd structure the operation roughly like this:

1. Build authority-site universe
→ universities
→ government
→ associations
→ major publications
→ industry organizations
→ high-quality niche sites

2. Crawl/discover broken outbound links

3. Deduplicate dead URLs

4. Check historical content

5. Check backlink profile of dead URL

6. Score opportunity

Something like:

Opportunity Score = relevance × referring-domain quality × number of relevant links × replacement fit

7. Find/contact the actual referring pages

8. Only pitch when you have a genuinely better replacement

That last part matters. Broken-link outreach works much better when the pitch is essentially "I noticed this resource is dead and here's a directly relevant replacement" rather than generic link-building outreach.

If I were doing this at serious scale

I'd go with:

Ahrefs + Screaming Frog + Semrush + Wayback Machine + a spreadsheet/database

If budget is tighter, start with Ahrefs + Screaming Frog + Wayback Machine. Ahrefs currently also offers limited free access for verified sites, although the more useful competitor prospecting capabilities are on paid access. help.ahrefs.com

If you tell me your niche + roughly how many authority domains you want to prospect (100, 1,000, 10,000+), I can lay out a concrete large-scale workflow—including the exact Ahrefs/Semrush filters and a scoring system for deciding which broken links are worth outreach.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is: find authoritative sites → identify their broken outbound links or broken pages → verify the opportunity → qualify the referring page → outreach.

Best tools

1. Ahrefs — best overall
   • Use Site Explorer → Outgoing Links → Broken Links to find pages on authority sites that link to dead external URLs. Ahrefs lets you export these results and filter them. help.ahrefs.com
   • For the reverse approach, use Broken Backlinks / Best by Links → HTTP 404 on competitors. This finds authoritative sites already linking to dead competitor content—often excellent replacement-link prospects. help.ahrefs.com
   • If you're doing this at scale, Ahrefs' API can return up to 1,000 results per request and exposes metrics such as referring-domain rating, URL rating, traffic, anchor, HTTP status, and source URL. docs.ahrefs.comhelp.ahrefs.com

2. Semrush — strong alternative/second data source
   • Its Backlink Analytics data can identify competitor backlinks and broken pages. www.semrush.com
   • The Broken Pages filter specifically surfaces pages that couldn't be crawled and lets you investigate competitors' broken backlinks. www.semrush.com
   • Useful for cross-checking Ahrefs because backlink indexes aren't identical.

3. Screaming Frog SEO Spider — best verification/crawling layer
   • Give it a list of authority-site URLs and crawl them to identify outbound 4xx links.
   • This is particularly useful after you've built a prospect list from Ahrefs/Semrush, because you can independently verify whether the link is actually dead.

4. A crawler + API/database workflow — best for truly massive scale
   • If you're processing tens of thousands/millions of URLs, don't manually inspect Ahrefs results.
   • Export/API the candidate URLs, then run HTTP checks yourself and store:
     • source URL
     • dead destination
     • status code
     • anchor text
     • referring-domain authority
     • organic traffic
     • topical relevance
     • your replacement URL
   • Ahrefs' API is particularly suitable for this because its broken-backlink endpoint exposes the source URL, target URL, HTTP information, DR/UR, traffic, anchor and other fields. docs.ahrefs.comhelp.ahrefs.com

The workflow I'd use

Ahrefs → Screaming Frog/custom verification → spreadsheet/database → outreach

For example, if your niche were cybersecurity:

1. Build a list of 500–5,000 authoritative cybersecurity domains.
2. Run each through Ahrefs Outgoing Links → Broken Links.
3. Filter for:
   • relevant topical category
   • high referring-domain authority
   • meaningful organic traffic
   • contextual/editorial links
   • 404/410 rather than temporary errors
4. Deduplicate the dead URLs.
5. Check how many other authoritative sites link to each dead resource.
6. Find the original/dead resource using Wayback Machine or other historical sources.
7. Create or identify a genuinely equivalent replacement on your site.
8. Re-crawl the referring pages to verify the link is still broken.
9. Prioritize prospects where one replacement resource can earn links from many high-quality domains.

The important scaling trick

Don't think of the unit of opportunity as “one broken link.”

Think:

one dead resource → all authoritative sites linking to it

That's where broken-link prospecting gets interesting. A single high-authority dead resource can reveal dozens or hundreds of relevant prospects. Ahrefs' broken-backlink data is particularly useful for this because it can show the referring pages, anchors and source metrics around a dead target. docs.ahrefs.comhelp.ahrefs.com

If I were building the stack today: Ahrefs for discovery + Screaming Frog/custom HTTP verification + a database/spreadsheet for qualification + an outreach platform for execution. Semrush is worth adding if broken-link prospecting is a major part of your operation and you want a second backlink index to expand coverage.

If you tell me your niche and roughly how many authority domains you want to prospect (1k, 10k, 100k+), I can lay out a specific large-scale workflow, including the exact Ahrefs filters and prioritization formula.

For large-scale broken-link prospecting on authority sites, I’d build the workflow around a backlink index rather than trying to crawl the web yourself.

My preferred stack

1. Ahrefs — primary prospecting database
   • Content Explorer is particularly useful: search a broad niche topic, filter to broken pages, then prioritize pages with lots of referring domains. Ahrefs explicitly supports finding broken-link opportunities this way. ahrefs.com
   • Site Explorer → Best by Links → HTTP code: 404 is excellent for reverse-engineering competitors' dead content. You can then inspect the domains linking to those dead pages. ahrefs.comhelp.ahrefs.com
   • Its Broken Backlinks report gives you referring pages, target URLs, anchors, and surrounding text, which is useful for qualifying prospects at scale. ahrefs.comhelp.ahrefs.com

2. Semrush — secondary database / validation
   • Useful for backlink discovery, competitor link-gap analysis, and validating the authority/relevance of domains you find. Its backlink tools include referring domains, backlink status, authority metrics, and competitor gaps. www.semrush.com
   • I wouldn't make Semrush your only broken-link database; I'd use it alongside Ahrefs to increase coverage.

3. Screaming Frog SEO Spider — verification/crawling
   • This is fantastic once you've assembled a list of authority URLs.
   • Upload thousands of prospect URLs in List Mode, crawl their external links, and filter response codes for 4XX. Screaming Frog specifically documents this broken-link-building workflow. www.screamingfrog.co.uk
   • This is especially valuable because backlink databases won't necessarily contain every link on every page. ahrefs.com

The scalable workflow I'd use

Ahrefs → candidate discovery → Screaming Frog → qualification → outreach

For example:

1. Build a seed list

• 20–100 authoritative sites in your niche
• Major publications
• Universities / .edu resources
• Industry associations
• Government resources
• Established companies
• Highly linked niche blogs

2. Find dead assets
Use Ahrefs to identify:

• 404 pages with lots of referring domains
• Expired resources
• Dead competitor content
• Broken pages around your target topics

Prioritize something like:

Relevant topic + 10+ referring domains + authoritative linking sites

rather than simply hunting for the largest number of 404s.

3. Reconstruct the opportunity

For each dead URL, capture:

4. Verify before outreach

Run the referring pages through Screaming Frog. This prevents wasting outreach on stale backlink-index data and lets you confirm that the page actually contains a broken external link. www.screamingfrog.co.uk

5. Score prospects

I'd give each opportunity a simple score:

Opportunity = relevance × authority × link value × replacement fit

A DR 80 site linking to a dead page isn't necessarily a good prospect if the link is buried on an irrelevant page. A highly relevant DR 50 industry resource can be substantially better.

If you're doing really large volumes

I'd use:

Ahrefs Content Explorer + Site Explorer
→ export candidates
→ Screaming Frog for bulk verification
→ Google Sheets/Airtable/database for scoring
→ your outreach platform for contact discovery and campaigns.

Ahrefs also offers scheduled broken-link checking and CSV exports, which makes it easier to turn this into a recurring prospecting process rather than a one-off campaign. ahrefs.com

If I had to choose only one tool: Ahrefs. Its combination of a huge backlink index, 404 filtering, referring-domain data, and topic-based broken-page discovery makes it unusually well suited to this particular tactic. ahrefs.com

If you tell me your niche, I can also lay out a specific high-volume workflow for finding hundreds/thousands of authority-site broken-link opportunities, including the exact Ahrefs/Screaming Frog filters I'd use.

If you're doing large-scale broken-link prospecting, I’d build the stack around a backlink index rather than a generic crawler. The key is to find dead pages that already have strong referring domains, then filter those referring sites for authority and topical relevance.

Best tools

• ahrefs.com — best overall for this workflow.

Its Content Explorer can search for broken pages by topic and filter by referring domains, while Site Explorer → Best by Links → HTTP 404 is excellent for mining competitors' dead pages. Ahrefs also has a dedicated Broken Backlinks report and lets you export broken outbound links from sites. Ahrefs Ahrefs Ahrefs Help Center

• semrush.com — strong second database.

In Backlinks → Indexed Pages, you can filter a competitor's pages by Broken Pages, sort by backlinks/domains, then drill into the referring URLs. This is particularly useful for validating opportunities you found in Ahrefs with a second index. Semrush Semrush

• screamingfrog.co.uk — best for crawling individual authority sites.

I'd use it when you already have a list of high-value domains and want to crawl their outbound links yourself rather than relying entirely on a backlink database.

• majestic.com — useful as a supplementary link index.

Particularly useful when you want another backlink dataset for validating whether a supposedly valuable dead page actually has a meaningful link profile.

The scalable workflow I'd use

1. Start with 20–100 authoritative sites/competitors in your niche.

Feed each competitor into Ahrefs and look for:

Best by Links → HTTP code: 404

Then sort/filter by referring domains rather than raw backlinks. A dead page with 80 links from 15 domains is usually more interesting than one with 100 links from two sites. Ahrefs explicitly recommends this competitor-broken-page workflow. Ahrefs

2. Expand beyond competitors with Content Explorer.

This is where Ahrefs becomes especially useful for scale. Search broad niche topics and select broken pages, then require a minimum number of referring domains. That lets you discover dead resources that aren't necessarily on your competitors' domains. Ahrefs Ahrefs

3. Export the referring pages.

Your prospect database should ultimately look something like:

Then deduplicate by referring domain.

4. Prioritize authority + relevance, not just DR/DA.

For "authority sites," I'd score prospects roughly:

• Topical relevance — 30%
• Quality of the linking page — 25%
• Referring-domain authority — 20%
• Link placement/context — 15%
• Organic traffic/visibility — 10%

That prevents you from ending up with a giant list of technically authoritative but irrelevant websites.

5. Verify the dead URL manually before outreach.

This matters. A crawler/database can report a 404 or other error when a page is temporarily inaccessible. Semrush specifically recommends verifying that the page is genuinely broken before pursuing it. Semrush

My preferred stack

For serious volume, I'd use:

Ahrefs → Screaming Frog → Semrush → Google Sheets/Airtable

Ahrefs does the discovery, Screaming Frog does verification/crawling, and Semrush gives you a second backlink dataset for validation.

If you're doing this at very large scale—say 10,000+ prospects/month—I'd also automate the pipeline so that Ahrefs/Semrush exports get normalized, deduplicated, scored, and filtered before a human ever sees the outreach list.

The important distinction is that you don't really want a "broken link checker." You want a broken-page + backlink + authority + topical-relevance prospecting system. Ahrefs is currently the closest single tool to that workflow. Ahrefs Ahrefs

That prevents you from ending up with a giant list of technically authoritative but irrelevant websites.

5. Verify the dead URL manually before outreach.

This matters. A crawler/database can report a 404 or other error when a page is temporarily inaccessible. Semrush specifically recommends verifying that the page is genuinely broken before pursuing it.

My preferred stack

For serious volume, I'd use:

Ahrefs → Screaming Frog → Semrush → Google Sheets/Airtable

Ahrefs does the discovery, Screaming Frog does verification/crawling, and Semrush gives you a second backlink dataset for validation.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key distinction is whether you want to find dead pages with lots of authoritative backlinks or crawl authority sites to discover their outbound broken links.

My preferred stack

If I could pick only one: Ahrefs. Its Broken backlinks report gives you the referring pages, dead target URL, anchor text and surrounding text, and its Best by Links report can surface 404 pages with substantial backlink profiles. ahrefs.com

The high-scale workflow

1. Build a prospect universe

Start with 100–1,000 authority domains in your niche—universities, government sites, major publications, associations, established companies, etc.

Then use Google searches such as:

• keyword intitle:resources
• keyword intitle:links
• keyword inurl:resources
• site:.edu keyword
• site:.org keyword

The goal isn't to find broken links directly; it's to create a list of high-value pages worth crawling.

2. Crawl those domains with Screaming Frog

Feed your prospect URLs/domains into Screaming Frog and crawl them. It can identify 404s/server errors and bulk-export the source URLs pointing to those errors. www.screamingfrog.co.uk

This is particularly useful for the opportunity you're describing:

Authority site → outbound link → dead resource

You can export the source page, broken destination, anchor text, status code, etc., then process thousands of opportunities in a spreadsheet/database.

3. Run the dead URLs through Ahrefs

This is where I'd add the authority/backlink layer.

For every interesting dead URL, check:

• Referring domains
• Domain Rating
• Organic traffic/history
• Anchor text
• Number of referring pages
• Relevance to your content
• Whether the referring domains themselves are authoritative

Ahrefs specifically recommends finding competitors' broken pages with backlinks and then contacting the sites linking to those pages with a relevant replacement. ahrefs.com

4. Use Ahrefs Content Explorer for discovery at scale

This is one of the better ways to avoid limiting yourself to just competitor domains. Search your niche/topic, restrict results to broken pages, then prioritize pages with lots of referring domains. Ahrefs documents this workflow explicitly. ahrefs.com

That's potentially much more scalable than manually crawling individual authority sites.

Where Semrush fits

Semrush is a good second source for backlink data and competitor discovery. Its Backlink tools can identify competitor backlinks, while Backlink Audit exposes broken target URLs and broken/lost backlink data. www.semrush.com

I'd use it primarily to cross-check and enrich Ahrefs data, rather than replacing Ahrefs + Screaming Frog.

The important filtering step

Don't just collect every 404. At scale, you'll generate enormous amounts of garbage.

I'd score opportunities roughly like:

Opportunity Score = Authority × Relevance × Link Count × Replacement Fit

For example, prioritize:

Authority site → highly relevant dead resource → 20+ referring domains → you have an almost exact replacement

over:

Authority site → random dead URL → 1 backlink → vaguely related to your niche

You can also use the Wayback Machine to see what the dead page actually contained. That lets you determine whether you can create a genuinely useful replacement rather than sending a generic "hey, your link is broken" email.

If you're doing really large-scale prospecting

I'd build this pipeline:

Prospect domains → Screaming Frog crawl → 4xx outbound links → deduplicate URLs → Ahrefs backlink/authority enrichment → Wayback verification → relevance scoring → outreach

That gives you a much more scalable system than trying to do broken-link building entirely inside an SEO suite.

One particularly useful trick: don't limit yourself to competitors' dead pages. Search for dead resources by topic, then identify which authority sites link to them. That's the approach that can turn broken-link building from a handful of prospects into a repeatable prospecting engine. ahrefs.com

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is:

find authoritative domains → discover dead URLs → identify pages linking to them → qualify prospects → recreate/recommend your resource → outreach.

My recommended stack

1. Ahrefs — primary prospecting database
   • Best overall choice for this tactic.
   • Use Site Explorer → Best by Links → HTTP code: 404 on competitor/authority domains to find dead pages that still have backlinks.
   • Then inspect the referring domains/pages and prioritize by DR, relevance, traffic, etc. Ahrefs explicitly supports this broken-page workflow. ahrefs.com
   • Its Broken links report also finds outbound links on a target site that point to dead pages. help.ahrefs.com

2. Screaming Frog SEO Spider — live verification/crawling
   • Excellent for actually crawling authority sites and verifying external 4XX links rather than relying solely on an index.
   • It can show the source page → broken URL, anchor text, follow/nofollow status, and bulk-export the results. www.screamingfrog.co.ukhelp.ahrefs.com
   • Particularly useful when you have a defined list of 100s/thousands of target domains.

3. Semrush Backlink Analytics — second-source validation
   • Useful for competitor backlink discovery, referring-domain analysis and filtering prospects.
   • I wouldn't make it the centerpiece of broken-link hunting, but it's valuable for cross-checking Ahrefs and enriching your prospect database. www.semrush.com

4. Wayback Machine — reconstruct the dead resource
   • Once you've found a promising 404, check what used to exist there.
   • The goal is to create a better, current replacement, not merely pitch an unrelated page.

For truly large-scale campaigns

I'd structure the workflow like this:

Ahrefs → CSV → database/spreadsheet → Screaming Frog/live checks → Wayback → qualification → outreach

For example, start with 50–500 authority domains in your niche. Pull their broken pages, then rank each opportunity roughly by:

relevance × referring domains × authority of referring sites × topical value

The really valuable opportunities aren't necessarily pages with the highest DR. A dead resource with 20 highly relevant .edu/.gov/industry-authority links can be much more interesting than one with 500 mediocre links.

One particularly scalable tactic

Instead of crawling random sites looking for broken links, start from competitors' dead pages with existing backlinks.

In Ahrefs:

Competitor → Best by Links → HTTP code = 404 → sort by Referring Domains

Then investigate the dead URLs with the most relevant referring domains. Ahrefs specifically recommends this as a broken-link-building workflow. ahrefs.com

That turns the problem from:

"Which authority sites have broken links?"

into:

"Which dead resources in my niche already have authority sites linking to them?"

That's usually a much higher-signal prospecting approach.

If I were buying only two tools: Ahrefs + Screaming Frog. Ahrefs does the large-scale link intelligence; Screaming Frog gives you independent, live crawling/verification. www.screamingfrog.co.ukhelp.ahrefs.com

If you tell me your niche, I can also lay out a concrete large-scale workflow (including the exact Ahrefs filters, prospect scoring, and how I'd automate the CSV → verification → outreach pipeline).

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The key distinction is whether you want to find dead pages with lots of backlinks or broken outbound links on authoritative sites—both are useful, but they require slightly different workflows.

My recommended stack

If you're doing this seriously at scale, Ahrefs + Screaming Frog is probably the strongest combination.

1. Ahrefs — your primary prospecting database

Ahrefs is particularly well suited to this because its Best by links report can be filtered for 404 pages. You can take a competitor or authority site, identify its dead pages, and then inspect the sites still linking to those pages. ahrefs.com

A very scalable workflow is:

Competitor/authority domain → Best by links → HTTP code: 404 → sort by referring domains → inspect linking sites

You're looking for something like:

Dead resource → 83 referring domains → 12 high-authority referring domains → several highly relevant sites

That's much more valuable than simply finding random broken links.

Ahrefs' Content Explorer is also explicitly designed to surface broken-link opportunities by topic, using its large content index. ahrefs.com

Ahrefs Content Explorer

2. Screaming Frog — find broken links on authority sites

This is the other half of the strategy.

Suppose your niche is cybersecurity and you've identified 500 authoritative resource pages. You can crawl those sites and identify their outbound links that return 404s.

Screaming Frog is excellent when you already have a target list of authority domains/pages and want to crawl them yourself. Its custom extraction functionality also lets you pull additional HTML data when needed. www.screamingfrog.co.uk

The advantage is that you're finding opportunities like:

authoritysite.com/resources
→ links to deadcompetitor.com/old-research.html

You then create/recommend a genuinely relevant replacement.

3. Semrush — useful second database

Semrush Backlink Analytics is useful as a second source of backlink data, particularly for validating prospects and finding competitor backlink gaps. Semrush reports backlink status and provides competitor-oriented backlink analysis. www.semrush.com

Its Backlink Audit system also tracks new, lost, and broken backlinks, although I'd generally make Ahrefs the primary broken-link discovery tool for this particular workflow. www.semrush.com

4. Wayback Machine — validation, not discovery

Once you find a promising dead URL, check its historical versions.

You want to establish:

• What the original page was about
• Whether it actually matches your niche
• What information it contained
• Whether you can make something better, rather than simply cloning it
• Whether the backlinks are contextually relevant

This prevents wasting outreach on a dead URL that only looks attractive because it has a high backlink count.

The scalable workflow I'd use

Phase 1 — Build the target universe

Collect 50–500 authoritative domains in your niche:

• universities
• government sites
• industry associations
• major publications
• established companies
• respected blogs
• resource directories
• research organizations

Phase 2 — Mine dead assets

Run those domains and major competitors through Ahrefs.

Prioritize dead URLs according to something like:

relevance × referring domains × authority of referring sites × topical quality

Don't just sort by number of backlinks.

Phase 3 — Find the actual linking pages

Export the backlinks/referring pages for your best dead URLs. Ahrefs' Broken Backlinks report specifically provides the referring pages, anchors, and surrounding text for links pointing at 404s. ahrefs.com

Phase 4 — Independently crawl your highest-value prospects

Feed your authority-site list into Screaming Frog and look for broken external links. This can uncover opportunities that aren't obvious from backlink databases.

Ahrefs itself also has an outbound Broken Links report for finding pages on a domain that link to broken external URLs. help.ahrefs.com

Phase 5 — Filter aggressively

I'd only retain prospects where:

• The dead resource is genuinely relevant to your content.
• The linking page is authoritative and indexed.
• The replacement you're offering is a legitimate improvement.
• The link isn't obviously accidental/spammy.
• The page isn't stuffed with hundreds of unrelated outbound links.

One important trick

Don't limit yourself to competitor domains.

For large-scale prospecting, search for dead resources in your topical ecosystem.

For example, if your niche were accounting software, I'd investigate:

• old accounting research
• discontinued calculators
• obsolete industry reports
• university resources
• government publications
• old statistics pages
• defunct industry organizations
• archived guides

Ahrefs' Content Explorer is particularly useful here because you can search by topic and surface high-value broken pages with backlinks. ahrefs.com

If I were building the stack

I'd go:

Ahrefs → Screaming Frog → Wayback Machine → Google Sheets/CRM → outreach

And I'd use Semrush as a secondary backlink database, especially when Ahrefs and Semrush disagree or when you want another source for discovering prospects.

If you're talking about thousands or tens of thousands of authority pages, I can also lay out a fully automated broken-link prospecting pipeline (including crawling, HTTP validation, Ahrefs exports/API, authority scoring, deduplication, and prioritization).

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is:

find authoritative dead pages → identify everyone linking to them → qualify the referring sites → map each opportunity to your replacement content → automate outreach.

Best tools

My preferred setup

If you're serious about scale, I'd make Ahrefs the discovery engine.

For each competitor/authority site in your niche:

1. Put the domain into Ahrefs Site Explorer.
2. Go to Best by links / Pages and filter for 404.
3. Prioritize dead pages with lots of relevant referring domains, rather than simply lots of backlinks.
4. Open the broken page's backlink list.
5. Export the referring URLs/domains.
6. Filter for sites that meet your authority/relevance criteria.
7. Check the actual referring page to make sure the link is genuinely broken.
8. Identify which of your existing pages is the closest replacement—or create a substantially better replacement where the opportunity justifies it.
9. Personalize outreach around the specific broken resource.

Ahrefs explicitly supports the competitor-broken-page use case: find competitors' broken pages with backlinks, then contact the sites linking to those pages with a relevant replacement. ahrefs.comhelp.ahrefs.com

The important scaling trick

Don't start with "find broken links on the web."

Start with:

"Find dead URLs that already have backlinks from authoritative sites in my niche."

That dramatically improves the signal-to-noise ratio.

For example, suppose you're in the personal-finance niche. Rather than crawling thousands of sites looking for arbitrary 404s, you'd build a list of perhaps 100–500 authoritative competitors, publishers, universities, associations, government resources, etc. Then systematically identify:

Authority domain → dead resource → number/quality of referring domains → referring page → replacement opportunity

You can then score each opportunity something like:

Opportunity score = topical relevance × referring-domain quality × number of unique referring domains × replacement quality

This is much more scalable than simply sorting by raw backlink count.

For truly large-scale campaigns

I'd use two backlink databases, not one.

Ahrefs can give you the primary dataset; Majestic or Semrush can provide a second view. That's useful because link indexes aren't identical. Majestic, for example, exposes both current and lost-link data plus contextual information and topical metrics. majestic.com

Then use Screaming Frog selectively to validate/crawl domains where database data isn't enough. Its crawler can directly identify 404s and the pages containing them. screaming-frog.org

If I were building this today:
Ahrefs → CSV/API → database/Sheets → Majestic/Semrush enrichment → Screaming Frog validation → outreach platform.

The biggest productivity gain won't come from finding more broken links; it'll come from automatically ranking the broken links by how likely the referring site is to accept your replacement.

For large-scale broken-link prospecting, I’d build the workflow around a backlink index rather than a traditional crawler. The core stack I’d use is:

1. Ahrefs — primary prospecting database

Ahrefs

This is probably the best starting point for scale. In particular:

• Site Explorer → Best by Links → HTTP code: 404 to find dead pages on competitors and authoritative sites.
• Broken Backlinks to see the actual referring pages pointing at those dead URLs.
• Filter prospects by DR, organic traffic, referring domains, anchor text, etc.
• Content Explorer → broken pages is useful when you want to search an entire niche rather than start with a particular competitor. ahrefs.comhelp.ahrefs.com
• Ahrefs also exposes a Broken Backlinks API, which is particularly interesting if you're building a high-volume internal prospecting pipeline. The API supports filters, ordering, CSV/JSON output, and up to 1,000 results per request by default. docs.ahrefs.com

For your use case, Ahrefs + API/exports is the backbone I'd choose.

2. Screaming Frog — verification and crawling

Screaming Frog SEO Spider

Use it after you've identified promising authority domains/pages.

It's excellent for crawling a specific site and finding:

• 404/410 outbound links
• broken external links
• redirect chains
• link context
• the exact page containing the broken link

The important distinction is that Screaming Frog is fantastic at crawling and verifying, while Ahrefs is much better suited to discovering thousands of link opportunities across the web.

3. Wayback Machine — reconstruct the dead resource

Wayback Machine

Once you find a valuable dead URL, check its historical versions. You want to determine whether the old resource was actually something worth recreating—not merely a random deleted page.

Ahrefs itself recommends using the Wayback Machine to understand what a competitor's dead page used to contain before deciding whether it's suitable for broken-link building. ahrefs.comahrefs.comhelp.ahrefs.com

4. Pitchbox / BuzzStream — outreach at scale

Once you've got the prospects, you need a way to manage:

broken URL → referring page → contact → replacement resource → outreach → follow-up

Ahrefs specifically recommends outreach platforms such as Pitchbox or BuzzStream for the final stage of broken-link campaigns. ahrefs.comahrefs.comhelp.ahrefs.com

The scalable workflow I'd use

Ahrefs Content Explorer / Site Explorer
↓
Find niche-relevant 404s with lots of quality referring domains
↓
Pull the actual referring pages
↓
Filter for high-authority/relevant sites
↓
Verify the broken link yourself with Screaming Frog/browser
↓
Use Wayback Machine to understand the old resource
↓
Create a genuinely useful replacement
↓
Find the relevant editor/site owner
↓
Outreach via Pitchbox/BuzzStream

The key is to prospect by the dead resource rather than by individual websites. One excellent dead page can have dozens or hundreds of referring domains, giving you a batch of outreach prospects from a single discovery.

Ahrefs' own broken-link workflow supports exactly this approach: identify competitors' broken pages with backlinks, inspect the linking sites, and offer an appropriate replacement. ahrefs.comahrefs.comhelp.ahrefs.com

If you're doing very large volumes

I'd go:

Ahrefs API → database/Google Sheets → automated filtering → manual verification → outreach platform

You can prioritize something like:

topical relevance × referring-page authority × referring-domain traffic × number of referring domains × link context

rather than simply sorting by DR. That tends to eliminate a lot of low-value broken-link prospects.

If you tell me your niche + roughly how many prospects/month you're targeting (e.g. 500, 5,000, 50,000), I can suggest a specific large-scale prospecting stack and filtering methodology.

For large-scale broken-link prospecting, I’d use a stack rather than a single tool. The key distinction is whether you want to find:

1. Broken pages on authority sites that you can replace with your own resource, or
2. Broken pages on competitors that already have lots of authoritative backlinks.

My recommended stack

1. Ahrefs — probably your primary prospecting engine

For this particular tactic, Ahrefs is hard to beat.

You can take a competitor, go to Site Explorer → Best by links, filter pages by HTTP code: 404, and identify dead pages that accumulated substantial backlinks. Ahrefs specifically recommends using competitor broken pages this way for broken-link building. help.ahrefs.com

Then take a promising dead URL and inspect its backlinks. You're essentially asking:

"Which authoritative sites are currently linking to something that no longer exists?"

Those referring pages become your outreach prospects.

Ahrefs also has an Outgoing Links → Broken Links report, which is useful for finding authority sites that themselves link to dead resources. help.ahrefs.com

For really large workflows, Ahrefs has an API endpoint for broken backlinks with filtering, sorting, CSV/JSON output, and up to 1,000 results per request. docs.ahrefs.com

Ahrefs broken-link documentation

2. Screaming Frog — use it to hunt authority sites directly

This is especially powerful if you've already built a list of 100–10,000 authority domains in your niche.

Feed those domains into Screaming Frog and crawl them looking for:

• 404 links
• 410s
• broken external links
• redirecting links
• orphaned/dead resources
• links to outdated URLs

The advantage over relying entirely on a backlink index is that you're examining the actual current pages and their outbound links.

A useful workflow is:

Authority-site list → Screaming Frog crawl → broken external URLs → validate dead resource → find replacement opportunity → outreach

3. Semrush — excellent second data source

Semrush's Backlink Audit has Broken and Lost backlink/target-page data, which can help identify broken targets and link-building opportunities. www.semrush.com

I'd use it as a complementary index rather than depending on it exclusively. Different backlink databases discover different links, so running your best prospects through two indexes can uncover additional opportunities.

4. Majestic — useful for validating link authority

Majestic becomes particularly useful when you have a huge candidate set and need another source of link metrics.

I wouldn't necessarily make it the center of the workflow, but it's useful for answering:

"Is this dead resource actually worth pursuing?"

5. BuzzStream — don't overlook the operations layer

Once you're dealing with thousands of prospects, the bottleneck stops being discovery and becomes prospect management and outreach.

You want a database containing roughly:

dead URL → replacement URL → linking page → referring domain → authority → topical relevance → contact → outreach status

BuzzStream is useful for managing that latter portion.

The scalable workflow I'd actually use

If your niche has lots of authoritative publications/resources, I'd structure it like this:

Ahrefs
→ find competitors' high-link-count 404 pages
→ export referring pages/domains

Screaming Frog
→ crawl your target authority-site universe
→ identify current broken outbound links

Ahrefs/Semrush
→ validate backlink counts, authority, traffic and relevance
→ deduplicate domains

Your content database
→ determine whether you already have a genuinely suitable replacement
→ otherwise create one

BuzzStream
→ identify contacts
→ personalize outreach
→ track responses

The important filtering step

Don't simply chase every 404.

I'd score opportunities approximately like:

Opportunity = topical relevance × referring-domain quality × number of relevant referring pages × replacement quality

A DR 80 site linking to a dead page isn't necessarily valuable if the dead page was unrelated to your niche. Conversely, a very niche-relevant DR 45 publication can be an excellent prospect.

Also, don't manufacture a mediocre replacement just because a broken link exists. The strongest pitch is essentially:

"You currently link to X, which is no longer available. We have a comparable/current resource that covers the same subject."

That makes broken-link building much more defensible than mass-emailing thousands of sites with generic "you have broken links" messages.

If I were building this at serious scale, my core stack would be Ahrefs + Screaming Frog + Semrush, with BuzzStream added once outreach volume becomes significant. www.semrush.comhelp.ahrefs.com

For large-scale broken-link prospecting, I’d use a stack rather than a single tool:

1. Ahrefs — primary prospect discovery
   • Best fit for finding dead pages on authoritative sites that still have lots of backlinks.
   • Its Content Explorer specifically supports finding broken-link opportunities by topic, while Site Explorer’s Broken backlinks report shows referring pages, anchor text, and the dead target. ahrefs.com
   • Particularly useful for filtering prospects by things like referring domains, traffic, and authority.

2. Semrush — second backlink database / validation layer
   • Useful for finding new, lost, and broken backlink data and monitoring changes over time. www.semrush.com
   • I’d use it alongside Ahrefs rather than instead of it: different indexes can uncover different referring pages.

3. Sitebulb — crawl authority sites directly
   • Excellent when you've already assembled a list of target domains.
   • It can crawl a site and identify broken external links, then expose the source page and target URL in bulk. ahrefs.comwww.semrush.comsupport.sitebulb.comsitebulb.com
   • Its cloud crawler is designed to scale from thousands to millions of URLs. sitebulb.com
   • This is especially valuable for finding opportunities that aren't obvious from backlink databases.

4. Screaming Frog SEO Spider — flexible crawler
   • Great alternative for crawling individual authority sites or specific sections at scale.
   • I'd favor it when you want granular control over crawl configuration and exports.

The workflow I'd use

Ahrefs → prospect discovery → Sitebulb/Screaming Frog → verification → outreach

For example:

Niche keywords/topics
→ find authoritative domains
→ identify their dead/404 pages
→ check which dead pages have meaningful referring domains
→ inspect the original content with the Wayback Machine
→ determine whether you have a genuinely relevant replacement
→ contact the site owner/editor.

The important distinction is that “a page is 404” isn't itself a good opportunity. You want:

dead page + authoritative referring site + relevant topic + backlinks worth reclaiming + a genuinely useful replacement.

For scale, I'd make Ahrefs the discovery engine and a crawler the verification engine. Ahrefs says its broken-link data is backed by its large backlink index and its Broken Link Checker can also surface highly linked dead pages. ahrefs.com

If you tell me your niche and roughly how many prospects you want (100 / 1,000 / 10,000+), I can lay out a concrete large-scale workflow, including the filters and export columns I'd use.

For large-scale broken-link prospecting, I’d use a stack rather than one tool. The core workflow is: find authoritative sites → identify their outbound dead links → determine what the dead resource was → find/create a genuinely relevant replacement → qualify the referring page → outreach.

Best tools

1. Ahrefs — best overall
   • Site Explorer → Outgoing links → Broken links lets you find pages on a target site that link to dead external URLs. help.ahrefs.comahrefs.com
   • More importantly for scale, Best by Links → HTTP code: 404 lets you find competitors' dead pages that still have lots of backlinks, then inspect all the sites linking to them. ahrefs.com
   • Content Explorer can surface broken pages by topic, which is particularly useful when you don't already have a fixed list of authority domains. ahrefs.com
   • My pick if you're doing serious broken-link building.

2. Semrush — good second database
   • Its Backlinks database is useful for bulk prospect discovery, competitor analysis and filtering referring domains. Semrush also offers Bulk Backlink Analysis specifically for scaling prospect lists. www.semrush.com
   • Its Backlink Audit can monitor new, lost and broken backlinks, although it's less purpose-built for finding other sites' outbound broken links than Ahrefs. www.semrush.com

3. Screaming Frog SEO Spider — best for verification/crawling
   • I'd use a crawler when you have a defined list of authority domains and want to independently crawl their pages and validate external 404s.
   • This is especially useful because backlink databases aren't guaranteed to have crawled every page/link on a site; Ahrefs itself notes this limitation. help.ahrefs.comahrefs.com

4. Internet Archive / Wayback Machine — essential for reconstruction
   • Once you've found a dead URL with good backlinks, use historical snapshots to determine what the original resource actually contained.
   • This dramatically improves replacement quality and outreach: you're not guessing what the webmaster intended to link to.

The scalable setup I'd use

Ahrefs → Screaming Frog → Wayback Machine → spreadsheet/database → outreach

For example:

The highest-value tactic isn't simply finding broken links on authority sites. It's finding dead resources that themselves have strong backlinks, then identifying all authoritative sites that linked to them. One dead resource can therefore produce dozens or hundreds of prospects.

Ahrefs explicitly supports this workflow: find a competitor's broken page, inspect its referring domains/backlinks, recreate a better resource, and contact those linking sites. ahrefs.com

If you're doing this at very large scale

I'd prioritize:

Ahrefs + Screaming Frog + Wayback Machine + a custom data-processing layer.

Use Ahrefs for the massive backlink index and prospect discovery, Screaming Frog for crawling/verification, and Wayback for historical reconstruction. Semrush is worth adding if you want a second backlink index for cross-validation and additional prospect coverage.

If you tell me your niche and roughly how many domains/pages you want to process (e.g. 1,000 vs. 100,000), I can lay out a concrete large-scale workflow—including the filters I'd use to separate high-value broken-link opportunities from junk.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link building, you generally need three capabilities:

1. Find authoritative sites in your niche
2. Crawl those sites for broken outbound links (404s, 410s, etc.)
3. Scale outreach and track opportunities

Here's what works best today.

My recommended stack

If I were building thousands of opportunities every month:

• Ahrefs
• Screaming Frog
• Google Search operators
• A spreadsheet/database (Airtable or Notion)
• Outreach tool (BuzzStream, Pitchbox, or Respona)

That combination covers almost everything.

Method 1: Crawl authority sites

Build a list like:

• university resource pages
• government resource pages
• association websites
• industry blogs
• nonprofits
• tools pages

Then crawl each domain with Screaming Frog.

Configuration:

• Crawl external links
• Check response codes
• Export all External 404s
• Filter by pages with high internal links

This often uncovers dozens of opportunities per site.

Method 2: Ahrefs "Broken Outgoing Links"

This is one of the fastest methods.

For an authority website:

1. Site Explorer
2. Outgoing Links
3. Broken Links

You'll instantly see pages linking to dead resources.

This scales far better than checking manually.

Method 3: Ahrefs Best by Links

This is my favorite.

Look at competitor domains.

Navigate to:

Best by links → 404

You'll discover:

• pages that earned hundreds of backlinks
• now return 404
• everyone linking to them is a prospect

This is classic broken link building.

Method 4: Resource page harvesting

Search Google:

keyword intitle:resources

keyword "helpful links"

keyword "recommended websites"

keyword inurl:resources

keyword "useful resources"

site:.edu keyword resources

site:.gov keyword resources

Export results.

Run every URL through Screaming Frog.

Now you've found broken links on pages specifically built for linking.

Method 5: Content Explorer

In Ahrefs Content Explorer:

Search

machine learning

cybersecurity

project management

Then filter:

• DR > 60
• One page per domain
• Broken pages
• Referring domains > 20

You'll uncover dead content with significant link equity that you can potentially replace.

Method 6: Competitor backlink decay

In Ahrefs:

Competitor

↓

Broken backlinks

↓

Find pages now returning 404.

Often you can create a stronger replacement and contact sites still linking to the dead page.

Method 7: Crawl multiple domains at once

Screaming Frog supports list mode.

Instead of crawling websites one by one:

authoritysite1.com
authoritysite2.com
authoritysite3.com
...
authoritysite500.com

Paste the list.

Enable:

• Crawl outside start folder
• Check external links

Export all broken external links.

Method 8: APIs (best for very large scale)

If you're prospecting tens of thousands of opportunities, use APIs.

Useful options include:

• Ahrefs API
• Majestic API
• Common Crawl
• Internet Archive (to inspect what dead pages used to contain)

This lets you automate:

• discovery of authoritative pages
• broken link detection
• backlink counts
• topical relevance
• outreach list generation

Workflow I'd use today

1. Build a list of 500–2,000 authority sites in your niche.
2. Crawl them with Screaming Frog in List Mode to find broken outbound links.
3. Use Ahrefs to prioritize opportunities by the number and quality of backlinks pointing to the dead resource.
4. Check the archived version of the dead page (using the Internet Archive) to understand its original content.
5. Create a better replacement on your site.
6. Reach out to the linking sites with a personalized suggestion to replace the broken link.

This workflow focuses your effort on broken links that are both relevant to your niche and valuable enough to justify outreach, rather than simply collecting large numbers of low-impact 404s.

For large-scale broken link prospecting, you'll typically combine three capabilities:

1. Crawling websites to find broken outbound links.
2. Finding authoritative websites in your niche.
3. Managing and prioritizing prospects for outreach.

A stack that many SEO professionals use looks like this:

My favorite workflow (most scalable)

Method 1: Dead resource pages (high success rate)

Search for resource pages like:

• intitle:resources "your keyword"
• inurl:links your keyword
• intitle:recommended sites
• your keyword useful links

Crawl the resulting domains with Screaming Frog configured to:

• Check external links
• Report 404s
• Export broken external links

This quickly surfaces pages linking to dead resources.

Method 2: Find dead pages with backlinks (my preferred approach)

Instead of crawling millions of pages, start with backlink data.

Using Ahrefs:

1. Search your competitors.
2. Go to Best by links.
3. Filter:
   • HTTP code = 404
   • Referring domains > 20 (or another threshold)
4. Export.

Now you have dead pages that dozens or hundreds of websites still link to.

This is often far more efficient than crawling random authority sites.

Method 3: Crawl authority sites directly

If you have a list of authority sites:

• Crawl with Screaming Frog.
• Enable:
  • Check external links
  • Store HTML
  • Follow external links (if appropriate)
• Export:
  • Broken external links
  • Source page
  • Anchor text

You now know exactly which page contains each broken link.

Method 4: Build a custom crawler

For very large campaigns, many SEO teams build a pipeline like:

Google search →
URL collection →
Crawler →
Broken link detection →
Page authority scoring →
Email finder →
CRM

Typical tools:

• n8n
• Python (Scrapy or Playwright)
• PostgreSQL
• OpenAI (for content categorization and prioritization)

This approach scales to hundreds of thousands of pages.

Prioritize prospects

Don't chase every broken link. Score opportunities using factors like:

• Domain authority or equivalent authority metrics
• Topical relevance
• Number of organic visitors
• Whether the linking page is indexed
• Number of outbound links on the page
• Whether the dead resource closely matches content you can provide

A site with moderate authority but a highly relevant, curated resource page is often a better target than a very large site with low editorial attention.

At enterprise scale

If you're looking at 100,000+ pages, a common workflow is:

• Use Ahrefs or Semrush to identify authoritative sites and dead pages with backlinks.
• Feed target URLs into Screaming Frog (or a cloud crawler) in List Mode.
• Store results in a database.
• Use automation (such as n8n) to enrich each opportunity with authority metrics, contact information, and topical relevance.
• Push qualified prospects into an outreach platform like BuzzStream.

This avoids wasting time crawling the entire web and focuses effort on pages that are both authoritative and likely to yield valuable link opportunities.

If your goal is millions of prospects across an entire niche, the workflow changes significantly from desktop SEO tools to distributed crawling and automation. I can outline an enterprise-scale pipeline that agencies use for that level of prospecting.

If you're doing broken link building at scale, you'll typically need four capabilities:

1. Find authoritative sites in your niche.
2. Crawl those sites for broken outbound links.
3. Evaluate the SEO value of the opportunities.
4. Manage outreach.

The best stack depends on your budget and technical comfort.

Budget-friendly workflow

If you're a solo SEO:

• Ahrefs
• Screaming Frog
• Hunter
• BuzzStream

This combination covers nearly everything without paying for multiple enterprise platforms.

Agency-scale workflow

For hundreds or thousands of prospects each month:

• Ahrefs for prospect discovery
• Screaming Frog in List Mode
• Custom exports into Google Sheets or Airtable
• BuzzStream or Pitchbox for outreach
• Optional Python automation for filtering and prioritization

How to find opportunities

A common workflow looks like this:

1. Find resource pages in your niche.
   • intitle:resources
   • inurl:links
   • helpful links
   • recommended websites

2. Export candidate domains.

3. Crawl them with Screaming Frog.

4. Filter:
   • External links
   • 404 responses
   • High-authority domains
   • Relevant topic

5. Verify the dead page with the Internet Archive to understand what the original content covered.

6. Create or improve a replacement resource.

7. Reach out to the webmaster explaining the broken link and suggesting your relevant replacement.

At very large scale (10,000+ sites)

Many advanced SEOs automate the process:

• Crawl target domains with Screaming Frog CLI or custom crawlers.
• Use APIs from Ahrefs or Semrush to enrich results with authority metrics.
• Check HTTP status codes programmatically.
• Query the Wayback Machine API to identify the original content.
• Store opportunities in a database (PostgreSQL, BigQuery, or Airtable).
• Prioritize by domain authority, topical relevance, and estimated traffic.
• Feed prioritized prospects into an outreach platform.

This approach can uncover tens of thousands of broken-link opportunities while allowing you to focus outreach on the highest-value prospects.

My recommended stack by budget

• Under $100/month: Screaming Frog + Hunter + Google Search operators.
• $200–500/month: Ahrefs + Screaming Frog + BuzzStream.
• Agency/enterprise: Ahrefs + Screaming Frog + BuzzStream or Pitchbox, with custom automation for crawling, filtering, and outreach.

One additional tactic that often produces higher-quality opportunities than generic resource pages is competitor broken backlink analysis. Using Ahrefs, you can identify pages on competitors' sites that have gone offline but still attract backlinks, then create a superior replacement and reach out to the sites linking to the dead page. This tends to yield more relevant and authoritative prospects than searching for random broken external links.

For large-scale broken link prospecting, you'll usually want a stack rather than a single tool. The best workflow combines a crawler, a backlink database, and automation.

Here's a practical setup depending on your budget.

The highest-ROI workflow

This approach scales well into tens of thousands of prospects.

1. Build a list of authority domains in your niche.
   • Competitors
   • Industry associations
   • Universities
   • Government resources
   • Resource pages
   • Blogs

2. Crawl them with Screaming Frog.
   • Find:
     • 404 pages
     • Broken outbound links
     • Resource pages

3. Take every discovered 404 URL and check it in Ahrefs.
   • How many referring domains?
   • How many backlinks?
   • Is it still getting links?

If a dead page has 100+ referring domains, it's often a valuable opportunity.

4. Export all pages linking to that dead URL.

5. Qualify prospects.
   • DR/DA
   • Traffic
   • Relevance
   • Language
   • Spam score

6. Reach out suggesting your replacement content.

An even faster method (my favorite)

In Ahrefs:

Best by Links
→ filter

• HTTP Code = 404
• Referring Domains > 20
• Language = English
• One niche at a time

This immediately surfaces dead pages that still have lots of links.

Then:

Dead page
↓
Backlinks report
↓
Every linking page becomes an outreach prospect.

This often uncovers opportunities in minutes instead of crawling hundreds of sites yourself.

At enterprise scale

People doing broken link building across hundreds of thousands of pages often automate the process:

Seed domains
      ↓
Crawler
      ↓
Extract every 404
      ↓
Check backlinks via Ahrefs API
      ↓
Filter by referring domains
      ↓
Find contact emails
      ↓
Push to Airtable/Sheets
      ↓
Outreach sequence

This lets you continuously discover new opportunities instead of running one-off campaigns.

Useful Google search operators

To find pages likely to contain external links:

intitle:resources "keyword"

inurl:links keyword

"helpful links" keyword

"recommended resources" keyword

site:.edu keyword resources

site:.gov keyword resources

"useful websites" keyword

Resource pages tend to accumulate outbound links over time, making them good candidates for broken link opportunities.

If you have a larger budget

A stack like this covers most needs:

• Ahrefs for backlink discovery and prioritization
• Screaming Frog SEO Spider for crawling
• Hunter.io for contact discovery
• Airtable for managing prospects
• n8n for automation

If you're comfortable with Python, you can automate much of this using the Ahrefs API (or another backlink provider's API), a crawler like crawl4ai or Scrapy, and an orchestration tool such as n8n. That combination can scale to processing hundreds of thousands of URLs with relatively little manual work.

For large-scale broken link building, the most effective workflow is usually a combination of a backlink database, a crawler, and automation. No single tool does everything well.

Here's a stack that scales from hundreds to hundreds of thousands of prospects.