ChatGPT answers, unedited

Yes. For reproducing crawler findings locally, I’d use a real browser + a raw HTTP client, because they expose different failure modes.

Best tools

Playwright is probably the best starting point. Its network API exposes request/response events, failed requests, and explicitly links redirecting requests through redirectedFrom() / redirectedTo(), so you can reconstruct an entire redirect chain programmatically. playwright.devplaywright.dev

For example, a minimal diagnostic harness can log every request:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });
  const page = await browser.newPage();

  page.on('request', req =>
    console.log('>>', req.method(), req.url())
  );

  page.on('response', res =>
    console.log('<<', res.status(), res.url())
  );

  page.on('requestfailed', req =>
    console.log('XX', req.url(), req.failure()?.errorText)
  );

  const response = await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log('final:', response?.status(), response?.url());

  await browser.close();
})();

For a redirect specifically, walk backward from the final request:

let req = response?.request();

while (req) {
  console.log(req.url());
  req = req.redirectedFrom();
}

That is useful for distinguishing something like:

http://example.com/foo
  ↓ 301
https://example.com/foo
  ↓ 302
https://www.example.com/foo
  ↓ 200

from a crawler finding a broken final resource after JavaScript executes.

Chrome DevTools for manual investigation

Open Network, enable Preserve log, then navigate to the problematic URL. DevTools records requests while it is open, and the Network panel exposes status, initiator, timing, response headers/body, etc. developer.chrome.comdeveloper.chrome.com

The particularly useful fields are:

• Status — 3xx/4xx/5xx and browser-level failures
• Initiator — what caused the request
• Headers — especially Location, caching, CORS, content type
• Response — what the server actually returned
• Timing — where a request stalled
• Initiator chain — useful for finding which script/resource caused it

DevTools also lets you block individual requests, which is handy for reproducing "works without X, breaks with X" crawler findings. developer.chrome.comdeveloper.chrome.com

curl as the control experiment

Before blaming JavaScript, run the URL through curl. This tells you what the HTTP layer itself does:

curl -I https://example.com/foo

Then follow redirects:

curl -IL --max-redirs 10 https://example.com/foo

curl can cap the number of redirects with --max-redirs, which is useful for exposing loops. curl.se

This gives you a useful diagnostic split:

curl says 200
        ↓
Playwright says 404 after JS
        → client-side / rendered-link problem

curl says 301 → 302 → 404
        ↓
Playwright agrees
        → server-side redirect-chain problem

curl says 200
Playwright says requestfailed
        → browser/network/CORS/TLS/etc. issue

If the crawler is sophisticated

I'd reproduce it in three passes:

1. curl — establish the raw HTTP redirect/status behavior.
2. Playwright — render JavaScript and capture every request/response.
3. Chrome DevTools — interactively inspect the exact problematic request and its initiator.

If you need to reproduce unusual crawler behavior—custom headers, blocked resources, modified responses, authentication, or service-worker effects—Playwright can intercept and modify browser traffic as well. Its documentation specifically notes that service workers can hide requests from normal routing interception, and provides a serviceWorkers: 'block' option for that situation. playwright.devplaywright.dev

For a crawler-vs-browser discrepancy, Playwright + curl is the combination I'd reach for first.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. Redirect Radar Mozilla Add-ons

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling TechRadar

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. TechRadar

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. Redirect Radar Mozilla Add-ons

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling TechRadar

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. TechRadar

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what a crawler saw, it's useful to separate two questions:

1. What URL did the crawler actually request, and what redirects occurred?
2. What did the page look like after JavaScript executed?

Different tools excel at different parts of that workflow.

1. Inspect redirect chains

With curl you can follow redirects while seeing each step:

curl -IL https://example.com/page

or more verbosely:

curl -Lv https://example.com/page

This helps answer questions like:

• Was it a 301 or 302?
• Did it bounce between HTTP and HTTPS?
• Was there an unexpected redirect loop?

2. Simulate JavaScript rendering

If the crawler rendered JavaScript (as Googlebot and many SEO crawlers do), use Playwright or Puppeteer.

A minimal Playwright example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  page.on("response", response => {
    console.log(response.status(), response.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(page.url());

  await browser.close();
})();

This lets you:

• wait for SPA routing
• inspect final URLs
• capture failed requests
• see dynamically injected links

3. Inspect every redirect visually

Chrome DevTools' Network tab is one of the best debugging tools.

Enable:

• Preserve log
• Disable cache (optional)

Then reload the page.

You can inspect:

• each redirect
• response headers
• initiator
• timing
• failed resources
• JavaScript-triggered navigations

4. Reproduce crawler behavior

If you're comparing against an SEO crawler:

• Enable JavaScript rendering.
• Use a desktop Chromium user agent first.
• Compare rendered HTML (not just the initial response).
• Check whether links are injected after hydration.
• Look for client-side redirects (window.location, router navigation).

Many "broken links" are actually caused by:

• relative URLs resolving differently
• client-side routing
• hydration failures
• conditional rendering
• redirects depending on cookies or authentication

5. Capture the full redirect chain programmatically

Playwright exposes navigation requests:

const response = await page.goto(url);

let req = response.request();

while (req.redirectedFrom()) {
  req = req.redirectedFrom();
}

console.log("Original request:", req.url());
console.log("Final URL:", page.url());

You can also log every response:

page.on("response", r => {
  console.log(r.status(), r.url());
});

6. Compare rendered vs. raw HTML

Sometimes the crawler reports links that don't exist in the server-rendered HTML because JavaScript added them.

Compare:

curl https://example.com

against the DOM after rendering:

const html = await page.content();
console.log(html);

If the rendered DOM differs substantially, the issue is likely in client-side code rather than the initial HTTP response.

A practical debugging workflow

1. Verify the reported URL with curl -Lv to inspect the HTTP redirect chain.
2. Open the page in Chrome with DevTools → Network (Preserve log).
3. Repeat the navigation in Playwright using waitUntil: "networkidle".
4. Compare the raw HTML (curl) with the rendered DOM (page.content()).
5. If the crawler found links that you can't reproduce manually, check for JavaScript-generated links, SPA route changes, service workers, or environment-specific behavior (cookies, authentication, locale, or feature flags).

That combination typically provides enough visibility to determine whether a reported broken link originated from the server's responses, a redirect sequence, or JavaScript-rendered content.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If your goal is to reproduce what a crawler saw, you generally want a combination of a browser that executes JavaScript, HTTP debugging tools, and a crawler that exposes redirect and rendering details.

Here's a practical toolkit:

1. Reproduce the redirect chain

For a single URL:

curl -I https://example.com/page

To follow redirects:

curl -L -v https://example.com/page

The verbose output shows:

• every request
• every response
• each Location header
• final status code

If you want each hop separately:

curl --max-redirs 0 -I https://example.com/page

Then manually request the URL in the Location header and repeat.

2. Compare pre-render vs post-render HTML

Some crawlers execute JavaScript while others don't.

With Playwright:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

This lets you inspect:

• injected links
• SPA routing
• dynamically generated URLs
• missing anchors

3. Watch the network requests

Chrome DevTools → Network tab

Enable:

• Preserve log
• Disable cache

Reload the page.

You'll see:

• redirects
• failed requests
• JS bundles
• API calls
• images
• CSS
• 404s

Clicking any request shows:

• Request URL
• Response headers
• Redirect source
• Timing
• Initiator

4. Capture every response with Playwright

page.on("response", async response => {
  console.log(
    response.status(),
    response.url()
  );
});

Or capture redirects:

page.on("requestfinished", request => {
  console.log(request.url());
});

5. Compare crawler behavior

SEO crawlers often have settings for:

• JavaScript rendering
• User-Agent
• robots.txt handling
• canonical handling
• redirect limits

If your production crawler reports a broken link, verify that your local crawl matches those settings. A discrepancy in rendering or user agent can lead to different results.

6. Look for common causes

Typical reasons a crawler reports a broken link include:

• Relative URLs resolving differently than expected.
• JavaScript generating malformed URLs.
• Redirect loops.
• Mixed HTTP/HTTPS redirects.
• Incorrect <base> tags.
• URLs requiring cookies or authentication.
• Soft 404 pages that return HTTP 200.
• Client-side routing that depends on browser state.

7. Use HAR files for comparison

Export a HAR from Chrome DevTools or record one with Playwright:

const context = await browser.newContext({
  recordHar: {
    path: "session.har"
  }
});

A HAR contains the full request/response sequence, making it much easier to compare what your browser observed against what a crawler reported.

A solid workflow is:

1. Verify the URL with curl -L -v.
2. Open it in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright (waitUntil: "networkidle").
4. Compare the raw HTML to the rendered DOM.
5. If the issue only appears in the crawler, align its rendering mode, user agent, and crawl settings with your local reproduction. This usually narrows down whether the problem is in server-side redirects, client-side rendering, or crawler configuration.

If you're trying to reproduce what a web crawler found, the most useful approach is to combine a browser automation tool (to execute JavaScript) with network inspection tools (to see every request, redirect, and response). Different crawlers render pages differently, so matching the crawler's rendering mode is important.

Here's a comparison of commonly used options:

Simulating JavaScript rendering

For modern sites, Playwright is one of the best choices because it uses real browser engines.

Typical workflow:

1. Open the page.
2. Wait for network activity or a specific DOM element.
3. Record every request and response.
4. Check the final DOM after scripts execute.
5. Compare that with what your crawler reported.

Playwright lets you inspect:

• HTTP status codes
• redirect URLs
• failed requests
• console errors
• JavaScript exceptions
• final rendered HTML

Inspecting redirect chains

Redirects often differ depending on:

• HTTP vs HTTPS
• trailing slash normalization
• locale detection
• authentication
• cookies
• user agent
• JavaScript navigation

A few useful approaches:

curl

curl -IL https://example.com/page

or

curl -Lv https://example.com/page

This clearly shows every 301/302/307/308 hop.

Chrome DevTools

In the Network panel:

• Preserve log
• Disable cache
• Reload
• Click the document request

The request details show:

• each redirect
• response headers
• Location headers
• timing
• initiator

Capturing every network request

Playwright and Puppeteer can log:

• outgoing requests
• responses
• failures
• redirects

This is useful when a crawler reports a broken asset that only appears after JavaScript executes.

Comparing with crawler behavior

Many crawler discrepancies come from environmental differences. Check whether your crawler:

• executes JavaScript
• waits for network idle or only the initial HTML
• uses a desktop or mobile user agent
• blocks cookies
• ignores robots directives during testing
• limits render time
• blocks third-party resources

Matching these settings locally can make reproduction much more reliable.

Common causes of "crawler found broken link, browser looks fine"

• JavaScript inserts an incorrect URL after page load.
• A redirect loop only affects certain user agents.
• Relative URLs resolve differently because of a malformed <base> tag.
• A CDN returns different responses depending on headers.
• An API request fails, preventing links from rendering.
• Hydration replaces server-rendered links with incorrect client-side routes.

A practical debugging workflow

1. Verify the URL with curl to inspect the raw redirect chain.
2. Load the page in Chrome DevTools with "Preserve log" enabled.
3. Repeat using Playwright while logging every request and response.
4. Compare the final rendered DOM with the crawler's reported HTML or extracted links.
5. If the crawler supports exporting crawl data (such as the referring page and discovered URL), use that information to trace exactly where the broken link originated.

Using these tools together gives you visibility into both the server-side redirect behavior and the client-side JavaScript execution that often explains why an automated crawler reports a broken link that isn't obvious during a normal browser visit.