ChatGPT answers, unedited

For reproducing crawler broken-link findings locally, I’d use a two-layer setup: a real browser for JS-rendered behavior, plus an HTTP client for isolating redirects.

• playwright.dev is the strongest all-around choice. It runs Chromium/Firefox/WebKit, executes JS, lets you change user-agent and other browser conditions, and exposes request, response, requestfailed, redirectedFrom() and redirectedTo(). That makes it particularly useful for determining whether a crawler's “broken link” exists in the original HTML, appears after JS rendering, or fails only after navigation. Playwright Playwright Playwright
• Chrome DevTools → Network is best for interactive investigation. Turn on Preserve log, reload, then inspect the document requests and their Location headers/statuses. Exporting a HAR gives you a reproducible record of the network sequence; Chrome's DevTools APIs represent network requests in HAR format as well. Chrome for Developers
• curl.se is ideal for separating server-side redirects from browser/JS behavior. For example, curl -v exposes request/response headers, while --trace/--trace-ascii gives considerably more debugging information. Curl Curl

A useful workflow is therefore:

Crawler reports /old-page as broken
        ↓
curl /old-page
        ↓
inspect HTTP 301/302/307/308 chain
        ↓
Playwright loads /old-page in Chromium
        ↓
inspect browser requests + JS execution
        ↓
Chrome DevTools/HAR when manual inspection is needed

For example, Playwright can log what the rendered browser actually encounters:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: false });

const context = await browser.newContext({
  userAgent: 'YOUR-CRAWLER-UA'
});

const page = await context.newPage();

page.on('request', req => {
  console.log('>>', req.method(), req.url());
});

page.on('response', res => {
  console.log('<<', res.status(), res.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

const response = await page.goto('https://example.com/problem-url', {
  waitUntil: 'load'
});

let req = response?.request();

console.log('\nRedirect chain:');

const chain = [];
while (req) {
  chain.unshift(req);
  req = req.redirectedFrom();
}

for (const r of chain) {
  const res = await r.response();
  console.log(res?.status(), r.url());
}

await page.waitForTimeout(3000);
await browser.close();

Playwright explicitly links redirected requests through redirectedFrom()/redirectedTo(), so you can reconstruct a multi-hop server redirect rather than seeing only the final navigation. Also note that a 404/503 generates a normal HTTP response in Playwright; requestfailed is for failures where the browser couldn't successfully obtain an HTTP response, such as certain network errors. Playwright

One particularly useful diagnostic is to run the same URL twice: JS enabled and JS disabled. Playwright supports javaScriptEnabled: false. If the crawler's finding only reproduces with JS enabled, you're probably dealing with client-side routing/navigation or dynamically generated links rather than a straightforward HTTP broken link. Playwright

If you're trying to match a specific crawler closely, also reproduce its user-agent, cookies/auth, viewport, HTTP method, headers, robots behavior, JS wait time, and redirect limit. Differences there are a common reason why curl says “200 OK” while an SEO crawler reports a broken URL. Playwright's request API can even set maxRedirects: 0, which is handy when you want to examine each redirect hop independently instead of automatically following the chain. Playwright

For reproducing crawler broken-link findings locally, I’d use a two-layer setup: a real browser for JS-rendered behavior, plus an HTTP client for isolating redirects.

• playwright.dev is the strongest all-around choice. It runs Chromium/Firefox/WebKit, executes JS, lets you change user-agent and other browser conditions, and exposes request, response, requestfailed, redirectedFrom() and redirectedTo(). That makes it particularly useful for determining whether a crawler's “broken link” exists in the original HTML, appears after JS rendering, or fails only after navigation. Playwright Playwright Playwright
• Chrome DevTools → Network is best for interactive investigation. Turn on Preserve log, reload, then inspect the document requests and their Location headers/statuses. Exporting a HAR gives you a reproducible record of the network sequence; Chrome's DevTools APIs represent network requests in HAR format as well. Chrome for Developers
• curl.se is ideal for separating server-side redirects from browser/JS behavior. For example, curl -v exposes request/response headers, while --trace/--trace-ascii gives considerably more debugging information. Curl Curl

A useful workflow is therefore:

Crawler reports /old-page as broken
        ↓
curl /old-page
        ↓
inspect HTTP 301/302/307/308 chain
        ↓
Playwright loads /old-page in Chromium
        ↓
inspect browser requests + JS execution
        ↓
Chrome DevTools/HAR when manual inspection is needed

For example, Playwright can log what the rendered browser actually encounters:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: false });

const context = await browser.newContext({
  userAgent: 'YOUR-CRAWLER-UA'
});

const page = await context.newPage();

page.on('request', req => {
  console.log('>>', req.method(), req.url());
});

page.on('response', res => {
  console.log('<<', res.status(), res.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

const response = await page.goto('https://example.com/problem-url', {
  waitUntil: 'load'
});

let req = response?.request();

console.log('\nRedirect chain:');

const chain = [];
while (req) {
  chain.unshift(req);
  req = req.redirectedFrom();
}

for (const r of chain) {
  const res = await r.response();
  console.log(res?.status(), r.url());
}

await page.waitForTimeout(3000);
await browser.close();

Playwright explicitly links redirected requests through redirectedFrom()/redirectedTo(), so you can reconstruct a multi-hop server redirect rather than seeing only the final navigation. Also note that a 404/503 generates a normal HTTP response in Playwright; requestfailed is for failures where the browser couldn't successfully obtain an HTTP response, such as certain network errors. Playwright

One particularly useful diagnostic is to run the same URL twice: JS enabled and JS disabled. Playwright supports javaScriptEnabled: false. If the crawler's finding only reproduces with JS enabled, you're probably dealing with client-side routing/navigation or dynamically generated links rather than a straightforward HTTP broken link. Playwright

If you're trying to match a specific crawler closely, also reproduce its user-agent, cookies/auth, viewport, HTTP method, headers, robots behavior, JS wait time, and redirect limit. Differences there are a common reason why curl says “200 OK” while an SEO crawler reports a broken URL. Playwright's request API can even set maxRedirects: 0, which is handy when you want to examine each redirect hop independently instead of automatically following the chain. Playwright

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• Playwright — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. playwright.dev

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. github.com

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. developer.chrome.com

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing crawler broken-link reports locally, I’d use two layers: a real browser for JavaScript behavior, and an HTTP client for exact redirect/header inspection.

• playwright.dev — best overall choice. It runs Chromium/Firefox/WebKit, executes client-side JS, and exposes request/response events. Importantly, Playwright links redirects via redirectedFrom() / redirectedTo(), so you can reconstruct the entire server-side redirect chain programmatically. Playwright

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

page.on('response', async response => {
  const req = response.request();
  console.log(response.status(), req.method(), response.url());
});

page.on('requestfailed', req => {
  console.log('FAILED', req.url(), req.failure());
});

await page.goto('https://example.com', {
  waitUntil: 'networkidle'
});

await browser.close();

This is especially useful when the crawler says a link is broken but curl says it's fine—the URL may be generated, modified, or requested only after JavaScript runs.

• curl.se — best for dissecting HTTP redirects. Start without following redirects so you can inspect each hop manually:

curl -sv -o /dev/null https://example.com/old-url

Look for:

< HTTP/2 301
< location: /new-url

Then request that Location yourself. Once you've understood the individual hops, compare with automatic following:

curl -Lsv -o /dev/null https://example.com/old-url

For especially confusing cases, --trace-ascii and --trace-time expose substantially more of the client/server exchange. GitHub Curl

• Chrome DevTools Network panel — best interactive debugger. Turn on Preserve log before navigating; otherwise redirects/navigation can wipe the evidence you're trying to inspect. You can then inspect status, headers, initiators, timing, and export the session as HAR. Chrome Developers Chrome Developers Chrome Developers

A good reproduction workflow is therefore:

crawler finding
     │
     ▼
curl without -L
     │
     ├── HTTP redirect/error? ──► inspect each Location hop
     │
     ▼
Playwright/Chromium
     │
     ├── execute JS
     ├── capture navigation
     ├── capture fetch/XHR
     └── inspect final DOM links
     │
     ▼
Chrome DevTools/HAR
     │
     └── manually compare browser behavior
     ▼
crawler behavior

The big thing to match is the crawler's environment. A discrepancy may come from user-agent, cookies, authentication, JS execution, timeout, robots behavior, HTTP vs HTTPS, HEAD vs GET, geography, or whether the crawler considers a redirect chain itself an error. Also test curl -A 'crawler-user-agent' ... when you know the crawler's UA.

If I were building a repeatable local debugging harness, I'd choose Playwright + curl: Playwright for rendered-page/link discovery and browser requests, curl for surgical HTTP-level verification.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

For reproducing a crawler’s broken-link findings locally, I’d use a two-layer setup: a real browser for JS/rendering behavior, plus an HTTP-level tool for isolating redirects.

• playwright.dev — best default. It runs real Chromium/Firefox/WebKit, executes JS, lets you wait for hydration/network idle, and exposes every request/response. Crucially, redirectedFrom() / redirectedTo() let you reconstruct an entire server-side redirect chain. Playwright Playwright
• Chrome DevTools Network panel — best interactive debugger. Enable Preserve log and Disable cache, reload, then inspect Status, Location headers, Initiator, timing, and the waterfall. The Initiator column distinguishes redirects from parser- or script-triggered navigation. Chrome for Developers
• Chrome DevTools Protocol (CDP) — best when Playwright’s abstraction hides something. Network.requestWillBeSent includes redirectResponse, and the Fetch domain exposes redirectedRequestId, so you can instrument redirects almost exactly as Chromium sees them. Chrome DevTools Chrome DevTools
• curl — useful as the non-JS control case. Run something like curl -v -L URL and compare its behavior against Chromium. If curl succeeds but your browser reproduction fails, JS, cookies, service workers, client-side routing, headers, or browser-specific behavior are likely involved.
• HAR exports — useful for comparing your reproduction against crawler evidence. Playwright can record browser traffic, and DevTools can export its Network log.

A particularly useful Playwright diagnostic is:

page.on('request', req =>
  console.log('>>', req.method(), req.url())
);

page.on('response', res =>
  console.log('<<', res.status(), res.url(),
              res.headers()['location'] ?? '')
);

page.on('requestfailed', req =>
  console.log('XX', req.url(), req.failure())
);

await page.goto(url, { waitUntil: 'networkidle' });

Playwright treats an HTTP 404/503 as a received HTTP response rather than a transport-level requestfailed; that distinction matters when matching crawler “broken link” reports. Playwright

For step-by-step redirects, I’d start with curl -v/DevTools to understand the HTTP chain, then reproduce with Playwright Chromium using the crawler’s exact user-agent, cookies, viewport, locale, headers, and JS wait strategy. Also try blocking service workers if results don't match—Playwright specifically notes that service workers can make network events appear missing. Playwright

One common gotcha: distinguish HTTP redirects (301/302/303/307/308) from JS redirects (location.href, SPA routing) and <meta http-equiv="refresh">. An HTTP-only checker may report something quite different from a rendered crawler.

Yes. For reproducing a crawler’s broken-link findings, I’d use a real browser + network interception, rather than relying on curl alone.

Best local stack

• playwright.dev — best choice for simulating a JS-capable crawler. It runs Chromium/Firefox/WebKit, waits for JS execution, follows browser navigation, and lets you observe requests/responses. Its trace viewer can give you a replayable, step-by-step record of a run. GitHub
• developer.chrome.com — excellent for manually reproducing a single failure. Turn on Preserve log, reload, and you can retain requests across navigations. The Network panel exposes status, initiator, headers, timing, and response information; Redirect is explicitly shown as an initiator type. Chrome for Developers Chrome for Developers
• mitmproxy.org — useful when you want to see and manipulate the actual HTTP exchanges independently of browser DevTools. It can intercept HTTPS, save complete conversations, replay them, and script request/response modifications. mitmproxy Documentation mitmproxy Documentation

A practical reproduction workflow

I'd structure the test harness like this:

crawler URL
    │
    ▼
Playwright Chromium
    │
    ├── JS execution
    ├── DOM mutations
    ├── navigation
    └── request/response events
             │
             ▼
          mitmproxy
             │
             ├── 301/302/303/307/308
             ├── Location headers
             ├── 4xx/5xx
             ├── TLS/connection errors
             └── rewritten/mock responses

In Playwright, record every request and response, including the URL, status, Location header, resource type, and whether the request was caused by navigation or by JavaScript. Then reconstruct a chain such as:

https://site.test/old
  └─ 301 Location: /new
       └─ 302 Location: https://www.site.test/final
            └─ 200

That lets you distinguish "the crawler found a broken final URL" from "the crawler reported an intermediate redirect as broken."

For browser-side behavior, also record links after the page has rendered—not just the original HTML. This catches cases such as:

document.body.innerHTML += '<a href="/generated-page">...</a>';

or links produced after an API call.

When I'd use each tool

One particularly useful trick is to disable automatic redirect following in a separate HTTP-level test. That exposes each hop individually, whereas browser navigation normally proceeds through the chain automatically. Then compare that result with the browser run. This often reveals why a crawler and a browser disagree.

If you're trying to reproduce a specific crawler (Screaming Frog, Sitebulb, Ahrefs, Googlebot, custom crawler, etc.), the exact Playwright configuration—user agent, JavaScript timing, robots handling, redirect limits, canonical handling, and HTTP cache behavior—matters quite a bit.

To reproduce a crawler’s broken-link findings locally, you usually need to separate HTTP-level crawling from browser-rendered crawling. A lot of “broken links” only appear after JavaScript executes, redirects fire, client-side routing happens, or lazy-loaded content is injected.

Here are the tools I’d use:

1. Browser automation (JS rendering + network capture)

playwright.dev

Best general-purpose choice for reproducing modern crawler behavior.

Useful for:

• Rendering pages exactly like a browser
• Waiting for JS execution
• Capturing failed requests
• Inspecting redirects
• Running Chromium/Firefox/WebKit
• Emulating devices and user agents Playwright

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("requestfailed", req => {
  console.log("FAILED", req.url(), req.failure());
});

page.on("response", res => {
  if (res.status() >= 300) {
    console.log(res.status(), res.url());
  }
});

await page.goto("https://example.com", {
  waitUntil: "networkidle"
});

await browser.close();

For debugging, Playwright’s inspector/trace tooling lets you step through actions and inspect network activity, DOM state, and console output. Playwright

pptr.dev

Similar idea, Chrome-focused.

Good if your crawler already uses Chrome DevTools Protocol.

2. Step-by-step redirect chain inspection

Chrome DevTools Network panel

Open DevTools → Network → reload the page.

You can inspect:

• 301/302/307/308 responses
• Location headers
• request initiators
• timing
• cookies
• final resources

Chrome’s Network panel shows request details and initiator chains, which is useful for finding what caused a failing request. Chrome for Developers

Useful workflow:

1. Enable Preserve log
2. Enable Disable cache
3. Reload
4. Click the document request
5. Follow: - Headers → redirect response
6. Initiator → what triggered it
7. Timing → where delays/errors occur

Redirect Trace browser extension

A browser extension can expose redirect hops including:

• server redirects
• HSTS upgrades
• meta refresh
• JavaScript redirects
• History API navigation

Chrome Web StoreThis is closer to what SEO crawlers often report because it follows browser behavior rather than only HTTP responses.

3. CLI tools for raw HTTP redirect debugging

curl

For server-side redirect chains:

curl -ILv https://example.com/broken-link

Shows:

• every HTTP hop
• status codes
• Location headers
• TLS details

To stop automatic following:

curl -v https://example.com

Then manually inspect each redirect.

wget

wget --max-redirect=10 --server-response --spider https://example.com

Useful for bulk checking.

4. Crawl locally like an SEO crawler

screamingfrog.co.uk

Good for comparing against commercial crawler reports.

Features:

• broken internal/external links
• redirect chains
• response codes
• JavaScript rendering mode
• crawl exports

crawler.siteone.io

Open-source option.

Useful for:

• recursive crawling
• broken links
• performance checks
• running locally in CI

5. If you want to reproduce a crawler’s exact behavior

I’d build a small harness:

HTTP layer

• curl / Node fetch
• record: - URL
• status
• headers
• redirect chain

Browser layer

• Playwright
• record: - console errors
• failed requests
• JS exceptions
• final URL
• DOM links after rendering

Example data structure:

{
  "source": "https://site.com/page",
  "link": "https://site.com/missing",
  "http_chain": [
    {
      "url": "https://site.com/missing",
      "status": 301,
      "location": "/new"
    },
    {
      "url": "https://site.com/new",
      "status": 404
    }
  ],
  "browser_result": {
    "console_errors": [],
    "request_failed": true
  }
}

That usually explains crawler discrepancies:

• crawler says 404, browser sees JS redirect
• crawler misses links inserted after rendering
• crawler follows redirects differently
• authenticated/session-dependent URLs fail locally

A strong local reproduction stack would be:

Playwright + Chrome DevTools Network + curl + Screaming Frog/SiteOne for comparison.

Yes. For reproducing crawler findings locally, I’d use a real browser plus a raw HTTP client, because crawler “broken links” can arise either before rendering or only after JavaScript executes.

Best tools

• Playwright — probably the best fit for an automated reproduction harness. It runs Chromium/Firefox/WebKit, executes JavaScript, and exposes request/response events. Crucially, Playwright links redirect hops through request.redirectedFrom() / redirectedTo(), so you can reconstruct the complete server-side redirect chain programmatically. Playwright Playwright

playwright.dev

• Chrome DevTools Network panel — best for interactive, step-by-step investigation. Turn on Preserve log, reload, and inspect each request's Status, Initiator, Headers, Response, and Timing. The Initiator view is particularly useful for distinguishing a normal link, redirect, parser request, or JavaScript-triggered request. Chrome for Developers Chrome for Developers

developer.chrome.com

• curl — use this as the “crawler without a browser” baseline. curl -I/headers lets you see the individual HTTP responses, while -L follows Location redirects and --max-redirs lets you deliberately stop after N hops. curl

curl.se

A useful reproduction workflow

1. Run curl first against the suspected URL and record every HTTP hop.
2. Run Playwright against the same URL with request/response logging.
3. Compare: - HTTP status at every hop
4. Location headers
5. final URL
6. whether JavaScript generated the request
7. whether the request is a document, XHR/fetch, image, stylesheet, etc.
8. whether the browser reports an actual network failure versus an HTTP 404/500
9. In DevTools, enable Preserve log and inspect the request's Initiator chain interactively. Chrome for Developers Playwright

One important distinction: a 404 isn't a browser-level request failure. Playwright's requestfailed is for failures such as connection problems; HTTP error responses such as 404/503 still produce normal response/request-completion events. That distinction can explain why a crawler and your first reproduction appear to disagree. Playwright

For a developer building a repeatable local reproducer, Playwright + curl is the combination I'd start with: curl establishes the raw redirect behavior, while Playwright establishes what a JavaScript-capable crawler actually sees.

To reproduce a crawler’s broken-link findings locally, you usually need to mimic two different things:

1. HTTP-level crawling (status codes, 301/302/307/308, DNS/TLS failures, headers)
2. Browser-level crawling (JavaScript execution, SPA routing, meta refresh, lazy-loaded links)

A good local stack combines a few tools:

1. Headless browser: reproduce JS rendering

Microsoft Playwright

Best general-purpose choice for crawler reproduction.

Use it to:

• execute JavaScript
• wait for network idle
• inspect the final DOM after rendering
• capture navigation events
• record failed requests
• emulate Chromium/Firefox/WebKit

Example:

npm install playwright

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  page.on("requestfailed", req => {
    console.log("FAILED", req.url(), req.failure());
  });

  page.on("response", res => {
    if (res.status() >= 400) {
      console.log("HTTP ERROR", res.status(), res.url());
    }
  });

  page.on("framenavigated", frame => {
    console.log("NAV", frame.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

This catches many failures that a plain HTTP crawler misses.

Puppeteer

Similar approach, especially if you want Chrome-only automation.

Useful APIs:

• page.goto()
• page.on('response')
• page.on('requestfailed')
• page.evaluate()

2. Browser DevTools: manually inspect what happened

Chrome/Edge DevTools Network panel is still one of the best ways to compare against a crawler.

Enable:

• Preserve log
• Disable cache
• filter: status-code:404
• inspect:
  • Initiator tab → what JS caused the request
  • Timing tab → where delays occur
  • Headers → redirect locations

For redirect debugging, DevTools shows the browser’s actual behavior, including JS-triggered navigation.

3. Redirect-chain tracing

Command line

curl is the baseline:

curl -ILv https://example.com/broken-link

or:

curl -L -v https://example.com/broken-link

You can see:

HTTP/2 301
location: https://example.com/new

HTTP/2 302
location: https://example.com/login

HTTP/2 200

This reveals server-side redirects before JavaScript runs. curl -L -v is commonly used because it exposes the redirect hops and headers directly. wuxwebtools.com

For structured output:

curl -s -o /dev/null \
-w "%{http_code} %{url_effective}\n" \
-L https://example.com

4. Capture everything as a HAR

For comparing with crawler reports:

• Chrome DevTools → Network → Export HAR
• Playwright tracing:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

Then:

await context.tracing.stop({
  path: "trace.zip"
});

HAR files let you inspect:

• redirects
• failed resources
• timing
• headers
• cookies

5. Intercept traffic

mitmproxy

Useful when the crawler differs because of:

• headers
• cookies
• user-agent
• compression
• CDN behavior

Run:

mitmproxy

Then point Playwright/Chrome at the proxy.

You can inspect the exact HTTP exchange.

6. Crawl like a search bot

If the crawler is SEO-oriented, emulate:

User agents

Example:

curl -A "Mozilla/5.0" https://example.com

or:

curl -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com

Also test:

• no JS
• JS enabled
• cookies disabled
• logged-out state
• mobile viewport

Many "broken links" only appear under one of these conditions.

7. Build a local crawler harness

A practical stack:

A useful pattern:

URL list
   |
   +--> HTTP crawler
   |       |
   |       +--> status codes
   |       +--> redirect chain
   |
   +--> Playwright crawler
           |
           +--> rendered DOM links
           +--> JS navigations
           +--> failed requests

Then diff the results.

For a crawler’s broken-link report specifically, I would start with Playwright + curl + HAR capture. That combination usually reveals whether the crawler found:

• a real 404,
• a redirect chain problem,
• a JS-generated dead link,
• a SPA route failure,
• or a crawler/browser mismatch.

Yes. For reproducing crawler findings locally, I’d use a real browser + a raw HTTP client, because they expose different failure modes.

Best tools

Playwright is probably the best starting point. Its network API exposes request/response events, failed requests, and explicitly links redirecting requests through redirectedFrom() / redirectedTo(), so you can reconstruct an entire redirect chain programmatically. playwright.devplaywright.dev

For example, a minimal diagnostic harness can log every request:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });
  const page = await browser.newPage();

  page.on('request', req =>
    console.log('>>', req.method(), req.url())
  );

  page.on('response', res =>
    console.log('<<', res.status(), res.url())
  );

  page.on('requestfailed', req =>
    console.log('XX', req.url(), req.failure()?.errorText)
  );

  const response = await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log('final:', response?.status(), response?.url());

  await browser.close();
})();

For a redirect specifically, walk backward from the final request:

let req = response?.request();

while (req) {
  console.log(req.url());
  req = req.redirectedFrom();
}

That is useful for distinguishing something like:

http://example.com/foo
  ↓ 301
https://example.com/foo
  ↓ 302
https://www.example.com/foo
  ↓ 200

from a crawler finding a broken final resource after JavaScript executes.

Chrome DevTools for manual investigation

Open Network, enable Preserve log, then navigate to the problematic URL. DevTools records requests while it is open, and the Network panel exposes status, initiator, timing, response headers/body, etc. developer.chrome.comdeveloper.chrome.com

The particularly useful fields are:

• Status — 3xx/4xx/5xx and browser-level failures
• Initiator — what caused the request
• Headers — especially Location, caching, CORS, content type
• Response — what the server actually returned
• Timing — where a request stalled
• Initiator chain — useful for finding which script/resource caused it

DevTools also lets you block individual requests, which is handy for reproducing "works without X, breaks with X" crawler findings. developer.chrome.comdeveloper.chrome.com

curl as the control experiment

Before blaming JavaScript, run the URL through curl. This tells you what the HTTP layer itself does:

curl -I https://example.com/foo

Then follow redirects:

curl -IL --max-redirs 10 https://example.com/foo

curl can cap the number of redirects with --max-redirs, which is useful for exposing loops. curl.se

This gives you a useful diagnostic split:

curl says 200
        ↓
Playwright says 404 after JS
        → client-side / rendered-link problem

curl says 301 → 302 → 404
        ↓
Playwright agrees
        → server-side redirect-chain problem

curl says 200
Playwright says requestfailed
        → browser/network/CORS/TLS/etc. issue

If the crawler is sophisticated

I'd reproduce it in three passes:

1. curl — establish the raw HTTP redirect/status behavior.
2. Playwright — render JavaScript and capture every request/response.
3. Chrome DevTools — interactively inspect the exact problematic request and its initiator.

If you need to reproduce unusual crawler behavior—custom headers, blocked resources, modified responses, authentication, or service-worker effects—Playwright can intercept and modify browser traffic as well. Its documentation specifically notes that service workers can hide requests from normal routing interception, and provides a serviceWorkers: 'block' option for that situation. playwright.devplaywright.dev

For a crawler-vs-browser discrepancy, Playwright + curl is the combination I'd reach for first.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. Redirect Radar Mozilla Add-ons

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling TechRadar

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. TechRadar

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. Redirect Radar Mozilla Add-ons

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling TechRadar

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. TechRadar

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce exactly why a crawler reported a broken link, it's helpful to separate the problem into three layers:

1. HTTP behavior (status codes, redirect chains, headers)
2. Browser rendering (JavaScript execution, DOM changes)
3. Crawler behavior (what the crawler extracted versus what a browser sees)

Here's a workflow that works well.

1. Playwright (best overall for JS-rendered sites)

Playwright gives you a headless (or headed) Chromium browser that behaves much like modern crawlers that render JavaScript.

Useful capabilities:

• Wait for network idle before inspecting the page.
• Record every request and response.
• Capture console errors and failed network requests.
• Compare the initial HTML to the final rendered DOM.

Example:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

page.on("response", r => {
  console.log(r.status(), r.url());
});

page.on("requestfailed", r => {
  console.log("FAILED:", r.url(), r.failure());
});

await page.goto(url, {
  waitUntil: "networkidle"
});

console.log(await page.content());

This is usually my first choice for reproducing crawler findings.

2. Chrome DevTools

If the crawler reports a broken link from a page:

• Network tab
• Preserve log
• Disable cache
• Reload
• Click the suspected link

You'll see:

• every redirect
• timing
• headers
• cookies
• failed requests
• JS-triggered navigations

The Initiator column is especially useful because it shows whether navigation came from HTML, JavaScript, or another resource.

3. Redirect-chain inspectors

For debugging redirect hops one at a time, browser extensions such as Redirect Radar or Redirect Inspector show each 301/302 (and many JavaScript or meta-refresh redirects), along with headers, timing, cookies, and export options like cURL or HAR. redirect-radar.com

These are much easier than manually following redirects in DevTools when you're investigating dozens of URLs.

4. curl (to reproduce crawler HTTP behavior)

Many "broken links" are actually redirect or server issues.

curl -IL https://example.com/page

or

curl -v https://example.com/page

Helpful flags:

curl -L      # follow redirects
curl -I      # headers only
curl -v      # verbose
curl --max-redirs 20

This reproduces what a non-rendering crawler would see before JavaScript executes.

5. Puppeteer

Very similar to Playwright.

Advantages:

• huge ecosystem
• Chrome DevTools Protocol access
• request interception

Good if your existing tooling already uses Puppeteer.

6. Screaming Frog SEO Spider

If the original crawler is SEO-oriented, Screaming Frog is excellent because it can:

• crawl with JavaScript rendering enabled
• report broken links
• show redirect chains
• export the exact path from source page → broken URL
• compare rendered vs non-rendered crawling www.techradar.com

This often reproduces findings from commercial SEO crawlers surprisingly well.

7. Browser automation with HAR recording

Playwright can save a HAR file:

const browser = await chromium.launch();

const context = await browser.newContext({
  recordHar: {
    path: "crawl.har"
  }
});

You can then inspect:

• redirect sequence
• response headers
• timing
• failed resources

This is great for sharing reproducible bug reports.

8. Diff raw HTML vs rendered DOM

Many "broken links" only appear after hydration.

Compare:

await page.goto(url);

const rendered = await page.content();

against:

curl https://example.com

If the bad URL exists only after rendering, the issue is likely in client-side code rather than the server response.

A practical debugging workflow

1. Reproduce the URL in Playwright with JS enabled.
2. Log every request/response and any failed requests.
3. Save a HAR file.
4. Use curl -IL to verify the HTTP redirect chain independently.
5. Open Chrome DevTools with Preserve log enabled to inspect the exact navigation.
6. Compare the initial HTML (curl) with the rendered DOM (page.content()).
7. If reproducing an SEO crawler's perspective, run a JavaScript-enabled crawl in Screaming Frog and compare its findings with your browser session. www.techradar.com

This combination gives you visibility into both the server-side behavior (redirects, status codes) and the client-side behavior (JavaScript-generated links, hydration, and SPA routing), which are the two most common sources of discrepancies between crawler reports and what you observe manually.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a cloud crawler) found, it's useful to separate the problem into three layers:

1. Does the server respond correctly? (redirects, status codes, headers)
2. Does the browser render the page and create links with JavaScript?
3. Does the crawler discover those links the same way?

Here's a toolkit that covers those layers.

1. Inspect redirects step by step

For a single URL, curl is often the quickest.

See only headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

See every request and response:

curl -Lv https://example.com/page

This lets you verify things like:

• HTTP → HTTPS
• www → non-www
• trailing slash redirects
• unexpected 302s
• redirect loops

2. Simulate JavaScript rendering

If the crawler reports links that only exist after JavaScript executes, a headless browser is ideal.

Playwright is an excellent choice because it closely matches Chromium.

Example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on("response", r => {
    console.log(r.status(), r.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

You can then inspect:

• rendered HTML
• dynamically inserted links
• lazy-loaded navigation
• client-side redirects

3. Watch every network request

Chrome DevTools' Network tab is invaluable.

Enable:

• Preserve log
• Disable cache

Then reload.

You'll see:

• redirect chain
• failed resources
• JS-generated requests
• document requests
• timing
• response headers

Clicking an individual request exposes:

• Initiator
• Headers
• Preview
• Timing

This often reveals why a crawler encountered a broken link.

4. Compare rendered vs. raw HTML

Sometimes the crawler is finding links because JS injected them.

Compare:

Raw HTML:

curl https://example.com/page

Rendered HTML:

await page.content()

Differences often explain crawler discrepancies.

5. Export all links after rendering

In Playwright:

const links = await page.$$eval("a", as =>
  as.map(a => ({
    text: a.textContent.trim(),
    href: a.href
  }))
);

console.log(links);

This lets you compare against the crawler's discovered URLs.

6. Record a trace

Playwright can capture a complete execution trace:

await context.tracing.start({
  screenshots: true,
  snapshots: true
});

// reproduce

await context.tracing.stop({
  path: "trace.zip"
});

Viewing the trace shows:

• DOM state
• network timeline
• console output
• screenshots
• redirects

This is extremely helpful for intermittent issues.

7. Run an SEO crawler locally

If you want behavior that's closer to an SEO audit than a browser:

• Screaming Frog SEO Spider — configurable JavaScript rendering, response-code reports, redirect chains, and exportable crawl data.
• Sitebulb — similar capabilities with strong visualizations and rendering diagnostics.

You can compare their output directly against your production crawler.

A practical debugging workflow

1. Confirm the URL's HTTP behavior with curl -Lv.
2. Open the same URL in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright using waitUntil: "networkidle".
4. Compare the rendered DOM to the raw HTML.
5. Extract rendered links and verify which ones return non-2xx responses.
6. If the issue is still unclear, capture a Playwright trace and inspect the request sequence.

This workflow usually distinguishes whether the "broken link" is caused by:

• an HTTP redirect or status code,
• JavaScript-generated navigation,
• a race condition during rendering,
• or a crawler-specific behavior (such as rendering timeouts or robots handling).

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If you're trying to reproduce what a crawler saw, it's useful to separate two questions:

1. What URL did the crawler actually request, and what redirects occurred?
2. What did the page look like after JavaScript executed?

Different tools excel at different parts of that workflow.

1. Inspect redirect chains

With curl you can follow redirects while seeing each step:

curl -IL https://example.com/page

or more verbosely:

curl -Lv https://example.com/page

This helps answer questions like:

• Was it a 301 or 302?
• Did it bounce between HTTP and HTTPS?
• Was there an unexpected redirect loop?

2. Simulate JavaScript rendering

If the crawler rendered JavaScript (as Googlebot and many SEO crawlers do), use Playwright or Puppeteer.

A minimal Playwright example:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  page.on("response", response => {
    console.log(response.status(), response.url());
  });

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(page.url());

  await browser.close();
})();

This lets you:

• wait for SPA routing
• inspect final URLs
• capture failed requests
• see dynamically injected links

3. Inspect every redirect visually

Chrome DevTools' Network tab is one of the best debugging tools.

Enable:

• Preserve log
• Disable cache (optional)

Then reload the page.

You can inspect:

• each redirect
• response headers
• initiator
• timing
• failed resources
• JavaScript-triggered navigations

4. Reproduce crawler behavior

If you're comparing against an SEO crawler:

• Enable JavaScript rendering.
• Use a desktop Chromium user agent first.
• Compare rendered HTML (not just the initial response).
• Check whether links are injected after hydration.
• Look for client-side redirects (window.location, router navigation).

Many "broken links" are actually caused by:

• relative URLs resolving differently
• client-side routing
• hydration failures
• conditional rendering
• redirects depending on cookies or authentication

5. Capture the full redirect chain programmatically

Playwright exposes navigation requests:

const response = await page.goto(url);

let req = response.request();

while (req.redirectedFrom()) {
  req = req.redirectedFrom();
}

console.log("Original request:", req.url());
console.log("Final URL:", page.url());

You can also log every response:

page.on("response", r => {
  console.log(r.status(), r.url());
});

6. Compare rendered vs. raw HTML

Sometimes the crawler reports links that don't exist in the server-rendered HTML because JavaScript added them.

Compare:

curl https://example.com

against the DOM after rendering:

const html = await page.content();
console.log(html);

If the rendered DOM differs substantially, the issue is likely in client-side code rather than the initial HTTP response.

A practical debugging workflow

1. Verify the reported URL with curl -Lv to inspect the HTTP redirect chain.
2. Open the page in Chrome with DevTools → Network (Preserve log).
3. Repeat the navigation in Playwright using waitUntil: "networkidle".
4. Compare the raw HTML (curl) with the rendered DOM (page.content()).
5. If the crawler found links that you can't reproduce manually, check for JavaScript-generated links, SPA route changes, service workers, or environment-specific behavior (cookies, authentication, locale, or feature flags).

That combination typically provides enough visibility to determine whether a reported broken link originated from the server's responses, a redirect sequence, or JavaScript-rendered content.

If you're trying to reproduce what an SEO crawler (like Screaming Frog, Sitebulb, or a search engine) is seeing, you'll usually want two capabilities:

1. A browser that executes JavaScript, so you can see the final DOM and network activity.
2. A way to inspect every HTTP redirect hop, including headers and status codes.

Here's a practical toolkit.

Browser-based tools (best for JS-rendered pages)

• Chrome or Edge DevTools
  • Open Network and reload the page.
  • Enable Preserve log so redirects remain visible.
  • Inspect each request's:
    • Status code (301, 302, 307, 308, etc.)
    • Location header
    • Response headers
    • Initiator
    • Timing
  • Use the Elements panel to compare the rendered DOM with the original HTML.

• Playwright
  • Automates a real Chromium/Firefox/WebKit browser.
  • Lets you wait for client-side rendering and capture all network requests.
  • Great for reproducing crawler behavior programmatically.

• Puppeteer
  • Similar to Playwright for Chromium.
  • Good for examining JS-rendered navigation and resource loading.

Command-line tools (best for redirect chains)

curl

Show headers:

curl -I https://example.com/page

Follow redirects:

curl -IL https://example.com/page

Show verbose redirect behavior:

curl -Lv https://example.com/page

This lets you inspect every hop in the redirect chain.

wget

wget --server-response --max-redirect=20 https://example.com/page

Useful for seeing server responses sequentially.

HTTP inspection tools

• mitmproxy
  • Intercepts browser traffic.
  • Excellent for watching every redirect and request.
  • Lets you inspect headers, cookies, caching, and HTTP/2 behavior.

• Fiddler
  • Similar capabilities with a graphical interface.

• Charles Proxy
  • Popular cross-platform HTTP debugging proxy.
  • Makes redirect chains easy to visualize.

Comparing rendered vs. raw HTML

Sometimes a crawler reports a broken link because JavaScript rewrites URLs.

Compare:

curl https://example.com/page

with what appears after rendering in Chrome DevTools (Elements panel) or Playwright.

Playwright example

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });

  const page = await browser.newPage();

  page.on('response', response => {
    console.log(
      response.status(),
      response.url()
    );
  });

  await page.goto('https://example.com', {
    waitUntil: 'networkidle'
  });

  console.log(await page.title());

  await browser.close();
})();

This runs a real browser, executes JavaScript, and logs every response received.

If you're trying to match an SEO crawler

Pay attention to differences in:

• User-Agent (Googlebot vs. Chrome vs. crawler default)
• JavaScript rendering enabled/disabled
• Robots.txt handling
• Cookie/session state
• Authentication
• Canonical URLs
• Meta refresh redirects
• Client-side redirects (e.g., window.location or SPA routing)
• HTTP vs. HTTPS
• Mobile vs. desktop rendering

A practical debugging workflow

1. Verify the URL with curl -Lv to understand the raw HTTP redirect chain.
2. Load the page in Chrome DevTools with Preserve log enabled.
3. Watch the Network panel for additional JavaScript-triggered navigations or failed resource requests.
4. If the issue only appears during rendering, reproduce it with Playwright while logging all requests and responses.
5. Compare the final rendered DOM against the original HTML to determine whether the broken link originates in the server response or is introduced by client-side code.

Using both an HTTP-level tool (curl or mitmproxy) and a real browser renderer (Chrome DevTools or Playwright) usually provides enough visibility to reproduce and explain nearly all crawler-reported broken-link issues.

If your goal is to reproduce what a crawler saw, you generally want a combination of a browser that executes JavaScript, HTTP debugging tools, and a crawler that exposes redirect and rendering details.

Here's a practical toolkit:

1. Reproduce the redirect chain

For a single URL:

curl -I https://example.com/page

To follow redirects:

curl -L -v https://example.com/page

The verbose output shows:

• every request
• every response
• each Location header
• final status code

If you want each hop separately:

curl --max-redirs 0 -I https://example.com/page

Then manually request the URL in the Location header and repeat.

2. Compare pre-render vs post-render HTML

Some crawlers execute JavaScript while others don't.

With Playwright:

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto("https://example.com", {
    waitUntil: "networkidle"
  });

  console.log(await page.content());

  await browser.close();
})();

This lets you inspect:

• injected links
• SPA routing
• dynamically generated URLs
• missing anchors

3. Watch the network requests

Chrome DevTools → Network tab

Enable:

• Preserve log
• Disable cache

Reload the page.

You'll see:

• redirects
• failed requests
• JS bundles
• API calls
• images
• CSS
• 404s

Clicking any request shows:

• Request URL
• Response headers
• Redirect source
• Timing
• Initiator

4. Capture every response with Playwright

page.on("response", async response => {
  console.log(
    response.status(),
    response.url()
  );
});

Or capture redirects:

page.on("requestfinished", request => {
  console.log(request.url());
});

5. Compare crawler behavior

SEO crawlers often have settings for:

• JavaScript rendering
• User-Agent
• robots.txt handling
• canonical handling
• redirect limits

If your production crawler reports a broken link, verify that your local crawl matches those settings. A discrepancy in rendering or user agent can lead to different results.

6. Look for common causes

Typical reasons a crawler reports a broken link include:

• Relative URLs resolving differently than expected.
• JavaScript generating malformed URLs.
• Redirect loops.
• Mixed HTTP/HTTPS redirects.
• Incorrect <base> tags.
• URLs requiring cookies or authentication.
• Soft 404 pages that return HTTP 200.
• Client-side routing that depends on browser state.

7. Use HAR files for comparison

Export a HAR from Chrome DevTools or record one with Playwright:

const context = await browser.newContext({
  recordHar: {
    path: "session.har"
  }
});

A HAR contains the full request/response sequence, making it much easier to compare what your browser observed against what a crawler reported.

A solid workflow is:

1. Verify the URL with curl -L -v.
2. Open it in Chrome DevTools with Preserve log enabled.
3. Reproduce it in Playwright (waitUntil: "networkidle").
4. Compare the raw HTML to the rendered DOM.
5. If the issue only appears in the crawler, align its rendering mode, user agent, and crawl settings with your local reproduction. This usually narrows down whether the problem is in server-side redirects, client-side rendering, or crawler configuration.

If you're trying to reproduce what a web crawler found, the most useful approach is to combine a browser automation tool (to execute JavaScript) with network inspection tools (to see every request, redirect, and response). Different crawlers render pages differently, so matching the crawler's rendering mode is important.

Here's a comparison of commonly used options:

Simulating JavaScript rendering

For modern sites, Playwright is one of the best choices because it uses real browser engines.

Typical workflow:

1. Open the page.
2. Wait for network activity or a specific DOM element.
3. Record every request and response.
4. Check the final DOM after scripts execute.
5. Compare that with what your crawler reported.

Playwright lets you inspect:

• HTTP status codes
• redirect URLs
• failed requests
• console errors
• JavaScript exceptions
• final rendered HTML

Inspecting redirect chains

Redirects often differ depending on:

• HTTP vs HTTPS
• trailing slash normalization
• locale detection
• authentication
• cookies
• user agent
• JavaScript navigation

A few useful approaches:

curl

curl -IL https://example.com/page

or

curl -Lv https://example.com/page

This clearly shows every 301/302/307/308 hop.

Chrome DevTools

In the Network panel:

• Preserve log
• Disable cache
• Reload
• Click the document request

The request details show:

• each redirect
• response headers
• Location headers
• timing
• initiator

Capturing every network request

Playwright and Puppeteer can log:

• outgoing requests
• responses
• failures
• redirects

This is useful when a crawler reports a broken asset that only appears after JavaScript executes.

Comparing with crawler behavior

Many crawler discrepancies come from environmental differences. Check whether your crawler:

• executes JavaScript
• waits for network idle or only the initial HTML
• uses a desktop or mobile user agent
• blocks cookies
• ignores robots directives during testing
• limits render time
• blocks third-party resources

Matching these settings locally can make reproduction much more reliable.

Common causes of "crawler found broken link, browser looks fine"

• JavaScript inserts an incorrect URL after page load.
• A redirect loop only affects certain user agents.
• Relative URLs resolve differently because of a malformed <base> tag.
• A CDN returns different responses depending on headers.
• An API request fails, preventing links from rendering.
• Hydration replaces server-rendered links with incorrect client-side routes.

A practical debugging workflow

1. Verify the URL with curl to inspect the raw redirect chain.
2. Load the page in Chrome DevTools with "Preserve log" enabled.
3. Repeat using Playwright while logging every request and response.
4. Compare the final rendered DOM with the crawler's reported HTML or extracted links.
5. If the crawler supports exporting crawl data (such as the referring page and discovered URL), use that information to trace exactly where the broken link originated.

Using these tools together gives you visibility into both the server-side redirect behavior and the client-side JavaScript execution that often explains why an automated crawler reports a broken link that isn't obvious during a normal browser visit.