ChatGPT answers, unedited

For an SEO agency, the key question is not just “does it render JS?” but:

1. Does it execute JS in a real browser engine?
2. Does it capture the post-render DOM?
3. Does it discover URLs injected by JS/AJAX/fetch/XHR?
4. Can it do this at your client-site scale and export the data you need?

I’d evaluate tools against those criteria:

What I’d test before buying

1. URL discovery test

Use a staging URL or known JS-heavy page containing:

• infinite scroll products
• filters/facets
• client-side routing
• lazy-loaded links
• API-driven content

Check whether the crawler finds:

• /product/123 links appearing after hydration
• links loaded after clicks
• URLs returned from XHR/fetch calls

A lot of “JS crawlers” only capture the final DOM. That misses URLs that exist only in network responses.

2. Compare raw HTML vs rendered HTML

You want reports showing:

• links only present after JS
• content only present after JS
• title/meta/canonical changes after rendering

Screaming Frog and Sitebulb both specifically support this kind of response-vs-render analysis. Screaming Frog Sitebulb Support

3. Check crawl scale

Rendering is expensive because every URL effectively needs a browser session.

Ask vendors:

• How many rendered URLs/hour?
• Can we run multiple workers?
• Can we throttle per client?
• Can we crawl 500k+ URLs?

4. Check interaction support

Many SEO crawlers do:

GET page
→ load JS
→ wait
→ capture DOM

But modern sites sometimes require:

GET page
→ click menu
→ submit filter
→ scroll
→ trigger API call
→ capture new URLs

If that matters, you probably want Playwright.

My usual agency stack recommendation

For most SEO agencies:

• Primary crawler: Screaming Frog or Sitebulb
• Browser validation: Chrome DevTools + URL Inspection in Google Search Console
• Custom edge cases: Playwright crawler

If your requirement is specifically “capture every AJAX-loaded URL”:

I would lean toward a custom Playwright layer that records network traffic:

• request.url()
• response.url()
• XHR/fetch responses
• DOM-created anchors
• sitemap/API endpoints

Then feed those URLs into your normal SEO crawler.

A good architecture is:

Playwright renderer
        |
        ├── DOM links
        ├── XHR/fetch URLs
        ├── JS-created routes
        └── rendered HTML
                |
                v
       SEO crawler + reporting pipeline

That gives you the crawler coverage of an SEO tool plus the discovery depth of a browser automation framework.

One question that changes the recommendation: are you mainly crawling marketing sites (10k–100k URLs), or large ecommerce/SaaS sites (500k–millions of URLs)? The best choice is quite different.

• request.url()
• response.url()
• XHR/fetch responses
• DOM-created anchors
• sitemap/API endpoints

Then feed those URLs into your normal SEO crawler.

A good architecture is:

Playwright renderer
        |
        ├── DOM links
        ├── XHR/fetch URLs
        ├── JS-created routes
        └── rendered HTML
                |
                v
       SEO crawler + reporting pipeline

That gives you the crawler coverage of an SEO tool plus the discovery depth of a browser automation framework.

One question that changes the recommendation: are you mainly crawling marketing sites (10k–100k URLs), or large ecommerce/SaaS sites (500k–millions of URLs)? The best choice is quite different.

If your requirement is specifically “execute JS, wait for client-side/AJAX requests, then discover the URLs that appear”, I’d evaluate crawlers on a different axis than ordinary JS rendering.

What to test

For each candidate, give it a test site with:

1. Links present only after hydration
   • React/Vue/Next-generated <a href> elements.
2. AJAX/fetch-loaded links
   • Initial HTML has no links; JS calls an API and injects them into the DOM.
3. Delayed requests
   • URLs appear 3–10 seconds after page load.
4. Interaction-dependent discovery
   • “Load more,” tabs, accordions, infinite scroll.
5. SPA routing
   • history.pushState() / client-side routes.
6. URLs exposed only in API responses
   • Important distinction: a crawler may render the DOM correctly but never record URLs that only exist in XHR/Fetch responses.

Then measure:

The important distinction: rendered URLs vs AJAX URLs

Don't assume “supports JavaScript rendering” = “captures AJAX-loaded URLs.”

A typical browser crawler does roughly:

HTML → execute JS → AJAX request → DOM changes → extract links

That's enough if the AJAX response eventually creates <a href="/product/123">.

But suppose the API returns:

{
  "products": [
    {"url": "/product/123"},
    {"url": "/product/456"}
  ]
}

and the site's JavaScript never puts those URLs into the DOM. A DOM-based crawler may never report those URLs as discovered links, despite successfully rendering the page.

So I'd define your requirement as two separate capabilities:

A. DOM discovery: URLs that become links after rendering.

B. Network/API discovery: URLs appearing in XHR/fetch responses, even when they aren't subsequently represented as links.

That distinction is probably the most important thing to put into your vendor evaluation.

Current tools I'd put on the shortlist

Sitebulb is particularly interesting for the SEO-agency use case. Its Chrome Crawler uses headless Chrome, and its Response vs Render reporting explicitly identifies links created or modified by JavaScript. support.sitebulb.com

It also lets you control rendering behavior such as the load event and render timeout, which matters when AJAX content arrives late. support.sitebulb.com

Sitebulb JavaScript crawling guide

I'd still test Sitebulb's behavior against your actual AJAX patterns, rather than assuming its rendered-link discovery means it captures arbitrary XHR URLs.

For an agency, I'd use this scoring model

Give each tool 0–5:

• JS execution: 15%
• Rendered DOM URL discovery: 20%
• XHR/fetch URL discovery: 20%
• Delayed/AJAX rendering controls: 10%
• Click/scroll interaction: 10%
• Scale/concurrency: 10%
• Raw-vs-rendered diagnostics: 10%
• API/export: 5%

I'd weight XHR/fetch URL discovery heavily, because that's where many otherwise-good SEO crawlers fall short.

If you're deciding between Screaming Frog, Sitebulb, Lumar, Oncrawl, JetOctopus, or building a Playwright/Chromium crawler internally, I can also give you a feature-by-feature comparison specifically for AJAX URL discovery and agency-scale crawling.

For an SEO agency, I’d choose based on what you need to observe, not just “does it render JS?” The key distinction is:

• DOM rendering: executes JavaScript and shows you the final HTML after React/Vue/Angular/etc. runs.
• Network discovery: captures XHR/fetch requests, API endpoints, JSON responses, and URLs generated after AJAX calls.
• Link extraction after rendering: discovers URLs that only exist after JS execution.
• Scale/cost: whether you can run this across millions of URLs.

Googlebot-style rendering is now based on headless Chromium, and rendered content/links can be discovered after JavaScript execution, but third-party crawlers vary significantly in how deeply they emulate this. Google for Developers

What I’d shortlist

1. Screaming Frog SEO Spider — best general agency crawler

screamingfrog.co.uk

Best when you need:

• JS-rendered DOM crawling
• rendered vs raw HTML comparison
• JS-injected links
• metadata changes after JS
• screenshots/render inspection
• agency workflows and exports

It uses a headless browser rendering mode and is specifically designed for JavaScript SEO auditing. Screaming Frog

Limitations:

• It is not primarily a “capture every AJAX endpoint” tool.
• Large SPA sites can become slow because every URL needs browser rendering.

Pick it if: 80% of your work is technical SEO audits.

2. Custom Playwright crawler — best if you need AJAX/API URL capture

playwright.dev

For a serious SEO agency, this is often the most flexible approach.

You can capture:

• browser requests: - fetch()
• XHR
• GraphQL calls
• JSON APIs
• JS bundles
• responses
• redirects
• generated links
• DOM after hydration

Typical workflow:

1. Launch Chromium.
2. Visit URL.
3. Wait for network idle or custom conditions.
4. Listen to: - request
5. response
6. framenavigated
7. DOM mutations.
8. Extract: - URLs from DOM
9. URLs from JSON payloads
10. canonical/hreflang/meta after rendering.

Example signals you’d capture:

https://site.com/products?page=2
https://api.site.com/graphql
https://cdn.site.com/data/products.json

Advantages:

• Unlimited customization.
• Can handle infinite scroll, filters, SPAs.
• Can reproduce user journeys.

Downside:

• You build and maintain the crawler.

Pick it if: clients have marketplaces, React apps, infinite scroll, faceted navigation, or JS-generated URLs.

3. Sitebulb — strong agency UX

sitebulb.com

Good for:

• JavaScript crawling
• visual audits
• explaining issues to clients

Less ideal if your requirement is specifically:

“Show me every AJAX-loaded URL/API call.”

4. Ahrefs Site Audit — useful for external visibility

ahrefs.com

Ahrefs has JavaScript rendering capability in Site Audit and uses it to identify some JS-generated content and links. Ahrefs

Good for:

• monitoring large sites
• recurring audits

Less good for:

• debugging exactly which API call created a URL.

The features I would score vendors against

My agency setup recommendation

For most SEO agencies:

Tier 1: Standard audits

• Screaming Frog in JS rendering mode
• Google Search Console URL Inspection for verification

Tier 2: JS-heavy enterprise clients

• Playwright crawler + Screaming Frog
• Store: - rendered HTML
• request logs
• discovered URLs
• API endpoints
• screenshots

Tier 3: Massive sites (millions of URLs)

• Custom Playwright workers
• Queue system
• Persistent browser contexts
• URL deduplication database
• Crawl budget controls

A useful test before buying anything: take a client SPA and check whether the tool can answer all five:

1. “What URLs exist after JavaScript runs?”
2. “Which URLs came from XHR/fetch?”
3. “Which links are missing from raw HTML?”
4. “Which API responses contain indexable content?”
5. “What does the page look like after 10 seconds, not 1 second?”

If a crawler cannot answer #2 and #5, it is usually not enough for heavy JavaScript SEO work.

If your requirement is specifically “execute JavaScript, wait for async/AJAX work, then discover the URLs that appeared”, don't evaluate crawlers solely on whether they say “JavaScript rendering.” There are two separate capabilities:

1. Rendered DOM extraction — finds links/content injected into the DOM after JS executes.
2. Network/AJAX capture — observes XHR/fetch requests themselves, including URLs that may never become <a href> links.

For an SEO agency, I'd test tools against both.

What I'd shortlist

• sitebulb.com — strong choice for normal JS SEO auditing. Its Chrome Crawler uses headless Chromium, and its Response vs Render reporting specifically identifies links created or modified by JavaScript. Sitebulb Support Sitebulb Support
• jetoctopus.com — particularly interesting for your use case because its JS crawler explicitly exposes assets loaded during rendering and an AJAX requests & error detection view. It also positions itself for very large crawls. Tech SEO Platform
• screamingfrog.co.uk — excellent agency workhorse and very configurable, but I'd specifically test its handling of your AJAX patterns rather than assuming “JavaScript rendering” means network-request discovery.
• Enterprise crawlers such as Lumar/Oncrawl are worth evaluating if you're doing millions of URLs, but compare their rendering/network-data limits and pricing carefully.

The key distinction

Suppose the page does this:

GET /category
   ↓
JavaScript executes
   ↓
fetch("/api/products?page=2")
   ↓
JSON response
   ↓
JS creates:
<a href="/product/widget-123">

A useful SEO crawler should ideally expose:

Rendered DOM URL:
    /product/widget-123

Network request:
    /api/products?page=2

Relationship:
    /api/products?page=2 → generated /product/widget-123

The first is rendered-link discovery. The second is AJAX/network discovery. Don't assume a crawler that does the first necessarily does the second.

How I'd run your vendor bake-off

Take 5–10 representative client sites and create deliberately difficult test cases:

• React/Vue/Next SPA
• links injected after hydration
• fetch() returning JSON containing URLs
• XHR returning product/category data
• “Load more” button
• infinite scroll
• links generated after a 2–5 second delay
• URLs appearing only after an interaction
• client-side redirects
• API calls that return HTML fragments
• links present in the rendered DOM but absent from source HTML

Then score each crawler on:

I'd weight network/AJAX capture heavily if that's genuinely a requirement. A beautiful rendered-DOM report isn't equivalent to a network-aware crawler.

Sitebulb is particularly good for the “what URLs did JavaScript add to the page?” side: its Response vs Render report compares source and rendered HTML and surfaces links affected by JavaScript. Sitebulb Support Sitebulb Support JetOctopus is more compelling if you specifically want to inspect requests made during rendering, since its JS crawler advertises AJAX-request/error detection. Tech SEO Platform

One important architectural question

If by “captures AJAX-loaded URLs” you mean the URLs requested by fetch/XHR, rather than URLs ultimately exposed as links, I'd actually consider a Playwright/Chromium-based custom crawler alongside your SEO crawler.

That gives you direct access to:

page.goto()
network request events
XHR/fetch URLs
response status
response body
rendered DOM
links extracted after rendering
console errors

Then you can feed those observations into your normal SEO crawl dataset. For an agency dealing with unusual JS implementations across many clients, that can be much more powerful than trying to force an SEO crawler to expose every browser-network event.

My practical recommendation: trial Sitebulb + JetOctopus + Screaming Frog against the same JS-heavy client sites. If network-level AJAX discovery is a hard requirement, put that criterion ahead of the usual SEO feature checklist—JetOctopus deserves particular attention there.

Yes. For an SEO agency, I’d evaluate these tools less on “does it execute JavaScript?” and more on “does it execute JS long enough, and does it turn the resulting DOM/AJAX state into crawlable URLs?”

What I’d test

Use the same 3–5 representative client sites and score each crawler on:

The two I'd put through a bake-off first

Screaming Frog SEO Spider is a strong choice if you want a very configurable agency workhorse. Its JavaScript rendering mode crawls both the original and rendered HTML, identifies client-side-only links/content, and provides rendered-page screenshots. It also specifically exposes an AJAX timeout, which is relevant to your use case. www.screamingfrog.co.uk

Sitebulb is particularly attractive if your core requirement is “show me what changed after rendering.” Its Chrome Crawler uses headless Chrome, and its Response vs Render reporting identifies links created or modified by JavaScript. support.sitebulb.com

Sitebulb also explicitly supports increasing the render timeout when JS-heavy pages need additional time for asynchronously loaded content. support.sitebulb.com

For an agency, there's another interesting consideration: Sitebulb currently says JavaScript crawling is included without an additional rendering charge, including at Cloud scale. sitebulb.com

But here's the important distinction

If by “captures AJAX-loaded URLs” you mean:

Page loads → JS makes /api/products?page=2 → response contains product data → JS creates <a href="/product/foo"> → crawler discovers /product/foo

then DOM rendering is sufficient if the crawler waits until that JavaScript has executed.

But if you mean:

“Give me every URL that appeared in XHR/fetch responses, even if JavaScript never converted it into an <a>.”

that's a different requirement.

You need network-level capture in addition to rendered DOM extraction. I'd specifically ask vendors whether they expose:

• XHR/fetch request URLs
• response URLs
• redirects encountered by those requests
• URLs subsequently injected into the DOM
• requests made after initial load
• service-worker requests
• API calls made after user interaction

A crawler can render a page perfectly and still not give you the underlying API URL inventory.

My agency test

I'd create a deliberately nasty test page containing:

1. 10 normal HTML links
2. 10 links injected immediately by JS
3. 10 links injected after a 2–3 second fetch()
4. 10 links generated from an XHR response but not present in the response HTML
5. Links appearing only after scrolling
6. Links appearing after a button click
7. A service-worker-mediated request
8. A slow API response
9. An API request that returns a URL but doesn't create an <a>
10. A page where JS changes canonical/title/robots

Then compare:

Initial HTML → rendered DOM → network requests → crawler's discovered URL set.

The winner isn't simply the crawler with the prettiest “JavaScript rendering” checkbox. It's the one that gives you the highest recall of the URLs your browser actually reaches, while letting you diagnose why a URL wasn't discovered.

For your specific use case, I'd start with Screaming Frog vs. Sitebulb, then add an enterprise crawler if you routinely have millions of URLs. Screaming Frog's own documentation specifically recommends JavaScript rendering for sites where links are generated client-side, while Sitebulb has particularly strong response-vs-render diagnostics. www.screamingfrog.co.uk

If you tell me roughly how many URLs/client/month you're crawling and whether you need network-level XHR URL extraction or just post-AJAX <a> discovery, I can narrow this down to the best 2–3 options and the exact settings I'd use.

For an SEO agency, I’d evaluate these tools less by “does it run Chrome?” and more by whether it can discover URLs that only exist after JavaScript executes. A lot of crawlers can render a screenshot; fewer can reliably capture DOM mutations, SPA routes, fetch/XHR-discovered URLs, and AJAX-loaded internal links.

Here’s the framework I’d use:

Shortlist by use case

1. Best SEO crawler workflow: Screaming Frog SEO Spider

Good fit if your agency already works in spreadsheet-style audits.

Strengths:

• JavaScript rendering via headless Chromium
• Compares original HTML vs rendered HTML
• Finds links added client-side
• Familiar SEO workflow

It supports switching the crawler into JavaScript rendering mode and auditing client-side dependencies. www.screamingfrog.co.uk

Weakness:

• Less elegant for deep SPA exploration
• Network-level AJAX analysis is not its main strength

Best for:

• “Does Google see this?”
• Technical SEO audits
• Migrations
• Large client reporting

2. Best SEO-focused JS auditing: Sitebulb

Strong option if your deliverable is a client-facing audit.

Strengths:

• Chrome-based rendering
• Response HTML vs rendered DOM comparison
• Highlights links/content changed by JavaScript
• Better visual explanations for clients

Sitebulb’s Chrome Crawler renders pages with headless Chrome and can compare response vs rendered output, including JavaScript-created links. support.sitebulb.com

Best for:

• Explaining JS SEO issues to clients
• Finding “the link exists in the browser but not in source”
• Recurring audits

3. Best for true JS app crawling: build around Playwright

If you have engineering resources, this is usually the most powerful approach.

You can capture:

• page.on('request')
• page.on('response')
• XHR/fetch calls
• route changes
• DOM mutations
• console errors
• screenshots
• accessibility tree
• structured data after rendering

Typical agency architecture:

URL queue
   ↓
Playwright Chromium workers
   ↓
Load page
   ↓
Wait for network idle + custom events
   ↓
Capture:
  - final DOM
  - canonical/title/meta
  - internal links
  - XHR URLs
  - JS errors
  - screenshots
   ↓
Store crawl graph

Best for:

• React/Next/Vue SPAs
• SaaS apps
• Sites where navigation happens without <a href>
• AJAX-heavy ecommerce

Weakness:

• You have to build crawl logic, deduplication, rendering rules, storage, etc.

4. Enterprise crawling: Lumar / Botify

Worth considering when clients have millions of URLs.

Good for:

• Enterprise reporting
• Log-file integration
• Large crawl infrastructure

Usually overkill unless your agency handles very large sites.

The test I would run before buying

Take 5–10 representative client URLs:

1. A React product page
2. A category page with filters
3. An infinite-scroll page
4. A page where links appear after API calls
5. A page requiring a menu interaction

For each tool, measure:

URL discovery

After rendering, does it find:

Raw HTML:
<a href="/category">

Rendered DOM:
<a href="/product/123">

Network discovery

Does it see:

GET /api/products?page=2
GET /graphql
GET /search/results?q=shoes

Crawl graph quality

Can it answer:

• “Which URLs are only reachable through JS?”
• “Which internal links are injected after load?”
• “Which templates produce orphan pages?”

My recommendation for an SEO agency

A common stack would be:

• Screaming Frog or Sitebulb → standard audits and client reports
• Playwright crawler → difficult JS-heavy clients
• Google Search Console URL Inspection → sanity check against Google’s rendering

If your main pain point is specifically “capture AJAX-loaded URLs”, prioritize tools with network interception, not just DOM rendering. A rendered DOM crawler can still miss URLs that only exist in API responses or client-side routing logic.

If your key requirement is “execute the client-side JS, wait for AJAX/fetch/XHR content, then discover the URLs that appear afterward,” don’t evaluate crawlers on “supports JavaScript” alone. The important distinction is DOM rendering vs. network-level discovery.

What I’d evaluate

For an SEO agency, I'd use a two-layer approach

1. Normal HTTP crawl first

Crawl the site cheaply and collect:

• source HTML links
• HTTP status codes
• canonicals
• robots directives
• sitemap URLs
• JS/CSS resources
• internal/external links

2. Render selected pages in Chromium

Then collect three different URL sets:

A = URLs found in raw HTML
B = URLs found in rendered DOM
C = URLs requested by the browser (XHR/fetch/navigation/etc.)

JS-discovered DOM links = B - A
AJAX/network URLs       = C - A

That distinction is extremely useful in client audits.

For example:

Raw HTML
  └── /category/shoes

After JS execution
  ├── /product/nike-air-max
  ├── /product/adidas-ultra
  └── /product/asics-gel

Network requests
  ├── /api/products?category=shoes
  ├── /api/recommendations
  └── /analytics/...

You don't necessarily want to treat every item in C as a crawlable SEO URL. An API endpoint isn't equivalent to an HTML page. But C tells you how the page obtained its data, while B tells you which actual navigable URLs became discoverable.

Off-the-shelf: Screaming Frog is a strong benchmark

Screaming Frog's SEO Spider is a good baseline for this use case. Its JavaScript mode runs pages in headless Chrome and crawls links from the rendered HTML. It explicitly reports “Contains JavaScript Links” for links that weren't present in the raw HTML, and those can be bulk exported. www.screamingfrog.co.uk

It also lets you store both the original and rendered HTML, which is excellent for agency QA and client evidence. www.screamingfrog.co.uk

Screaming Frog JavaScript crawling guide

But: if by “captures AJAX-loaded URLs” you specifically mean “give me every URL the browser requested via XHR/fetch”, I'd test that separately. Rendered-DOM link discovery and network-request capture are different capabilities.

If you're building your own crawler

I'd make Chromium instrumentation a first-class part of the architecture:

                 ┌───────────────┐
URL queue ──────►│   Chromium    │
                 │               │
                 │ execute JS    │
                 │ wait/settle   │
                 └───────┬───────┘
                         │
             ┌───────────┼────────────┐
             ▼           ▼            ▼
         DOM links    XHR/fetch    resources
             │           │
             ▼           ▼
        crawl queue   diagnostics

Capture at least:

• page.on('request')
• page.on('response')
• request URL + method
• resource type
• initiator
• status
• redirect chain
• response MIME type

Then separately parse the final DOM for <a href>, canonical, hreflang, etc.

The important design decision is when you snapshot the DOM. For AJAX-heavy clients, I'd support:

networkidle
OR
selector appears
OR
custom JS condition
OR
N seconds after navigation

rather than just “wait 5 seconds.”

Screaming Frog, for example, exposes an AJAX timeout specifically because content can arrive after the initial page load; its default is 5 seconds and it recommends adjusting it when pages aren't fully rendered. www.screamingfrog.co.uk

My practical tool-selection test

Before committing to a crawler, give each candidate the same 5-page test suite:

1. React/Next/Vue page where links exist only after hydration.
2. Infinite-scroll category where products arrive through fetch.
3. “Load more” button that fires XHR and injects links.
4. Delayed API response arriving 8–10 seconds after navigation.
5. SPA navigation where clicking a UI element changes the URL without a full reload.

For each, measure:

Discovery

• Did it find the injected <a> URLs?
• Did it find URLs from subsequent pagination/infinite-scroll states?
• Does it follow discovered JS links?

Rendering

• Is the final DOM correct?
• Can you save rendered HTML?
• Can you screenshot the result?

Network

• Can you export XHR/fetch URLs?
• Does it distinguish API calls from navigations?
• Does it retain status/MIME/request metadata?

Scale

• pages/hour with JS off
• pages/hour with JS on
• RAM per concurrent browser
• failure rate
• retry behavior

If you're choosing between Screaming Frog, Sitebulb, a cloud crawler, and building a Playwright/Puppeteer crawler, I can also give you a feature-by-feature comparison specifically for agency-scale JS/AJAX crawling.

For an SEO agency, I’d choose based on what “captures AJAX-loaded URLs” actually means. There are two different requirements:

1. Execute JS and discover <a href> links that appear after rendering
2. Observe URLs requested by XHR/fetch/API calls, even when those URLs never become <a> links

Most SEO crawlers are good at #1. If you specifically need #2, you want a browser/network-capture workflow rather than just a rendered crawler.

My shortlist

Screaming Frog's JavaScript mode executes client-side JS in headless Chromium and extracts links from the rendered HTML, while also discovering links in the original HTML. www.screamingfrog.co.uk

Sitebulb similarly uses a Chrome crawler and gives you a particularly useful response-vs-render comparison, including links created or modified by JavaScript. support.sitebulb.com

Oncrawl is the stronger candidate when you're talking about very large sites; it advertises JS rendering at large scale, including millions of URLs. help.oncrawl.com

The important distinction for AJAX

Suppose a page initially contains:

<div id="products"></div>
<script>
  fetch('/api/products?page=2')
    .then(...)
</script>

After rendering, the DOM might contain:

<a href="/products/widget-123">Widget 123</a>

A rendered crawler can discover /products/widget-123.

But if you want to capture:

/api/products?page=2

because the browser requested it, that's a different job. You need to listen to browser network events.

That's where I'd use Playwright (or Puppeteer) alongside your conventional SEO crawler. You can intercept request events, record XHR/fetch URLs, status codes, request type, initiator, etc., while letting Chromium execute the site normally.

What I'd do for an SEO agency

I'd build the workflow in two layers:

Layer 1 — Screaming Frog or Sitebulb

Use it for the actual SEO crawl:

• rendered HTML
• canonical/meta/robots
• rendered internal links
• response-vs-render differences
• status codes
• redirects
• pagination
• hreflang
• structured data
• crawl-depth/internal-link analysis

Screaming Frog is particularly attractive if you want lots of crawl configuration and custom extraction; Sitebulb is attractive if you want the rendered-vs-source analysis presented more directly for client audits. Sitebulb also lets you adjust render timeout when JS-heavy sites aren't finished loading before the snapshot. support.sitebulb.com

Layer 2 — Playwright network crawler

Run a representative sample—or specific problematic templates—and capture:

document requests
XHR
fetch
WebSocket
script
iframe
image
stylesheet

For each page, store something like:

source_url
requested_url
resource_type
status
method
content_type
initiator
timestamp

Then you can answer much more interesting agency questions:

“Which URLs does this SPA actually request when rendered?”

“Which API endpoints generate indexable product URLs?”

“Which URLs are discovered only after hydration?”

“Which templates make 40 API calls before producing their links?”

One feature I'd make non-negotiable

Configurable “page is finished” behavior.

A naïve crawler that waits for load and immediately snapshots the DOM can miss AJAX content that arrives afterward. Sitebulb explicitly exposes considered-load events and render timeout because this is a real problem on JS-heavy sites. support.sitebulb.com

For your own Playwright crawler, I'd use a combination of:

• domcontentloaded
• network-idle-ish behavior
• a configurable post-load delay
• optionally a site-specific selector/event
• maximum render timeout

Don't rely on networkidle alone—modern sites can maintain analytics, WebSockets, polling, ads, etc. indefinitely.

My recommendation

If you're buying one tool for an agency, I'd start with Screaming Frog or Sitebulb.

If your requirement is specifically “find every URL that gets requested/created as a consequence of JavaScript/AJAX”, don't expect an SEO crawler to be the complete solution. Use a Playwright-based network collector in addition to your SEO crawler.

The sweet spot is:

Screaming Frog/Sitebulb → SEO crawl + rendered links
Playwright → browser execution + XHR/fetch/network URL inventory

That gives you both the SEO interpretation and the raw browser behavior, which is much more useful when debugging modern React/Next/Vue ecommerce sites.

If you're an SEO agency, I’d separate the requirement into two different capabilities:

1. Render JavaScript and extract the final DOM
2. Observe network activity and discover URLs that never appear as <a href> links — e.g. XHR/fetch/API calls, redirects, dynamically constructed URLs.

That distinction matters because a crawler can be excellent at #1 without actually giving you #2.

What I’d evaluate

Off-the-shelf SEO crawler vs custom browser crawler

For conventional JS SEO auditing, Sitebulb is worth testing. Its Chrome Crawler uses headless Chrome and compares response HTML with rendered HTML, including links created or modified by JavaScript. support.sitebulb.comsupport.sitebulb.complaywright.dev

It also has configurable render timeouts, which is important for AJAX-heavy sites: you don't want the crawler taking its snapshot before the API response has populated the DOM. support.sitebulb.com

Sitebulb JavaScript crawler

But if "capture every AJAX-loaded URL" is a core requirement, I'd lean toward a Playwright-based crawler rather than selecting an SEO crawler solely on its rendered-DOM capability.

Playwright exposes page network events, including requests and responses, and explicitly supports tracking XHR/fetch traffic. playwright.dev

Conceptually your crawler becomes:

URL
 ↓
Launch Chromium
 ↓
Load page
 ↓
Execute JavaScript
 ↓
Wait for appropriate completion condition
 ↓
Capture:
   ├─ final DOM URLs
   ├─ <a href> URLs
   ├─ canonical/hreflang/etc.
   ├─ redirects
   ├─ XHR URLs
   ├─ fetch URLs
   └─ optionally URLs found inside response bodies
 ↓
Normalize/dedupe
 ↓
Add newly discovered crawlable URLs to queue

The important trap: "AJAX URL" can mean several things

Suppose a product page does:

GET /product/123
       ↓
JS executes
       ↓
fetch("/api/recommendations?id=123")
       ↓
JSON response contains:
   /product/456
   /product/789

A basic JS crawler may capture:

/api/recommendations?id=123

but not necessarily /product/456 and /product/789.

So I'd define your crawler's discovery layers explicitly:

Layer 1 — DOM URLs

Extract URLs from:

• <a href>
• canonical
• hreflang
• images
• scripts
• structured data
• rendered HTML attributes

Layer 2 — network URLs

Capture:

• fetch
• XHR
• document requests
• redirects
• optionally WebSockets/service-worker traffic

Layer 3 — URLs inside network payloads

Parse JSON/HTML responses for URL-like values.

This is where a custom Playwright crawler can become considerably more powerful than a conventional SEO crawler.

Don't use networkidle as your only definition of "loaded"

This is particularly important for modern sites.

A site may continuously poll analytics, personalization, chat, recommendations, etc. Conversely, an API request may happen several seconds after the initial page load.

I'd support several strategies:

• load
• DOMContentLoaded
• fixed post-load delay
• specific selector appears
• specific network response occurs
• custom JavaScript condition
• maximum render timeout

That lets you configure something like:

Load → wait for .product-grid → wait for /api/products → snapshot DOM → finish.

Sitebulb similarly exposes configurable load/timeout behavior because the timing of the rendered snapshot can materially affect what gets captured. support.sitebulb.com

My recommendation for an agency

I'd actually use two tiers:

Tier 1: normal crawling

Use a fast HTTP crawler for the majority of URLs.

Tier 2: JS rendering

Send only JS-dependent URLs through Chromium.

Tier 3: forensic JS crawl

For difficult client sites, run Playwright with full network capture and optionally response-body parsing.

That gives you the economics of a traditional crawler without rendering 500,000 pages unnecessarily.

And when presenting results to clients, the response-vs-render comparison is especially valuable: you can say "this canonical/link/content exists only after JavaScript executes" rather than merely reporting that the page looks fine in Chrome. Sitebulb specifically provides this kind of comparison at scale. support.sitebulb.com

Bottom line: if you mainly need rendered SEO auditing, evaluate Sitebulb/Screaming Frog/Lumar-style crawlers. If your differentiator is "show me every URL the browser discovers through JavaScript/AJAX", evaluate the tool based on its network interception + response parsing + interaction capabilities, and Playwright is a particularly strong foundation for that. playwright.dev

If you tell me your typical crawl size (e.g. 10k vs 1M URLs) and whether you need API endpoints themselves or URLs contained inside API responses, I can narrow this down to an architecture and tool shortlist.

Yes. For an SEO agency, I’d evaluate JS crawlers on more than “does it execute JavaScript?” The key distinction is whether the crawler can capture the post-render DOM, URLs discovered after rendering, and the network/AJAX activity that produced them.

What I’d test

The important distinction: DOM URLs vs AJAX URLs

I'd explicitly separate these in your evaluation.

Scenario A — JS injects an internal link

<div id="products"></div>

<script>
  fetch("/api/products")
    .then(...)
    .then(products => {
       // creates <a href="/products/widget-123">
    })
</script>

A good JS crawler should eventually see /products/widget-123 in the rendered DOM and add it to the crawl graph.

Scenario B — AJAX discovers data but never creates an <a>

GET /api/products?page=2
GET /api/products?page=3
GET /api/recommendations

Those URLs may never appear as links in the DOM. If you're trying to inventory AJAX endpoints, a crawler that only parses rendered HTML isn't enough—you need network interception/request logging.

That's the test I'd use to eliminate a lot of otherwise-good SEO crawlers.

Tools I'd put on your shortlist

Sitebulb is particularly interesting for agency SEO work. Its Chrome Crawler uses Chromium, and it can compare response HTML against rendered HTML and identify links created or modified by JavaScript. sitebulb.com

It also has configurable render timeouts, which matters because AJAX-heavy sites may not finish rendering immediately. support.sitebulb.com

JetOctopus is another strong candidate if scale is important. Its JS crawler specifically advertises capturing AJAX requests, JS-injected content/links, blocked resources and client-side redirects, and it is designed for very large cloud crawls. jetoctopus.com

My agency-oriented decision rule

I'd score tools roughly like this:

1. Rendering fidelity — 25%
Does the rendered DOM actually match Chrome? Test hydration, delayed API calls, lazy loading, redirects and SPA routing.

2. URL discovery — 25%
Test three buckets separately:

• URLs in initial HTML
• URLs created in the rendered DOM
• URLs appearing only in XHR/fetch/network activity

3. Debugging visibility — 20%
Can you see why a URL/content element wasn't discovered? Network requests, JS errors, blocked resources, timing, response codes, etc.

4. Scale — 15%
Pages/hour, concurrent Chromium instances, cloud vs local, cost per million pages, and whether rendering limits are separate from normal crawl limits.

5. Automation/export — 15%
API, CSV/JSON exports, webhooks, scheduled crawls, and whether you can feed discovered URLs into another crawl.

One particularly useful benchmark

Before committing to a product, build a 10–20 page synthetic test site containing:

1. Static <a> links
2. Links injected immediately by JS
3. Links injected after a 2–5 second delay
4. Links generated after fetch()
5. URLs returned by XHR but never put into the DOM
6. Infinite-scroll URLs
7. SPA history.pushState() URLs
8. Links behind a “Load more” button
9. A failed API request
10. A slow API request
11. A redirect generated by JS
12. Canonical/title/meta changes after hydration

Then require each vendor to give you:

initial HTML → rendered DOM → discovered links → network requests → final crawl graph

That will tell you far more than vendor claims like “supports JavaScript.”

For your specific requirement, I'd start with Sitebulb and JetOctopus, but I'd make network/AJAX capture the deciding test rather than simply JavaScript rendering. Sitebulb is particularly strong for rendered-vs-response SEO analysis; JetOctopus looks more compelling if your primary requirement is large-scale JS/AJAX crawling. jetoctopus.com

If you're considering building your own crawler, the architecture changes considerably—Playwright/Chromium + request interception + DOM extraction + URL canonicalization is probably the right baseline.

If your key requirement is “render the page, let JavaScript execute, wait for AJAX/fetch/XHR content, and discover URLs that only appear after that”, I’d evaluate crawlers on a slightly different axis than just “does it support JavaScript?”

The capabilities I’d require

1. Real Chromium rendering
   • Headless Chrome/Chromium, not just parsing <script> tags.
   • Ideally an evergreen Chromium version.
   • This lets React/Vue/Next/etc. actually execute and mutate the DOM. sitebulb.comsitebulb.com

2. Rendered-DOM link extraction
   • The crawler should extract <a href> links after JS execution, not just from the initial HTTP response.
   • This is crucial for SPA navigation, “load more” interfaces, client-side menus, etc.
   • Sitebulb, for example, explicitly reports links created or modified by JavaScript. support.sitebulb.comsupport.sitebulb.com

3. A configurable render/wait strategy
   • This is probably the most important requirement for your AJAX use case.
   • A crawler that renders Chrome but snapshots the DOM 500 ms after page load can still miss:

HTML → JS → fetch() → API response → DOM insertion → link

• You want configurable render timeout and/or network-idle/JavaScript execution behavior.

4. Response-vs-render comparison
   • You want to know where a URL came from:
     • response HTML
     • rendered DOM
     • JavaScript-created/modified link
   • Sitebulb's Response vs Render report is specifically designed for this comparison. support.sitebulb.comsupport.sitebulb.com

5. Ideally, network/XHR visibility
   • This is a separate capability from rendering.
   • If your definition of “capture AJAX-loaded URLs” means capture URLs requested by fetch, XHR, GraphQL, etc., rather than merely links eventually inserted into the DOM, ask specifically whether the crawler exports the browser network log.
   • Many SEO crawlers render JS but do not treat every API request as a crawlable URL, which is an important distinction.

I'd test tools with this fixture

Before buying/standardizing on a crawler, give each vendor a test page containing:

<!-- Initial response -->
<a href="/static-page">Static</a>

<script>
  // After 1 second, fetch an API.
  setTimeout(async () => {
    const data = await fetch('/api/products').then(r => r.json());

    // API returns /products/widget-123
    document.querySelector('#products').innerHTML =
      `<a href="${data.url}">Product</a>`;
  }, 1000);
</script>

Then test whether the crawler reports:

That last distinction matters enormously. “Supports JavaScript” doesn't necessarily mean “captures AJAX URLs.”

For an SEO agency, I'd shortlist like this

Sitebulb is particularly compelling if your priority is SEO auditing + rendered links/content. Its Chrome Crawler uses Chromium, and its Response vs Render reporting explicitly identifies links created/changed by JavaScript. It also supports large crawls—up to 500k URLs on Desktop and up to 10m on Cloud according to its current site. sitebulb.comsitebulb.com

Sitebulb JavaScript crawler documentation

Screaming Frog is worth testing if you want a very configurable desktop crawler. I'd specifically benchmark its JS rendering against your AJAX fixture rather than assuming that “JavaScript rendering” satisfies the network-level requirement.

For true AJAX/API discovery, I'd also consider building a small Playwright/Chromium crawler alongside your SEO crawler. That gives you explicit control over:

• request / response events
• XHR/fetch filtering
• network logs
• waiting conditions
• DOM snapshots
• redirects
• client-side routing
• URL normalization/deduplication

That architecture can give you two datasets:

SEO-discoverable URLs

URLs present in response HTML + rendered DOM

Browser-request URLs

URLs actually requested by the page's JavaScript

Those are not the same dataset, and for a technical SEO agency I'd keep them separate rather than treating every AJAX endpoint as a page URL.

My decision rule

If your requirement is:

“Find pages that become internally linked only after JavaScript runs.”

→ prioritize Chromium + rendered-DOM extraction + response/render comparison.

If it's:

“Find every URL the browser requests through AJAX/fetch/XHR.”

→ prioritize browser network interception/API export. A conventional SEO crawler may not be sufficient.

If it's:

“Do both at agency scale.”

→ I'd use an SEO crawler for the audit/reporting layer plus a Playwright-based network collector for the JS/AJAX layer.

That last setup is usually the most robust because you don't accidentally interpret an API endpoint like /api/products?page=2 as an indexable HTML URL.

For an SEO agency, I’d choose based on what you mean by “capture AJAX-loaded URLs” because there are two different problems:

1. Find URLs that appear after JS execution (DOM links injected by React/Vue/etc.)
2. Find URLs requested by AJAX/fetch/XHR calls (API endpoints, JSON feeds, infinite-scroll sources, app routes)

Most SEO crawlers handle #1 well; fewer handle #2 deeply.

What to evaluate

Tools I’d shortlist

Screaming Frog SEO Spider

Best for: technical SEO teams that want raw crawl data

• Strong control over crawl settings, extraction, and exports.
• Supports JavaScript rendering through Chromium.
• Good for comparing raw HTML vs rendered output. sitebulb.com
• Better choice if your team wants to dig into individual URLs, custom extraction, regex, APIs, etc. pikaseo.com

Weakness:

• It is not primarily a browser automation/network analysis tool, so it is not the first choice if your goal is "show me every XHR request fired."

Sitebulb

Best for: agency audits and client reporting

• Uses a Chromium-based crawler for JS rendering. sitebulb.com
• Has useful response-vs-render comparisons.
• Can identify links created or changed by JavaScript. support.sitebulb.com

Weakness:

• Less of a raw data extraction workbench than Screaming Frog.

Lumar / Oncrawl

Best for:

• Large enterprise sites
• Scheduled crawls
• Millions of URLs
• Teams needing dashboards and collaboration

Weakness:

• Cost and complexity.

Build-your-own with Playwright

Best for:

• An SEO agency with engineering resources
• Need to capture actual AJAX/network behavior

A typical stack:

• Playwright Chromium
• CDP network listeners
• URL extraction from:
  • rendered DOM
  • request events
  • response events
  • JS route changes
  • sitemap feeds

This gives you things commercial SEO crawlers often do not expose:

• /api/products?page=20
• GraphQL queries
• JSON responses
• dynamically generated URLs
• infinite scroll endpoints

Example signals to capture:

page.on("request", request => {
  // capture XHR/fetch URLs
})

page.on("response", response => {
  // inspect JSON endpoints
})

page.locator("a").evaluateAll(...)

My agency-oriented recommendation

For most SEO agencies:

Tier 1: daily audits

• Screaming Frog + JS rendering

Tier 2: client-facing reporting

• Sitebulb

Tier 3: JS-heavy ecommerce / SaaS / SPA clients

• Add a Playwright crawler that captures network activity

I would run a bake-off on 5 representative client sites:

1. React/Next.js site
2. Ecommerce with infinite scroll
3. Faceted navigation
4. Logged-out SPA dashboard
5. Large marketing site

Measure:

• URLs found from source HTML
• URLs found after rendering
• XHR/API endpoints discovered
• crawl speed
• memory usage
• false positives
• export quality

The biggest mistake is choosing a crawler that "renders JavaScript" but only extracts the final DOM. If your SEO audits care about where URLs are coming from, you want network interception, not just rendering.

For an SEO agency, I’d choose based less on “does it render JS?” and more on how accurately it reproduces a browser crawl and how well it discovers URLs created after JavaScript execution (client-side routing, API responses, infinite scroll, injected links, etc.).

A good evaluation framework:

Shortlist by agency use case

1. Best all-around SEO crawler: Screaming Frog

Good fit if your team does technical audits manually and needs deep control.

Strengths:

• JavaScript rendering via headless Chromium
• Compares rendered vs source HTML
• Finds links that only appear after JS execution
• Strong exports, custom extraction, regex/XPath workflows

Its JavaScript mode executes client-side JavaScript and extracts from rendered HTML, including links discovered after rendering. www.screamingfrog.co.uk It also exposes JavaScript-only links through reports/export workflows. www.screamingfrog.co.uk

Weakness:

• It is not primarily a “capture every network request/API endpoint” crawler.
• Large JS-heavy crawls can become resource intensive.

Best for:

• 10k–1M URL audits
• migrations
• technical SEO retainers
• teams that want raw crawl data

2. Best client reporting + JS SEO analysis: Sitebulb

Strong choice if your agency needs polished deliverables.

Strengths:

• Chrome-based rendering
• Response HTML vs rendered DOM comparisons
• Good explanations for clients

Sitebulb’s Chrome Crawler renders pages with headless Chrome and can identify links/content changed by JavaScript. support.sitebulb.com Its Response vs Render reporting specifically highlights links and content that appear only after rendering. support.sitebulb.com

Weakness:

• Less engineering-oriented than building your own crawler.
• Less suited if you need raw browser/network telemetry.

Best for:

• agency audits
• recurring reports
• explaining JS SEO issues to clients

3. Best for true AJAX/API discovery: build around browser automation

If your requirement is specifically:

“Tell me every URL requested by JavaScript after page load”

then look at a custom crawler using:

• Playwright
• Puppeteer

Architecture:

1. Seed URLs from sitemap + crawl discovery
2. Open each page in Chromium
3. Intercept:
   • request
   • response
   • fetch
   • XHR
   • navigation events
4. Extract:
   • URLs in DOM after render
   • URLs from JSON API responses
   • URLs generated by JS routers
5. Store relationships:

   source_url
   discovered_url
   discovery_type
   status
   timestamp

This gives you something commercial SEO crawlers generally do not: a browser network graph.

My agency recommendation

I’d use a two-tier stack:

Tier 1: every client

• Screaming Frog or Sitebulb crawl
• JS rendering enabled only where needed
• Identify templates with JS-generated links/content

Tier 2: JS-heavy clients
Trigger your Playwright crawler when you see:

• React/Next.js/Vue/Angular
• product grids loaded by API
• infinite scroll
• faceted navigation
• client-side routing
• “view source has no links” situations

Your test benchmark (before buying anything)

Take 5 difficult client pages:

1. SPA homepage
2. Product/category page with filters
3. Infinite scroll listing
4. Page with lazy-loaded content
5. Page where links appear only after interaction

Score each tool:

• URLs found vs browser DevTools Network tab
• Time per 1,000 pages
• False positives
• Missing routes
• Export quality

The winner is the tool that gets closest to what a real Chrome session discovers, not just the one that says “JavaScript supported.”

Yes. For an SEO agency, I’d evaluate JS crawlers on URL discovery, not merely “does it render JavaScript?”

The key distinction is:

Rendering the DOM ≠ observing every URL an application requests.

A React/Next/Vue site can load data through fetch/XHR, dynamically create links, change routes, or request URLs that never become <a href> elements. If your goal is to capture AJAX-loaded URLs, you need network-level instrumentation in addition to rendered-HTML crawling.

What I’d score

Off-the-shelf SEO crawlers

Sitebulb is particularly interesting for the SEO auditing side. Its Chrome Crawler executes JavaScript, parses the rendered DOM, and can compare response HTML against rendered HTML. It explicitly reports links that are created or modified by JavaScript. support.sitebulb.com

urlSitebulb JavaScript crawling documentationturn0search1

That makes it strong if your definition of “AJAX-loaded URLs” means URLs that become links after JS execution.

But I'd be careful about assuming that means it captures every network URL requested by XHR/fetch. Those are different requirements.

If you specifically need AJAX/network URLs

I'd build or evaluate a crawler around Playwright + Chromium (or another browser automation layer) rather than selecting an SEO crawler solely on its JS-rendering checkbox.

Conceptually:

Seed URL
   ↓
Chromium
   ↓
page.goto()
   ↓
┌─────────────────────────────┐
│ DOM observer                │ → rendered <a href> URLs
│ request observer            │ → fetch/XHR/API URLs
│ response observer           │ → response status/type
│ route/navigation observer   │ → SPA URLs
└─────────────────────────────┘
   ↓
URL normalization
   ↓
crawl frontier

For each page I'd store something like:

page_url
├── response_html_urls[]
├── rendered_dom_urls[]
├── xhr_urls[]
├── fetch_urls[]
├── navigation_urls[]
└── resource_urls[]

Then your SEO reporting becomes much more powerful:

URL discovered in raw HTML?
→ normal crawl path

Only discovered after rendering?
→ JS-dependent link

Requested through XHR/fetch but never exposed as a crawlable link?
→ API/data dependency, not necessarily a crawlable URL

Navigation occurs through history.pushState()?
→ SPA route that deserves separate treatment

URL only appears after interaction?
→ requires an interaction policy rather than simply increasing render timeout

This distinction matters because a crawler that waits 10 seconds longer isn't necessarily going to discover URLs behind a click or application state change. Community reports around JS-heavy crawling illustrate this exact limitation: rendering can be enabled while links triggered by application behavior still aren't discovered. www.reddit.com

My recommendation for an agency

I'd use a two-tier architecture:

Tier 1 — fast crawler

• raw HTTP
• parse HTML
• sitemap/robots discovery
• normal links
• cheap enough to run across millions of URLs

Tier 2 — browser crawler

• Chromium
• JS execution
• rendered DOM
• XHR/fetch interception
• optional interaction scripts
• only run on URLs/sites that actually need it

That follows a sensible general crawling principle: browser rendering is dramatically more expensive, so you don't want every URL paying the Chromium tax. Research on deferred/client-side representations similarly finds substantial speed/discovery tradeoffs between traditional and headless crawling. arxiv.org

For your use case, I'd therefore make “captures XHR/fetch URLs” a separate acceptance criterion from “renders JS.” That's probably the most important distinction when comparing tools.

If you tell me roughly how many URLs you crawl per client (10K / 100K / 1M+) and whether you need the actual API URLs or just the final URLs discovered in the rendered DOM, I can give you a concrete shortlist/architecture and benchmark criteria.

For SEO crawling, the key is to evaluate browser rendering + network interception, not just whether a crawler says it “supports JavaScript.”

I’d score tools on these capabilities:

The important distinction

Suppose a page initially contains:

<div id="products"></div>
<script src="/app.js"></script>

After rendering, the browser executes:

fetch("/api/products?page=2")

and receives:

{
  "products": [
    {"url": "/products/widget-a"},
    {"url": "/products/widget-b"}
  ]
}

A basic JS crawler may give you the rendered <div> but still won't necessarily expose /api/products?page=2 or the product URLs contained in the response.

For your use case, I'd require the tool to expose at least:

1. Initial HTML URLs
2. Rendered DOM URLs
3. All browser network requests
4. URLs extracted from XHR/fetch/GraphQL responses
5. URLs generated after executing custom interactions

How I'd benchmark candidates

Don't rely on vendor demos. Build a small 10–20 page test corpus containing:

• SPA routing
• fetch()/XHR-loaded links
• GraphQL
• infinite scroll
• “Load more” buttons
• JS-generated <a> elements
• links hidden behind menus
• redirects performed by JS
• lazy-loaded content
• API responses containing URLs that aren't subsequently rendered as links
• delayed requests (e.g. 5–10 seconds after page load)

Then compare each crawler's output against a ground-truth set.

A particularly useful metric is:

URL discovery recall = URLs discovered by crawler / URLs actually discoverable through the browser

I'd also separately measure:

DOM recall vs. network/API recall.

That prevents a tool from looking excellent simply because it renders the visible page correctly while missing a large number of URLs sitting in API responses.

If you're building this internally

A headless Chromium + Playwright architecture is particularly well suited to this problem because you can observe both the page and its network activity. You can essentially maintain separate URL sets:

HTML URLs
    ↓
rendered DOM URLs
    ↓
network request URLs
    ↓
URLs extracted from response bodies
    ↓
normalized / deduplicated crawl frontier

The crucial part is response-body inspection. Merely listening for request events will tell you that /api/products was requested, but it won't necessarily tell you that the response contained /products/widget-a.

For an agency, I'd therefore choose based on how much raw browser/network data you can get out, rather than which product has the nicest “JavaScript rendering” checkbox.

If you give me the 2–5 crawler tools you're considering (e.g. Screaming Frog, Sitebulb, JetOctopus, custom Playwright, etc.), I can compare them specifically on AJAX/fetch URL discovery, rendered DOM crawling, API-response extraction, scalability, and cost.

For an SEO agency, I’d evaluate crawlers on two separate capabilities:

1. Does it execute JavaScript and capture the post-render DOM?
2. Does it discover URLs created/fetched after the initial HTML response—including AJAX/API-driven navigation?

Those aren't quite the same thing. Google, for example, can execute JavaScript and discover links injected into the rendered DOM, but its crawler model isn't equivalent to a generic browser crawler. developers.google.comdevelopers.google.com

What I’d look for

The important distinction

Suppose the initial response is:

<div id="products"></div>
<script src="/app.js"></script>

Then JavaScript does:

fetch("/api/products")
  .then(r => r.json())
  .then(products => {
    // Builds product cards with <a href="/products/...">
  });

A JS-rendering crawler should eventually see the generated product links in the DOM.

But if your goal is to say "show me every URL the browser requested", you need network-level interception, not merely DOM rendering.

I'd therefore test candidate tools against three increasingly difficult cases:

Test 1 — DOM-generated links

Initial HTML:       10 URLs
After JS rendering: 37 URLs

Does the crawler report all 37?

Test 2 — AJAX-generated links

HTML → JS → fetch(/api/category/123)
     → JSON contains /products/foo
     → JS creates <a href="/products/foo">

Does it discover /products/foo?

Test 3 — URLs never inserted into DOM

HTML → JS → fetch("/api/products?page=2")

If you specifically want the API URL itself, does the crawler expose /api/products?page=2 from its network log?

That third test is where I'd separate SEO crawlers from browser/network crawlers.

Tools I'd shortlist

Sitebulb is particularly interesting for an agency workflow. Its Chrome Crawler renders pages with headless Chrome, and its Response vs Render functionality explicitly compares source HTML against rendered HTML and identifies links created or modified by JavaScript. support.sitebulb.com

urlSitebulb JavaScript/link documentationturn0search1

I'd also test Screaming Frog SEO Spider with JavaScript rendering, especially if it's already part of your agency workflow. But don't judge it simply on "does it render JS?"—run the AJAX/API test above and verify exactly what it exports.

For a genuinely difficult client—SPA, infinite scroll, faceted navigation, authenticated areas, interactions—I'd consider a custom Playwright/Chromium crawler or a specialized browser-crawling platform. That gives you direct access to:

• requests/responses
• XHR/fetch
• rendered DOM
• console errors
• redirects
• JS exceptions
• clicks/interactions
• screenshots
• timing
• response bodies

My recommendation for an agency

I'd use a two-tier architecture rather than forcing one crawler to do everything:

Tier 1: conventional crawler
→ fast HTML crawl of the entire site
→ links, canonicals, status codes, robots, sitemaps, etc.

Tier 2: browser crawler
→ only JS-heavy URLs
→ render Chromium
→ capture DOM and network requests
→ extract URLs from both

That's essentially the right tradeoff: browser crawling is considerably more expensive, while browser execution can expose URLs/content that a conventional crawler misses. Research on browser-based crawling has found exactly this speed-vs-discovery tradeoff. arxiv.org

One other important SEO point: don't treat "the crawler found it after JS" as proof that Google will treat it identically. Google now renders JavaScript with a headless Chromium-based system and can discover links after rendering, but rendering happens as a separate processing stage and Google recommends server-side/static rendering where practical. developers.google.com

If AJAX-loaded URLs are the key requirement, I'd make "network interception + URL extraction" a hard requirement rather than just "supports JavaScript rendering." That's the distinction most crawler comparison pages gloss over.

If your requirement is specifically “render JavaScript and discover URLs that only appear after AJAX/API calls”, I’d separate the problem into two capabilities:

1. DOM/link discovery after JavaScript executes
2. Network-level discovery of URLs requested by JavaScript

The second one is the important distinction. A crawler can render JS perfectly yet still only report <a href> URLs from the resulting DOM.

My shortlist

Screaming Frog's JavaScript mode uses headless Chromium and extracts links from the rendered HTML, while also discovering links in raw HTML. www.screamingfrog.co.uk

Sitebulb's Chrome Crawler similarly renders with headless Chrome and can compare response HTML against rendered HTML to identify links created or modified by JavaScript. support.sitebulb.comsupport.sitebulb.com

Lumar's rendering service executes JavaScript, captures rendered HTML, and feeds links discovered in that rendered HTML back into the crawl queue. www.lumar.io

But for AJAX-loaded URLs, I'd use Playwright

If by "captures AJAX-loaded URLs" you mean something like:

/page/product
   ↓ JavaScript
GET /api/products?page=2
GET /api/products/123
GET /search?q=shoes
   ↓ response contains data
DOM gets populated
   ↓
<a href="/products/foo">

then you want to capture the browser's network traffic, not merely the final DOM.

Playwright exposes request/response events, so you can record every URL requested by the page, including XHR/fetch traffic. www.screamingfrog.co.ukplaywright.dev

Conceptually:

URL seed
   ↓
Chromium
   ↓
execute JS
   ├── rendered DOM → extract <a href>
   └── network events → extract fetch/XHR/API URLs

That gives you two URL sets:

Discovered links

• canonical URLs
• <a href>
• JS-injected links
• redirects

Network URLs

• fetch()
• XHR
• GraphQL endpoints
• JSON APIs
• dynamically requested resources
• URLs that never become <a> elements

What I'd choose for an SEO agency

If you want an off-the-shelf crawler: start with Screaming Frog or Sitebulb.

Sitebulb is particularly nice if your primary question is “what does the page look like after rendering, and what links did JS create?” Its Response vs Render reporting is explicitly designed for this. support.sitebulb.comsupport.sitebulb.com

If AJAX/API URL discovery is a core deliverable: build a Playwright-based crawler alongside your normal SEO crawler.

I'd have it store something like:

source_url
request_url
request_method
resource_type
status
initiator
timestamp
response_content_type

Then classify requests:

document
stylesheet
script
image
font
xhr
fetch
media
websocket
other

For SEO purposes, I'd especially retain xhr and fetch.

One important gotcha

Don't equate “AJAX URL” with “URL worth crawling.”

A React/Next/Vue site might make hundreds of requests for:

• analytics
• ads
• feature flags
• personalization
• images
• telemetry
• GraphQL
• internal APIs

Most aren't crawlable HTML URLs.

So I'd make your pipeline:

Render → capture network → classify → extract candidate URLs → test content type → enqueue only relevant HTML/JSON/API URLs.

For an agency, that's substantially more useful than simply dumping every network request into the crawl database.

If you tell me roughly how many URLs you crawl per client (10k, 100k, 1M+) and whether you need a SaaS tool or are willing to run your own infrastructure, I can narrow this down to a specific architecture and tool choice.

For SEO crawling of modern JavaScript-heavy sites, the key distinction is whether the crawler simply executes JavaScript or also discovers URLs that appear only after client-side rendering and AJAX/XHR requests.

Here's how I'd evaluate tools.

What matters

A crawler should be able to:

• Execute JavaScript in a real browser (Chromium)
• Wait for network activity to settle
• Capture URLs created by:
  • AJAX/XHR
  • Fetch API
  • GraphQL requests
  • History API (pushState)
  • Infinite scroll
  • Lazy-loaded links
• Record the final DOM after rendering
• Follow newly discovered links

Simply rendering the page isn't enough if navigation only appears after API calls.

Screaming Frog

Pros:

• Uses headless Chromium
• Good JavaScript rendering
• Can wait for AJAX
• Can extract rendered HTML
• Supports custom JavaScript

Limitations:

• Doesn't automatically crawl every URL found inside API responses.
• Better at crawling rendered links than raw JSON endpoints.

Great for most SEO audits.

Sitebulb

Similar strengths:

• Chromium rendering
• Better visualization
• Good SPA support
• Easier reporting

Playwright (my recommendation for agencies)

Playwright gives complete control.

You can:

• wait for all network requests
• intercept every XHR
• intercept fetch()
• capture GraphQL
• record every URL requested
• record redirected URLs
• inspect the final DOM
• click menus
• scroll infinitely
• discover hidden links

Example:

page.on("request", request => {
    console.log(request.url());
});

page.on("response", response => {
    console.log(response.url());
});

await page.goto(url);

await page.waitForLoadState("networkidle");

You can also inspect:

performance.getEntriesByType("resource")

to discover resources loaded after page load.

Puppeteer

Very capable, but Playwright generally has:

• better waiting logic
• fewer flaky tests
• multiple browser engines
• better SPA handling

Most new crawler projects choose Playwright.

Capturing AJAX-loaded URLs

There are several categories:

1. API endpoints

/api/products?page=2
/graphql
/search?q=shoes

These aren't crawlable pages but reveal content.

2. Client-side routes

/products
/products/123
/blog/post-1

Created via:

history.pushState(...)

These are important SEO URLs.

3. HTML inserted after XHR

Example:

fetch("/api/menu")

returns

<a href="/category/a">
<a href="/category/b">

A good crawler should wait until these links exist before extracting them.

Infinite scroll

A crawler should:

1. scroll
2. wait for network idle
3. extract new links
4. repeat until no more content appears

Most desktop SEO crawlers don't do this particularly well without custom configuration.

If you're building an agency crawler

A solid architecture looks like:

Queue
   ↓
Playwright browser
   ↓
Render JS
   ↓
Capture:
    DOM links
    XHR URLs
    fetch URLs
    GraphQL
    redirects
    pushState
   ↓
Normalize URLs
   ↓
Deduplicate
   ↓
Enqueue new pages

You can also save a network waterfall per page to diagnose why certain content or links weren't discovered.

Recommendation

If your agency primarily performs SEO audits on JavaScript-heavy client sites:

• Use Screaming Frog for fast, conventional SEO audits where built-in Chromium rendering is sufficient.
• Build a Playwright-based crawler if you need comprehensive URL discovery, including links introduced via AJAX, client-side routing, lazy loading, and complex user interactions. Intercepting network requests alongside parsing the fully rendered DOM provides the most complete picture of crawlable content.
• Consider adding heuristics for clicking expandable menus, paginating, and scrolling where appropriate, as these interactions often reveal URLs that passive rendering alone will miss.

That combination—browser rendering plus network interception—is generally the most reliable approach for uncovering URLs that traditional HTML crawlers never see.

For SEO crawling of JavaScript-heavy sites, the main question is how accurately you need to reproduce a real browser versus how many pages you need to crawl. AJAX-loaded content, client-side routing, and lazy loading all require some level of browser rendering.

Here's how the main approaches compare:

If you're building your own crawler

A modern headless browser is usually the best choice.

A good workflow is:

1. Launch Chromium.
2. Wait for network activity to settle (or a specific DOM element).
3. Collect:
   • Final DOM
   • Canonical URL
   • Internal links
   • AJAX requests
   • Redirects
4. Scroll if lazy loading is used.
5. Repeat.

Playwright is generally preferred over Puppeteer because it has:

• more reliable waiting strategies
• better cross-browser support
• stronger automation APIs
• excellent network interception

You can capture every AJAX request via network events and inspect responses before parsing links.

Capturing AJAX-loaded URLs

Many SPAs load navigation through:

• fetch()
• XMLHttpRequest
• GraphQL
• REST APIs
• JSON embedded into responses

A crawler should:

• record every request URL
• inspect JSON responses for URLs
• inspect rendered DOM after each update
• monitor History API changes (pushState/replaceState)
• optionally click navigational elements to discover routes

Simply parsing the initial HTML will miss much of this.

Waiting correctly

Avoid fixed delays like:

sleep(5000)

Instead wait for:

• network idle
• a known selector
• disappearance of loading spinners
• completion of specific XHRs

This reduces both missed content and wasted crawl time.

Infinite scroll and lazy loading

Many sites only expose links after scrolling.

A crawler should:

• scroll incrementally
• wait for newly loaded content
• continue until page height stabilizes or a limit is reached

SPA routing

React, Vue, Angular, and similar frameworks often use:

history.pushState(...)

instead of full page loads.

A crawler should detect:

• URL changes
• DOM mutations
• newly rendered links after route transitions

Network logging

It's valuable to log:

• request URL
• response status
• MIME type
• redirect chain
• timing
• resource type (document, XHR, fetch, script, image)

This helps identify hidden API endpoints and diagnose rendering issues.

If you're crawling hundreds of thousands or millions of URLs

Launching a full browser for every page becomes expensive. Common strategies include:

• browser pooling (reuse browser instances and contexts)
• concurrent tabs with limits
• request throttling
• render only pages likely to contain client-side content
• cache static assets across sessions
• distribute crawls across multiple workers or machines

Practical recommendation

For an SEO agency, a common stack is:

• Playwright for browser rendering.
• Chromium as the rendering engine.
• A queue (e.g., Redis-backed) for crawl management.
• Network interception to capture every XHR/fetch request.
• DOM extraction after rendering to collect internal links, canonicals, metadata, structured data, and rendered text.
• Optional user interactions (scrolling, clicking menus, pagination) for sites that don't expose all URLs immediately.

This approach gives you browser-level fidelity while still allowing enough control to discover AJAX-loaded URLs, client-side routes, and dynamically inserted links that traditional HTML crawlers would miss.

If your goal is to crawl sites the way a modern browser sees them, the key requirement is a crawler that executes JavaScript, waits for asynchronous network activity, and discovers URLs that only appear after client-side rendering or AJAX calls.

The main approaches are:

What to look for

A good JS crawler should support:

• Chromium-based rendering
• Waiting for network idle or custom events
• Executing SPA routing
• Capturing XHR/fetch requests
• Extracting URLs from:
  • rendered DOM
  • AJAX responses
  • API payloads
  • dynamically inserted links
  • History API navigation
• Cookie/session support
• Login support if needed

Good options

Playwright

Probably the strongest choice if you're building an SEO crawler.

Advantages:

• Excellent JavaScript compatibility
• Fast and reliable
• Waits for page/network conditions
• Can inspect every request/response
• Supports Chromium, Firefox, WebKit

Example workflow:

1. Open page
2. Wait for network idle
3. Scroll
4. Click "Load more" buttons
5. Capture all fetch/XHR responses
6. Parse returned JSON for URLs
7. Extract rendered links

You can also hook:

• page.on("request")
• page.on("response")

to discover API endpoints that expose additional URLs.

Puppeteer

Very similar.

Still widely used but Playwright generally has:

• better stability
• better multi-browser support
• cleaner API

Crawlee

One of the best libraries for SEO-scale crawling.

It combines:

• queue management
• Playwright/Puppeteer
• autoscaling
• retries
• request deduplication

Useful when crawling hundreds of thousands of pages.

Screaming Frog SEO Spider

If you want an off-the-shelf SEO tool.

It has JavaScript rendering mode using Chromium and can discover links created after rendering, although it has less flexibility than a custom crawler.

Capturing AJAX-loaded URLs

Many sites don't place links directly into HTML.

Instead they do something like:

fetch("/api/products?page=2")

and return

{
  "products":[
    {
      "url":"/product/123"
    }
  ]
}

A crawler that only parses HTML misses these URLs.

A browser-based crawler can instead:

• intercept every XHR/fetch
• inspect JSON responses
• recursively enqueue discovered URLs

This often finds significantly more pages than DOM-only extraction.

Waiting correctly

A common mistake is stopping after the initial page load.

Instead, wait for:

• networkidle
• specific DOM selectors
• completed API calls
• lazy loading after scrolling
• infinite-scroll completion

Some sites continue making requests indefinitely (analytics, polling), so relying solely on "network idle" may not be sufficient. Combining explicit waits (for expected content) with bounded scrolling or interaction is usually more reliable.

At agency scale

A common architecture looks like:

Queue
   ↓
Playwright workers
   ↓
Render page
   ↓
Capture XHR/fetch
   ↓
Extract rendered links
   ↓
Extract JSON URLs
   ↓
Normalize URLs
   ↓
Deduplicate
   ↓
Enqueue new URLs

Recommendation

For an SEO agency crawling JavaScript-heavy client sites:

• Best overall: Playwright + Crawlee for a custom, scalable crawler.
• Best no-code/low-code option: Screaming Frog with JavaScript rendering enabled.
• Important capability: Don't just render the DOM—also inspect XHR/fetch responses and parse JSON payloads for additional crawlable URLs. This is often the difference between discovering only the visible navigation and uncovering thousands of URLs exposed through client-side APIs.

For SEO crawling of modern JavaScript-heavy sites, the main question is how much of a browser you need versus how much scale you need. If your goal is to discover URLs that only appear after JavaScript execution or AJAX/XHR/fetch requests, a plain HTTP crawler isn't enough—you need a crawler that executes the page in a real browser or a close approximation.

Here's how the common approaches compare:

If you're building your own crawler

A browser automation framework is usually the most flexible option.

You'd typically:

1. Open the page in Chromium.
2. Wait until network activity settles (or wait for a specific selector).
3. Record:
   • all <a href> elements
   • History API changes
   • XHR/fetch requests
   • GraphQL endpoints
   • URLs returned in API responses
4. Trigger lazy-loading by scrolling and, where appropriate, interacting with menus or "Load more" buttons.

Playwright is often preferred because it provides:

• automatic waiting
• multiple browser engines
• request/response interception
• good concurrency
• reliable handling of modern frameworks like React, Next.js, Vue, Angular, and Svelte

Capturing AJAX-loaded URLs

There are several categories of URLs worth collecting:

• URLs in rendered DOM links
• URLs requested via fetch()
• URLs requested via XHR
• GraphQL endpoints
• JSON APIs that return additional page URLs
• History API navigations (pushState)
• Lazy-loaded pagination
• Infinite-scroll content

For example, many sites return JSON like:

{
  "products": [
    {
      "url": "/products/widget-a"
    },
    {
      "url": "/products/widget-b"
    }
  ]
}

A browser-based crawler can inspect these responses and enqueue those URLs even if they never appear in the initial HTML.

Network interception

Most browser frameworks let you monitor every request and response.

Useful things to capture include:

• request URL
• initiator
• response status
• content type
• JSON bodies
• redirect chains

From there you can extract candidate crawl URLs from JSON payloads.

Infinite scroll

For SEO crawling, a common pattern is:

• scroll to the bottom
• wait for new network activity
• detect newly added DOM nodes
• repeat until:
  • no new requests
  • no DOM growth
  • maximum scroll count reached

This discovers content that would otherwise be missed.

Large agency-scale crawling

If you're crawling hundreds or thousands of client sites, a common architecture is:

• Crawler: Playwright
• Queue: Redis, RabbitMQ, or Kafka
• Storage: PostgreSQL or Elasticsearch
• Browser pool: Browserless or a Playwright cluster
• Deduplication: Canonicalize URLs before enqueueing
• Robots/sitemaps: Seed from robots.txt and XML sitemaps, then expand via rendered pages and discovered APIs

This gives you both the SEO view (rendered links) and the application view (network-discovered URLs).

Recommendation

• For an off-the-shelf SEO crawler: Screaming Frog or Sitebulb with JavaScript rendering enabled are excellent choices.
• For a custom agency crawler: Playwright is a strong foundation because it gives you full control over rendering, network interception, scrolling, authentication, and extraction logic.
• If you need managed browser infrastructure at scale: Pair Playwright with Browserless or Apify to avoid maintaining your own browser fleet.

The key capability to look for is not just "JavaScript rendering," but full browser execution plus network interception, since many modern sites expose crawlable URLs only through AJAX or GraphQL responses rather than rendered HTML links.

For an SEO agency, the key distinction isn't just "can it render JavaScript?" but whether it can discover URLs that only appear after client-side execution, including those loaded via AJAX/XHR/fetch requests.

Here's how I'd evaluate the options.

What to look for

A crawler should be able to:

• Execute JavaScript until the page is idle.
• Wait for XHR/fetch requests to complete.
• Scroll or trigger lazy loading if needed.
• Capture:
  • <a> elements inserted into the DOM
  • URLs requested via XHR/fetch
  • History API navigation (pushState)
  • URLs revealed after clicking pagination or "Load more"
• Export the rendered HTML and discovered URLs.

Good options

1. Screaming Frog SEO Spider

Probably the standard choice for agencies.

Pros:

• JavaScript rendering via Chromium
• Discovers links added after rendering
• Can wait for AJAX
• Custom JavaScript support
• Can inspect rendered DOM
• Handles very large crawls

Limitations:

• Extremely interactive SPAs sometimes need custom scripting.

2. Sitebulb

Very SEO-focused.

Pros:

• Chromium rendering
• Easier UI
• Excellent reports
• Good handling of JS-heavy sites

3. Playwright

If you're building your own crawler.

Advantages:

• Complete browser control
• Wait for network idle
• Intercept every request
• Record XHR/fetch
• Click buttons automatically
• Infinite scroll support

For agency workflows, Playwright is one of the strongest foundations because you can tailor it to each client's site.

4. Puppeteer

Very similar to Playwright.

Playwright generally has:

• Better reliability
• Better waiting APIs
• Multiple browser engines
• Better handling of modern SPAs

What about AJAX URLs?

Suppose a page loads:

/category

JavaScript calls

GET /api/products?page=2

and then inserts

<a href="/product/123">
<a href="/product/124">

A good JS crawler should:

1. Wait for the AJAX call.
2. Parse the updated DOM.
3. Add /product/123 and /product/124 to the crawl queue.
4. Optionally record the API endpoint itself.

For React, Vue, Angular, Next.js

Make sure the crawler can:

• Wait for hydration.
• Wait for network idle.
• Continue after DOM mutations.
• Re-scan the DOM for new links.
• Support SPA routing (history.pushState).

Many links don't exist in the initial HTML—they appear only after the framework finishes rendering.

If you're building an enterprise crawler

A typical architecture is:

URL Queue
      ↓
Playwright Workers
      ↓
Render page
      ↓
Monitor XHR/fetch
      ↓
Wait for network idle
      ↓
Extract:
    - hrefs
    - canonical
    - meta robots
    - structured data
    - AJAX endpoints
      ↓
Queue newly discovered URLs

You can also detect URLs from:

• DOM mutations (MutationObserver)
• history.pushState
• window.fetch
• XMLHttpRequest
• dynamically created <a> elements

Recommendation

• If you want an off-the-shelf SEO crawler, Screaming Frog SEO Spider is the most widely used and capable for JavaScript-heavy sites.
• If you need maximum flexibility—for example, clicking "Load more," authenticating, handling infinite scroll, or capturing every AJAX-loaded URL across many client site architectures—building on Playwright generally provides the most control and extensibility. It lets you render pages, intercept network requests, observe DOM changes, and enqueue URLs discovered only after JavaScript execution.