ChatGPT answers, unedited

If you're choosing a crawler for an SEO agency, I’d treat “renders JavaScript” and “discovers URLs from AJAX/API activity” as two separate capabilities. A crawler can execute JS perfectly well while still missing URLs that only appear in XHR/fetch responses.

What I’d evaluate

Google itself renders JavaScript with an evergreen Chromium-based renderer, and after rendering it can discover links that JavaScript injected into the DOM. Google for Developers Google for Developers

But there's an important distinction: Google's rendering behavior isn't the same thing as “crawl every URL appearing in every API response.” Google says its renderer processes XHR requests to understand page content, while URL discovery is primarily based on crawlable links, sitemaps, and redirects. Google for Developers Google for Developers

The architecture I'd want

For an agency crawler, I'd favor something conceptually like:

Seed URLs
   ↓
HTTP crawler
   ↓
Chromium renderer
   ├── final DOM
   ├── rendered links
   ├── JS console/errors
   └── network events
          ├── document
          ├── fetch/XHR
          ├── GraphQL
          └── other requests
                    ↓
             URL extraction
                    ↓
        ┌───────────┴───────────┐
        ↓                       ↓
  crawlable URLs          API-discovered URLs
        ↓                       ↓
       queue              discovery dataset

That last distinction is important. Don't automatically treat every network URL as a crawl target. Otherwise you'll end up crawling analytics endpoints, JSON APIs, image CDNs, telemetry, personalization calls, and dozens of irrelevant resources.

Instead, I'd maintain at least two datasets:

• SEO URLs — URLs found in rendered <a href>, canonical tags, structured data, sitemaps, redirects, etc.
• Network-discovered URLs — URLs observed in XHR/fetch/GraphQL/network responses, with the source page and request type recorded.

Then you can report something like:

/category/shoes → rendered page → XHR /api/products?page=2 → response contains 48 product URLs → 46 aren't present as HTML links.

That's much more useful in an SEO audit than simply saying “JavaScript is enabled.”

One particularly important test

Build a small benchmark suite of deliberately nasty client sites:

1. React SPA where links appear only after hydration.
2. Infinite scroll where subsequent products arrive through XHR.
3. Load More button requiring interaction.
4. GraphQL endpoint returning product/category URLs.
5. JSON API containing URLs but never rendering them as links.
6. Client-side router using history.pushState.
7. Lazy-loaded content triggered by viewport visibility.
8. JS-generated canonical/meta/structured data.
9. Authentication/session-dependent rendering.
10. A site with a never-ending analytics connection, to test whether your wait strategy hangs.

For each, record:

raw HTML URLs
rendered DOM URLs
network request URLs
URLs extracted from network responses
URLs discovered after interactions

Then compare the crawler's output against a known ground truth.

The key question when comparing products

Don't ask vendors:

“Does your crawler support JavaScript?”

Ask:

“Can your Chromium crawler capture fetch/XHR/GraphQL requests, inspect their response bodies, extract URLs from those responses, and put those URLs into a separate crawl/discovery queue?”

That question will separate genuine browser-based crawlers from tools that merely render the page.

Also test how they wait for rendering. A fixed 5-second delay is much less useful than configurable conditions such as DOM selectors, network activity, or custom browser-side JavaScript.

And for SEO validation, I'd retain a Google-equivalent rendered view as a separate comparison. Google explicitly notes that rendering can have limitations and that its renderer is stateless between page loads, so “works in Chrome” doesn't automatically mean “Google can process it.” Google for Developers Google for Developers

If you're deciding between Screaming Frog, Sitebulb, JetOctopus, Lumar, Oncrawl, or building your own Playwright/Chromium crawler, I can also give you a feature-by-feature comparison specifically for AJAX/XHR URL discovery at agency scale.

Then compare the crawler's output against a known ground truth.

The key question when comparing products

Don't ask vendors:

“Does your crawler support JavaScript?”

Ask:

“Can your Chromium crawler capture fetch/XHR/GraphQL requests, inspect their response bodies, extract URLs from those responses, and put those URLs into a separate crawl/discovery queue?”

That question will separate genuine browser-based crawlers from tools that merely render the page.

Also test how they wait for rendering. A fixed 5-second delay is much less useful than configurable conditions such as DOM selectors, network activity, or custom browser-side JavaScript.

And for SEO validation, I'd retain a Google-equivalent rendered view as a separate comparison. Google explicitly notes that rendering can have limitations and that its renderer is stateless between page loads, so “works in Chrome” doesn't automatically mean “Google can process it.”

If you're choosing a crawler for an SEO agency, I’d evaluate it less on “does it render JavaScript?” and more on what it does after JavaScript executes—especially whether it discovers links/URLs injected by XHR/fetch/AJAX.

What I’d test

Use a representative client site with at least one page where:

• The initial HTML contains few/no product/content links.
• JavaScript makes an XHR/fetch request.
• The response is used to build DOM elements containing <a href="...">.
• Those URLs aren't present in the original HTML.
• Content appears after a delay or after hydration.
• Ideally, test an SPA where navigation uses history.pushState().

Then score each crawler on these capabilities:

Two obvious tools to put through the test

Screaming Frog SEO Spider explicitly supports JavaScript rendering using headless Chromium. In JavaScript mode it extracts content and links from rendered HTML as well as discovering links in the raw HTML. It also has a specific workflow for identifying links that exist only after JavaScript executes. Screaming Frog Screaming Frog

That makes it particularly interesting for your “AJAX-loaded URLs” requirement: its documentation describes exporting JavaScript-only links and adjusting the AJAX timeout (5 seconds by default). Screaming Frog

Sitebulb has a Chrome Crawler that renders pages with headless Chrome and compares response HTML against rendered HTML. Its Response vs Render functionality specifically identifies links created or modified by JavaScript. Sitebulb Support Sitebulb Support

The important distinction

Don't confuse:

“The crawler can render AJAX content.”

with:

“The crawler will discover URLs produced by AJAX and recursively crawl them.”

Those are separate capabilities.

For example:

GET /category
       │
       └── initial HTML
              │
              └── app.js executes
                     │
                     └── fetch("/api/products")
                              │
                              └── JSON response
                                     │
                                     └── JS creates:
                                         <a href="/product/123">
                                         <a href="/product/456">

You want the crawler to ultimately produce:

/category
/product/123
/product/456

—not merely show /product/123 in a rendered screenshot or rendered DOM while never putting it into the crawl frontier.

My recommended agency benchmark

Before committing to a platform, build a 10–20 URL JavaScript test suite and run every candidate against exactly the same site.

Include:

1. Links in initial HTML
2. Links inserted synchronously by JS
3. Links inserted after fetch()
4. Links inserted after XHR
5. Links appearing 2–5 seconds after load
6. Links generated after React/Vue hydration
7. SPA pushState() URLs
8. Infinite-scroll URLs
9. Links behind “Load more”
10. Links whose existence depends on an API response

Then measure:

A. Discovery

• Did it find the URL?

B. Crawl expansion

• Did it actually schedule the discovered URL for crawling?

C. Timing

• How long did it wait for the URL to appear?

D. Fidelity

• Did it reproduce what a real Chrome session sees?

E. Scale

• How many rendered URLs/hour can you realistically process?

That last one is important: rendering is substantially more expensive than fetching raw HTML. Sitebulb, for example, notes that its Chrome crawler downloads page resources and therefore takes considerably longer than its HTML crawler. Sitebulb Support

One trap I'd specifically test

Don't rely on a fixed AJAX timeout alone.

Suppose your client has:

fetch('/api/products')
  .then(...)
  .then(renderProducts)

If the crawler snapshots the DOM at 1 second but the API responds at 1.8 seconds, it can incorrectly conclude that the page has no product links.

Screaming Frog exposes an AJAX timeout specifically for this scenario, while Sitebulb exposes more granular Chrome crawler/load settings. Screaming Frog Sitebulb Support

For an agency, I'd therefore prioritize render lifecycle controls + rendered-link extraction + crawl-frontier behavior over raw “JavaScript support.”

If you tell me roughly how many URLs you crawl per client (10k / 100k / 1M+) and whether you need an API, I can narrow this into a practical Screaming Frog vs Sitebulb vs Lumar/enterprise vs building your own Playwright crawler comparison.

B. Crawl expansion

• Did it actually schedule the discovered URL for crawling?

C. Timing

• How long did it wait for the URL to appear?

D. Fidelity

• Did it reproduce what a real Chrome session sees?

E. Scale

• How many rendered URLs/hour can you realistically process?

That last one is important: rendering is substantially more expensive than fetching raw HTML. Sitebulb, for example, notes that its Chrome crawler downloads page resources and therefore takes considerably longer than its HTML crawler.

If your requirement is specifically “execute the client’s JavaScript, wait for AJAX/API activity, and discover URLs that only appear after rendering,” I’d evaluate crawlers on a slightly different axis than ordinary SEO crawling.

What I’d test for

The key distinction is:

Rendered DOM links ≠ AJAX/network URLs.

A crawler can execute JS and discover <a href> elements inserted into the DOM, while completely missing URLs that exist only as API/XHR/fetch responses.

So I’d score tools against these capabilities:

Off-the-shelf SEO crawlers

Screaming Frog and Sitebulb are good choices if your primary objective is SEO auditing of rendered pages. Both use Chromium-based rendering and can identify links that appear only after JavaScript executes. Screaming Frog can also store/export rendered HTML and compare it with the original HTML. Screaming Frog Screaming Frog

Sitebulb similarly has a Chrome Crawler and a Response vs Render report specifically designed to identify links created or modified by JavaScript. Sitebulb Support Sitebulb Support

But there's an important caveat for your use case:

Don't assume “JavaScript rendering” means “capture every AJAX URL.”

For example, imagine:

/page/category
    ↓
React executes
    ↓
fetch("/api/products?page=2")
    ↓
JSON response
    ↓
React creates:
<a href="/product/widget-123">

A conventional SEO crawler may discover /product/widget-123 from the rendered DOM.

But if you specifically need:

/api/products?page=2

then you want network-level instrumentation, not merely rendered-DOM crawling.

If AJAX URLs are a first-class requirement

I'd seriously consider building your crawler around Playwright/Chromium rather than trying to force an SEO crawler to expose something it wasn't designed to capture.

The architecture can be:

URL frontier
    ↓
Chromium / Playwright
    ↓
page.goto()
    ↓
execute JS
    ↓
capture:
  ├── rendered DOM links
  ├── fetch()
  ├── XHR
  ├── document navigations
  ├── redirects
  └── optionally WebSockets
    ↓
URL normalizer
    ↓
dedupe / canonicalization
    ↓
crawl frontier

Then classify discovered URLs:

DISCOVERY_TYPE

html
rendered_dom
xhr
fetch
redirect
history_api
script
stylesheet
image

That classification becomes extremely useful for an SEO agency because you can tell a client:

/products/widget-123 was discovered only after JavaScript execution

versus:

/api/products?page=2 was requested by the browser but never exposed as a crawlable HTML link.

The test I would use when selecting a tool

Don't run a generic benchmark. Give every candidate the same deliberately nasty test site:

1. Static HTML contains /static-product.
2. JS injects /js-product after 2 seconds.
3. fetch() retrieves JSON containing /api-product.
4. Clicking “Load more” triggers another API request.
5. Infinite scroll loads another batch.
6. history.pushState() creates /spa-product.
7. One API request redirects.
8. One request is blocked until a cookie/consent action.
9. Some content arrives after 5–10 seconds.
10. One link exists only inside the API JSON and never becomes an <a>.

Then require the crawler to produce:

URL                         Found via
/static-product             HTML
/js-product                 rendered DOM
/api/products?page=2       fetch
/spa-product                history API
/api/redirect               XHR

That will expose the difference between “has JS rendering” and “actually gives us the network/discovery data we need.”

My practical decision rule

For an SEO agency, I'd use a two-tier setup:

• Screaming Frog / Sitebulb for normal client audits, rendered HTML, JS-only links, technical SEO reporting, and raw-vs-rendered comparisons. Both explicitly support these workflows. Screaming Frog Sitebulb Support
• A Playwright/Chromium crawler for your difficult clients where you need to capture everything the browser requests, particularly AJAX/fetch URLs, SPA navigation, interaction-triggered requests, and API-discovered URLs.

The latter also gives you much more control over when a page is considered “finished.” That's important because a fixed render timeout can snapshot the DOM before late AJAX content appears; Screaming Frog, for example, exposes an AJAX timeout specifically for this reason. Screaming Frog

If you tell me roughly how many URLs you crawl per client (10k / 100k / 1M+) and whether you need the AJAX/API URLs themselves or just the final URLs they produce, I can give you a concrete shortlist and architecture.

Off-the-shelf SEO crawlers

Screaming Frog and Sitebulb are good choices if your primary objective is SEO auditing of rendered pages. Both use Chromium-based rendering and can identify links that appear only after JavaScript executes. Screaming Frog can also store/export rendered HTML and compare it with the original HTML. Screaming Frog Screaming Frog

Sitebulb similarly has a Chrome Crawler and a Response vs Render report specifically designed to identify links created or modified by JavaScript. Sitebulb Support Sitebulb Support

But there's an important caveat for your use case:

Don't assume “JavaScript rendering” means “capture every AJAX URL.”

For example, imagine:

/page/category
    ↓
React executes
    ↓
fetch("/api/products?page=2")
    ↓
JSON response
    ↓
React creates:
<a href="/product/widget-123">

A conventional SEO crawler may discover /product/widget-123 from the rendered DOM.

But if you specifically need:

/api/products?page=2

then you want network-level instrumentation, not merely rendered-DOM crawling.

If AJAX URLs are a first-class requirement

I'd seriously consider building your crawler around Playwright/Chromium rather than trying to force an SEO crawler to expose something it wasn't designed to capture.

The architecture can be:

URL frontier
    ↓
Chromium / Playwright
    ↓
page.goto()
    ↓
execute JS
    ↓
capture:
  ├── rendered DOM links
  ├── fetch()
  ├── XHR
  ├── document navigations
  ├── redirects
  └── optionally WebSockets
    ↓
URL normalizer
    ↓
dedupe / canonicalization
    ↓
crawl frontier

Then classify discovered URLs:

DISCOVERY_TYPE

html
rendered_dom
xhr
fetch
redirect
history_api
script
stylesheet
image

That classification becomes extremely useful for an SEO agency because you can tell a client:

/products/widget-123 was discovered only after JavaScript execution

versus:

/api/products?page=2 was requested by the browser but never exposed as a crawlable HTML link.

The test I would use when selecting a tool

Don't run a generic benchmark. Give every candidate the same deliberately nasty test site:

1. Static HTML contains /static-product.
2. JS injects /js-product after 2 seconds.
3. fetch() retrieves JSON containing /api-product.
4. Clicking “Load more” triggers another API request.
5. Infinite scroll loads another batch.
6. history.pushState() creates /spa-product.
7. One API request redirects.
8. One request is blocked until a cookie/consent action.
9. Some content arrives after 5–10 seconds.
10. One link exists only inside the API JSON and never becomes an <a>.

Then require the crawler to produce:

URL                         Found via
/static-product             HTML
/js-product                 rendered DOM
/api/products?page=2       fetch
/spa-product                history API
/api/redirect               XHR

That will expose the difference between “has JS rendering” and “actually gives us the network/discovery data we need.”

My practical decision rule

For an SEO agency, I'd use a two-tier setup:

If your requirement is specifically “execute the client-side app, wait for AJAX/API content, then discover URLs that appear only after rendering,” I’d evaluate crawlers on more than just whether they say “JavaScript rendering.”

What I’d test

Google's own rendering process uses a headless Chromium renderer and extracts links again from the rendered HTML, so that is a useful baseline for what you're trying to emulate. Google for Developers

For an SEO agency, I'd start with Screaming Frog

Screaming Frog's SEO Spider is particularly well suited to this use case because its JavaScript mode renders pages in Chromium and crawls both the original HTML and the rendered HTML. It explicitly identifies links that exist only after JavaScript execution. Screaming Frog Screaming Frog

For example, it can flag “Contains JavaScript Links” and let you inspect which outlinks came from rendered HTML; those links can also be bulk exported. Screaming Frog

The configurable AJAX timeout is particularly relevant to your requirement. Its default is 5 seconds, but you can increase it when API calls/content arrive later. Screaming Frog

So I'd use this as your baseline test:

1. Give the crawler a page whose initial HTML contains no product/category links.
2. Have JS call an API.
3. Have the API response populate <a href="/product/...">.
4. Delay the API response by 3–10 seconds.
5. See whether the crawler discovers those URLs.
6. Check whether it records the source page → discovered URL relationship.
7. Repeat with infinite scroll, client-side routing, lazy-loaded content, and cookie-dependent API calls.

One important distinction

“Captures AJAX-loaded URLs” can mean two different things:

A. URLs inserted into the rendered DOM

HTML
  ↓
JavaScript executes
  ↓
fetch("/api/products")
  ↓
JS creates <a href="/products/widget">
  ↓
crawler extracts /products/widget

Most good JS SEO crawlers can handle this. Screaming Frog explicitly does. Screaming Frog Screaming Frog

B. URLs that only appear as API/network requests

HTML
  ↓
JavaScript
  ↓
GET /api/products?page=2
  ↓
JSON response

Those /api/... URLs aren't necessarily links that a search engine would crawl. If your agency wants to inventory the network layer itself, you need a crawler that exposes browser network requests (or build one around Playwright/Puppeteer). That's a different requirement from discovering SEO crawlable links.

For SEO auditing, I'd prioritize A, while having B available for debugging.

My selection rule

I'd score candidate tools like this:

40% — rendering correctness

• Chromium
• JS execution
• AJAX/fetch
• SPA routing
• delayed/lazy content
• cookies

25% — URL discovery

• rendered <a href>
• source-page attribution
• canonical/hreflang discovery
• redirects
• links loaded after interaction

20% — agency scalability

• concurrency
• memory consumption
• crawl scheduling
• API/CLI
• cloud vs desktop
• project persistence

15% — debugging/export

• screenshots
• raw vs rendered HTML
• network information
• custom extraction
• CSV/database/API export

And I'd never choose based on the vendor's “supports JavaScript” checkbox alone. The critical test is whether it waits long enough and extracts the post-AJAX DOM.

One other SEO nuance: Google recommends that important links ultimately exist as normal <a href> links, even though Google can discover links injected by JavaScript. Google for Developers So your crawler should report “JS-only discovered links” rather than simply treating them as equivalent to links present in the original HTML.

If you're comparing Screaming Frog vs Sitebulb vs a custom Playwright crawler vs cloud crawlers, I can give you a practical agency-oriented comparison, including rendering accuracy, AJAX handling, scale, proxy support, cost, and API/automation.

• concurrency
• memory consumption
• crawl scheduling
• API/CLI
• cloud vs desktop
• project persistence

15% — debugging/export

• screenshots
• raw vs rendered HTML
• network information
• custom extraction
• CSV/database/API export

And I'd never choose based on the vendor's “supports JavaScript” checkbox alone. The critical test is whether it waits long enough and extracts the post-AJAX DOM.

For an SEO agency crawling JavaScript-heavy client sites, I would not choose based on “does it render JS?”—many tools do. The important question is:

Can it execute a real browser session, wait for async activity, discover URLs created by JS/AJAX, and give you the evidence needed to diagnose SEO impact?

Here’s how I’d evaluate tools.

1. Define the exact JS discovery problem

Different tools solve different problems:

A lot of SEO crawlers will render the DOM but won’t expose every network request. If “capture AJAX-loaded URLs” is a core requirement, you should test specifically for that.

2. Shortlist tools by use case

Best general SEO crawler: Screaming Frog

screamingfrog.co.uk

Good fit if your agency has technical SEOs who want raw data.

Strengths:

• Chromium-based JS rendering
• Custom extraction (XPath/CSS/regex)
• Strong exports/API workflows
• Good for one-off forensic audits
• Can compare source HTML vs rendered output

Weakness:

• Less “crawl what the browser requested” oriented. You may need custom workflows for AJAX endpoint discovery.

Screaming Frog and Sitebulb both support JavaScript crawling/rendering, but they emphasize different workflows: Screaming Frog tends toward raw crawl data and extraction; Sitebulb emphasizes guided audits and reporting. PikaSEO Sitebulb

Best for agency reporting: Sitebulb

sitebulb.com

Good fit if client-facing audits matter.

Strengths:

• Chromium rendering
• Response vs rendered comparisons
• Good explanations of JS SEO problems
• Easier for junior analysts/client reports

Useful for catching:

• links appearing only after JS
• titles/canonicals changed by JS
• content injected after load

Sitebulb specifically highlights response-vs-render differences, including JS-created or modified links and metadata. Sitebulb Support

Weakness:

• Less of a “developer debugging browser” than a custom Playwright setup.

Best for AJAX/API discovery: build around Playwright

For your exact phrase — “captures AJAX-loaded URLs” — I would seriously consider a custom crawler layer.

A Playwright crawler can:

• launch real Chromium
• intercept request / response
• record XHR/fetch URLs
• wait for network idle
• click menus, filters, pagination
• scroll infinite feeds
• capture generated routes

Example architecture:

URL queue
   ↓
Playwright Chromium worker
   ↓
Page render
   ↓
Capture:
  - DOM links
  - href changes
  - XHR/fetch URLs
  - JSON API responses
  - redirects
  - canonical/meta robots
   ↓
SEO database

This is closer to what you need if clients have:

• ecommerce filters
• JS search
• marketplaces
• travel sites
• SaaS apps
• SPAs

3. Questions to ask vendors before buying

I would run a bake-off with 3–5 difficult URLs and ask:

Rendering

• Does it use real Chromium?
• Can I set a render delay?
• Can it wait for specific selectors?

URL discovery

• Does it extract URLs only from DOM anchors?
• Does it capture fetch() and XHR endpoints?
• Can it record URLs created after user interaction?
• Can it crawl client-side routes?

Scale

• How many rendered pages/hour?
• How many concurrent browsers?
• Can I run multiple projects?
• Is there an API?

SEO validation

Can it report:

• source HTML vs rendered HTML?
• rendered internal links?
• rendered canonicals?
• rendered robots directives?
• rendered structured data?

4. My agency-oriented recommendation

A practical stack:

Most agencies:

• Screaming Frog + custom extraction
• Sitebulb for client reporting
• Playwright for difficult JS investigations

Enterprise SEO agency:

• A cloud crawler for routine monitoring
• Playwright-based crawler for JS edge cases
• Store crawl events in your own database

If AJAX URL discovery is a core selling point:

Skip relying only on an SEO crawler. Use a browser automation crawler that treats network activity as crawl data.

The key test: take a client page where URLs appear only after clicking filters or loading more products. If the tool cannot show you those URLs without manual interaction, it probably is not the right tool for that workload.

The key test: take a client page where URLs appear only after clicking filters or loading more products. If the tool cannot show you those URLs without manual interaction, it probably is not the right tool for that workload.

If your requirement is specifically “execute the site’s JavaScript and discover URLs that appear only after AJAX/fetch/XHR/client-side rendering”, I’d evaluate crawlers on a different axis than ordinary SEO crawling.

What I’d test

The critical capability isn't just “supports JavaScript.” You want:

1. Real browser rendering — Chromium/Chrome, not merely parsing scripts.
2. Rendered-DOM link extraction — URLs inserted after JS execution must become crawl candidates.
3. Delayed AJAX support — it should wait long enough for fetch(), XHR, hydration, infinite-scroll components, etc.
4. Configurable load/wait conditions — load, network-idle-like behavior, custom delays, or equivalent.
5. Raw-vs-rendered comparison — useful for proving which URLs were discovered only after JS.
6. Scale/concurrency — browser rendering is dramatically more expensive than HTTP crawling.
7. URL provenance — ideally you can tell where a URL came from: source HTML, rendered DOM, sitemap, redirect, JS/network activity, etc.
8. Network visibility, if you're doing serious forensic JS SEO — capture XHR/fetch requests, response URLs, status codes, and ideally request/response bodies.

My shortlist

Screaming Frog Sitebulb Support Oncrawl Help### For an SEO agency, I'd start with Screaming Frog or Sitebulb

Screaming Frog is particularly straightforward: its JavaScript rendering mode crawls both the original and rendered HTML and can identify links that exist only in the rendered HTML. Screaming Frog

Sitebulb does something similar with its Chrome Crawler and Response vs Render reporting. It specifically surfaces links created or modified by JavaScript. Sitebulb Support Sitebulb Support

Sitebulb also exposes render-time settings, which matters for AJAX-heavy sites: the crawler can wait beyond the load event before taking its HTML snapshot, giving asynchronous content more time to appear. Sitebulb Support

screamingfrog.co.uk
sitebulb.com

But there's an important distinction

Suppose the page does this:

Initial HTML
   ↓
React hydrates
   ↓
fetch("/api/products?page=2")
   ↓
JSON arrives
   ↓
JS creates <a href="/product/123">
   ↓
Crawler discovers /product/123

A conventional rendered-DOM crawler should discover /product/123.

But it doesn't necessarily mean the crawler will give you a useful record of:

GET /api/products?page=2
200
→ JSON contained product URL
→ JS transformed it into /product/123

If capturing the AJAX/network URLs themselves is part of your requirement, rather than merely discovering the resulting <a> URLs, I'd move toward a Playwright/Chromium-based crawler.

That's where you can instrument the browser's network layer and record:

• fetch() requests
• XHR requests
• API endpoints
• redirects
• response status
• response URLs
• requests triggered by scrolling/clicking
• URLs subsequently inserted into the DOM

That gives you two separate discovery graphs:

NETWORK DISCOVERY
page
 ├── API request
 ├── API request
 └── JS asset

DOM DISCOVERY
page
 ├── /category/a
 ├── /product/123
 └── /product/456

For an agency building a repeatable crawling platform, that's considerably more powerful than simply turning on “JavaScript rendering.”

The benchmark I'd use

Before buying/standardizing on anything, give each crawler 5–10 deliberately nasty URLs:

• links present in initial HTML
• links injected immediately by JS
• links injected after a 2–5 second AJAX request
• links appearing after fetch() returns
• infinite-scroll URLs
• links behind a “Load more” interaction
• SPA route changes
• URLs generated from API JSON
• content/links inside Shadow DOM
• service-worker-mediated content

Then compare:

A. Raw HTML URLs

B. Rendered DOM URLs

C. Network-request URLs

D. Final crawl graph

The killer metric for your use case is:

What percentage of URLs discoverable by a fully instrumented browser does the commercial crawler actually discover?

That's much more meaningful than a vendor saying “supports JavaScript.”

Oncrawl, for example, explicitly supports JS crawling and has API configuration for enabling JS rendering, while also exposing crawl-speed controls. Oncrawl Help Oncrawl Developer

My recommendation

If you need an agency tool today: test Screaming Frog + Sitebulb first.

If you're crawling millions of URLs or need centralized/cloud crawling: evaluate Oncrawl alongside them.

If “capture every AJAX/API URL and understand how it produced the final URL graph” is a core product requirement: build/evaluate a Playwright/Chromium crawler rather than relying solely on an SEO crawler.

The key question I'd ask vendors is not “Do you render JavaScript?” but:

“After rendering, exactly what constitutes a discovered URL, how long do you wait for asynchronous requests, can I configure the wait condition, and can I export the network requests separately from DOM-discovered links?”

That question will expose most of the meaningful differences very quickly.

The killer metric for your use case is:

What percentage of URLs discoverable by a fully instrumented browser does the commercial crawler actually discover?

That's much more meaningful than a vendor saying “supports JavaScript.”

Oncrawl, for example, explicitly supports JS crawling and has API configuration for enabling JS rendering, while also exposing crawl-speed controls.

My recommendation

If your agency’s requirement is specifically “execute JS, wait for async/AJAX work, then discover the URLs that appear”, I’d evaluate crawlers on a different axis than simply “does it render JavaScript?”

What I’d test

• Rendered DOM URL discovery — Does it extract <a href> URLs after JavaScript executes, not just from the initial HTML?
• Async/AJAX wait behavior — Can you control how long the browser waits after load? This matters for APIs that populate content 1–5 seconds after the initial page load.
• Network activity — Ideally you can inspect XHR/fetch requests separately from links inserted into the DOM. These are different datasets.
• SPA navigation — Test history.pushState, client-side routing, infinite scroll, “Load more,” and click-triggered content.
• Rendering at scale — A tool that renders one page perfectly but takes 20× longer per URL can be painful for agency-sized crawls.
• Raw vs rendered comparison — You want to prove which URLs only exist after rendering, rather than simply getting a larger URL list.
• Custom interaction support — If discovering URLs requires clicking tabs/buttons or scrolling, basic headless rendering may not be sufficient.
• Export/API access — Important if you want to feed discovered URLs into your own SEO pipeline.

My shortlist

Screaming Frog is a strong choice if you want maximum control on the desktop. Its JavaScript rendering mode uses headless Chromium and extracts content/links from rendered HTML, while also discovering links from the raw HTML. Screaming Frog Screaming Frog

Sitebulb is particularly attractive for an agency workflow because its Chrome Crawler compares response HTML with rendered HTML and explicitly reports links created or modified by JavaScript. Sitebulb Support Sitebulb Support It also exposes render timeout and load-event controls, which are important for AJAX-heavy sites. Sitebulb Support Sitebulb Support

The important distinction: AJAX URLs vs AJAX requests

I'd make this a hard requirement in your evaluation.

Suppose the initial HTML is:

<div id="products"></div>
<script src="/app.js"></script>

Then JS does:

GET /api/products?page=1
GET /api/products?page=2

and injects:

<a href="/products/widget-a">Widget A</a>
<a href="/products/widget-b">Widget B</a>

There are three different things you might want to capture:

1. /products/widget-a and /products/widget-b — URLs discovered in the rendered DOM.
2. /api/products?page=1 — network/AJAX endpoint requested by the browser.
3. The data returned by that endpoint.

Most SEO crawlers are primarily solving #1, not acting as a full browser network recorder for #2/#3. That's why I'd run your own acceptance test rather than relying on a vendor saying “JavaScript supported.”

A practical agency bake-off

Take 5–10 representative client sites and create a test set containing:

• React/Next/Vue pages
• Links injected after API calls
• Links appearing 2–5 seconds after DOMContentLoaded
• Infinite-scroll URLs
• “Load more” pagination
• Client-side routing with pushState
• Links that exist only after a click
• API calls returning URLs that never become <a> elements

Then score each tool:

If your core goal is SEO URL discovery, I'd start with Screaming Frog vs. Sitebulb. Both genuinely execute Chromium-based JavaScript; Sitebulb has particularly nice response-vs-render reporting, while Screaming Frog gives you a lot of low-level crawl configuration and JavaScript diagnostics. Screaming Frog Sitebulb Support

If by “captures AJAX-loaded URLs” you specifically mean “show me every URL requested by fetch()/XHR, including endpoints that never appear as links”, that's a different requirement. In that case I'd look at a Playwright/Puppeteer-based crawler with network interception, rather than choosing an SEO crawler solely on its JS-rendering feature.

For an SEO agency, the key distinction is “does it render JavaScript?” vs “does it behave like a browser and discover URLs created after JS execution?”. A lot of crawlers can render a DOM snapshot, but fewer are good at capturing AJAX/fetch/XHR-generated URLs, client-side routing, infinite scroll, and interaction-triggered links.

I’d evaluate tools against these capabilities:

1. Define your crawl requirements first

Make a checklist:

Many SEO crawlers will satisfy the first two but not the network-level discovery requirement. Screaming Frog support.sitebulb.com

2. Shortlist by use case

Best all-around SEO crawler: Screaming Frog SEO Spider

Good fit if you need:

• agency-standard audits
• rendered HTML comparisons
• JavaScript SEO diagnostics
• exports into Excel/BigQuery/custom pipelines

It has a JavaScript rendering mode using a browser engine and can compare original HTML vs rendered output to identify JS-created links and content. Screaming Frog

Watch-outs:

• It is not primarily a network inspector.
• If your goal is “capture every API endpoint called by the app,” you may need something alongside it.

Best SEO-focused JS auditing: Sitebulb

Strong choice for client-facing audits.

Advantages:

• Chromium-based rendering
• response vs rendered HTML comparisons
• identifies links changed or created by JavaScript
• better explanations/reporting for clients

Its Chrome Crawler renders pages and can surface links affected by JavaScript. support.sitebulb.com support.sitebulb.com

Watch-outs:

• Like most SEO crawlers, it is not a full browser automation/network analysis platform.

Best for enterprise-scale crawling: Lumar or Botify

Consider these if you have:

• millions of URLs
• scheduled enterprise crawls
• log file integration
• multiple client accounts

They are better operationally, but verify exactly what JS rendering features are included in your plan. Some enterprise crawlers treat rendering as an advanced capability. Sitebulb

3. If you specifically need AJAX-loaded URLs, add a browser crawler

For your exact requirement (“captures AJAX-loaded URLs”), I would not rely only on an SEO crawler.

Look at browser automation:

• Playwright
• Puppeteer

A custom crawler can:

1. Open the page in Chromium.
2. Wait for hydration/network idle.
3. Listen to: - request
4. response
5. fetch
6. XHR calls
7. WebSocket traffic
8. Extract: - API endpoints
9. JSON responses
10. dynamically generated URLs
11. SPA routes

Example workflow:

URL list
   ↓
Playwright Chromium
   ↓
Render JS
   ↓
Capture network events
   ↓
Extract:
 - hrefs
 - API URLs
 - JSON payload URLs
 - canonical changes
 - rendered DOM links
   ↓
SEO crawl database

This is usually what agencies build when crawling very JS-heavy ecommerce, marketplaces, or SaaS apps.

4. Test vendors with a “JS torture page”

Before buying, create a test site containing:

• Links injected after 5 seconds
• Products loaded via fetch()
• Infinite scroll
• React router navigation
• “Load more” buttons
• API calls returning URLs
• Canonical/title changes after JS

Then compare:

• URLs discovered
• rendered HTML
• API calls captured
• crawl depth reached

A tool that finds the visible content but misses the API-driven URL graph will give you false crawl coverage.

My practical agency stack

For most SEO agencies:

• Routine audits: Screaming Frog or Sitebulb
• Large scheduled audits: Lumar/Botify
• Hard JS apps: custom Playwright crawler feeding your SEO pipeline

If your clients are mostly React/Next.js ecommerce sites, I would prioritize network interception + rendered crawling rather than just “JS rendering.” That is the difference between seeing the page and actually understanding the application.

• URLs discovered
• rendered HTML
• API calls captured
• crawl depth reached

A tool that finds the visible content but misses the API-driven URL graph will give you false crawl coverage.

My practical agency stack

For most SEO agencies:

• Routine audits: Screaming Frog or Sitebulb
• Large scheduled audits: Lumar/Botify
• Hard JS apps: custom Playwright crawler feeding your SEO pipeline

If your clients are mostly React/Next.js ecommerce sites, I would prioritize network interception + rendered crawling rather than just “JS rendering.” That is the difference between seeing the page and actually understanding the application.

If you're choosing a crawler for an SEO agency, I’d evaluate it less on “does it execute JavaScript?” and more on “does it discover the URLs that exist only after JavaScript executes?”

Google itself renders pages with headless Chromium and then parses the rendered HTML for additional links, so that’s a useful benchmark for your crawler. Google Developers

What I’d test

For each candidate tool, run the same deliberately difficult client site and measure:

• Initial HTML URLs — links present in the HTTP response.
• Rendered DOM URLs — links appearing after JS execution.
• AJAX/API discovery — whether content loaded by fetch, XHR, GraphQL, etc. ultimately produces crawlable URLs.
• Delayed rendering — whether it waits long enough for hydration/API calls to finish.
• Interaction-dependent URLs — URLs exposed only after clicking “Load more,” expanding menus, scrolling, tabs, etc.
• URL extraction from the rendered DOM — especially dynamically inserted <a href> elements.
• Response vs. rendered comparison — extremely useful for agency audits.
• Rendering failures — JS errors, blocked resources, timeouts, failed API calls.
• Scale — URLs/hour when Chromium rendering is enabled.
• Concurrency/resource controls — number of browser instances, threads, memory limits, etc.
• Export/API access — whether you can dump discovered URLs into your own pipeline.

The important distinction is that seeing AJAX requests in the Network tab isn't the same thing as discovering the URLs generated by those requests. If an API returns 500 products but the browser turns those into <a> elements, your SEO crawler needs to capture the resulting links if your goal is site discovery.

Tools I'd shortlist

1. Sitebulb — strong agency-oriented choice

sitebulb.com has a dedicated Chrome Crawler that uses headless Chrome and renders JavaScript. It also compares response HTML with rendered HTML and can identify links created or modified by JavaScript. Sitebulb Support Sitebulb Support

That's particularly attractive for an agency because you can answer client questions such as:

“These 4,200 URLs aren't present in the source HTML; they're being created after JavaScript execution.”

Sitebulb also exposes rendering timeout and Chrome resource controls, which matter considerably on slow SPAs. Sitebulb Support Sitebulb Support

I'd put this near the top if you want an SEO product rather than a crawling framework.

2. Screaming Frog — excellent for hands-on technical SEO

screamingfrog.co.uk is another obvious benchmark. Its JavaScript rendering mode uses a browser to execute JS, making it useful when you're investigating exactly what gets added after load.

I'd particularly test it against your hardest clients rather than assuming that “JavaScript rendering = AJAX discovery.”

3. Build your own Playwright crawler — best if this is core infrastructure

If your agency wants maximum control, I'd seriously consider building the rendering layer around playwright.dev.

You can capture:

HTTP response
      ↓
initial HTML
      ↓
Chromium / Playwright
      ↓
JS execution
      ↓
fetch/XHR/GraphQL activity
      ↓
rendered DOM
      ↓
<a href> extraction
      ↓
new URL queue

That gives you substantially more control than an SEO crawler over things like:

• waiting for specific API responses
• network-idle vs. DOM-ready behavior
• clicking UI elements
• infinite scroll
• authentication
• cookies/local storage
• custom JavaScript
• intercepting requests
• recording failed requests
• extracting URLs directly from API responses
• reproducing Googlebot-like rendering conditions

The tradeoff is that you now own browser orchestration, queueing, deduplication, retries, resource management, proxying, and crawl politeness.

The test I'd actually run

I'd create a JS torture-test page for your evaluation:

<!-- URL 1: present in source -->
<a href="/products/a">A</a>

<script>
  // URL 2: inserted immediately
  document.body.insertAdjacentHTML(
    "beforeend",
    '<a href="/products/b">B</a>'
  );

  // URL 3: inserted after API response
  fetch("/api/products")
    .then(r => r.json())
    .then(products => {
      products.forEach(p => {
        document.body.insertAdjacentHTML(
          "beforeend",
          `<a href="/products/${p.slug}">${p.name}</a>`
        );
      });
    });

  // URL 4: only appears after interaction
  document.querySelector("#load-more").onclick = () => {
    // fetch and insert more links
  };
</script>

Then give every crawler the same starting URL.

Your results matrix should look something like:

This test will tell you far more than vendor feature pages.

One subtle but important SEO point

Don't define success as “we captured every URL requested by AJAX.”

A crawler could capture:

/api/products?page=17
/api/recommendations?id=123
/graphql
/analytics/event

Those aren't necessarily pages you want in your crawl graph.

For SEO crawling, I'd define the primary output as:

URLs that become discoverable through browser rendering and could constitute crawlable resources/pages.

In particular, prioritize URLs represented by actual links in the rendered DOM. Google says it parses rendered HTML for links after rendering, which makes that a much more meaningful benchmark than simply recording network traffic. Google Developers

My practical recommendation

For an agency, I'd start with Sitebulb and Screaming Frog as your off-the-shelf comparison, and run a small Playwright proof-of-concept if AJAX-heavy sites are a major part of your business.

If the requirement is specifically “crawl huge JS sites and build the most complete URL graph possible,” I'd lean toward a custom Chromium/Playwright crawler. If the requirement is “produce client-ready technical SEO audits with strong JS diagnostics,” I'd lean toward Sitebulb. Sitebulb specifically exposes JS-created/modified links and response-vs-render differences, which maps unusually well to agency SEO work. Sitebulb Support Sitebulb Support

developers.google.com is also worth using as the baseline for your crawler's rendering/discovery behavior.

I'd put this near the top if you want an SEO product rather than a crawling framework.

2. Screaming Frog — excellent for hands-on technical SEO

screamingfrog.co.uk is another obvious benchmark. Its JavaScript rendering mode uses a browser to execute JS, making it useful when you're investigating exactly what gets added after load.

I'd particularly test it against your hardest clients rather than assuming that “JavaScript rendering = AJAX discovery.”

3. Build your own Playwright crawler — best if this is core infrastructure

If your agency wants maximum control, I'd seriously consider building the rendering layer around playwright.dev.

You can capture:

HTTP response
      ↓
initial HTML
      ↓
Chromium / Playwright
      ↓
JS execution
      ↓
fetch/XHR/GraphQL activity
      ↓
rendered DOM
      ↓
<a href> extraction
      ↓
new URL queue

That gives you substantially more control than an SEO crawler over things like:

For an SEO agency, I’d choose based less on “does it render JS?” (many do now) and more on whether it can discover the URLs created after rendering. You want a crawler that behaves like a browser, captures network activity, and lets you export the discovered URL graph. Google itself also separates crawling and rendering, so a tool that only fetches raw HTML can miss links/content generated after JavaScript execution. Google Developers

The capabilities I would require

1. Full browser rendering (not just HTML parsing)

Look for:

• Chromium-based rendering
• Execution of: - React/Vue/Angular hydration
• fetch()
• XHR
• GraphQL calls
• lazy-loaded components
• client-side routing
• Ability to compare: - raw HTML
• rendered DOM
• extracted links

A “JS enabled” checkbox is not enough. You want evidence that the crawler waits for the application to settle.

2. Network request capture

This is the big differentiator for AJAX-heavy sites.

The crawler should expose:

• requests made after page load
• API endpoints called
• response URLs
• redirects
• JSON payload URLs
• resources loaded after interaction

Useful data points:

Initial URL
    ↓
Render page
    ↓
XHR:
  /api/products?page=2
  /graphql/query
  /search/results
    ↓
DOM mutation
    ↓
New URLs discovered:
  /product/a
  /product/b

Many SEO crawlers only see the final DOM links. For diagnosing crawlability problems, the network layer is often more valuable.

3. Interaction support

For modern sites, ask whether it can:

• scroll
• click menus
• expand accordions
• submit forms
• trigger filters
• wait for infinite scroll

A product listing that only appears after clicking “Load more” is a classic failure case.

4. Crawl scale

Your agency use case changes the answer.

Tools I’d shortlist

Best technical control: custom Playwright crawler

For an SEO agency that audits difficult JS sites, this is often the gold standard.

Stack:

• Playwright
• Chromium
• request interception
• DOM snapshots
• URL extraction
• queue management

You can capture:

• every network request
• every DOM link
• SPA route changes
• API endpoints

Downside: you own infrastructure and reporting.

Best agency crawler: Screaming Frog SEO Spider

Strong choice for technical audits. It can render JavaScript using Chromium and is widely used for JS-heavy SEO investigations. theStacc

Good for:

• client audits
• comparing rendered vs source HTML
• extracting custom data
• finding JS-generated links

Weakness:

• not designed as a distributed browser crawler
• huge JS sites can become resource intensive

Best cloud enterprise options

For large clients, look at:

• Lumar — enterprise crawling, rendering, automation
• Oncrawl — strong when you combine crawling with log analysis
• JetOctopus — useful for large-scale SEO crawling and analytics

These are better when you need scheduled crawls across many client sites rather than one-off investigations. AI Agents Directory

Good if you already use the ecosystem

Ahrefs Site Audit supports JavaScript rendering and can compare rendered output against raw HTML. Ahrefs

My agency-oriented decision framework

I’d test candidates with a deliberately difficult benchmark site:

1. A React SPA with client-side routing
2. A product grid loaded via API
3. Infinite scroll
4. Filters that change URLs
5. Content hidden behind tabs
6. JS-generated canonical/meta tags

Score each tool:

If I were building an SEO agency stack

I’d probably use:

• Screaming Frog → daily technical audits and client work
• Playwright crawler → hard JS debugging and custom audits
• Lumar/Oncrawl/JetOctopus → enterprise recurring crawls
• Google Search Console URL Inspection → final “what Google sees” validation

The key question I’d ask vendors is:

“Can your crawler give me the complete URL set discovered after JavaScript execution, including URLs only reachable through XHR/fetch responses and client-side routing?”

If the answer is “we render JS,” ask the follow-up. Rendering alone does not guarantee URL discovery.

For an SEO agency, the key question is not just “does it render JS?” — it’s what exactly does it capture after rendering? Many tools execute JavaScript but differ on whether they:

• discover links injected after load
• capture fetch/XHR/AJAX-loaded URLs
• wait long enough for hydration/infinite scroll/client routing
• compare raw HTML vs rendered DOM
• scale to thousands/millions of URLs

A good selection framework:

1. Define your crawl requirement first

For heavy JS clients, you usually need one or more of these:

A crawler that only gives you rendered HTML may still miss the network layer where URLs are discovered.

2. Shortlist tools by use case

Best all-around SEO agency crawler: Screaming Frog

Good fit if you need:

• technical SEO audits
• rendered HTML extraction
• JS-injected links
• exports for client reports

It supports JavaScript rendering through a headless Chromium engine and can identify content and links that only appear after JS execution. Screaming Frog

Watch-outs:

• Large JS sites can become resource-heavy.
• It is more “SEO spreadsheet” oriented than “browser debugging” oriented.

Best for JS SEO diagnostics and client reporting: Sitebulb

Strong choice for agencies because it emphasizes:

• response HTML vs rendered HTML comparisons
• identifying JS-generated titles, canonicals, links, and content
• visual explanations clients understand

Its Chrome Crawler uses headless Chrome rendering and can compare response vs rendered output. Sitebulb Support Sitebulb Support

Watch-outs:

• Not the first choice if your main goal is crawling every AJAX endpoint at huge scale.

Best for enterprise JS crawling: JetOctopus

Consider it if you have:

• large ecommerce sites
• millions of URLs
• cloud crawling requirements
• recurring automated audits

It focuses heavily on JS crawling, including raw vs rendered comparisons and detecting JS-generated content/links. SEO Platform

Best if you need total control: custom Playwright/Puppeteer crawler

For an SEO agency doing advanced audits, this is often the missing piece.

A custom crawler can:

• launch Chromium
• intercept: - request
• response
• XHR
• fetch
• GraphQL
• wait for network idle or custom events
• capture: - discovered URLs
• rendered DOM
• screenshots
• console errors
• failed resources

Example workflow:

URL list
   ↓
Playwright Chromium
   ↓
Capture network requests
   ↓
Capture rendered DOM
   ↓
Extract links
   ↓
Store crawl graph

This is what I would build if your agency sells “JS SEO audits” as a specialty.

3. Test tools against a real client site

Before committing, take 5–10 difficult URLs:

Include:

• React/Vue/Angular pages
• infinite scroll pages
• product/category pages
• pages where links load after API calls
• pages with client-side redirects

Measure:

Link discovery test

Raw HTML contains:

0 internal links

Rendered page contains:

500 internal links

Does the crawler find those 500?

AJAX discovery test

Open Chrome DevTools → Network → Fetch/XHR.

Look for:

/api/products?page=2
/graphql
/search?query=

Does your crawler expose or capture these?

Timing test

Some sites render in stages:

0s HTML shell
2s React hydration
5s API data
8s recommendations widget

A crawler with a 2-second timeout may produce false SEO issues.

4. My typical agency stack recommendation

For a professional SEO agency:

Small/medium client audits

• Screaming Frog + Sitebulb

Enterprise clients

• JetOctopus or another cloud crawler
• plus custom Playwright scripts for edge cases

JS-heavy SPA specialists

• Build your own Playwright-based crawler layer
• feed results into your reporting system

One important distinction: if your goal is specifically “capture AJAX-loaded URLs”, prioritize tools that expose browser network events, not only rendered HTML. Many SEO crawlers are good at seeing the final DOM but are not designed as API/request discovery tools. SEO Platform

A useful vendor test question is:

“Can your crawler export every URL requested through XHR/fetch during rendering, including URLs that never appear as links in the final DOM?”

If the answer is no, it may not solve the hardest JS crawl cases.

Look for:

/api/products?page=2
/graphql
/search?query=

Does your crawler expose or capture these?

Timing test

Some sites render in stages:

0s HTML shell
2s React hydration
5s API data
8s recommendations widget

A crawler with a 2-second timeout may produce false SEO issues.

4. My typical agency stack recommendation

For a professional SEO agency:

Small/medium client audits

• Screaming Frog + Sitebulb

Enterprise clients

• JetOctopus or another cloud crawler
• plus custom Playwright scripts for edge cases

JS-heavy SPA specialists

• Build your own Playwright-based crawler layer
• feed results into your reporting system

One important distinction: if your goal is specifically “capture AJAX-loaded URLs”, prioritize tools that expose browser network events, not only rendered HTML. Many SEO crawlers are good at seeing the final DOM but are not designed as API/request discovery tools.

A useful vendor test question is:

“Can your crawler export every URL requested through XHR/fetch during rendering, including URLs that never appear as links in the final DOM?”

Yes. For an SEO agency, I’d evaluate JS crawlers less on “does it run Chrome?” and more on “what exactly does it discover after Chrome runs, and can I prove where each URL came from?”

Google itself renders pages in headless Chromium and parses the rendered HTML for additional links, so that’s the behavior you generally want to approximate. Google for Developers

The capabilities I'd require

The important distinction: AJAX-loaded URLs

There are actually three different things people mean by this:

1. An AJAX request returns data containing URLs

Example: /api/products returns 100 product records.
A crawler needs to execute the JS and/or inspect network responses if those URLs never become DOM links.

2. JS takes AJAX data and creates <a href> elements

This is much easier. A rendered crawler can execute the page and extract the resulting links.

3. URLs only appear after an interaction

Example: clicking “Load more” triggers an API request and inserts another 50 links.
Merely rendering the page isn't enough—you need interaction automation.

That distinction should be part of your vendor test.

My shortlist

1. screamingfrog.co.uk — best for granular technical investigation

This would be my first choice if your agency wants to debug exactly what's happening.

Its JavaScript mode runs pages in headless Chrome and extracts content/links from the rendered HTML while also discovering links in the raw HTML. Screaming Frog

Particularly relevant to your use case: Screaming Frog can identify links that exist only in rendered HTML, and you can bulk-export those JavaScript-dependent links. Screaming Frog

It also exposes an AJAX timeout, which determines how long Chrome is allowed to execute before the rendered snapshot is taken. Screaming Frog

I'd choose it when: your SEO team wants maximum control and detailed debugging.

2. sitebulb.com — best for agency audits/reporting

Sitebulb's Chrome Crawler uses Chromium and compares response HTML with rendered HTML. Its Response vs Render reporting specifically surfaces links created or modified by JavaScript. Sitebulb Support Sitebulb Support

That's a very nice workflow for client work:

Response HTML → render → compare → identify JS-dependent URLs/content → explain SEO impact.

It also has desktop and cloud options, which is useful if you're moving from individual audits toward agency-scale crawling. Sitebulb

I'd choose it when: presentation, diagnostics and repeatable client audits matter as much as raw crawling control.

3. Enterprise crawlers

For very large clients, I'd also evaluate Lumar, Botify, and Oncrawl. The question there isn't simply whether they render JS—it is how much control they give you over rendering, crawl scheduling, extraction, network behavior, and scale. JavaScript rendering can have significant resource implications, so enterprise architecture becomes important. Sitebulb

The test I'd run before buying anything

Don't give vendors a normal HTML site. Build a JS torture-test site containing:

/page-1
  ├── normal HTML link
  ├── JS-injected <a>
  ├── AJAX → JSON → <a>
  ├── delayed AJAX → <a>
  ├── link behind "Load more"
  ├── link behind click interaction
  ├── URL generated only in JavaScript
  ├── link added after 10 seconds
  └── API response containing URLs that never enter the DOM

Then measure:

Discovery accuracy

Expected URLs: 1,000
Discovered from raw HTML: 400
Discovered after rendering: 950
Discovered from network/API only: 50

You want the crawler to make those distinctions visible.

Also test:

• Does it wait for network idle or just a fixed timeout?
• Can you increase the render timeout?
• Can it execute clicks?
• Can it scroll?
• Can it handle infinite scroll?
• Does it extract links from the rendered DOM?
• Can it capture XHR/fetch responses?
• Can it export those network URLs?
• Does it distinguish href links from URLs merely present in JSON?
• Can you run 10,000+ rendered pages without the crawl becoming impractical?
• Can you throttle requests to protect clients?
• Can you authenticate into staging sites?
• Can you supply custom headers/cookies?
• Can you save screenshots and rendered HTML?

One particularly important trap

Don't equate “renders JavaScript” with “captures AJAX URLs.”

A conventional JS crawler may do:

GET page
    ↓
launch Chromium
    ↓
execute JS
    ↓
AJAX request
    ↓
JSON returned
    ↓
JS creates <a href="/product/123">
    ↓
crawler extracts /product/123

Great.

But this:

GET page
    ↓
launch Chromium
    ↓
AJAX request
    ↓
JSON contains "/product/123"
    ↓
JS uses URL internally
    ↓
NO <a> element

is a different problem.

If your agency specifically wants URLs discovered from API/XHR responses, ask vendors whether they expose network interception / request-response logging. That's beyond ordinary rendered-HTML crawling.

For pure SEO crawling, however, I'd generally prioritize URLs that become crawlable links in the rendered DOM, because that's much closer to how Google discovers links after rendering. Google for Developers

My practical recommendation

For an agency, I'd probably standardize on:

Screaming Frog → deep technical JS investigation and URL-level debugging
Sitebulb → recurring client audits, JS-vs-response reporting and easier presentation
Custom Playwright/Chromium crawler → only when you specifically need network-level URL capture, complex interactions, or a bespoke pipeline

And I'd make “response HTML vs rendered DOM vs network URLs” three separate datasets in your internal crawling architecture. That separation makes JS SEO problems dramatically easier to diagnose.

Don't give vendors a normal HTML site. Build a JS torture-test site containing:

/page-1
  ├── normal HTML link
  ├── JS-injected <a>
  ├── AJAX → JSON → <a>
  ├── delayed AJAX → <a>
  ├── link behind "Load more"
  ├── link behind click interaction
  ├── URL generated only in JavaScript
  ├── link added after 10 seconds
  └── API response containing URLs that never enter the DOM

Then measure:

Discovery accuracy

Expected URLs: 1,000
Discovered from raw HTML: 400
Discovered after rendering: 950
Discovered from network/API only: 50

You want the crawler to make those distinctions visible.

Also test:

• Does it wait for network idle or just a fixed timeout?
• Can you increase the render timeout?
• Can it execute clicks?
• Can it scroll?
• Can it handle infinite scroll?
• Does it extract links from the rendered DOM?
• Can it capture XHR/fetch responses?
• Can it export those network URLs?
• Does it distinguish href links from URLs merely present in JSON?
• Can you run 10,000+ rendered pages without the crawl becoming impractical?
• Can you throttle requests to protect clients?
• Can you authenticate into staging sites?
• Can you supply custom headers/cookies?
• Can you save screenshots and rendered HTML?

One particularly important trap

Don't equate “renders JavaScript” with “captures AJAX URLs.”

A conventional JS crawler may do:

GET page
    ↓
launch Chromium
    ↓
execute JS
    ↓
AJAX request
    ↓
JSON returned
    ↓
JS creates <a href="/product/123">
    ↓
crawler extracts /product/123

Great.

But this:

GET page
    ↓
launch Chromium
    ↓
AJAX request
    ↓
JSON contains "/product/123"
    ↓
JS uses URL internally
    ↓
NO <a> element

is a different problem.

If your agency specifically wants URLs discovered from API/XHR responses, ask vendors whether they expose network interception / request-response logging. That's beyond ordinary rendered-HTML crawling.

For pure SEO crawling, however, I'd generally prioritize URLs that become crawlable links in the rendered DOM, because that's much closer to how Google discovers links after rendering.

My practical recommendation

For an agency, I'd probably standardize on:

If you're an SEO agency, I’d evaluate this less as “does it render JavaScript?” and more as “does it reproduce the crawl/discovery consequences of JavaScript?” AJAX-loaded URLs are the important distinction.

What I’d put on the shortlist

The tests I'd use to choose

Don't benchmark them on a normal WordPress site. Give each candidate a deliberately nasty JS site and measure these:

1. Rendered DOM discovery

• Page initially contains no <a> to /products/widget-a.
• JS fetches an API response.
• JS creates the product links.
• Does the crawler discover /products/widget-a?
• Network/AJAX discovery

• The page makes fetch()/XHR calls returning JSON containing URLs.
• Determine whether the tool merely renders the resulting DOM or also exposes the network requests and their responses.
• This matters because an API can contain URLs that never become <a> elements.
• Delayed rendering

• API response arrives after 3–10 seconds.
• Test configurable render/wait time.
• A crawler that snapshots too early will give you false negatives.
• Interactions

• “Load more”
• Infinite scroll
• Tabs
• Click-to-expand navigation
• Cookie/consent overlays
• Login/session-dependent content
• URL attribution

You want to know not merely that a URL was found, but:

• raw HTML link
• rendered HTML link
• JS-generated link
• HTTP redirect
• canonical
• sitemap
• perhaps network/API discovery
• Raw vs rendered comparison

This is crucial for SEO audits. You want both versions so you can say, for example:

“2,413 internal links exist only after JavaScript execution.”

My first two tests would be Screaming Frog and Sitebulb

Screaming Frog SEO Spider explicitly identifies links that exist only after JavaScript execution and lets you export those URLs in bulk. Its workflow can distinguish rendered-H​​TML link origins from the raw HTML. Screaming Frog

Sitebulb is particularly interesting for an agency because its Chrome Crawler renders the DOM and compares it against the original response, including links created or modified by JavaScript. Sitebulb Support Sitebulb Support

Sitebulb also has a cloud version capable of crawling up to 10 million URLs per audit, which becomes relevant if you're doing this across many large client sites. Sitebulb

For very large clients, I'd also benchmark JetOctopus. Its JS crawler specifically exposes AJAX requests, JS-injected content, blocked resources and client-side redirects, and is designed for cloud-scale crawling. Tech SEO Platform

One important distinction

If by “captures AJAX-loaded URLs” you mean:

“The AJAX response contains /category/shoes, and I want the crawler to discover that URL even though the page never creates an <a href> for it”

then don't assume a JavaScript-rendering crawler does this.

There are actually three levels:

Level 1 — raw HTML

HTML → <a href="/products/a">

Level 2 — rendered DOM

HTML
  ↓
JavaScript executes
  ↓
<a href="/products/a">
  ↓
crawler discovers URL

Level 3 — network/API discovery

HTML
  ↓
JavaScript executes
  ↓
fetch("/api/products")
  ↓
JSON contains "/products/a"
  ↓
crawler records URL even though no <a> exists

For SEO, Level 2 is usually the key requirement because that's how JS-generated internal links become crawlable links. But Level 3 is extremely useful diagnostically because it lets you identify URLs/content being fetched but never exposed as crawlable HTML links.

For an agency, I'd score tools like this

I'd make a 100-point internal benchmark:

• 25 — rendered-link discovery accuracy
• 20 — AJAX/network visibility
• 15 — raw-vs-rendered comparison
• 15 — configurable render/wait behavior
• 10 — scale/crawl speed
• 5 — custom JS/interactions
• 5 — exports/API
• 5 — client-friendly reporting

And don't benchmark just crawl completion. Compare the discovered URL set against a ground-truth URL set. That's the metric that will tell you whether you're actually capturing the site architecture.

If your clients are predominantly React/Next/Vue/Angular sites with API-driven product/category discovery, I'd lean toward a Screaming Frog + Sitebulb/JetOctopus bake-off, with the AJAX/network-discovery requirement tested separately rather than treating “supports JavaScript” as sufficient.

If by “captures AJAX-loaded URLs” you mean:

“The AJAX response contains /category/shoes, and I want the crawler to discover that URL even though the page never creates an <a href> for it”

then don't assume a JavaScript-rendering crawler does this.

There are actually three levels:

Level 1 — raw HTML

HTML → <a href="/products/a">

Level 2 — rendered DOM

HTML
  ↓
JavaScript executes
  ↓
<a href="/products/a">
  ↓
crawler discovers URL

Level 3 — network/API discovery

HTML
  ↓
JavaScript executes
  ↓
fetch("/api/products")
  ↓
JSON contains "/products/a"
  ↓
crawler records URL even though no <a> exists

For SEO, Level 2 is usually the key requirement because that's how JS-generated internal links become crawlable links. But Level 3 is extremely useful diagnostically because it lets you identify URLs/content being fetched but never exposed as crawlable HTML links.

For an agency, I'd score tools like this

I'd make a 100-point internal benchmark:

• 25 — rendered-link discovery accuracy
• 20 — AJAX/network visibility
• 15 — raw-vs-rendered comparison
• 15 — configurable render/wait behavior
• 10 — scale/crawl speed
• 5 — custom JS/interactions
• 5 — exports/API
• 5 — client-friendly reporting

And don't benchmark just crawl completion. Compare the discovered URL set against a ground-truth URL set. That's the metric that will tell you whether you're actually capturing the site architecture.

If your clients are predominantly React/Next/Vue/Angular sites with API-driven product/category discovery, I'd lean toward a Screaming Frog + Sitebulb/JetOctopus bake-off, with the AJAX/network-discovery requirement tested separately rather than treating “supports JavaScript” as sufficient.

If your agency’s requirement is specifically “execute JS and discover URLs that only appear after XHR/fetch/AJAX calls,” don’t evaluate crawlers on “does it render JavaScript?” alone. That’s only half the requirement.

What I’d evaluate

Use a test site/client site with these cases:

• Links present in initial HTML.
• <a> links injected after hydration.
• URLs returned by fetch() / XHR but never inserted into the DOM.
• Infinite-scroll/pagination that makes API calls.
• Links revealed after a click.
• SPA route changes using pushState.
• JSON API responses containing product/article URLs.
• Delayed requests, e.g. 5–10 seconds after page load.
• JS-generated canonical/title/meta.
• Cross-origin API calls.

Then score each crawler on three separate capabilities:

That last distinction is important. A crawler can be excellent at rendering and still not give you a complete list of network-discovered URLs.

My shortlist

Screaming Frog SEO Spider is a strong agency workhorse. Its JavaScript mode uses headless Chromium and extracts links from rendered HTML as well as raw HTML. Screaming Frog Screaming Frog

Sitebulb is another strong option, particularly if you want client-friendly auditing and cloud-scale crawling. Its Chrome Crawler uses Evergreen Chromium and lets you compare response HTML with rendered HTML. Sitebulb Sitebulb Support

But for your particular requirement—“show me every URL the browser requested, including AJAX/fetch URLs”—I'd put browser/network instrumentation ahead of traditional SEO-crawler features.

If AJAX URLs are the priority

I'd seriously consider building a small crawler around Playwright. Its network API can monitor HTTP/HTTPS traffic, including XHR and fetch() requests. Playwright

Conceptually:

URL
 ↓
Playwright/Chromium
 ↓
execute JS
 ↓
 ├── rendered DOM → extract <a href>
 ├── network events → capture XHR/fetch URLs
 ├── redirects → capture
 └── SPA navigation → capture
 ↓
normalize URLs
 ↓
deduplicate
 ↓
crawl newly discovered URLs

That gives you two discovery sets:

DOM URLs
+
Network URLs
=
your candidate URL frontier

I'd not automatically treat every network URL as a crawlable page, though. Your network stream will contain analytics, JS/CSS, images, API endpoints, telemetry, GraphQL requests, etc. You need URL classification rules.

For example:

https://client.com/products/123       → crawl
https://api.client.com/products?page=2 → inspect/classify
https://cdn.client.com/app.js          → resource
https://analytics.example.com/event    → ignore

The agency-friendly approach I'd use

Use Screaming Frog/Sitebulb for standard SEO audits, and a Playwright-based crawler for JS/AJAX discovery when the site warrants it.

That avoids forcing an expensive browser crawl onto every client while giving your technical team a much deeper mode for SPAs and heavily dynamic sites.

One particularly useful acceptance test is:

Can the tool export the complete browser network log, including XHR/fetch requests, and distinguish those from ordinary page resources?

If the answer is no, I wouldn't select it as your primary AJAX URL discovery solution, regardless of how good its rendered-HTML reporting is.

If you're an SEO agency, I’d choose based on what you mean by “captures AJAX-loaded URLs.” There are really two separate capabilities:

1. Render the DOM after JavaScript executes → discovers <a href> links injected by JS.
2. Observe network/API activity → captures URLs requested by fetch/XHR, even when those URLs never become links in the DOM.

Most SEO crawlers are good at #1. Far fewer are designed to give you a comprehensive #2 dataset.

What I'd shortlist

Screaming Frog executes client-side JavaScript in headless Chrome and extracts content and links from the rendered HTML, while also discovering links in the raw HTML. Screaming Frog

Sitebulb is particularly nice for SEO diagnostics because its Chrome Crawler compares response HTML with rendered DOM and explicitly identifies links created or altered by JavaScript. Sitebulb Support Sitebulb Support It also uses an Evergreen Chromium engine. Sitebulb

Oncrawl supports JavaScript crawling and even has an option related to Shadow DOM, which can matter on modern JS applications. Oncrawl Help

But if AJAX URLs are the important requirement…

I'd seriously consider building a small Playwright-based crawler alongside your normal SEO crawler.

The architecture I'd use is:

Seed URLs
   ↓
HTTP fetch
   ↓
Launch Chromium
   ↓
Load page
   ↓
Execute JS
   ↓
┌─────────────────────────────┐
│ Capture rendered DOM links  │
│ Capture fetch/XHR requests  │
│ Capture redirects           │
│ Capture canonical/hreflang  │
│ Capture JS-generated URLs   │
└─────────────────────────────┘
   ↓
Normalize + deduplicate URLs
   ↓
Crawl queue

That distinction is important.

Suppose a page does this:

fetch('/api/products?page=2')
  .then(...)

The API URL might never appear as an <a>, so a conventional rendered crawler doesn't necessarily treat /api/products?page=2 as a discovered crawl URL.

With browser automation you can listen to network requests and classify them:

document navigation
stylesheet
image
script
XHR
fetch
WebSocket
iframe

Then you can decide which ones become part of your SEO discovery dataset.

I'd actually maintain two URL graphs

SEO URL graph

HTML links
+ rendered DOM links
+ canonical URLs
+ hreflang URLs
+ redirects
+ sitemap URLs

Network graph

page
 ├── XHR → /api/products?page=2
 ├── fetch → /api/category/shoes
 ├── fetch → https://cdn.example.com/...
 └── XHR → /search?q=running

That lets you answer much more interesting client questions:

• “Which indexable pages depend on JS to expose internal links?”
• “Which pages make API requests to endpoints that return crawlable content?”
• “Are infinite-scroll products discoverable without executing JS?”
• “Which URLs exist only in client-side routing?”
• “What URLs does the application request that aren't represented in the crawl graph?”
• “Does rendering generate thousands of parameterized URLs?”

The test I'd use when evaluating vendors

Don't judge them on a marketing demo. Give each crawler a deliberately nasty test site containing:

• React/Vue/Next SPA navigation
• links inserted after hydration
• fetch()-loaded content
• XHR-loaded pagination
• infinite scroll
• links appearing after a 2–5 second delay
• client-side redirects
• links inside Shadow DOM
• links generated after a click
• URLs generated from API responses
• history.pushState() routing
• canonical/hreflang generated by JS

Then measure:

Discovery completeness

URLs found by crawler
--------------------- × 100
URLs actually generated

Rendering fidelity

Compare:

raw response HTML
        vs
post-JS DOM

Network visibility

Ask whether you can export something equivalent to:

source_page
request_url
request_method
resource_type
status
initiator
timestamp

Control

This is crucial for agency work. You want configurable:

• render timeout
• wait conditions
• user agent
• viewport/device
• cookies
• authentication
• JavaScript execution
• robots handling
• URL extraction rules
• crawl concurrency
• network request logging

Sitebulb, for example, exposes render-timeout and Chrome crawler settings specifically because when the DOM isn't ready before the snapshot, you can miss JS-generated content/links. Sitebulb Support Sitebulb

My recommendation

If you want an off-the-shelf agency crawler, I'd start with Screaming Frog + Sitebulb and compare them on your test corpus.

If your differentiator is specifically “we discover every URL the application exposes through JavaScript/AJAX”, I'd use an SEO crawler for the conventional audit and Playwright/Chromium as a custom discovery layer. That gives you substantially more control than choosing a crawler solely on whether it has a checkbox called “JavaScript rendering.”

If you tell me roughly how many URLs you crawl per client (10k / 100k / 1M+) and whether you need API/XHR URLs themselves or only URLs that become internal links, I can give you a concrete architecture and tool shortlist.

The API URL might never appear as an <a>, so a conventional rendered crawler doesn't necessarily treat /api/products?page=2 as a discovered crawl URL.

With browser automation you can listen to network requests and classify them:

document navigation
stylesheet
image
script
XHR
fetch
WebSocket
iframe

Then you can decide which ones become part of your SEO discovery dataset.

I'd actually maintain two URL graphs

SEO URL graph

HTML links
+ rendered DOM links
+ canonical URLs
+ hreflang URLs
+ redirects
+ sitemap URLs

Network graph

page
 ├── XHR → /api/products?page=2
 ├── fetch → /api/category/shoes
 ├── fetch → https://cdn.example.com/...
 └── XHR → /search?q=running

That lets you answer much more interesting client questions:

• “Which indexable pages depend on JS to expose internal links?”
• “Which pages make API requests to endpoints that return crawlable content?”
• “Are infinite-scroll products discoverable without executing JS?”
• “Which URLs exist only in client-side routing?”
• “What URLs does the application request that aren't represented in the crawl graph?”
• “Does rendering generate thousands of parameterized URLs?”

The test I'd use when evaluating vendors

Don't judge them on a marketing demo. Give each crawler a deliberately nasty test site containing:

• React/Vue/Next SPA navigation
• links inserted after hydration
• fetch()-loaded content
• XHR-loaded pagination
• infinite scroll
• links appearing after a 2–5 second delay
• client-side redirects
• links inside Shadow DOM
• links generated after a click
• URLs generated from API responses
• history.pushState() routing
• canonical/hreflang generated by JS

Then measure:

Discovery completeness

URLs found by crawler
--------------------- × 100
URLs actually generated

Rendering fidelity

Compare:

raw response HTML
        vs
post-JS DOM

Network visibility

Ask whether you can export something equivalent to:

source_page
request_url
request_method
resource_type
status
initiator
timestamp

Control

This is crucial for agency work. You want configurable:

• render timeout
• wait conditions
• user agent
• viewport/device
• cookies
• authentication
• JavaScript execution
• robots handling
• URL extraction rules
• crawl concurrency
• network request logging

Sitebulb, for example, exposes render-timeout and Chrome crawler settings specifically because when the DOM isn't ready before the snapshot, you can miss JS-generated content/links.

My recommendation

If you want an off-the-shelf agency crawler, I'd start with Screaming Frog + Sitebulb and compare them on your test corpus.

If you're choosing a crawler specifically for JS-heavy client sites where URLs appear only after AJAX/fetch/XHR execution, I’d evaluate it differently from a normal SEO crawler.

The key distinction is:

“Renders JavaScript” ≠ “discovers every URL generated by JavaScript.”

You want a tool that actually executes the page, waits appropriately for asynchronous work, parses the resulting DOM, and ideally gives you visibility into the network activity.

What I’d test

Use a representative client site with all of these:

• Links present only after React/Vue/etc. hydration
• URLs injected after fetch()/XHR calls
• “Load more” pagination
• Infinite scroll
• Content that appears 2–5 seconds after initial load
• Links generated from API responses
• Client-side route changes
• Links on a different subdomain
• Potentially authenticated/API-backed content

Then score each crawler on these dimensions:

The big test: AJAX-loaded URLs

Don't just check whether the crawler reports "JavaScript rendered."

Give it a page where:

Initial HTML
   ↓
JavaScript executes
   ↓
fetch("/api/products")
   ↓
API returns products
   ↓
JS creates <a href="/product/foo">
   ↓
Crawler should discover /product/foo

A good crawler should ultimately discover /product/foo as a crawlable URL.

For example, screamingfrog.co.uk specifically describes executing client-side JavaScript and extracting content and links from the rendered HTML. Its rendering configuration also says the JavaScript mode uses headless Chrome and extracts links from the rendered HTML. Screaming Frog

sitebulb.com similarly uses a headless Chrome crawler and compares response HTML with rendered HTML. Sitebulb explicitly reports links created or modified by JavaScript. Sitebulb Support

I'd separate two use cases

1. "Find the URLs Google could discover"

For this, rendered-DOM extraction is usually what you want.

A crawler like Screaming Frog or Sitebulb can execute JS and then extract the links that exist in the resulting DOM. That's sufficient for many React/Next/Vue sites.

Sitebulb is particularly interesting for an agency because its Response vs Render reporting can show which links exist only after rendering. Sitebulb Support Sitebulb Support

support.sitebulb.com

2. "Tell me every URL the application requested/generated"

That's a different requirement.

Suppose the page does:

fetch("https://api.example.com/products?page=2")

and the response contains:

{
  "products": [
    {"url": "/products/widget-a"},
    {"url": "/products/widget-b"}
  ]
}

but the application never creates <a> elements for those URLs.

A normal SEO crawler parsing rendered HTML may never report those product URLs.

For that requirement, I'd want network-level capture in addition to DOM extraction.

That means your crawler should expose something equivalent to:

Document request
XHR/fetch
WebSocket
resource requests
redirects
response status
response URL
request initiator

This is where I'd seriously consider building an agency-specific crawler around Chromium + Playwright rather than relying exclusively on an off-the-shelf SEO crawler.

A strong architecture for an SEO agency

I'd actually use two layers:

             ┌──────────────────┐
             │ URL Frontier     │
             └────────┬─────────┘
                      ↓
             ┌──────────────────┐
             │ Chromium         │
             │ + Playwright     │
             └────────┬─────────┘
                      ↓
          ┌───────────┴───────────┐
          ↓                       ↓
   Rendered DOM              Network log
          ↓                       ↓
   <a href> URLs          XHR/fetch URLs
          │                       │
          └───────────┬───────────┘
                      ↓
              URL normalization
                      ↓
                URL frontier

Then classify discoveries:

• HTML_LINK
• JS_DOM_LINK
• XHR_REQUEST
• FETCH_REQUEST
• REDIRECT
• CANONICAL
• SITEMAP
• STRUCTURED_DATA
• etc.

That gives you much more useful SEO data than simply saying "JavaScript: yes/no."

One important trap: waiting

This is probably the most important thing I'd test.

A crawler can execute JavaScript perfectly and still miss AJAX URLs if it snapshots the page too early:

0 ms       HTML arrives
100 ms     React hydrates
800 ms     fetch starts
1,400 ms   API responds
1,500 ms   links inserted
1,600 ms   crawler snapshots DOM   ← good

versus:

0 ms       HTML arrives
100 ms     React hydrates
800 ms     fetch starts
1,400 ms   API responds
1,500 ms   links inserted
500 ms     crawler snapshots DOM   ← missed

Sitebulb exposes render timing/timeout controls specifically because render completion matters for JS-heavy sites. Sitebulb Support

So I'd benchmark crawlers using slow API responses intentionally. Add 500 ms, 2 seconds, and 5 seconds of artificial delay and see how URL discovery changes.

My shortlist

For an SEO agency, I'd test these first:

• Screaming Frog — excellent general-purpose SEO crawler with Chromium JS rendering; particularly attractive if your team already uses it. Screaming Frog Screaming Frog
• Sitebulb — particularly strong if you want to diagnose what JavaScript changed, because of its response-vs-render and JS-created-link reporting. Sitebulb Support Sitebulb Support
• A custom Playwright/Chromium crawler — best if "capture every AJAX/fetch URL" is a core agency capability rather than just an SEO audit feature.

I'd not select based on a vendor saying "supports JavaScript." Make vendors pass your actual benchmark.

The benchmark I'd give vendors

Give each tool the same 10-page test site and ask it to return:

1. URLs found in raw HTML
2. URLs found only after rendering
3. URLs requested through XHR/fetch
4. URLs added after a delayed API response
5. URLs exposed by infinite scroll
6. URLs exposed by "Load more"
7. URLs generated through SPA routing
8. HTTP status for every discovered URL
9. Time from page load → URL discovery
10. Evidence showing how each URL was discovered

#10 is the differentiator. If you're doing this at agency scale, knowing why a URL entered the crawl graph is incredibly valuable for debugging JS SEO.

If you tell me your expected crawl volume (e.g. 100k, 1M, or 10M URLs/month) and whether you want SaaS vs self-hosted, I can narrow this to a practical shortlist and architecture.

If you tell me your expected crawl volume (e.g. 100k, 1M, or 10M URLs/month) and whether you want SaaS vs self-hosted, I can narrow this to a practical shortlist and architecture.

For this, rendered-DOM extraction is usually what you want.

A crawler like Screaming Frog or Sitebulb can execute JS and then extract the links that exist in the resulting DOM. That's sufficient for many React/Next/Vue sites.

Sitebulb is particularly interesting for an agency because its Response vs Render reporting can show which links exist only after rendering. Sitebulb Support Sitebulb Support

support.sitebulb.com

2. "Tell me every URL the application requested/generated"

That's a different requirement.

Suppose the page does:

fetch("https://api.example.com/products?page=2")

and the response contains:

{
  "products": [
    {"url": "/products/widget-a"},
    {"url": "/products/widget-b"}
  ]
}

but the application never creates <a> elements for those URLs.

A normal SEO crawler parsing rendered HTML may never report those product URLs.

For that requirement, I'd want network-level capture in addition to DOM extraction.

That means your crawler should expose something equivalent to:

Document request
XHR/fetch
WebSocket
resource requests
redirects
response status
response URL
request initiator

This is where I'd seriously consider building an agency-specific crawler around Chromium + Playwright rather than relying exclusively on an off-the-shelf SEO crawler.

A strong architecture for an SEO agency

I'd actually use two layers:

             ┌──────────────────┐
             │ URL Frontier     │
             └────────┬─────────┘
                      ↓
             ┌──────────────────┐
             │ Chromium         │
             │ + Playwright     │
             └────────┬─────────┘
                      ↓
          ┌───────────┴───────────┐
          ↓                       ↓
   Rendered DOM              Network log
          ↓                       ↓
   <a href> URLs          XHR/fetch URLs
          │                       │
          └───────────┬───────────┘
                      ↓
              URL normalization
                      ↓
                URL frontier

Then classify discoveries:

• HTML_LINK
• JS_DOM_LINK
• XHR_REQUEST
• FETCH_REQUEST
• REDIRECT
• CANONICAL
• SITEMAP
• STRUCTURED_DATA
• etc.

That gives you much more useful SEO data than simply saying "JavaScript: yes/no."

One important trap: waiting

This is probably the most important thing I'd test.

A crawler can execute JavaScript perfectly and still miss AJAX URLs if it snapshots the page too early:

0 ms       HTML arrives
100 ms     React hydrates
800 ms     fetch starts
1,400 ms   API responds
1,500 ms   links inserted
1,600 ms   crawler snapshots DOM   ← good

versus:

0 ms       HTML arrives
100 ms     React hydrates
800 ms     fetch starts
1,400 ms   API responds
1,500 ms   links inserted
500 ms     crawler snapshots DOM   ← missed

If you're an SEO agency, I’d choose based on what you mean by “captures AJAX-loaded URLs.” There are actually two different requirements:

1. Render JS and extract links that appear in the rendered DOM.
2. Observe network/AJAX requests themselves and extract URLs that may never become <a href> links.

Most SEO crawlers are good at #1. Far fewer are true network-traffic crawlers for #2.

What I’d evaluate

For conventional JavaScript SEO: Sitebulb is worth testing

Sitebulb's Chrome Crawler uses headless Chromium and extracts links from the rendered DOM. It also compares response HTML against rendered HTML and specifically reports links created or modified by JavaScript. Sitebulb Support Sitebulb Support

That makes it particularly good for questions like:

“This React site has 50,000 URLs. Which internal links only exist after JavaScript executes?”

It can also adjust render timeout, which matters when AJAX requests arrive well after the initial page load. Sitebulb Support

sitebulb.com

For very large crawls: JetOctopus is another one to benchmark

JetOctopus explicitly markets its JavaScript crawler around detecting AJAX requests, JS-injected content, blocked resources and rendering errors, and supports large-scale cloud crawling. Tech SEO for Google & AI

jetoctopus.com

The important distinction: DOM URLs vs. AJAX URLs

Suppose the page does this:

GET /category/shoes
        ↓
JavaScript executes
        ↓
fetch("/api/products?page=2")
        ↓
JSON returned
        ↓
JS creates:
<a href="/product/nike-air-max">...</a>

A normal JS-enabled SEO crawler can potentially discover:

/product/nike-air-max

because it exists in the rendered DOM.

But if the site instead does:

fetch("/api/products?page=2")

and never creates a crawlable link, that's a different requirement. You need the crawler to record network requests, not merely parse the rendered DOM.

For an agency, I'd therefore run a vendor bake-off with three test cases:

• DOM injection: URL appears only after hydration.
• Delayed AJAX: URL/content appears 5–15 seconds after initial load.
• Network-only: URL is requested via fetch/XHR but never appears as an <a>.

Then compare the exported URL sets.

My practical decision rule

If your primary job is technical SEO auditing: start with Sitebulb. Its response-vs-render functionality is unusually useful for proving that JS created/changed internal links, titles, canonicals, etc. Sitebulb Support

If you're doing huge enterprise crawls and specifically care about AJAX/network activity: put JetOctopus into the bake-off and verify exactly what its AJAX reporting exports, rather than assuming “JavaScript crawler” means “capture every network URL.” Tech SEO for Google & AI

If “capture every XHR/fetch URL” is a hard requirement: I'd look beyond traditional SEO crawlers and evaluate a Playwright/Chromium-based crawler where you can explicitly listen to request/response events. That gives you deterministic control over what constitutes an AJAX URL, wait conditions, interactions, authentication, and exports.

The key question for your agency is therefore: do you want the URLs that become crawlable links after JS, or literally every URL requested by XHR/fetch? Those lead to rather different tool choices.

For very large crawls: JetOctopus is another one to benchmark

JetOctopus explicitly markets its JavaScript crawler around detecting AJAX requests, JS-injected content, blocked resources and rendering errors, and supports large-scale cloud crawling. jetoctopus.com

jetoctopus.com

The important distinction: DOM URLs vs. AJAX URLs

Suppose the page does this:

GET /category/shoes
        ↓
JavaScript executes
        ↓
fetch("/api/products?page=2")
        ↓
JSON returned
        ↓
JS creates:
<a href="/product/nike-air-max">...</a>

A normal JS-enabled SEO crawler can potentially discover:

/product/nike-air-max

because it exists in the rendered DOM.

But if the site instead does:

fetch("/api/products?page=2")

For an SEO agency, I’d choose based on what you need to observe, not just “does it run JavaScript?” A lot of crawlers execute JS but differ massively in whether they can capture post-render URLs, fetch/XHR endpoints, SPA routes, lazy-loaded links, and DOM changes.

Google itself renders JavaScript with an evergreen Chromium engine and can discover links after rendering, but your crawler needs to be much more configurable if you’re doing client audits at scale. Google for Developers

The capabilities I’d require

1. Full browser rendering

Must support:

• Chromium-based rendering
• waiting for network idle / custom delays
• executing React/Vue/Angular apps
• handling hydration
• running JS events

Avoid tools that only parse HTML + JS files. They will miss content injected after API calls.

2. Capture URLs from multiple sources

Look for extraction from:

• Rendered DOM: - <a href>
• canonical tags
• hreflang
• JSON-LD
• Browser network layer: - fetch()
• XHR
• GraphQL calls
• API endpoints
• JavaScript: - route definitions
• embedded JSON
• JS bundles
• Browser history: - pushState
• SPA route changes

For SEO, the important question is often not just “what links exist?” but “what URLs does the application generate?”

3. Crawl comparison modes

Useful features:

• raw HTML vs rendered HTML diff
• before/after JS execution
• screenshots
• rendered source export
• DOM snapshot export
• blocked resource reporting

4. Scale and control

For agency work, evaluate:

• authenticated crawling
• custom headers/user agents
• cookies/session handling
• crawl scheduling
• API access
• concurrency controls
• export formats

Tools I’d shortlist

Screaming Frog SEO Spider

Best general SEO agency choice.

Strong for:

• JavaScript rendering
• DOM extraction
• crawl reports
• client deliverables
• integrations

Good fit if your workflow is:

“Run technical SEO audits across hundreds of client sites.”

Weakness:

• browser automation is not as deep as a custom Playwright stack.

Sitebulb

Best if you want polished audits and visual explanations.

Strong for:

• JavaScript crawling
• issue prioritization
• client-facing reporting

Good fit:

Agencies that sell technical SEO audits rather than raw crawl data.

Oncrawl / Botify

Best for enterprise SEO programs.

Strong for:

• huge sites
• log analysis
• crawl budget analysis
• combining crawl + search data

Good fit:

Ecommerce, marketplaces, publishers.

Custom Playwright crawler

Best if “capture every AJAX URL” is the core requirement.

A custom crawler can:

• launch Chromium
• intercept every request
• record XHR/fetch URLs
• wait for specific selectors
• click menus/buttons
• scroll to trigger lazy loading
• capture SPA navigation

Example workflow:

URL list
   ↓
Playwright Chromium
   ↓
network listeners
   ↓
capture:
  - document URLs
  - XHR URLs
  - GraphQL calls
  - JS-created routes
  - DOM links
   ↓
SEO database

This is usually what I’d build for an agency whose differentiator is JS-heavy crawling.

My selection matrix

The test I’d run before buying

Take 5 difficult client sites:

1. React SPA
2. Infinite scroll ecommerce
3. Angular site
4. Site with lazy-loaded navigation
5. Site where URLs only appear after API calls

Measure:

• URLs found in raw HTML
• URLs found after rendering
• XHR/fetch endpoints captured
• JS errors detected
• crawl speed
• export quality

The winner is the tool that finds the largest meaningful URL set, not the one that simply says “JavaScript rendering enabled.”

For an SEO agency specifically, I’d probably use Screaming Frog/Sitebulb for routine audits plus a Playwright-based crawler for investigations and JS-heavy clients. That combination covers most cases.

The test I’d run before buying

Take 5 difficult client sites:

1. React SPA
2. Infinite scroll ecommerce
3. Angular site
4. Site with lazy-loaded navigation
5. Site where URLs only appear after API calls

Measure:

• URLs found in raw HTML
• URLs found after rendering
• XHR/fetch endpoints captured
• JS errors detected
• crawl speed
• export quality

The winner is the tool that finds the largest meaningful URL set, not the one that simply says “JavaScript rendering enabled.”

For an SEO agency specifically, I’d probably use Screaming Frog/Sitebulb for routine audits plus a Playwright-based crawler for investigations and JS-heavy clients. That combination covers most cases.

Yes. For an SEO agency, I’d evaluate JS crawlers on what they actually discover after rendering, not merely whether they “support JavaScript.”

The capabilities I’d require

1. Real browser rendering

• Chromium/Chrome-based headless browser.
• Executes client-side JS rather than just parsing <script> tags.
• Captures the post-JS DOM, not only the original HTTP response.
• Ideally supports mobile and desktop viewports.
• Rendered-link discovery

This is the big one for your use case. You want the crawler to discover:

Initial HTML
    ↓
Execute JavaScript
    ↓
API/AJAX requests
    ↓
DOM gets populated
    ↓
<a href="/product/123">
    ↓
Crawler adds /product/123 to crawl queue

A tool that merely renders the page but doesn't feed newly discovered DOM links back into its crawl queue isn't sufficient.

For example, Screaming Frog explicitly documents that its JavaScript mode crawls and extracts content and links from rendered HTML, and can identify links that exist only after JavaScript execution. Screaming Frog Screaming Frog

3. Raw HTML vs. rendered HTML

You want both versions stored/exportable. This lets you answer:

“Was this URL/link present in the server response, or did JavaScript create it?”

That's extremely valuable in client audits. Screaming Frog, for example, can store both the original and rendered HTML and compare them. Screaming Frog

4. AJAX/network waiting controls

Don't accept a simplistic “wait 5 seconds” implementation as the only option.

Look for:

• network-idle waiting
• configurable rendering timeout
• custom JavaScript delays
• ability to wait for a particular selector
• handling of late API responses

Otherwise you'll get false negatives from React/Vue/Next/etc. pages whose API calls finish after your snapshot is taken. Screaming Frog exposes an AJAX timeout specifically for this problem. Screaming Frog Screaming Frog

5. Network/request capture

For an agency, I'd actually make this a separate requirement from rendered HTML.

Ideally you can export:

This lets you distinguish:

• URL discovered as an HTML link
• URL discovered in rendered DOM
• URL requested by JavaScript
• URL returned inside an API response

Those are very different SEO signals.

The critical distinction: AJAX URLs vs. URLs in AJAX responses

I'd define your requirement very precisely before selecting a crawler.

There are actually three different things people mean by “captures AJAX-loaded URLs”:

A. DOM links created by AJAX

fetch('/api/products').then(...)

eventually produces:

<a href="/product/red-shoe">Red Shoe</a>

You need the crawler to render the page and discover /product/red-shoe.

B. API endpoints themselves

You want to record:

/api/products
/api/categories
/api/recommendations

even though they aren't HTML pages.

C. URLs contained inside API responses

For example:

{
  "products": [
    {"url": "/product/red-shoe"},
    {"url": "/product/blue-shoe"}
  ]
}

A normal SEO crawler may discover A, but not necessarily B or C.

If your agency specifically needs B/C, I'd prioritize a crawler with network/CDP request logging or a programmable browser layer, rather than choosing based on the marketing phrase “JavaScript rendering.”

A practical agency test

Before buying anything, give every candidate the same deliberately JS-heavy test site.

Build pages with:

• links present in initial HTML
• links inserted after fetch()
• links inserted after setTimeout()
• links returned from XHR JSON
• infinite-scroll links
• links generated after user interaction
• links inside an iframe
• links loaded only after 8–10 seconds
• links whose URLs are generated rather than literally present in the JS bundle

Then compare:

URLs in raw HTML
URLs in rendered DOM
URLs requested by browser
URLs discovered by crawler
URLs actually crawled

The last two columns are the ones I'd care about most.

What I'd choose

For a general-purpose SEO agency crawler, Screaming Frog is a strong baseline because its JS renderer uses Chromium and explicitly supports rendered-link discovery, rendered HTML extraction, screenshots, and JS-specific reporting. Screaming Frog Screaming Frog

But if your requirement is specifically “show me every URL the browser requested, including AJAX/fetch/API traffic, and then let me turn URLs from those responses into crawl targets,” I'd look beyond conventional SEO crawlers toward a programmable Playwright/Puppeteer/CDP-based crawler. That's a different requirement from ordinary JS SEO rendering.

My procurement checklist would therefore be:

• ✅ Chromium rendering
• ✅ Post-JS DOM extraction
• ✅ Rendered <a> discovery
• ✅ Rendered links added to crawl frontier
• ✅ Raw + rendered HTML export
• ✅ Network request logging
• ✅ XHR/fetch capture
• ✅ Response-body capture
• ✅ Configurable wait/network-idle behavior
• ✅ Custom JS/hooks
• ✅ Cookies/session support
• ✅ Mobile/desktop rendering
• ✅ Robots handling
• ✅ Crawl-scale concurrency
• ✅ Exportable URL → source-page relationships

If you tell me your crawl volume (e.g. 10k, 1M, 100M URLs/month) and whether you need API-request URLs as well as DOM-discovered URLs, I can narrow this down to the best 3–5 tools/architectures for an agency setup.

What I'd choose

For a general-purpose SEO agency crawler, Screaming Frog is a strong baseline because its JS renderer uses Chromium and explicitly supports rendered-link discovery, rendered HTML extraction, screenshots, and JS-specific reporting.

B. API endpoints themselves

You want to record:

/api/products
/api/categories
/api/recommendations

even though they aren't HTML pages.

C. URLs contained inside API responses

For example:

{
  "products": [
    {"url": "/product/red-shoe"},
    {"url": "/product/blue-shoe"}
  ]
}

A normal SEO crawler may discover A, but not necessarily B or C.

If your agency specifically needs B/C, I'd prioritize a crawler with network/CDP request logging or a programmable browser layer, rather than choosing based on the marketing phrase “JavaScript rendering.”

A practical agency test

Before buying anything, give every candidate the same deliberately JS-heavy test site.

Build pages with:

• links present in initial HTML
• links inserted after fetch()
• links inserted after setTimeout()
• links returned from XHR JSON
• infinite-scroll links
• links generated after user interaction
• links inside an iframe
• links loaded only after 8–10 seconds
• links whose URLs are generated rather than literally present in the JS bundle

Then compare:

URLs in raw HTML
URLs in rendered DOM
URLs requested by browser
URLs discovered by crawler
URLs actually crawled

The last two columns are the ones I'd care about most.

What I'd choose

If your requirement is specifically “execute JavaScript, let AJAX/API calls populate the page, then discover URLs that only exist after rendering,” I’d evaluate crawlers on more than just whether they say “JavaScript rendering.”

What I’d test

Run the same representative client site through each candidate and score these capabilities:

The key distinction: AJAX URLs vs rendered links

I'd actually separate your requirement into two tests.

1. “Does JS create links?”

For example:

<div id="products"></div>

<script>
fetch('/api/products')
  .then(r => r.json())
  .then(products => {
    document.querySelector('#products').innerHTML =
      products.map(p => `<a href="/product/${p.slug}">${p.name}</a>`).join('');
  });
</script>

A crawler needs to:

1. load the page in Chromium;
2. execute the JavaScript;
3. allow the fetch() to finish;
4. capture the resulting DOM;
5. extract /product/foo, /product/bar, etc.;
6. add those URLs to the crawl queue.

That's more demanding than simply saying “we render JavaScript.”

2. “Does JS request URLs that never become links?”

For example:

/page
   ↓
JS
   ↓
GET /api/products?page=2
   ↓
JSON returned

If by “captures AJAX-loaded URLs” you mean you want /api/products?page=2 itself captured as a URL, that's a different requirement. Most SEO crawlers primarily care about URLs exposed in the rendered page, rather than treating every XHR/fetch endpoint as a crawlable page.

So I'd ask vendors explicitly:

“Do you extract URLs from the post-JavaScript DOM, and do those discovered URLs get added to the crawl queue? Separately, can you export the network requests made during rendering, including XHR/fetch URLs?”

Two good benchmarks

Screaming Frog SEO Spider is a good baseline. Its JavaScript rendering mode uses headless Chromium and extracts content and links from the rendered HTML. It also specifically identifies links that exist only after JavaScript execution and can bulk-export them. Screaming Frog Screaming Frog

Sitebulb is another strong benchmark. Its Chrome Crawler renders pages and its Response vs Render report identifies links created or modified by JavaScript. It also has controls for render timeout and infinite scrolling. Sitebulb Support Sitebulb Support

Sitebulb's documentation is particularly relevant to your use case: it explicitly distinguishes links present in the response HTML from links that only appear after JavaScript execution. Sitebulb

For an SEO agency, I'd use this decision rule

If you mostly need JS-rendered SEO crawling:
Start with Screaming Frog or Sitebulb and benchmark them against your hardest client sites.

If you need massive-scale crawling:
Look at enterprise crawlers, but benchmark rendered URLs/hour per dollar, not just raw crawl speed. Chromium rendering is inherently much more resource-intensive than fetching HTML. Sitebulb Support

If you need to capture the actual network layer:
Don't assume an SEO crawler is the right tool. Consider a custom Playwright/Chromium crawler where you can record request, response, XHR/fetch, DOM mutations, and discovered anchors independently. That gives you much more control over exactly what “AJAX-loaded URL” means.

My agency benchmark

I'd create a synthetic test site containing:

• links in raw HTML;
• links inserted immediately by JS;
• links inserted after a 2–3 second fetch();
• links inserted after user-like scrolling;
• links inside an iframe;
• links generated from API JSON;
• URLs requested by XHR but never exposed as <a> links;
• pagination loaded through fetch();
• a SPA route changed with history.pushState();
• a service-worker-mediated request.

Then measure:

discovered URLs / expected URLs × 100

and separately:

network URLs captured / expected network URLs × 100

That will tell you very quickly whether a product merely renders JavaScript or actually solves your crawling problem.

If you tell me your typical crawl size (e.g. 10k, 100k, 1M+ URLs) and whether you mean AJAX endpoint URLs or page URLs revealed by AJAX, I can narrow this down to the best architecture/tooling for an agency.

Sitebulb is another strong benchmark. Its Chrome Crawler renders pages and its Response vs Render report identifies links created or modified by JavaScript. It also has controls for render timeout and infinite scrolling. Sitebulb Support Sitebulb Support

Sitebulb's documentation is particularly relevant to your use case: it explicitly distinguishes links present in the response HTML from links that only appear after JavaScript execution. sitebulb.com

For an SEO agency, I'd use this decision rule

If you mostly need JS-rendered SEO crawling:
Start with Screaming Frog or Sitebulb and benchmark them against your hardest client sites.

If you need massive-scale crawling:
Look at enterprise crawlers, but benchmark rendered URLs/hour per dollar, not just raw crawl speed. Chromium rendering is inherently much more resource-intensive than fetching HTML. Sitebulb Support

If you need to capture the actual network layer:
Don't assume an SEO crawler is the right tool. Consider a custom Playwright/Chromium crawler where you can record request, response, XHR/fetch, DOM mutations, and discovered anchors independently. That gives you much more control over exactly what “AJAX-loaded URL” means.

My agency benchmark

I'd create a synthetic test site containing:

If your requirement is specifically “execute JavaScript, let AJAX/fetch/XHR requests finish, then discover URLs that only appear afterward”, I’d evaluate crawlers on more than just “has JS rendering.”

What to test

I’d score each tool against these six capabilities:

1. Real browser rendering

• Chromium/headless Chrome rather than a JS parser.
• Ideally lets you control user agent and viewport.
• Screaming Frog and Sitebulb both use Chrome-based rendering. Screaming Frog Sitebulb Support
• Rendered-DOM link discovery

• Critical distinction: the crawler needs to parse <a href> elements after JS executes, not merely render a screenshot.
• Sitebulb explicitly compares response HTML against rendered HTML and identifies links created/modified by JavaScript. Sitebulb Support
• Screaming Frog likewise crawls/extracts links from the rendered HTML while also discovering raw-HTML links. Screaming Frog
• AJAX/fetch completion

• This is the big one for your use case.
• Test pages where: ```

Initial HTML
↓
React/Vue/etc. boots
↓
fetch('/api/products')
↓
JSON response
↓
JS creates <a href="/product/123">
↓
crawler discovers /product/123

- You want configurable **render timeout / end-render conditions**, because simply waiting for `DOMContentLoaded` can miss late API responses.
- Sitebulb, for example, exposes both a considered-load event and render timeout; it specifically recommends increasing the timeout when JS-heavy pages need more time. [Sitebulb Support](https://support.sitebulb.com/en/articles/13011308-troubleshooting-page-rendering-issues?utm_source=chatgpt.com)
- JetOctopus exposes JS execution time and a JS end-render event. [TechSEO AI](https://jetoctopus.com/how-to-configure-a-crawl-of-your-website/?utm_source=chatgpt.com)
4. **Network/AJAX visibility**

 - If your agency needs to diagnose *why* URLs are missing, don't settle for rendered HTML alone.
- Look for visibility into XHR/fetch requests, failed resources, blocked JS, hydration errors, etc.
- JetOctopus specifically advertises detection of AJAX requests, JS-injected links, blocked resources and hydration errors. [TechSEO AI](https://jetoctopus.com/js-seo/?utm_source=chatgpt.com)
5. **Response-vs-render comparison**

 - This is extremely useful for client reporting: - URL exists in source but disappears after JS
- URL appears only after JS
- title/canonical/robots changes
- content is injected client-side
- Sitebulb has a dedicated Response vs Render report, while Screaming Frog has a JavaScript tab and filters for these changes. [Sitebulb Support](https://support.sitebulb.com/en/articles/9857330-response-vs-render-report?utm_source=chatgpt.com) [Screaming Frog](https://www.screamingfrog.co.uk/seo-spider/user-guide/tabs/?utm_source=chatgpt.com)
6. **Scale/cost**

 - Browser crawling is dramatically more expensive than HTTP crawling because every URL effectively needs a browser session and its resources.
- For an agency, I'd want **two crawl modes**: - cheap/raw crawl across the entire site
- rendered crawl for JS-dependent templates/URLs
- Screaming Frog itself recommends selective JS crawling because rendering is slower and more resource intensive. [Screaming Frog](https://www.screamingfrog.co.uk/seo-spider/tutorials/crawl-javascript-seo/?utm_source=chatgpt.com)

### How I'd shortlist them

Tool | JS rendering | Rendered links | AJAX/JS diagnostics | Large-scale agency use
--- | --- | --- | --- | ---
**Screaming Frog** | Excellent | Excellent | Good | Good, especially desktop
**Sitebulb** | Excellent | Excellent | Good–very good | **Very good**, especially Cloud
**JetOctopus** | Excellent | Good | **Very good** | **Excellent** for large crawls
Custom Playwright crawler | **Full control** | **Full control** | **Full control** | Excellent if you can build/maintain it

For a typical SEO agency, I'd start with **Sitebulb vs JetOctopus**, with **Screaming Frog** as the benchmark/reference tool. Sitebulb has Chrome crawling plus explicit response/render link comparison; JetOctopus is particularly interesting if your problem is *network activity and AJAX-heavy applications at scale*. [Sitebulb Support](https://support.sitebulb.com/en/articles/9497252-find-links-created-or-altered-by-javascript?utm_source=chatgpt.com) [TechSEO AI](https://jetoctopus.com/js-seo/?utm_source=chatgpt.com)

### The test I'd actually run before buying

Don't evaluate them on a normal WordPress site. Give every vendor the **same deliberately hostile JS test site** containing:

- links present in initial HTML
- links generated synchronously by JS
- links generated after `fetch()`
- links generated after a 3–5 second delay
- pagination loaded by AJAX
- infinite scroll
- links inside JSON/API responses but converted to `<a>` elements by JS
- links injected after hydration
- failed API request
- API request to another subdomain
- content behind a cookie/consent state
- SPA route changes using `history.pushState()`

Then compare the final URL sets:

Raw HTML URLs
∪
Rendered DOM URLs
=
Expected crawl frontier

The most important metric isn't **“did it render the page?”** It's:

>  **Of the URLs that become discoverable after JavaScript executes, what percentage did the crawler actually add to its crawl frontier?**
That's the distinction between a crawler that's merely *JS-rendered* and one that's genuinely useful for **AJAX URL discovery**.

One other important nuance: Google retired the old special “AJAX crawling scheme”; modern Googlebot renders JavaScript itself, so you should test modern JS execution rather than looking for support for the old `?_escaped_fragment_=` mechanism. [developers.google.com](https://developers.google.com/search/blog/2017/12/rendering-ajax-crawling-pages?utm_source=chatgpt.com)

If you tell me roughly **how many URLs/client/month** you're crawling (e.g. 100k, 1M, 10M+) and whether you need **API-level AJAX request capture or just URLs appearing in the rendered DOM**, I can narrow this to the best 2–3 options and the exact evaluation criteria.

If your requirement is specifically “execute the site’s JavaScript, let AJAX/fetch/XHR requests finish, and discover URLs that only appear after those requests”, don’t choose a crawler based on “supports JavaScript” alone. You want to distinguish DOM/link discovery from network-request discovery.

What I’d evaluate

1. Real browser rendering

• Chromium/Chrome, not a lightweight JS interpreter.
• Ideally configurable user agent, viewport, cookies, wait conditions, and render timeout.
• This matters for React/Vue/Next/etc. and client-side navigation. Sitebulb Support Screaming Frog
• Rendered DOM link extraction

• Does it extract <a href> URLs that weren't present in the initial HTML?
• This catches things like menus, product grids, infinite-scroll implementations, and JS-generated links.
• Sitebulb and Screaming Frog explicitly compare response HTML against rendered HTML for this purpose. Sitebulb Support Screaming Frog
• Network-level URL capture — the important part for AJAX

• Ask whether the crawler records requests made by the browser, including: - fetch()
• XHR
• GraphQL
• JSON API endpoints
• dynamically requested HTML fragments
• redirects
• This is a different capability from merely extracting links from the rendered DOM.
• If your agency needs to discover URLs inside API responses, make this a hard requirement.
• Wait behavior

• A crawler that renders for 1 second can technically “support JS” while completely missing an API response that arrives after 2–5 seconds.
• Look for configurable render timeout and, ideally, network-idle/custom wait conditions. Sitebulb, for example, exposes render-timeout controls. Sitebulb
• Evidence/export

• You want exports containing at least: - source URL
• discovered URL
• discovery mechanism (HTML, rendered DOM, XHR, fetch, etc.)
• HTTP status
• content type
• request/response timing
• This turns “we found 40k extra URLs” into something your SEO team can actually audit.

Tools I'd put on your shortlist

Sitebulb's Chrome Crawler uses an Evergreen Chromium engine and explicitly reports links created or changed during JavaScript rendering. Sitebulb Sitebulb Support Screaming Frog likewise has a JavaScript rendering mode that crawls both original and rendered HTML. Screaming Frog Oncrawl supports full JavaScript rendering at large scale. Oncrawl - Technical SEO Data

But: I would not automatically choose any of these if capturing AJAX/API-discovered URLs is the core requirement. Their documented strengths are primarily rendered-page/DOM crawling. You should explicitly test their network-request visibility before buying.

A good agency bake-off

Take one nasty client site with:

• 100 URLs in normal HTML
• 500 URLs injected into the DOM
• 5,000 URLs returned by an XHR/fetch endpoint
• pagination implemented entirely through API calls
• an infinite-scroll product/category listing
• links generated only after hydration
• a deliberately slow API response

Run the same crawl through each candidate and measure:

Initial HTML URLs → rendered DOM URLs → network-request URLs → unique URLs ultimately discovered

That four-stage funnel will tell you much more than a vendor's “JavaScript rendering: Yes” checkbox.

If network URLs are genuinely first-class data you need, I’d also consider building a small Playwright/Chromium crawler layer alongside your SEO crawler. That gives you direct access to browser network events and lets you feed the resulting URL set into your normal crawler for status/indexability/canonical analysis.

In other words: use an SEO crawler for SEO analysis; use browser network instrumentation when “what did the JavaScript request?” is itself part of the crawl.

Yes. For an SEO agency, I’d evaluate these tools less on “does it render JavaScript?” and more on “what exactly happens after JavaScript runs, and what does the crawler put into its URL graph?”

What you actually need

For heavy-JS clients, your crawler should ideally do all of these:

1. Execute real Chromium/Chrome, not just parse HTML.
2. Wait for AJAX/fetch/XHR activity rather than immediately snapshotting the DOM.
3. Extract <a href> URLs from the post-JS DOM.
4. Detect URLs that only appear after JavaScript executes.
5. Ideally expose the network requests themselves—XHR/fetch endpoints, status codes, redirects, failures, etc.
6. Compare response HTML vs rendered DOM.
7. Let you control render timeout, user agent, viewport, cookies, and load/wait conditions.
8. Scale reasonably across hundreds of thousands/millions of URLs.

That distinction matters: capturing an AJAX request URL isn't necessarily the same thing as discovering the SEO URL created by that request. For SEO crawling, you usually care about both.

My shortlist

1. Sitebulb — best fit for agency-level JS SEO auditing

sitebulb.com is probably where I'd start.

Its Chrome Crawler uses headless Chromium, and it specifically compares response HTML with rendered HTML. It can identify links created or modified by JavaScript, including links that don't exist in the initial response. Sitebulb Support Sitebulb Support

That's particularly useful for your use case because you can answer:

“Was /product/123 actually discoverable in the initial HTML, or did JS create that link?”

It also has controls for render timeout and load events, which matter enormously for AJAX-heavy sites. Sitebulb Support

Sitebulb Cloud claims up to 10 million URLs per audit, while Desktop supports up to 500,000. Sitebulb

I'd choose it if: your agency's primary deliverable is technical SEO audits and you want strong rendered-vs-source diagnostics without building infrastructure.

2. JetOctopus — worth testing for very large JS crawls

jetoctopus.com is interesting if scale is your dominant concern.

Its JS crawler specifically advertises inspection of AJAX requests, JS-injected content and links, blocked resources, hydration errors, and client-side redirects. It also positions itself for million-page/day-scale crawling. Tech SEO Platform

I'd choose it if: you're dealing with huge ecommerce/client sites and want more visibility into the browser/network layer rather than simply rendered HTML.

3. Screaming Frog — excellent general-purpose agency crawler

I'd still keep Screaming Frog in the evaluation, particularly if your team already uses it.

The important test isn't whether it says “JavaScript rendering: yes.” It's whether its rendered crawl gives you the complete URL discovery behavior you need on your clients' particular implementations.

For example, test a client where:

Initial HTML
    ↓
React/Next/Vue executes
    ↓
fetch("/api/products?page=2")
    ↓
JSON response
    ↓
JS builds product cards
    ↓
<a href="/products/widget-2">

You want /products/widget-2 to enter the crawler's URL graph.

If it only records:

/api/products?page=2

you haven't necessarily solved the SEO crawling problem.

The test I'd use before buying

Don't evaluate these tools using a normal brochure-style website.

Take 3–5 genuinely difficult client URLs:

• SPA with client-side routing
• Infinite scroll
• “Load more” button
• Product/category grids populated by XHR/fetch
• Content that appears 2–10 seconds after initial load
• Links injected after an API response
• Ideally a site using service workers

Then create a known ground truth.

For example:

URL A
 ├── /category/shoes          [initial HTML]
 ├── /category/hats           [JS]
 ├── /product/red-shoe         [AJAX → JS]
 └── /product/blue-shoe        [AJAX → JS]

Run each crawler and compare:

I'd make URL discovery accuracy the pass/fail criterion.

One subtle but important distinction

For SEO, I'd actually maintain two URL datasets:

A. Browser/network URLs

https://client.com/api/products?page=2
https://client.com/api/categories/12

These tell you what the application is requesting.

B. SEO/document URLs

https://client.com/products/red-shoe
https://client.com/products/blue-shoe

These are the URLs you ultimately want in the crawl graph.

A crawler that gives you A but not B is useful for debugging the application, but isn't necessarily a good SEO crawler.

Sitebulb's explicit reporting of links created/altered during JavaScript rendering is therefore particularly relevant to your agency workflow. Sitebulb Support Sitebulb

If you're considering building your own

If your agency wants this as an internal crawling platform rather than buying a crawler, I'd use Playwright + Chromium and capture three things simultaneously:

                 ┌─ response HTML
URL ── Chromium ─┼─ rendered DOM
                 └─ network events
                         │
             ┌───────────┴───────────┐
             ↓                       ↓
       XHR/fetch URLs          rendered <a> URLs

Then store an edge like:

source_url
discovered_url
discovery_method
discovered_after_js
request_type
http_status
render_timestamp

That gives you a much richer SEO graph than a conventional crawler.

My practical recommendation: benchmark Sitebulb, JetOctopus, and Screaming Frog against the same nasty client site. If your biggest requirement is “find every SEO URL that appears because of AJAX/JS”, I'd give Sitebulb and JetOctopus the closest look, with Sitebulb having especially clear tooling around JS-created links and response-vs-render analysis. Sitebulb Support Tech SEO Platform

• SPA with client-side routing
• Infinite scroll
• “Load more” button
• Product/category grids populated by XHR/fetch
• Content that appears 2–10 seconds after initial load
• Links injected after an API response
• Ideally a site using service workers

Then create a known ground truth.

For example:

URL A
 ├── /category/shoes          [initial HTML]
 ├── /category/hats           [JS]
 ├── /product/red-shoe         [AJAX → JS]
 └── /product/blue-shoe        [AJAX → JS]

Run each crawler and compare:

I'd make URL discovery accuracy the pass/fail criterion.

One subtle but important distinction

For SEO, I'd actually maintain two URL datasets:

A. Browser/network URLs

https://client.com/api/products?page=2
https://client.com/api/categories/12

These tell you what the application is requesting.

B. SEO/document URLs

https://client.com/products/red-shoe
https://client.com/products/blue-shoe

These are the URLs you ultimately want in the crawl graph.

A crawler that gives you A but not B is useful for debugging the application, but isn't necessarily a good SEO crawler.

Sitebulb's explicit reporting of links created/altered during JavaScript rendering is therefore particularly relevant to your agency workflow.

If you're considering building your own

For an SEO agency, the key question is not just “does it render JS?” but:

1. Does it execute JS in a real browser engine?
2. Does it capture the post-render DOM?
3. Does it discover URLs injected by JS/AJAX/fetch/XHR?
4. Can it do this at your client-site scale and export the data you need?

I’d evaluate tools against those criteria:

What I’d test before buying

1. URL discovery test

Use a staging URL or known JS-heavy page containing:

• infinite scroll products
• filters/facets
• client-side routing
• lazy-loaded links
• API-driven content

Check whether the crawler finds:

• /product/123 links appearing after hydration
• links loaded after clicks
• URLs returned from XHR/fetch calls

A lot of “JS crawlers” only capture the final DOM. That misses URLs that exist only in network responses.

2. Compare raw HTML vs rendered HTML

You want reports showing:

• links only present after JS
• content only present after JS
• title/meta/canonical changes after rendering

Screaming Frog and Sitebulb both specifically support this kind of response-vs-render analysis. Screaming Frog Sitebulb Support

3. Check crawl scale

Rendering is expensive because every URL effectively needs a browser session.

Ask vendors:

• How many rendered URLs/hour?
• Can we run multiple workers?
• Can we throttle per client?
• Can we crawl 500k+ URLs?

4. Check interaction support

Many SEO crawlers do:

GET page
→ load JS
→ wait
→ capture DOM

But modern sites sometimes require:

GET page
→ click menu
→ submit filter
→ scroll
→ trigger API call
→ capture new URLs

If that matters, you probably want Playwright.

My usual agency stack recommendation

For most SEO agencies:

• Primary crawler: Screaming Frog or Sitebulb
• Browser validation: Chrome DevTools + URL Inspection in Google Search Console
• Custom edge cases: Playwright crawler

If your requirement is specifically “capture every AJAX-loaded URL”:

I would lean toward a custom Playwright layer that records network traffic:

• request.url()
• response.url()
• XHR/fetch responses
• DOM-created anchors
• sitemap/API endpoints

Then feed those URLs into your normal SEO crawler.

A good architecture is:

Playwright renderer
        |
        ├── DOM links
        ├── XHR/fetch URLs
        ├── JS-created routes
        └── rendered HTML
                |
                v
       SEO crawler + reporting pipeline

That gives you the crawler coverage of an SEO tool plus the discovery depth of a browser automation framework.

One question that changes the recommendation: are you mainly crawling marketing sites (10k–100k URLs), or large ecommerce/SaaS sites (500k–millions of URLs)? The best choice is quite different.

• request.url()
• response.url()
• XHR/fetch responses
• DOM-created anchors
• sitemap/API endpoints

Then feed those URLs into your normal SEO crawler.

A good architecture is:

Playwright renderer
        |
        ├── DOM links
        ├── XHR/fetch URLs
        ├── JS-created routes
        └── rendered HTML
                |
                v
       SEO crawler + reporting pipeline

That gives you the crawler coverage of an SEO tool plus the discovery depth of a browser automation framework.

One question that changes the recommendation: are you mainly crawling marketing sites (10k–100k URLs), or large ecommerce/SaaS sites (500k–millions of URLs)? The best choice is quite different.

If your requirement is specifically “execute JS, wait for client-side/AJAX requests, then discover the URLs that appear”, I’d evaluate crawlers on a different axis than ordinary JS rendering.

What to test

For each candidate, give it a test site with:

1. Links present only after hydration
   • React/Vue/Next-generated <a href> elements.
2. AJAX/fetch-loaded links
   • Initial HTML has no links; JS calls an API and injects them into the DOM.
3. Delayed requests
   • URLs appear 3–10 seconds after page load.
4. Interaction-dependent discovery
   • “Load more,” tabs, accordions, infinite scroll.
5. SPA routing
   • history.pushState() / client-side routes.
6. URLs exposed only in API responses
   • Important distinction: a crawler may render the DOM correctly but never record URLs that only exist in XHR/Fetch responses.

Then measure:

The important distinction: rendered URLs vs AJAX URLs

Don't assume “supports JavaScript rendering” = “captures AJAX-loaded URLs.”

A typical browser crawler does roughly:

HTML → execute JS → AJAX request → DOM changes → extract links

That's enough if the AJAX response eventually creates <a href="/product/123">.

But suppose the API returns:

{
  "products": [
    {"url": "/product/123"},
    {"url": "/product/456"}
  ]
}

and the site's JavaScript never puts those URLs into the DOM. A DOM-based crawler may never report those URLs as discovered links, despite successfully rendering the page.

So I'd define your requirement as two separate capabilities:

A. DOM discovery: URLs that become links after rendering.

B. Network/API discovery: URLs appearing in XHR/fetch responses, even when they aren't subsequently represented as links.

That distinction is probably the most important thing to put into your vendor evaluation.

Current tools I'd put on the shortlist

Sitebulb is particularly interesting for the SEO-agency use case. Its Chrome Crawler uses headless Chrome, and its Response vs Render reporting explicitly identifies links created or modified by JavaScript. support.sitebulb.com

It also lets you control rendering behavior such as the load event and render timeout, which matters when AJAX content arrives late. support.sitebulb.com

Sitebulb JavaScript crawling guide

I'd still test Sitebulb's behavior against your actual AJAX patterns, rather than assuming its rendered-link discovery means it captures arbitrary XHR URLs.

For an agency, I'd use this scoring model

Give each tool 0–5:

• JS execution: 15%
• Rendered DOM URL discovery: 20%
• XHR/fetch URL discovery: 20%
• Delayed/AJAX rendering controls: 10%
• Click/scroll interaction: 10%
• Scale/concurrency: 10%
• Raw-vs-rendered diagnostics: 10%
• API/export: 5%

I'd weight XHR/fetch URL discovery heavily, because that's where many otherwise-good SEO crawlers fall short.

If you're deciding between Screaming Frog, Sitebulb, Lumar, Oncrawl, JetOctopus, or building a Playwright/Chromium crawler internally, I can also give you a feature-by-feature comparison specifically for AJAX URL discovery and agency-scale crawling.

For an SEO agency, I’d choose based on what you need to observe, not just “does it render JS?” The key distinction is:

• DOM rendering: executes JavaScript and shows you the final HTML after React/Vue/Angular/etc. runs.
• Network discovery: captures XHR/fetch requests, API endpoints, JSON responses, and URLs generated after AJAX calls.
• Link extraction after rendering: discovers URLs that only exist after JS execution.
• Scale/cost: whether you can run this across millions of URLs.

Googlebot-style rendering is now based on headless Chromium, and rendered content/links can be discovered after JavaScript execution, but third-party crawlers vary significantly in how deeply they emulate this. Google for Developers

What I’d shortlist

1. Screaming Frog SEO Spider — best general agency crawler

screamingfrog.co.uk

Best when you need:

• JS-rendered DOM crawling
• rendered vs raw HTML comparison
• JS-injected links
• metadata changes after JS
• screenshots/render inspection
• agency workflows and exports

It uses a headless browser rendering mode and is specifically designed for JavaScript SEO auditing. Screaming Frog

Limitations:

• It is not primarily a “capture every AJAX endpoint” tool.
• Large SPA sites can become slow because every URL needs browser rendering.

Pick it if: 80% of your work is technical SEO audits.

2. Custom Playwright crawler — best if you need AJAX/API URL capture

playwright.dev

For a serious SEO agency, this is often the most flexible approach.

You can capture:

• browser requests: - fetch()
• XHR
• GraphQL calls
• JSON APIs
• JS bundles
• responses
• redirects
• generated links
• DOM after hydration

Typical workflow:

1. Launch Chromium.
2. Visit URL.
3. Wait for network idle or custom conditions.
4. Listen to: - request
5. response
6. framenavigated
7. DOM mutations.
8. Extract: - URLs from DOM
9. URLs from JSON payloads
10. canonical/hreflang/meta after rendering.

Example signals you’d capture:

https://site.com/products?page=2
https://api.site.com/graphql
https://cdn.site.com/data/products.json

Advantages:

• Unlimited customization.
• Can handle infinite scroll, filters, SPAs.
• Can reproduce user journeys.

Downside:

• You build and maintain the crawler.

Pick it if: clients have marketplaces, React apps, infinite scroll, faceted navigation, or JS-generated URLs.

3. Sitebulb — strong agency UX

sitebulb.com

Good for:

• JavaScript crawling
• visual audits
• explaining issues to clients

Less ideal if your requirement is specifically:

“Show me every AJAX-loaded URL/API call.”

4. Ahrefs Site Audit — useful for external visibility

ahrefs.com

Ahrefs has JavaScript rendering capability in Site Audit and uses it to identify some JS-generated content and links. Ahrefs

Good for:

• monitoring large sites
• recurring audits

Less good for:

• debugging exactly which API call created a URL.

The features I would score vendors against

My agency setup recommendation

For most SEO agencies:

Tier 1: Standard audits

• Screaming Frog in JS rendering mode
• Google Search Console URL Inspection for verification

Tier 2: JS-heavy enterprise clients

• Playwright crawler + Screaming Frog
• Store: - rendered HTML
• request logs
• discovered URLs
• API endpoints
• screenshots

Tier 3: Massive sites (millions of URLs)

• Custom Playwright workers
• Queue system
• Persistent browser contexts
• URL deduplication database
• Crawl budget controls

A useful test before buying anything: take a client SPA and check whether the tool can answer all five:

1. “What URLs exist after JavaScript runs?”
2. “Which URLs came from XHR/fetch?”
3. “Which links are missing from raw HTML?”
4. “Which API responses contain indexable content?”
5. “What does the page look like after 10 seconds, not 1 second?”

If a crawler cannot answer #2 and #5, it is usually not enough for heavy JavaScript SEO work.

If your requirement is specifically “execute JavaScript, wait for async/AJAX work, then discover the URLs that appeared”, don't evaluate crawlers solely on whether they say “JavaScript rendering.” There are two separate capabilities:

1. Rendered DOM extraction — finds links/content injected into the DOM after JS executes.
2. Network/AJAX capture — observes XHR/fetch requests themselves, including URLs that may never become <a href> links.

For an SEO agency, I'd test tools against both.

What I'd shortlist

• sitebulb.com — strong choice for normal JS SEO auditing. Its Chrome Crawler uses headless Chromium, and its Response vs Render reporting specifically identifies links created or modified by JavaScript. Sitebulb Support Sitebulb Support
• jetoctopus.com — particularly interesting for your use case because its JS crawler explicitly exposes assets loaded during rendering and an AJAX requests & error detection view. It also positions itself for very large crawls. Tech SEO Platform
• screamingfrog.co.uk — excellent agency workhorse and very configurable, but I'd specifically test its handling of your AJAX patterns rather than assuming “JavaScript rendering” means network-request discovery.
• Enterprise crawlers such as Lumar/Oncrawl are worth evaluating if you're doing millions of URLs, but compare their rendering/network-data limits and pricing carefully.

The key distinction

Suppose the page does this:

GET /category
   ↓
JavaScript executes
   ↓
fetch("/api/products?page=2")
   ↓
JSON response
   ↓
JS creates:
<a href="/product/widget-123">

A useful SEO crawler should ideally expose:

Rendered DOM URL:
    /product/widget-123

Network request:
    /api/products?page=2

Relationship:
    /api/products?page=2 → generated /product/widget-123

The first is rendered-link discovery. The second is AJAX/network discovery. Don't assume a crawler that does the first necessarily does the second.

How I'd run your vendor bake-off

Take 5–10 representative client sites and create deliberately difficult test cases:

• React/Vue/Next SPA
• links injected after hydration
• fetch() returning JSON containing URLs
• XHR returning product/category data
• “Load more” button
• infinite scroll
• links generated after a 2–5 second delay
• URLs appearing only after an interaction
• client-side redirects
• API calls that return HTML fragments
• links present in the rendered DOM but absent from source HTML

Then score each crawler on:

I'd weight network/AJAX capture heavily if that's genuinely a requirement. A beautiful rendered-DOM report isn't equivalent to a network-aware crawler.

Sitebulb is particularly good for the “what URLs did JavaScript add to the page?” side: its Response vs Render report compares source and rendered HTML and surfaces links affected by JavaScript. Sitebulb Support Sitebulb Support JetOctopus is more compelling if you specifically want to inspect requests made during rendering, since its JS crawler advertises AJAX-request/error detection. Tech SEO Platform

One important architectural question

If by “captures AJAX-loaded URLs” you mean the URLs requested by fetch/XHR, rather than URLs ultimately exposed as links, I'd actually consider a Playwright/Chromium-based custom crawler alongside your SEO crawler.

That gives you direct access to:

page.goto()
network request events
XHR/fetch URLs
response status
response body
rendered DOM
links extracted after rendering
console errors

Then you can feed those observations into your normal SEO crawl dataset. For an agency dealing with unusual JS implementations across many clients, that can be much more powerful than trying to force an SEO crawler to expose every browser-network event.

My practical recommendation: trial Sitebulb + JetOctopus + Screaming Frog against the same JS-heavy client sites. If network-level AJAX discovery is a hard requirement, put that criterion ahead of the usual SEO feature checklist—JetOctopus deserves particular attention there.

Yes. For an SEO agency, I’d evaluate these tools less on “does it execute JavaScript?” and more on “does it execute JS long enough, and does it turn the resulting DOM/AJAX state into crawlable URLs?”

What I’d test

Use the same 3–5 representative client sites and score each crawler on:

The two I'd put through a bake-off first

Screaming Frog SEO Spider is a strong choice if you want a very configurable agency workhorse. Its JavaScript rendering mode crawls both the original and rendered HTML, identifies client-side-only links/content, and provides rendered-page screenshots. It also specifically exposes an AJAX timeout, which is relevant to your use case. www.screamingfrog.co.uk

Sitebulb is particularly attractive if your core requirement is “show me what changed after rendering.” Its Chrome Crawler uses headless Chrome, and its Response vs Render reporting identifies links created or modified by JavaScript. support.sitebulb.com

Sitebulb also explicitly supports increasing the render timeout when JS-heavy pages need additional time for asynchronously loaded content. support.sitebulb.com

For an agency, there's another interesting consideration: Sitebulb currently says JavaScript crawling is included without an additional rendering charge, including at Cloud scale. sitebulb.com

But here's the important distinction

If by “captures AJAX-loaded URLs” you mean:

Page loads → JS makes /api/products?page=2 → response contains product data → JS creates <a href="/product/foo"> → crawler discovers /product/foo

then DOM rendering is sufficient if the crawler waits until that JavaScript has executed.

But if you mean:

“Give me every URL that appeared in XHR/fetch responses, even if JavaScript never converted it into an <a>.”

that's a different requirement.

You need network-level capture in addition to rendered DOM extraction. I'd specifically ask vendors whether they expose:

• XHR/fetch request URLs
• response URLs
• redirects encountered by those requests
• URLs subsequently injected into the DOM
• requests made after initial load
• service-worker requests
• API calls made after user interaction

A crawler can render a page perfectly and still not give you the underlying API URL inventory.

My agency test

I'd create a deliberately nasty test page containing:

1. 10 normal HTML links
2. 10 links injected immediately by JS
3. 10 links injected after a 2–3 second fetch()
4. 10 links generated from an XHR response but not present in the response HTML
5. Links appearing only after scrolling
6. Links appearing after a button click
7. A service-worker-mediated request
8. A slow API response
9. An API request that returns a URL but doesn't create an <a>
10. A page where JS changes canonical/title/robots

Then compare:

Initial HTML → rendered DOM → network requests → crawler's discovered URL set.

The winner isn't simply the crawler with the prettiest “JavaScript rendering” checkbox. It's the one that gives you the highest recall of the URLs your browser actually reaches, while letting you diagnose why a URL wasn't discovered.

For your specific use case, I'd start with Screaming Frog vs. Sitebulb, then add an enterprise crawler if you routinely have millions of URLs. Screaming Frog's own documentation specifically recommends JavaScript rendering for sites where links are generated client-side, while Sitebulb has particularly strong response-vs-render diagnostics. www.screamingfrog.co.uk

If you tell me roughly how many URLs/client/month you're crawling and whether you need network-level XHR URL extraction or just post-AJAX <a> discovery, I can narrow this down to the best 2–3 options and the exact settings I'd use.

For an SEO agency, I’d evaluate these tools less by “does it run Chrome?” and more by whether it can discover URLs that only exist after JavaScript executes. A lot of crawlers can render a screenshot; fewer can reliably capture DOM mutations, SPA routes, fetch/XHR-discovered URLs, and AJAX-loaded internal links.

Here’s the framework I’d use:

Shortlist by use case

1. Best SEO crawler workflow: Screaming Frog SEO Spider

Good fit if your agency already works in spreadsheet-style audits.

Strengths:

• JavaScript rendering via headless Chromium
• Compares original HTML vs rendered HTML
• Finds links added client-side
• Familiar SEO workflow

It supports switching the crawler into JavaScript rendering mode and auditing client-side dependencies. www.screamingfrog.co.uk

Weakness:

• Less elegant for deep SPA exploration
• Network-level AJAX analysis is not its main strength

Best for:

• “Does Google see this?”
• Technical SEO audits
• Migrations
• Large client reporting

2. Best SEO-focused JS auditing: Sitebulb

Strong option if your deliverable is a client-facing audit.

Strengths:

• Chrome-based rendering
• Response HTML vs rendered DOM comparison
• Highlights links/content changed by JavaScript
• Better visual explanations for clients

Sitebulb’s Chrome Crawler renders pages with headless Chrome and can compare response vs rendered output, including JavaScript-created links. support.sitebulb.com

Best for:

• Explaining JS SEO issues to clients
• Finding “the link exists in the browser but not in source”
• Recurring audits

3. Best for true JS app crawling: build around Playwright

If you have engineering resources, this is usually the most powerful approach.

You can capture:

• page.on('request')
• page.on('response')
• XHR/fetch calls
• route changes
• DOM mutations
• console errors
• screenshots
• accessibility tree
• structured data after rendering

Typical agency architecture:

URL queue
   ↓
Playwright Chromium workers
   ↓
Load page
   ↓
Wait for network idle + custom events
   ↓
Capture:
  - final DOM
  - canonical/title/meta
  - internal links
  - XHR URLs
  - JS errors
  - screenshots
   ↓
Store crawl graph

Best for:

• React/Next/Vue SPAs
• SaaS apps
• Sites where navigation happens without <a href>
• AJAX-heavy ecommerce

Weakness:

• You have to build crawl logic, deduplication, rendering rules, storage, etc.

4. Enterprise crawling: Lumar / Botify

Worth considering when clients have millions of URLs.

Good for:

• Enterprise reporting
• Log-file integration
• Large crawl infrastructure

Usually overkill unless your agency handles very large sites.

The test I would run before buying

Take 5–10 representative client URLs:

1. A React product page
2. A category page with filters
3. An infinite-scroll page
4. A page where links appear after API calls
5. A page requiring a menu interaction

For each tool, measure:

URL discovery

After rendering, does it find:

Raw HTML:
<a href="/category">

Rendered DOM:
<a href="/product/123">

Network discovery

Does it see:

GET /api/products?page=2
GET /graphql
GET /search/results?q=shoes

Crawl graph quality

Can it answer:

• “Which URLs are only reachable through JS?”
• “Which internal links are injected after load?”
• “Which templates produce orphan pages?”

My recommendation for an SEO agency

A common stack would be:

• Screaming Frog or Sitebulb → standard audits and client reports
• Playwright crawler → difficult JS-heavy clients
• Google Search Console URL Inspection → sanity check against Google’s rendering

If your main pain point is specifically “capture AJAX-loaded URLs”, prioritize tools with network interception, not just DOM rendering. A rendered DOM crawler can still miss URLs that only exist in API responses or client-side routing logic.

If your key requirement is “execute the client-side JS, wait for AJAX/fetch/XHR content, then discover the URLs that appear afterward,” don’t evaluate crawlers on “supports JavaScript” alone. The important distinction is DOM rendering vs. network-level discovery.

What I’d evaluate

For an SEO agency, I'd use a two-layer approach

1. Normal HTTP crawl first

Crawl the site cheaply and collect:

• source HTML links
• HTTP status codes
• canonicals
• robots directives
• sitemap URLs
• JS/CSS resources
• internal/external links

2. Render selected pages in Chromium

Then collect three different URL sets:

A = URLs found in raw HTML
B = URLs found in rendered DOM
C = URLs requested by the browser (XHR/fetch/navigation/etc.)

JS-discovered DOM links = B - A
AJAX/network URLs       = C - A

That distinction is extremely useful in client audits.

For example:

Raw HTML
  └── /category/shoes

After JS execution
  ├── /product/nike-air-max
  ├── /product/adidas-ultra
  └── /product/asics-gel

Network requests
  ├── /api/products?category=shoes
  ├── /api/recommendations
  └── /analytics/...

You don't necessarily want to treat every item in C as a crawlable SEO URL. An API endpoint isn't equivalent to an HTML page. But C tells you how the page obtained its data, while B tells you which actual navigable URLs became discoverable.

Off-the-shelf: Screaming Frog is a strong benchmark

Screaming Frog's SEO Spider is a good baseline for this use case. Its JavaScript mode runs pages in headless Chrome and crawls links from the rendered HTML. It explicitly reports “Contains JavaScript Links” for links that weren't present in the raw HTML, and those can be bulk exported. www.screamingfrog.co.uk

It also lets you store both the original and rendered HTML, which is excellent for agency QA and client evidence. www.screamingfrog.co.uk

Screaming Frog JavaScript crawling guide

But: if by “captures AJAX-loaded URLs” you specifically mean “give me every URL the browser requested via XHR/fetch”, I'd test that separately. Rendered-DOM link discovery and network-request capture are different capabilities.

If you're building your own crawler

I'd make Chromium instrumentation a first-class part of the architecture:

                 ┌───────────────┐
URL queue ──────►│   Chromium    │
                 │               │
                 │ execute JS    │
                 │ wait/settle   │
                 └───────┬───────┘
                         │
             ┌───────────┼────────────┐
             ▼           ▼            ▼
         DOM links    XHR/fetch    resources
             │           │
             ▼           ▼
        crawl queue   diagnostics

Capture at least:

• page.on('request')
• page.on('response')
• request URL + method
• resource type
• initiator
• status
• redirect chain
• response MIME type

Then separately parse the final DOM for <a href>, canonical, hreflang, etc.

The important design decision is when you snapshot the DOM. For AJAX-heavy clients, I'd support:

networkidle
OR
selector appears
OR
custom JS condition
OR
N seconds after navigation

rather than just “wait 5 seconds.”

Screaming Frog, for example, exposes an AJAX timeout specifically because content can arrive after the initial page load; its default is 5 seconds and it recommends adjusting it when pages aren't fully rendered. www.screamingfrog.co.uk

My practical tool-selection test

Before committing to a crawler, give each candidate the same 5-page test suite:

1. React/Next/Vue page where links exist only after hydration.
2. Infinite-scroll category where products arrive through fetch.
3. “Load more” button that fires XHR and injects links.
4. Delayed API response arriving 8–10 seconds after navigation.
5. SPA navigation where clicking a UI element changes the URL without a full reload.

For each, measure:

Discovery

• Did it find the injected <a> URLs?
• Did it find URLs from subsequent pagination/infinite-scroll states?
• Does it follow discovered JS links?

Rendering

• Is the final DOM correct?
• Can you save rendered HTML?
• Can you screenshot the result?

Network

• Can you export XHR/fetch URLs?
• Does it distinguish API calls from navigations?
• Does it retain status/MIME/request metadata?

Scale

• pages/hour with JS off
• pages/hour with JS on
• RAM per concurrent browser
• failure rate
• retry behavior

If you're choosing between Screaming Frog, Sitebulb, a cloud crawler, and building a Playwright/Puppeteer crawler, I can also give you a feature-by-feature comparison specifically for agency-scale JS/AJAX crawling.

For an SEO agency, I’d choose based on what “captures AJAX-loaded URLs” actually means. There are two different requirements:

1. Execute JS and discover <a href> links that appear after rendering
2. Observe URLs requested by XHR/fetch/API calls, even when those URLs never become <a> links

Most SEO crawlers are good at #1. If you specifically need #2, you want a browser/network-capture workflow rather than just a rendered crawler.

My shortlist

Screaming Frog's JavaScript mode executes client-side JS in headless Chromium and extracts links from the rendered HTML, while also discovering links in the original HTML. www.screamingfrog.co.uk

Sitebulb similarly uses a Chrome crawler and gives you a particularly useful response-vs-render comparison, including links created or modified by JavaScript. support.sitebulb.com

Oncrawl is the stronger candidate when you're talking about very large sites; it advertises JS rendering at large scale, including millions of URLs. help.oncrawl.com

The important distinction for AJAX

Suppose a page initially contains:

<div id="products"></div>
<script>
  fetch('/api/products?page=2')
    .then(...)
</script>

After rendering, the DOM might contain:

<a href="/products/widget-123">Widget 123</a>

A rendered crawler can discover /products/widget-123.

But if you want to capture:

/api/products?page=2

because the browser requested it, that's a different job. You need to listen to browser network events.

That's where I'd use Playwright (or Puppeteer) alongside your conventional SEO crawler. You can intercept request events, record XHR/fetch URLs, status codes, request type, initiator, etc., while letting Chromium execute the site normally.

What I'd do for an SEO agency

I'd build the workflow in two layers:

Layer 1 — Screaming Frog or Sitebulb

Use it for the actual SEO crawl:

• rendered HTML
• canonical/meta/robots
• rendered internal links
• response-vs-render differences
• status codes
• redirects
• pagination
• hreflang
• structured data
• crawl-depth/internal-link analysis

Screaming Frog is particularly attractive if you want lots of crawl configuration and custom extraction; Sitebulb is attractive if you want the rendered-vs-source analysis presented more directly for client audits. Sitebulb also lets you adjust render timeout when JS-heavy sites aren't finished loading before the snapshot. support.sitebulb.com

Layer 2 — Playwright network crawler

Run a representative sample—or specific problematic templates—and capture:

document requests
XHR
fetch
WebSocket
script
iframe
image
stylesheet

For each page, store something like:

source_url
requested_url
resource_type
status
method
content_type
initiator
timestamp

Then you can answer much more interesting agency questions:

“Which URLs does this SPA actually request when rendered?”

“Which API endpoints generate indexable product URLs?”

“Which URLs are discovered only after hydration?”

“Which templates make 40 API calls before producing their links?”

One feature I'd make non-negotiable

Configurable “page is finished” behavior.

A naïve crawler that waits for load and immediately snapshots the DOM can miss AJAX content that arrives afterward. Sitebulb explicitly exposes considered-load events and render timeout because this is a real problem on JS-heavy sites. support.sitebulb.com

For your own Playwright crawler, I'd use a combination of:

• domcontentloaded
• network-idle-ish behavior
• a configurable post-load delay
• optionally a site-specific selector/event
• maximum render timeout

Don't rely on networkidle alone—modern sites can maintain analytics, WebSockets, polling, ads, etc. indefinitely.

My recommendation

If you're buying one tool for an agency, I'd start with Screaming Frog or Sitebulb.

If your requirement is specifically “find every URL that gets requested/created as a consequence of JavaScript/AJAX”, don't expect an SEO crawler to be the complete solution. Use a Playwright-based network collector in addition to your SEO crawler.

The sweet spot is:

Screaming Frog/Sitebulb → SEO crawl + rendered links
Playwright → browser execution + XHR/fetch/network URL inventory

That gives you both the SEO interpretation and the raw browser behavior, which is much more useful when debugging modern React/Next/Vue ecommerce sites.

If you're an SEO agency, I’d separate the requirement into two different capabilities:

1. Render JavaScript and extract the final DOM
2. Observe network activity and discover URLs that never appear as <a href> links — e.g. XHR/fetch/API calls, redirects, dynamically constructed URLs.

That distinction matters because a crawler can be excellent at #1 without actually giving you #2.

What I’d evaluate

Off-the-shelf SEO crawler vs custom browser crawler

For conventional JS SEO auditing, Sitebulb is worth testing. Its Chrome Crawler uses headless Chrome and compares response HTML with rendered HTML, including links created or modified by JavaScript. support.sitebulb.comsupport.sitebulb.complaywright.dev

It also has configurable render timeouts, which is important for AJAX-heavy sites: you don't want the crawler taking its snapshot before the API response has populated the DOM. support.sitebulb.com

Sitebulb JavaScript crawler

But if "capture every AJAX-loaded URL" is a core requirement, I'd lean toward a Playwright-based crawler rather than selecting an SEO crawler solely on its rendered-DOM capability.

Playwright exposes page network events, including requests and responses, and explicitly supports tracking XHR/fetch traffic. playwright.dev

Conceptually your crawler becomes:

URL
 ↓
Launch Chromium
 ↓
Load page
 ↓
Execute JavaScript
 ↓
Wait for appropriate completion condition
 ↓
Capture:
   ├─ final DOM URLs
   ├─ <a href> URLs
   ├─ canonical/hreflang/etc.
   ├─ redirects
   ├─ XHR URLs
   ├─ fetch URLs
   └─ optionally URLs found inside response bodies
 ↓
Normalize/dedupe
 ↓
Add newly discovered crawlable URLs to queue

The important trap: "AJAX URL" can mean several things

Suppose a product page does:

GET /product/123
       ↓
JS executes
       ↓
fetch("/api/recommendations?id=123")
       ↓
JSON response contains:
   /product/456
   /product/789

A basic JS crawler may capture:

/api/recommendations?id=123

but not necessarily /product/456 and /product/789.

So I'd define your crawler's discovery layers explicitly:

Layer 1 — DOM URLs

Extract URLs from:

• <a href>
• canonical
• hreflang
• images
• scripts
• structured data
• rendered HTML attributes

Layer 2 — network URLs

Capture:

• fetch
• XHR
• document requests
• redirects
• optionally WebSockets/service-worker traffic

Layer 3 — URLs inside network payloads

Parse JSON/HTML responses for URL-like values.

This is where a custom Playwright crawler can become considerably more powerful than a conventional SEO crawler.

Don't use networkidle as your only definition of "loaded"

This is particularly important for modern sites.

A site may continuously poll analytics, personalization, chat, recommendations, etc. Conversely, an API request may happen several seconds after the initial page load.

I'd support several strategies:

• load
• DOMContentLoaded
• fixed post-load delay
• specific selector appears
• specific network response occurs
• custom JavaScript condition
• maximum render timeout

That lets you configure something like:

Load → wait for .product-grid → wait for /api/products → snapshot DOM → finish.

Sitebulb similarly exposes configurable load/timeout behavior because the timing of the rendered snapshot can materially affect what gets captured. support.sitebulb.com

My recommendation for an agency

I'd actually use two tiers:

Tier 1: normal crawling

Use a fast HTTP crawler for the majority of URLs.

Tier 2: JS rendering

Send only JS-dependent URLs through Chromium.

Tier 3: forensic JS crawl

For difficult client sites, run Playwright with full network capture and optionally response-body parsing.

That gives you the economics of a traditional crawler without rendering 500,000 pages unnecessarily.

And when presenting results to clients, the response-vs-render comparison is especially valuable: you can say "this canonical/link/content exists only after JavaScript executes" rather than merely reporting that the page looks fine in Chrome. Sitebulb specifically provides this kind of comparison at scale. support.sitebulb.com

Bottom line: if you mainly need rendered SEO auditing, evaluate Sitebulb/Screaming Frog/Lumar-style crawlers. If your differentiator is "show me every URL the browser discovers through JavaScript/AJAX", evaluate the tool based on its network interception + response parsing + interaction capabilities, and Playwright is a particularly strong foundation for that. playwright.dev

If you tell me your typical crawl size (e.g. 10k vs 1M URLs) and whether you need API endpoints themselves or URLs contained inside API responses, I can narrow this down to an architecture and tool shortlist.

Yes. For an SEO agency, I’d evaluate JS crawlers on more than “does it execute JavaScript?” The key distinction is whether the crawler can capture the post-render DOM, URLs discovered after rendering, and the network/AJAX activity that produced them.

What I’d test

The important distinction: DOM URLs vs AJAX URLs

I'd explicitly separate these in your evaluation.

Scenario A — JS injects an internal link

<div id="products"></div>

<script>
  fetch("/api/products")
    .then(...)
    .then(products => {
       // creates <a href="/products/widget-123">
    })
</script>

A good JS crawler should eventually see /products/widget-123 in the rendered DOM and add it to the crawl graph.

Scenario B — AJAX discovers data but never creates an <a>

GET /api/products?page=2
GET /api/products?page=3
GET /api/recommendations

Those URLs may never appear as links in the DOM. If you're trying to inventory AJAX endpoints, a crawler that only parses rendered HTML isn't enough—you need network interception/request logging.

That's the test I'd use to eliminate a lot of otherwise-good SEO crawlers.

Tools I'd put on your shortlist

Sitebulb is particularly interesting for agency SEO work. Its Chrome Crawler uses Chromium, and it can compare response HTML against rendered HTML and identify links created or modified by JavaScript. sitebulb.com

It also has configurable render timeouts, which matters because AJAX-heavy sites may not finish rendering immediately. support.sitebulb.com

JetOctopus is another strong candidate if scale is important. Its JS crawler specifically advertises capturing AJAX requests, JS-injected content/links, blocked resources and client-side redirects, and it is designed for very large cloud crawls. jetoctopus.com

My agency-oriented decision rule

I'd score tools roughly like this:

1. Rendering fidelity — 25%
Does the rendered DOM actually match Chrome? Test hydration, delayed API calls, lazy loading, redirects and SPA routing.

2. URL discovery — 25%
Test three buckets separately:

• URLs in initial HTML
• URLs created in the rendered DOM
• URLs appearing only in XHR/fetch/network activity

3. Debugging visibility — 20%
Can you see why a URL/content element wasn't discovered? Network requests, JS errors, blocked resources, timing, response codes, etc.

4. Scale — 15%
Pages/hour, concurrent Chromium instances, cloud vs local, cost per million pages, and whether rendering limits are separate from normal crawl limits.

5. Automation/export — 15%
API, CSV/JSON exports, webhooks, scheduled crawls, and whether you can feed discovered URLs into another crawl.

One particularly useful benchmark

Before committing to a product, build a 10–20 page synthetic test site containing:

1. Static <a> links
2. Links injected immediately by JS
3. Links injected after a 2–5 second delay
4. Links generated after fetch()
5. URLs returned by XHR but never put into the DOM
6. Infinite-scroll URLs
7. SPA history.pushState() URLs
8. Links behind a “Load more” button
9. A failed API request
10. A slow API request
11. A redirect generated by JS
12. Canonical/title/meta changes after hydration

Then require each vendor to give you:

initial HTML → rendered DOM → discovered links → network requests → final crawl graph

That will tell you far more than vendor claims like “supports JavaScript.”

For your specific requirement, I'd start with Sitebulb and JetOctopus, but I'd make network/AJAX capture the deciding test rather than simply JavaScript rendering. Sitebulb is particularly strong for rendered-vs-response SEO analysis; JetOctopus looks more compelling if your primary requirement is large-scale JS/AJAX crawling. jetoctopus.com

If you're considering building your own crawler, the architecture changes considerably—Playwright/Chromium + request interception + DOM extraction + URL canonicalization is probably the right baseline.

If your key requirement is “render the page, let JavaScript execute, wait for AJAX/fetch/XHR content, and discover URLs that only appear after that”, I’d evaluate crawlers on a slightly different axis than just “does it support JavaScript?”

The capabilities I’d require

1. Real Chromium rendering
   • Headless Chrome/Chromium, not just parsing <script> tags.
   • Ideally an evergreen Chromium version.
   • This lets React/Vue/Next/etc. actually execute and mutate the DOM. sitebulb.comsitebulb.com

2. Rendered-DOM link extraction
   • The crawler should extract <a href> links after JS execution, not just from the initial HTTP response.
   • This is crucial for SPA navigation, “load more” interfaces, client-side menus, etc.
   • Sitebulb, for example, explicitly reports links created or modified by JavaScript. support.sitebulb.comsupport.sitebulb.com

3. A configurable render/wait strategy
   • This is probably the most important requirement for your AJAX use case.
   • A crawler that renders Chrome but snapshots the DOM 500 ms after page load can still miss:

HTML → JS → fetch() → API response → DOM insertion → link

• You want configurable render timeout and/or network-idle/JavaScript execution behavior.

4. Response-vs-render comparison
   • You want to know where a URL came from:
     • response HTML
     • rendered DOM
     • JavaScript-created/modified link
   • Sitebulb's Response vs Render report is specifically designed for this comparison. support.sitebulb.comsupport.sitebulb.com

5. Ideally, network/XHR visibility
   • This is a separate capability from rendering.
   • If your definition of “capture AJAX-loaded URLs” means capture URLs requested by fetch, XHR, GraphQL, etc., rather than merely links eventually inserted into the DOM, ask specifically whether the crawler exports the browser network log.
   • Many SEO crawlers render JS but do not treat every API request as a crawlable URL, which is an important distinction.

I'd test tools with this fixture

Before buying/standardizing on a crawler, give each vendor a test page containing:

<!-- Initial response -->
<a href="/static-page">Static</a>

<script>
  // After 1 second, fetch an API.
  setTimeout(async () => {
    const data = await fetch('/api/products').then(r => r.json());

    // API returns /products/widget-123
    document.querySelector('#products').innerHTML =
      `<a href="${data.url}">Product</a>`;
  }, 1000);
</script>

Then test whether the crawler reports:

That last distinction matters enormously. “Supports JavaScript” doesn't necessarily mean “captures AJAX URLs.”

For an SEO agency, I'd shortlist like this

Sitebulb is particularly compelling if your priority is SEO auditing + rendered links/content. Its Chrome Crawler uses Chromium, and its Response vs Render reporting explicitly identifies links created/changed by JavaScript. It also supports large crawls—up to 500k URLs on Desktop and up to 10m on Cloud according to its current site. sitebulb.comsitebulb.com

Sitebulb JavaScript crawler documentation

Screaming Frog is worth testing if you want a very configurable desktop crawler. I'd specifically benchmark its JS rendering against your AJAX fixture rather than assuming that “JavaScript rendering” satisfies the network-level requirement.

For true AJAX/API discovery, I'd also consider building a small Playwright/Chromium crawler alongside your SEO crawler. That gives you explicit control over:

• request / response events
• XHR/fetch filtering
• network logs
• waiting conditions
• DOM snapshots
• redirects
• client-side routing
• URL normalization/deduplication

That architecture can give you two datasets:

SEO-discoverable URLs

URLs present in response HTML + rendered DOM

Browser-request URLs

URLs actually requested by the page's JavaScript

Those are not the same dataset, and for a technical SEO agency I'd keep them separate rather than treating every AJAX endpoint as a page URL.

My decision rule

If your requirement is:

“Find pages that become internally linked only after JavaScript runs.”

→ prioritize Chromium + rendered-DOM extraction + response/render comparison.

If it's:

“Find every URL the browser requests through AJAX/fetch/XHR.”

→ prioritize browser network interception/API export. A conventional SEO crawler may not be sufficient.

If it's:

“Do both at agency scale.”

→ I'd use an SEO crawler for the audit/reporting layer plus a Playwright-based network collector for the JS/AJAX layer.

That last setup is usually the most robust because you don't accidentally interpret an API endpoint like /api/products?page=2 as an indexable HTML URL.

For an SEO agency, I’d choose based on what you mean by “capture AJAX-loaded URLs” because there are two different problems:

1. Find URLs that appear after JS execution (DOM links injected by React/Vue/etc.)
2. Find URLs requested by AJAX/fetch/XHR calls (API endpoints, JSON feeds, infinite-scroll sources, app routes)

Most SEO crawlers handle #1 well; fewer handle #2 deeply.

What to evaluate

Tools I’d shortlist

Screaming Frog SEO Spider

Best for: technical SEO teams that want raw crawl data

• Strong control over crawl settings, extraction, and exports.
• Supports JavaScript rendering through Chromium.
• Good for comparing raw HTML vs rendered output. sitebulb.com
• Better choice if your team wants to dig into individual URLs, custom extraction, regex, APIs, etc. pikaseo.com

Weakness:

• It is not primarily a browser automation/network analysis tool, so it is not the first choice if your goal is "show me every XHR request fired."

Sitebulb

Best for: agency audits and client reporting

• Uses a Chromium-based crawler for JS rendering. sitebulb.com
• Has useful response-vs-render comparisons.
• Can identify links created or changed by JavaScript. support.sitebulb.com

Weakness:

• Less of a raw data extraction workbench than Screaming Frog.

Lumar / Oncrawl

Best for:

• Large enterprise sites
• Scheduled crawls
• Millions of URLs
• Teams needing dashboards and collaboration

Weakness:

• Cost and complexity.

Build-your-own with Playwright

Best for:

• An SEO agency with engineering resources
• Need to capture actual AJAX/network behavior

A typical stack:

• Playwright Chromium
• CDP network listeners
• URL extraction from:
  • rendered DOM
  • request events
  • response events
  • JS route changes
  • sitemap feeds

This gives you things commercial SEO crawlers often do not expose:

• /api/products?page=20
• GraphQL queries
• JSON responses
• dynamically generated URLs
• infinite scroll endpoints

Example signals to capture:

page.on("request", request => {
  // capture XHR/fetch URLs
})

page.on("response", response => {
  // inspect JSON endpoints
})

page.locator("a").evaluateAll(...)

My agency-oriented recommendation

For most SEO agencies:

Tier 1: daily audits

• Screaming Frog + JS rendering

Tier 2: client-facing reporting

• Sitebulb

Tier 3: JS-heavy ecommerce / SaaS / SPA clients

• Add a Playwright crawler that captures network activity

I would run a bake-off on 5 representative client sites:

1. React/Next.js site
2. Ecommerce with infinite scroll
3. Faceted navigation
4. Logged-out SPA dashboard
5. Large marketing site

Measure:

• URLs found from source HTML
• URLs found after rendering
• XHR/API endpoints discovered
• crawl speed
• memory usage
• false positives
• export quality

The biggest mistake is choosing a crawler that "renders JavaScript" but only extracts the final DOM. If your SEO audits care about where URLs are coming from, you want network interception, not just rendering.

For an SEO agency, I’d choose based less on “does it render JS?” and more on how accurately it reproduces a browser crawl and how well it discovers URLs created after JavaScript execution (client-side routing, API responses, infinite scroll, injected links, etc.).

A good evaluation framework:

Shortlist by agency use case

1. Best all-around SEO crawler: Screaming Frog

Good fit if your team does technical audits manually and needs deep control.

Strengths:

• JavaScript rendering via headless Chromium
• Compares rendered vs source HTML
• Finds links that only appear after JS execution
• Strong exports, custom extraction, regex/XPath workflows

Its JavaScript mode executes client-side JavaScript and extracts from rendered HTML, including links discovered after rendering. www.screamingfrog.co.uk It also exposes JavaScript-only links through reports/export workflows. www.screamingfrog.co.uk

Weakness:

• It is not primarily a “capture every network request/API endpoint” crawler.
• Large JS-heavy crawls can become resource intensive.

Best for:

• 10k–1M URL audits
• migrations
• technical SEO retainers
• teams that want raw crawl data

2. Best client reporting + JS SEO analysis: Sitebulb

Strong choice if your agency needs polished deliverables.

Strengths:

• Chrome-based rendering
• Response HTML vs rendered DOM comparisons
• Good explanations for clients

Sitebulb’s Chrome Crawler renders pages with headless Chrome and can identify links/content changed by JavaScript. support.sitebulb.com Its Response vs Render reporting specifically highlights links and content that appear only after rendering. support.sitebulb.com

Weakness:

• Less engineering-oriented than building your own crawler.
• Less suited if you need raw browser/network telemetry.

Best for:

• agency audits
• recurring reports
• explaining JS SEO issues to clients

3. Best for true AJAX/API discovery: build around browser automation

If your requirement is specifically:

“Tell me every URL requested by JavaScript after page load”

then look at a custom crawler using:

• Playwright
• Puppeteer

Architecture:

1. Seed URLs from sitemap + crawl discovery
2. Open each page in Chromium
3. Intercept:
   • request
   • response
   • fetch
   • XHR
   • navigation events
4. Extract:
   • URLs in DOM after render
   • URLs from JSON API responses
   • URLs generated by JS routers
5. Store relationships:

   source_url
   discovered_url
   discovery_type
   status
   timestamp

This gives you something commercial SEO crawlers generally do not: a browser network graph.

My agency recommendation

I’d use a two-tier stack:

Tier 1: every client

• Screaming Frog or Sitebulb crawl
• JS rendering enabled only where needed
• Identify templates with JS-generated links/content

Tier 2: JS-heavy clients
Trigger your Playwright crawler when you see:

• React/Next.js/Vue/Angular
• product grids loaded by API
• infinite scroll
• faceted navigation
• client-side routing
• “view source has no links” situations

Your test benchmark (before buying anything)

Take 5 difficult client pages:

1. SPA homepage
2. Product/category page with filters
3. Infinite scroll listing
4. Page with lazy-loaded content
5. Page where links appear only after interaction

Score each tool:

• URLs found vs browser DevTools Network tab
• Time per 1,000 pages
• False positives
• Missing routes
• Export quality

The winner is the tool that gets closest to what a real Chrome session discovers, not just the one that says “JavaScript supported.”

Yes. For an SEO agency, I’d evaluate JS crawlers on URL discovery, not merely “does it render JavaScript?”

The key distinction is:

Rendering the DOM ≠ observing every URL an application requests.

A React/Next/Vue site can load data through fetch/XHR, dynamically create links, change routes, or request URLs that never become <a href> elements. If your goal is to capture AJAX-loaded URLs, you need network-level instrumentation in addition to rendered-HTML crawling.

What I’d score

Off-the-shelf SEO crawlers

Sitebulb is particularly interesting for the SEO auditing side. Its Chrome Crawler executes JavaScript, parses the rendered DOM, and can compare response HTML against rendered HTML. It explicitly reports links that are created or modified by JavaScript. support.sitebulb.com

urlSitebulb JavaScript crawling documentationturn0search1

That makes it strong if your definition of “AJAX-loaded URLs” means URLs that become links after JS execution.

But I'd be careful about assuming that means it captures every network URL requested by XHR/fetch. Those are different requirements.

If you specifically need AJAX/network URLs

I'd build or evaluate a crawler around Playwright + Chromium (or another browser automation layer) rather than selecting an SEO crawler solely on its JS-rendering checkbox.

Conceptually:

Seed URL
   ↓
Chromium
   ↓
page.goto()
   ↓
┌─────────────────────────────┐
│ DOM observer                │ → rendered <a href> URLs
│ request observer            │ → fetch/XHR/API URLs
│ response observer           │ → response status/type
│ route/navigation observer   │ → SPA URLs
└─────────────────────────────┘
   ↓
URL normalization
   ↓
crawl frontier

For each page I'd store something like:

page_url
├── response_html_urls[]
├── rendered_dom_urls[]
├── xhr_urls[]
├── fetch_urls[]
├── navigation_urls[]
└── resource_urls[]

Then your SEO reporting becomes much more powerful:

URL discovered in raw HTML?
→ normal crawl path

Only discovered after rendering?
→ JS-dependent link

Requested through XHR/fetch but never exposed as a crawlable link?
→ API/data dependency, not necessarily a crawlable URL

Navigation occurs through history.pushState()?
→ SPA route that deserves separate treatment

URL only appears after interaction?
→ requires an interaction policy rather than simply increasing render timeout

This distinction matters because a crawler that waits 10 seconds longer isn't necessarily going to discover URLs behind a click or application state change. Community reports around JS-heavy crawling illustrate this exact limitation: rendering can be enabled while links triggered by application behavior still aren't discovered. www.reddit.com

My recommendation for an agency

I'd use a two-tier architecture:

Tier 1 — fast crawler

• raw HTTP
• parse HTML
• sitemap/robots discovery
• normal links
• cheap enough to run across millions of URLs

Tier 2 — browser crawler

• Chromium
• JS execution
• rendered DOM
• XHR/fetch interception
• optional interaction scripts
• only run on URLs/sites that actually need it

That follows a sensible general crawling principle: browser rendering is dramatically more expensive, so you don't want every URL paying the Chromium tax. Research on deferred/client-side representations similarly finds substantial speed/discovery tradeoffs between traditional and headless crawling. arxiv.org

For your use case, I'd therefore make “captures XHR/fetch URLs” a separate acceptance criterion from “renders JS.” That's probably the most important distinction when comparing tools.

If you tell me roughly how many URLs you crawl per client (10K / 100K / 1M+) and whether you need the actual API URLs or just the final URLs discovered in the rendered DOM, I can give you a concrete shortlist/architecture and benchmark criteria.

For SEO crawling, the key is to evaluate browser rendering + network interception, not just whether a crawler says it “supports JavaScript.”

I’d score tools on these capabilities:

The important distinction

Suppose a page initially contains:

<div id="products"></div>
<script src="/app.js"></script>

After rendering, the browser executes:

fetch("/api/products?page=2")

and receives:

{
  "products": [
    {"url": "/products/widget-a"},
    {"url": "/products/widget-b"}
  ]
}

A basic JS crawler may give you the rendered <div> but still won't necessarily expose /api/products?page=2 or the product URLs contained in the response.

For your use case, I'd require the tool to expose at least:

1. Initial HTML URLs
2. Rendered DOM URLs
3. All browser network requests
4. URLs extracted from XHR/fetch/GraphQL responses
5. URLs generated after executing custom interactions

How I'd benchmark candidates

Don't rely on vendor demos. Build a small 10–20 page test corpus containing:

• SPA routing
• fetch()/XHR-loaded links
• GraphQL
• infinite scroll
• “Load more” buttons
• JS-generated <a> elements
• links hidden behind menus
• redirects performed by JS
• lazy-loaded content
• API responses containing URLs that aren't subsequently rendered as links
• delayed requests (e.g. 5–10 seconds after page load)

Then compare each crawler's output against a ground-truth set.

A particularly useful metric is:

URL discovery recall = URLs discovered by crawler / URLs actually discoverable through the browser

I'd also separately measure:

DOM recall vs. network/API recall.

That prevents a tool from looking excellent simply because it renders the visible page correctly while missing a large number of URLs sitting in API responses.

If you're building this internally

A headless Chromium + Playwright architecture is particularly well suited to this problem because you can observe both the page and its network activity. You can essentially maintain separate URL sets:

HTML URLs
    ↓
rendered DOM URLs
    ↓
network request URLs
    ↓
URLs extracted from response bodies
    ↓
normalized / deduplicated crawl frontier

The crucial part is response-body inspection. Merely listening for request events will tell you that /api/products was requested, but it won't necessarily tell you that the response contained /products/widget-a.

For an agency, I'd therefore choose based on how much raw browser/network data you can get out, rather than which product has the nicest “JavaScript rendering” checkbox.

If you give me the 2–5 crawler tools you're considering (e.g. Screaming Frog, Sitebulb, JetOctopus, custom Playwright, etc.), I can compare them specifically on AJAX/fetch URL discovery, rendered DOM crawling, API-response extraction, scalability, and cost.

For an SEO agency, I’d evaluate crawlers on two separate capabilities:

1. Does it execute JavaScript and capture the post-render DOM?
2. Does it discover URLs created/fetched after the initial HTML response—including AJAX/API-driven navigation?

Those aren't quite the same thing. Google, for example, can execute JavaScript and discover links injected into the rendered DOM, but its crawler model isn't equivalent to a generic browser crawler. developers.google.comdevelopers.google.com

What I’d look for

The important distinction

Suppose the initial response is:

<div id="products"></div>
<script src="/app.js"></script>

Then JavaScript does:

fetch("/api/products")
  .then(r => r.json())
  .then(products => {
    // Builds product cards with <a href="/products/...">
  });

A JS-rendering crawler should eventually see the generated product links in the DOM.

But if your goal is to say "show me every URL the browser requested", you need network-level interception, not merely DOM rendering.

I'd therefore test candidate tools against three increasingly difficult cases:

Test 1 — DOM-generated links

Initial HTML:       10 URLs
After JS rendering: 37 URLs

Does the crawler report all 37?

Test 2 — AJAX-generated links

HTML → JS → fetch(/api/category/123)
     → JSON contains /products/foo
     → JS creates <a href="/products/foo">

Does it discover /products/foo?

Test 3 — URLs never inserted into DOM

HTML → JS → fetch("/api/products?page=2")

If you specifically want the API URL itself, does the crawler expose /api/products?page=2 from its network log?

That third test is where I'd separate SEO crawlers from browser/network crawlers.

Tools I'd shortlist

Sitebulb is particularly interesting for an agency workflow. Its Chrome Crawler renders pages with headless Chrome, and its Response vs Render functionality explicitly compares source HTML against rendered HTML and identifies links created or modified by JavaScript. support.sitebulb.com

urlSitebulb JavaScript/link documentationturn0search1

I'd also test Screaming Frog SEO Spider with JavaScript rendering, especially if it's already part of your agency workflow. But don't judge it simply on "does it render JS?"—run the AJAX/API test above and verify exactly what it exports.

For a genuinely difficult client—SPA, infinite scroll, faceted navigation, authenticated areas, interactions—I'd consider a custom Playwright/Chromium crawler or a specialized browser-crawling platform. That gives you direct access to:

• requests/responses
• XHR/fetch
• rendered DOM
• console errors
• redirects
• JS exceptions
• clicks/interactions
• screenshots
• timing
• response bodies

My recommendation for an agency

I'd use a two-tier architecture rather than forcing one crawler to do everything:

Tier 1: conventional crawler
→ fast HTML crawl of the entire site
→ links, canonicals, status codes, robots, sitemaps, etc.

Tier 2: browser crawler
→ only JS-heavy URLs
→ render Chromium
→ capture DOM and network requests
→ extract URLs from both

That's essentially the right tradeoff: browser crawling is considerably more expensive, while browser execution can expose URLs/content that a conventional crawler misses. Research on browser-based crawling has found exactly this speed-vs-discovery tradeoff. arxiv.org

One other important SEO point: don't treat "the crawler found it after JS" as proof that Google will treat it identically. Google now renders JavaScript with a headless Chromium-based system and can discover links after rendering, but rendering happens as a separate processing stage and Google recommends server-side/static rendering where practical. developers.google.com

If AJAX-loaded URLs are the key requirement, I'd make "network interception + URL extraction" a hard requirement rather than just "supports JavaScript rendering." That's the distinction most crawler comparison pages gloss over.

If your requirement is specifically “render JavaScript and discover URLs that only appear after AJAX/API calls”, I’d separate the problem into two capabilities:

1. DOM/link discovery after JavaScript executes
2. Network-level discovery of URLs requested by JavaScript

The second one is the important distinction. A crawler can render JS perfectly yet still only report <a href> URLs from the resulting DOM.

My shortlist

Screaming Frog's JavaScript mode uses headless Chromium and extracts links from the rendered HTML, while also discovering links in raw HTML. www.screamingfrog.co.uk

Sitebulb's Chrome Crawler similarly renders with headless Chrome and can compare response HTML against rendered HTML to identify links created or modified by JavaScript. support.sitebulb.comsupport.sitebulb.com

Lumar's rendering service executes JavaScript, captures rendered HTML, and feeds links discovered in that rendered HTML back into the crawl queue. www.lumar.io

But for AJAX-loaded URLs, I'd use Playwright

If by "captures AJAX-loaded URLs" you mean something like:

/page/product
   ↓ JavaScript
GET /api/products?page=2
GET /api/products/123
GET /search?q=shoes
   ↓ response contains data
DOM gets populated
   ↓
<a href="/products/foo">

then you want to capture the browser's network traffic, not merely the final DOM.

Playwright exposes request/response events, so you can record every URL requested by the page, including XHR/fetch traffic. www.screamingfrog.co.ukplaywright.dev

Conceptually:

URL seed
   ↓
Chromium
   ↓
execute JS
   ├── rendered DOM → extract <a href>
   └── network events → extract fetch/XHR/API URLs

That gives you two URL sets:

Discovered links

• canonical URLs
• <a href>
• JS-injected links
• redirects

Network URLs

• fetch()
• XHR
• GraphQL endpoints
• JSON APIs
• dynamically requested resources
• URLs that never become <a> elements

What I'd choose for an SEO agency

If you want an off-the-shelf crawler: start with Screaming Frog or Sitebulb.

Sitebulb is particularly nice if your primary question is “what does the page look like after rendering, and what links did JS create?” Its Response vs Render reporting is explicitly designed for this. support.sitebulb.comsupport.sitebulb.com

If AJAX/API URL discovery is a core deliverable: build a Playwright-based crawler alongside your normal SEO crawler.

I'd have it store something like:

source_url
request_url
request_method
resource_type
status
initiator
timestamp
response_content_type

Then classify requests:

document
stylesheet
script
image
font
xhr
fetch
media
websocket
other

For SEO purposes, I'd especially retain xhr and fetch.

One important gotcha

Don't equate “AJAX URL” with “URL worth crawling.”

A React/Next/Vue site might make hundreds of requests for:

• analytics
• ads
• feature flags
• personalization
• images
• telemetry
• GraphQL
• internal APIs

Most aren't crawlable HTML URLs.

So I'd make your pipeline:

Render → capture network → classify → extract candidate URLs → test content type → enqueue only relevant HTML/JSON/API URLs.

For an agency, that's substantially more useful than simply dumping every network request into the crawl database.

If you tell me roughly how many URLs you crawl per client (10k, 100k, 1M+) and whether you need a SaaS tool or are willing to run your own infrastructure, I can narrow this down to a specific architecture and tool choice.