ChatGPT answers, unedited

If the goal is specifically finding indexing problems on a JavaScript-heavy site, I’d narrow the field to Sitebulb and Screaming Frog, with enterprise crawlers like Oncrawl/Lumar becoming more relevant at very large scale.

My shortlist

Why I'd start with Sitebulb

For your particular problem, Sitebulb has an unusually useful response-vs-rendered HTML comparison. It can crawl the initial HTML, execute JavaScript in Chrome, and then identify what changed. Sitebulb Support Sitebulb

That's exactly what you want when investigating things such as:

• noindex appearing only after JavaScript executes
• canonical changing after rendering
• titles/meta descriptions being rewritten
• important content appearing only after JS execution
• internal links being injected by JS
• rendered content differing substantially from the server response
• pages becoming non-indexable after hydration

Sitebulb actually has dedicated rendered-HTML hints for things like canonical mismatches, rendered-only canonicals, rendered-only titles, JS-modified titles, rendered-only H1s, and JS links. Sitebulb

It also uses an Evergreen Chromium rendering engine, which makes the rendered crawl much more representative of modern Google rendering than an old-school HTML-only crawler. Sitebulb

Where Screaming Frog wins

I'd choose Screaming Frog if you're technically comfortable and want maximum control over the crawl.

It's particularly good if you want to combine:

response HTML → rendered HTML → custom extraction → JavaScript → API data → crawl data analysis

It's more of a technical toolbox, whereas Sitebulb is arguably better at turning the same crawl data into an understandable SEO diagnosis.

One important caveat

Don't treat "Google can render JavaScript" as equivalent to "JavaScript is safe for indexing."

The useful test is:

What does the bot receive before rendering, and what does it see after rendering?

For example, imagine your product page initially returns:

<title>Product 123</title>
<link rel="canonical" href="/product-123">

and JavaScript subsequently changes the DOM to:

<title>Product 123 | Sale</title>
<link rel="canonical" href="/category">

An HTML-only crawler can completely miss that problem. A rendered crawl exposes it. Sitebulb specifically recommends comparing response and rendered HTML for titles, canonicals, robots directives, links and content. Sitebulb Sitebulb Support

What I'd use in practice

For a serious JS SEO/indexing investigation, I'd use a three-layer approach:

1. Sitebulb — crawl the whole site with Chromium and identify response/render discrepancies.
2. Google Search Console URL Inspection — test representative URLs and compare Google's rendered view against your crawler.
3. Screaming Frog — use when you need deeper/custom extraction or want to investigate specific technical patterns.

Sitebulb's own guidance also recommends using Search Console's Test Live URL → View Tested Page to see Google's rendered representation. Sitebulb

If you tell me roughly how large the site is (e.g. 50k, 500k, 10M URLs), what framework it's using (Next.js, React, Angular, Vue, etc.), and whether you're mainly concerned with Google indexing or also AI crawlers, I can give you a much more specific recommendation and the exact crawl configuration I'd use.

response HTML → rendered HTML → custom extraction → JavaScript → API data → crawl data analysis

It's more of a technical toolbox, whereas Sitebulb is arguably better at turning the same crawl data into an understandable SEO diagnosis.

One important caveat

Don't treat "Google can render JavaScript" as equivalent to "JavaScript is safe for indexing."

The useful test is:

What does the bot receive before rendering, and what does it see after rendering?

For example, imagine your product page initially returns:

<title>Product 123</title>
<link rel="canonical" href="/product-123">

and JavaScript subsequently changes the DOM to:

<title>Product 123 | Sale</title>
<link rel="canonical" href="/category">

An HTML-only crawler can completely miss that problem. A rendered crawl exposes it. Sitebulb specifically recommends comparing response and rendered HTML for titles, canonicals, robots directives, links and content.

What I'd use in practice

For a serious JS SEO/indexing investigation, I'd use a three-layer approach:

For JavaScript-heavy sites where the goal is specifically to find indexing/rendering problems, I’d shortlist these:

1. Sitebulb — my first choice

Sitebulb

For your particular problem, Sitebulb has a very useful Response vs Render workflow. It crawls the original HTML, renders the page with Chromium, and compares the two. That lets you find things like:

• noindex appearing only after rendering
• canonical changing after JavaScript executes
• titles/meta descriptions being rewritten
• H1/content appearing only after JS
• internal links being injected or modified by JS
• important content disappearing during rendering

Those are exactly the sorts of issues that can produce confusing indexing behavior. sitebulb.com

It also uses an Evergreen Chromium engine intended to approximate Google's current rendering environment. sitebulb.com

If I were starting a JS SEO audit today, this is probably where I'd start.

2. Screaming Frog — best if you want maximum control

Screaming Frog SEO Spider

Screaming Frog is fantastic if you're comfortable configuring crawls and then digging into the data yourself. Its JavaScript rendering uses Chromium, and you can inspect rendered HTML, screenshots, links, resources, etc.

I'd favor it over Sitebulb if you need things like custom extraction, XPath/CSS selectors, regex, complicated crawl configurations, or integration with other technical SEO workflows.

For pure "show me where JS is changing indexability," though, Sitebulb's reporting is arguably more immediately useful.

3. Google Search Console — don't skip this

Google Search Console

The crawler tells you what a crawler sees. Search Console's URL Inspection → Test Live URL → View Tested Page is important for answering the more fundamental question: what can Google actually fetch and render?

Google explicitly notes that its search systems execute JavaScript, but there are limitations around which resources Googlebot fetches and how rendering works. developers.google.com

I'd therefore use Sitebulb/Screaming Frog for bulk detection + Search Console for verification of important URLs.

A particularly useful audit methodology

For a JS-heavy site, I'd run two crawls:

Crawl A — raw HTML

• No JS rendering
• Googlebot user agent
• Capture status codes, robots directives, canonical, links, content

Crawl B — rendered

• Chromium rendering
• Same URLs/user agent
• Capture rendered DOM, links, canonical, robots, content

Then compare:

Response HTML → Rendered HTML → Google's indexed version

That's much more informative than simply asking whether the page "works" in Chrome.

For example, a particularly nasty issue is:

Raw HTML:
<meta name="robots" content="index,follow">
<link rel="canonical" href="/product-a">

       ↓ JavaScript

Rendered HTML:
<meta name="robots" content="noindex">
<link rel="canonical" href="/product-b">

A conventional crawler can completely miss that. Sitebulb specifically surfaces these response-vs-render discrepancies. sitebulb.com

My practical stack would be:

Sitebulb + Google Search Console for most sites.

Add Screaming Frog if you're doing very technical/custom crawling, and Lumar/Oncrawl if you're dealing with a large enterprise site and need infrastructure/log-analysis capabilities.

If you tell me roughly how big the site is (e.g. 50k / 500k / 10M URLs) and whether it's React, Next.js, Angular, Vue, etc., I can narrow this down considerably.

For heavy JavaScript sites where the goal is to find indexing problems, I would prioritize crawlers that can run a real browser rendering engine (Chromium/headless Chrome) and compare raw HTML vs rendered DOM. A non-rendering crawler can miss content, links, canonicals, or robots directives that JavaScript changes after load. Sitebulb Sitebulb Support

My shortlist:

My typical stack for JS indexing audits

Small–medium site (<100k URLs):

1. Screaming Frog with JavaScript rendering enabled
2. Sitebulb for visual diagnosis and prioritized findings
3. Google Search Console URL Inspection for spot checks

Large ecommerce / millions of URLs:

1. Lumar or Oncrawl for scale
2. Screaming Frog for deep dives on problem URL sets
3. Log file analysis to see what Googlebot is actually crawling

Specific issues I’d test for

A good JS crawler should help you find:

• Content missing from rendered HTML - Product descriptions
• Category text
• Internal links
• Structured data

• JavaScript changing SEO directives - meta robots changing from index → noindex
• canonical tags being injected incorrectly
• hreflang changes
• pagination links disappearing Sitebulb Support

• Rendering failures - JavaScript console errors
• blocked API calls
• hydration failures
• timeout issues
• lazy-loaded content not appearing Sitebulb Support Screaming Frog

• Googlebot parity issues - Compare: - View Source
• Rendered DOM
• Google Search Console URL Inspection result

If I had to pick one tool specifically for finding JS-caused indexing problems, I’d start with Sitebulb because its response-vs-render workflow maps very closely to the question “what changed between what the server sent and what Google can render?” Sitebulb Support

If you need maximum crawling power and custom extraction, I’d pick Screaming Frog instead. Screaming Frog

For JavaScript-heavy sites where the goal is specifically to find indexing/rendering problems, I’d shortlist these:

1. Sitebulb — my first choice

Sitebulb

For your particular problem, Sitebulb has a very useful feature: Response vs Render. It crawls the original HTML and the JavaScript-rendered DOM and identifies changes to things such as:

• noindex / nofollow
• canonical tags
• titles and meta descriptions
• H1s
• internal/external links
• rendered content

That makes it particularly good at answering "what does Google get before JS executes vs. what exists after rendering?" support.sitebulb.com

It uses an Evergreen Chromium crawler designed to approximate Google's rendering environment. sitebulb.com

If I were auditing a React/Next/Vue/Angular site for indexing problems, this is where I'd start.

2. Screaming Frog — best for technical power users

Screaming Frog SEO Spider

I'd choose Screaming Frog if you want to get extremely granular with the crawl: custom extraction, JavaScript execution, XPath/CSS selectors, API integrations, URL inspection data, etc.

It's especially good when the question evolves from:

"Is Google seeing this content?"

to:

"Exactly which DOM element/API response/script is causing this 100,000-page indexing problem?"

3. Lumar — enterprise option

Lumar Website Crawler

Lumar is worth considering for very large sites and enterprise workflows. Its crawler uses a rendering/parsing engine intended to align closely with Google's crawling behavior and specifically emphasizes identifying rendering problems affecting search performance. www.lumar.io

One important caveat

Don't rely on a rendered crawl alone. Run both the raw HTML and rendered versions and compare them.

Google itself notes that Google Search executes JavaScript but that Googlebot/WRS has limitations around which resources it fetches. developers.google.com

For example, a particularly nasty problem can look like:

Server response:
200 OK
<title>Product X</title>
<link rel="canonical" href="/product-x">
<meta name="robots" content="index,follow">

        ↓ JavaScript executes

Rendered DOM:
<meta name="robots" content="noindex">
<link rel="canonical" href="/category">

A normal crawler can completely miss that. A response-vs-render comparison exposes it.

My practical stack

For a serious JS SEO/indexing investigation, I'd use:

Sitebulb → Google Search Console → Screaming Frog

• Sitebulb: bulk rendered-vs-unrendered comparison
• GSC URL Inspection: validate what Google actually fetched/rendered for representative URLs
• Screaming Frog: deep-dive into problematic templates, resources, links, and DOM/API behavior

Google's own documentation recommends using its rendering/debugging tools when investigating JavaScript problems. developers.google.com

If you tell me roughly how big the site is (e.g. 50k, 1M, 50M URLs) and whether it's React/Next/Vue/etc., I can narrow this down to the best tool and the exact crawl configuration I'd use.

For JavaScript-heavy sites where the goal is specifically to find indexing problems, I’d shortlist these:

1. Sitebulb — my first choice for most SEO teams

Sitebulb is particularly good at the “what did the server return vs. what exists after JavaScript executes?” question.

Its Chrome Crawler uses Evergreen Chromium, and its Response vs Render analysis compares things such as:

• title
• meta robots
• canonical
• content
• internal/external links
• other important DOM elements

That makes it very effective for finding situations where JS accidentally changes a canonical, injects noindex, removes links, or causes important content to appear only after rendering. support.sitebulb.com

It also lets you adjust render timeouts, which matters on very slow React/Next/Vue applications. support.sitebulb.com

Best for: diagnosing why JS is affecting indexability, not just discovering that something is wrong.

2. Screaming Frog SEO Spider — best for deep technical investigation

Screaming Frog is excellent if you want a very granular crawler and are comfortable working with lots of raw data.

Its JavaScript rendering mode uses Chrome and can capture:

• rendered HTML
• JavaScript errors
• rendered screenshots
• console messages
• dynamically generated links
• changes to SEO elements

For example, it specifically reports pages with JavaScript errors captured during rendering, which can help identify errors that interfere with search-engine rendering. www.screamingfrog.co.uk

Best for: technical SEOs who want maximum control, custom extraction, XPath/CSS selectors, log-style investigation, and detailed exports.

3. Google Search Console — essential validation, but not your bulk crawler

This is the one I'd use alongside either Sitebulb or Screaming Frog.

Google's URL Inspection lets you test a live URL and determine what Google currently knows about it. It's particularly useful for validating individual URLs where your crawler says something suspicious. support.google.com

Google's documentation confirms that Google Search goes through crawling → rendering → indexing, with JavaScript executed by a headless Chromium environment. developers.google.com

So a useful workflow is:

Bulk crawler → identify anomalies → Google URL Inspection → verify what Google actually sees.

What I'd choose

If I had to pick one: Sitebulb. Its Response vs Render functionality maps unusually well to the actual indexing questions you're trying to answer. Sitebulb also explicitly recommends its Chrome Crawler for JavaScript frameworks. support.sitebulb.com

One important caveat: a crawler rendering a page successfully doesn't prove Google will index it. Google itself notes that JavaScript has limitations and that server-side/static rendering can still be preferable. developers.google.com

If you're auditing something like a large React/Next.js ecommerce site, I can also give you a specific crawl configuration and the 10–15 reports/checks I'd run to uncover JS indexing failures.

For JavaScript-heavy sites where the goal is specifically finding indexing/rendering problems, I’d shortlist these:

1. Sitebulb — my first choice

Sitebulb is particularly good for this use case because it doesn't just render JavaScript; it gives you a Response vs Render comparison.

That lets you identify things like:

• noindex appearing only after JavaScript executes
• canonical changing during rendering
• titles/H1s being injected or modified by JS
• internal links appearing only in the rendered DOM
• important content missing from the initial HTML
• rendered vs. non-rendered word counts
• pages that become non-indexable after rendering

Sitebulb's Chrome Crawler uses an Evergreen Chromium engine and is specifically designed to approximate Google's rendering environment. support.sitebulb.com

Its rendered-specific diagnostics are unusually useful: it explicitly flags things such as "Noindex only in the HTTP response HTML," "Canonical mismatch between rendered and response HTML," and JavaScript-created links. sitebulb.comsupport.sitebulb.com

For diagnosing "Why isn't Google indexing this?" I'd start here.

2. Screaming Frog — best if you want maximum control

Screaming Frog SEO Spider is probably the most flexible option if you're comfortable configuring crawls yourself.

It's especially useful when you want to:

• switch between raw HTML and rendered HTML
• emulate Googlebot
• control JavaScript execution
• inspect rendered screenshots
• extract custom data
• run custom JavaScript
• connect crawl data with Search Console/GA/etc.
• export huge amounts of crawl data for analysis

I'd favor Screaming Frog over Sitebulb if your workflow involves highly customized extraction or scripting. Sitebulb is generally nicer when the goal is getting an SEO diagnosis quickly.

3. JetOctopus — excellent for very large JS sites

JetOctopus is worth looking at if you're dealing with millions of URLs.

Its JavaScript crawler specifically compares raw HTML vs. rendered DOM and can identify JS-injected content and links, blocked resources, hydration problems and client-side redirects. jetoctopus.com

I'd put it ahead of the desktop crawlers when scale and cloud-based crawling are major requirements.

4. Lumar — enterprise option

Lumar is another strong enterprise choice. Its crawler uses a rendering engine intended to closely align with Google's crawling/rendering behavior and is designed to identify rendering problems affecting search performance. www.lumar.io

It's more compelling if you're already operating a large technical SEO program rather than just needing a JS crawler for an individual audit.

What I'd actually use

For a serious JS SEO audit, I'd use two layers:

Sitebulb/Screaming Frog → Google Search Console → manual Google rendering tests

The important distinction is that "the page renders in Chrome" does not mean "the page is indexable."

I'd specifically compare:

HTTP response
      ↓
Raw HTML
      ↓
JavaScript execution
      ↓
Rendered DOM
      ↓
Indexability signals
      ↓
Google's indexed version

The biggest things I'd look for are:

1. Robots directives changing after rendering
   • noindex
   • nofollow

2. Canonical changes
   • response canonical ≠ rendered canonical

3. Content only appearing after JS
   • product/category text
   • editorial content
   • structured data
   • important page elements

4. Links created by JavaScript
   • particularly navigation and pagination
   • links that Google may not discover reliably

5. Client-side redirects

6. Hydration failures
   • especially React/Next.js/Vue applications

7. Resources blocked to Googlebot

8. Rendering timeouts
   • page looks fine to a human but the crawler captures an incomplete DOM

9. Different mobile vs. desktop rendering

10. Large discrepancies between rendered HTML and what Google actually has indexed

Sitebulb is particularly good at surfacing #1–#4 at scale. sitebulb.comsupport.sitebulb.com

Bottom line

If you're buying one tool specifically for JS indexing diagnostics: Sitebulb.

If you're a technical SEO who wants maximum crawling/extraction flexibility: Screaming Frog.

If you're crawling millions of URLs: JetOctopus or Lumar.

And for a particularly important site, I'd still validate representative URLs in Google Search Console's URL Inspection / live test, because a third-party crawler can approximate Google's rendering environment but can't tell you whether Google has actually indexed that particular URL.

For JavaScript-heavy sites where the goal is specifically to find indexing/rendering problems, I’d shortlist these:

1. Sitebulb — best overall for JS SEO audits

This would be my first choice for most technical SEO teams.

Its Chrome Crawler renders pages with headless Chromium, and its Response vs Render reporting is particularly useful: it can identify when JavaScript creates, modifies, or removes things like:

• meta robots
• canonicals
• titles
• meta descriptions
• internal links
• external links
• page content

That makes it very good at finding the subtle problems where the HTML looks fine until JavaScript executes. support.sitebulb.comsitebulb.com

It also has explicit hints for things like canonical mismatches, noindex appearing only in response/rendered HTML, and JS-modified titles. sitebulb.com

Best for: React/Next/Vue sites, large-scale rendering audits, comparing source vs rendered DOM, recurring technical SEO audits.

2. Screaming Frog SEO Spider — best for deep forensic crawling

Screaming Frog is probably the strongest alternative if you want to dig into individual URLs and diagnose exactly what's happening.

Its JavaScript rendering uses Chrome and can expose:

• JavaScript errors
• rendered HTML
• missing/changed links
• rendering problems
• screenshots
• resource loading
• console errors

For example, its crawler specifically records JavaScript errors from the Chrome console, which can reveal errors that prevent content from rendering correctly for search engines. www.screamingfrog.co.uk

Best for: SEO engineers, debugging individual rendering failures, highly customized crawls, very granular analysis.

3. Google Search Console — essential validation layer

I wouldn't use Search Console as your primary crawler, but I'd use it alongside one of the above.

The URL Inspection → Test Live URL → View Tested Page workflow lets you see how Google actually fetched/rendered a particular URL. Google's documentation confirms that Googlebot executes JavaScript, while also having limitations around resources and APIs. developers.google.com

This is particularly valuable for answering:

"My crawler says this page is indexable, but does Google actually see it that way?"

My practical stack

The key thing is don't just run a JavaScript-enabled crawl. The really valuable test is response HTML vs rendered HTML. That's where you can catch a page that initially returns an indexable canonical but changes it after hydration, injects a noindex, loses internal links, or only creates important content client-side. support.sitebulb.comsitebulb.com

If I were choosing one: Sitebulb for a repeatable JS-indexing audit; Screaming Frog if you're more technically hands-on and want maximum crawl/debugging control.

For JavaScript-heavy sites where the goal is specifically to diagnose indexing problems, I’d narrow the field to Sitebulb, Screaming Frog, and JetOctopus.

My shortlist

1. Sitebulb — my default choice

If your primary question is “What changes between the HTML Google receives and the DOM after JavaScript executes, and could that affect indexing?”, Sitebulb is probably the easiest tool to work with.

Its Chrome Crawler renders pages and its Response vs Render reporting specifically identifies things such as:

• noindex appearing/disappearing after rendering
• canonical changes
• titles changed by JS
• JS-generated content
• JS-created/modified links
• H1 changes
• differences in response vs. rendered HTML

Those are exactly the sorts of problems that can cause confusing indexing behavior. sitebulb.comsitebulb.com

It also uses an Evergreen Chromium-based crawler, which makes it a particularly good fit for modern React/Vue/Angular/Next-style sites. sitebulb.com

I'd pick Sitebulb if you're an SEO/technical SEO person who wants the findings surfaced clearly rather than spending hours digging through raw crawl data.

Sitebulb JavaScript SEO tools

2. Screaming Frog — best for hands-on technical debugging

Screaming Frog is excellent if you want maximum control and granular data.

You can enable JavaScript rendering and inspect:

• rendered HTML
• Chrome console errors
• JS-generated content
• rendered vs. non-rendered word counts
• rendering problems at the individual URL level

For example, it specifically flags pages where body content is only discovered after JavaScript execution, and it captures JavaScript errors from the Chrome console. www.screamingfrog.co.uk

The tradeoff is that its interface is more SEO-crawler/spreadsheet oriented than Sitebulb's diagnostic approach.

I'd choose Screaming Frog if you already use it heavily or your workflow involves exporting crawl data and doing your own analysis.

Screaming Frog SEO Spider

3. JetOctopus — for very large sites

If you're dealing with hundreds of thousands or millions of URLs, JetOctopus becomes more interesting.

Its JavaScript crawler is designed to compare raw HTML against rendered DOM and investigate things such as:

• JS-injected content
• JS-generated links
• hydration errors
• blocked assets
• client-side redirects
• rendering behavior at scale

jetoctopus.com

For a massive ecommerce or publishing site, I'd seriously consider it over a desktop crawler.

JetOctopus JavaScript SEO crawler

4. Don't skip Google Search Console

There's an important distinction here: a third-party crawler can tell you what happens when a page is rendered, but Search Console tells you what Google actually observed for a particular URL.

URL Inspection can show Google's indexed version, run a live test, show the rendered page/screenshot, loaded resources, JavaScript output, and Google's selected canonical. support.google.com

So my preferred workflow would be:

Sitebulb/Screaming Frog → identify patterns at scale → Google Search Console → validate representative URLs.

Google's own documentation confirms that Google Search processes JavaScript in three stages—crawling, rendering, and indexing—and that it uses a headless Chromium environment for rendering. developers.google.com

What I'd use in practice

For a typical JS-heavy ecommerce/content site:

Sitebulb + Google Search Console

For a technical SEO team that wants maximum crawl control:

Screaming Frog + Google Search Console

For a huge enterprise site:

JetOctopus + Google Search Console

And one particularly important test: don't just look at whether the rendered page looks correct. Compare response HTML → rendered DOM specifically for canonical, robots directives, internal links, primary content, title, and structured data. Those differences are where many JS indexing problems hide. sitebulb.com

If you tell me roughly how many URLs the site has and whether it's React, Next.js, Angular, Vue, etc., I can narrow this down to the best option and the exact crawl configuration I'd use.

For JavaScript-heavy sites where the goal is specifically to find indexing/rendering problems, I’d shortlist these:

My pick: Sitebulb

For your particular requirement, I'd start with Sitebulb. Its Chrome Crawler uses an Evergreen Chromium engine and is specifically designed to show what Google can see after JavaScript execution. sitebulb.comwww.screamingfrog.co.ukwww.oncrawl.comdevelopers.google.com

The particularly useful feature is Response vs Render. It lets you compare the raw HTML Google initially receives with the DOM after JavaScript executes. That catches issues such as:

• noindex appearing only after rendering
• canonical changing after JavaScript
• important content existing only in rendered HTML
• internal links being generated by JS
• titles/H1s being changed by JS
• rendered content disappearing or becoming substantially different
• JavaScript-generated redirects

Sitebulb explicitly flags things such as "Noindex only in the HTTP response HTML," "Canonical mismatch between rendered and response HTML," and content that exists only in rendered HTML. support.sitebulb.com

That's much more useful for an indexing investigation than simply knowing "the page rendered successfully."

Screaming Frog is a very close second

If you're already comfortable with Screaming Frog, there's little reason to switch. Its JavaScript rendering mode can compare HTML word counts before and after rendering and identify content discovered only after JS execution. www.screamingfrog.co.uk

I'd choose Screaming Frog over Sitebulb if you need things like:

• highly customized extraction
• XPath/CSS/regex extraction
• custom JavaScript
• extensive API integrations
• combining crawling with lots of bespoke technical checks

For very large sites

If you're talking millions of URLs, I'd look harder at Oncrawl. It supports full JavaScript rendering at large scale and is explicitly designed for complex/large websites. www.oncrawl.com

One important caveat

A rendered crawl isn't the same thing as proving that Google actually indexed a URL.

For an indexing investigation, I'd use a three-layer approach:

1. Crawler
→ Compare raw response HTML vs rendered HTML.

2. Google Search Console
→ Check URL Inspection / indexing status for representative URLs.

3. Crawl/indexation comparison
→ Compare your rendered crawl against indexed URLs, sitemap URLs, canonicals, internal links, and GSC data.

Google itself notes that JavaScript can introduce limitations and that some content may not appear in rendered HTML as expected; it recommends SSR/static rendering/hydration rather than relying on dynamic rendering as a long-term solution. developers.google.com

If you tell me roughly how big the site is (e.g. 50k, 500k, 10M URLs) and whether it's React/Next.js, Angular, Vue, etc., I can narrow this down to the best crawler and the exact crawl configuration I'd use.

For JavaScript-heavy sites where the goal is specifically to find indexing/rendering problems, I’d shortlist these:

My first choice: Sitebulb

For your particular use case, Sitebulb is probably the one I'd start with.

Its Chrome Crawler uses an Evergreen Chromium engine and lets you compare the initial HTTP response HTML against the post-JavaScript rendered HTML. That's extremely useful for finding things like:

• noindex appearing only after JavaScript executes
• canonicals changing during rendering
• titles/H1s being injected or modified by JS
• important content missing from the initial HTML
• internal links being created/removed by JS
• rendered pages having substantially different content from the response
• pages that appear indexable before rendering but aren't afterward

Sitebulb actually has specific "Rendered" hints for several of these scenarios, including noindex only in the HTTP response, canonical mismatches, JS-modified titles, and JS-created links. support.sitebulb.comsitebulb.com

It also gives you a Response vs Render report at scale rather than requiring you to inspect pages one at a time. support.sitebulb.com

Where Screaming Frog wins

I'd choose Screaming Frog if you want maximum control over the crawl and are comfortable doing your own analysis.

It's particularly good when you want to combine JS rendering with things like:

• custom extraction
• XPath/CSS selectors
• custom JavaScript
• very detailed URL-level exports
• crawl comparisons
• integration into a broader technical SEO workflow

The tradeoff is that its raw-vs-rendered information can feel more like a large dataset you need to analyze yourself, whereas Sitebulb tends to turn those differences into SEO diagnoses. Sitebulb itself describes Screaming Frog's JS audit workflow as providing raw-vs-render data in its traditional list/spreadsheet format. sitebulb.com

If this is an enterprise site

I'd look seriously at Oncrawl, particularly if you have access to server logs.

The really valuable combination is:

crawler + JS rendering + server logs + Search Console/indexation data

That lets you distinguish:

"Google can't render this page"

from

"Google can render it, but Googlebot isn't actually crawling/rendering these URLs frequently"

Oncrawl supports JavaScript rendering specifically for sites where important content or links are generated dynamically. help.oncrawl.com

Don't rely on a crawler alone

For indexing issues, I'd use a three-layer workflow:

1. Sitebulb/Screaming Frog
Find the problems at scale.

2. Google Search Console URL Inspection
Take representative problem URLs and use Google's live test to see what Googlebot can actually fetch/render.

3. Server logs
Determine whether Googlebot is actually requesting the affected JS, CSS, API endpoints, etc.

That's important because a third-party crawler can tell you "this breaks when rendered in Chromium" without necessarily telling you "Google has actually encountered this problem in production."

One particularly useful test

For a JS-heavy site, I'd run the same crawl twice:

Crawl A — raw HTML
→ What exists before JS executes?

Crawl B — rendered Chrome
→ What exists after JS executes?

Then compare:

HTML response → rendered DOM → indexability signals

I'd pay especially close attention to:

robots meta → canonical → title → H1 → main content → internal links → structured data

Those are exactly the sorts of elements that can change between the response and rendered versions. sitebulb.com

Bottom line: if you're primarily an SEO trying to diagnose "why aren't these JS-generated pages/content being indexed?", I'd start with Sitebulb + Google Search Console. If you're doing very deep technical crawling/extraction, add Screaming Frog. For a very large enterprise site where crawl behavior needs to be correlated with actual bot activity, Oncrawl + log files becomes more compelling.

If your main goal is finding SEO/indexing problems on JavaScript-heavy sites, I’d shortlist these:

My first choice: Sitebulb

For detecting indexing issues caused by JavaScript, Sitebulb is particularly well suited. Its Chrome Crawler uses headless Chrome to actually render JS, and its Response vs Render functionality lets you compare the HTML Google receives initially against the DOM after JavaScript executes. support.sitebulb.comsupport.sitebulb.com

That's extremely useful for finding things like:

• <title> or meta descriptions changed by JS
• H1/content appearing only after rendering
• canonical tags injected or modified by JS
• noindex appearing in the rendered DOM
• internal links that only exist after JS execution
• pagination/faceted navigation that isn't discoverable in raw HTML
• structured data injected client-side
• major differences in word count between source and rendered page
• content/API calls failing during rendering
• pages that return a 200 but effectively render empty
• JS redirects or client-side routing problems

Sitebulb also specifically provides a Response vs Render Report for detecting these differences at scale. support.sitebulb.com

Screaming Frog is the other one I'd definitely have

Screaming Frog is arguably more flexible if you're technically comfortable with it. Its JavaScript rendering plus custom extraction makes it excellent when you have a very specific question such as:

"Show me every URL where the rendered DOM contains noindex but the source HTML doesn't."

or:

"Find pages where the rendered canonical differs from the response canonical."

It's particularly strong for XPath/CSS/regex extraction and custom JavaScript, so you can build very targeted diagnostics rather than relying exclusively on predefined reports. www.techradar.com

One important distinction

Don't equate "the page renders in Chrome" with "Google can index everything on the page."

For a serious JS SEO audit, I'd run at least two passes:

1. Raw HTML crawl

Look at what arrives in the initial HTTP response.

2. Chromium-rendered crawl

Execute the site's JavaScript and inspect the resulting DOM.

Then compare:

                    Response HTML
                         │
                         ▼
                  ┌──────────────┐
                  │ JS execution │
                  └──────┬───────┘
                         │
                         ▼
                   Rendered DOM
                         │
                         ▼
              Indexability differences

The difference between those two states is often more informative than either crawl individually. Sitebulb explicitly exposes this comparison at scale. support.sitebulb.comsupport.sitebulb.com

If it's a really JS-heavy site

If you're dealing with something like Next.js/React/Angular/Vue + lots of API-driven content, I'd use:

Sitebulb → initial diagnostic
Screaming Frog → deep investigation/custom extraction
Google Search Console → verify actual indexing
Chrome DevTools / URL Inspection → investigate individual failures

And for an unusually complex site, I'd add a Playwright-based crawler so you can control things like wait conditions, network failures, cookies, authentication, viewport, user agent, and specific application states.

Bottom line: if you want to buy one tool specifically for "crawl this huge JS site and tell me what could be different between what the crawler receives and what gets rendered/indexed," I'd start with Sitebulb. If you're a technical SEO who wants maximum control and custom extraction, Screaming Frog is probably the better complement. sitebulb.com

If you tell me roughly how big the site is (e.g. 50k vs 10M URLs) and whether it's React/Next/Angular/Vue, I can narrow this down considerably.

For crawling JavaScript-heavy sites specifically to find SEO/indexing problems, I’d prioritize tools that can compare raw HTML vs rendered DOM, execute JavaScript, and expose what a search engine is likely to discover. A normal HTML crawler can miss critical issues like JS-injected canonicals, noindex tags, content, and links. sitebulb.comsupport.sitebulb.com

My shortlist:

growwithba.com

My typical stack for JS indexing investigations

1. Screaming Frog SEO Spider (first pass)

• Enable JavaScript rendering (Chrome/Chromium)
• Crawl with:
  • rendered HTML
  • screenshots
  • response codes
  • canonicals
  • hreflang
  • meta robots
  • internal links
• Compare:
  • "View Source" vs rendered DOM
  • pages where important content appears only after JS execution

It’s especially good when you need to answer:
"What exactly changed after JavaScript ran?" www.techradar.com

2. Sitebulb (second pass)
I’d use this when the goal is diagnosing rather than just collecting data. Its response-vs-render reports are built specifically around finding cases where JavaScript changes SEO-critical elements. sitebulb.comsupport.sitebulb.com

Look for:

• title changes after rendering
• missing H1/content
• links appearing only after JS
• canonical changes
• robots directives changing
• rendered pages with thin/empty content sitebulb.com

3. Google tools for confirmation
A crawler tells you what could happen; you still want to verify what Google sees:

• Google Search Console URL Inspection
• Rich Results Test
• Mobile-Friendly Test
• server logs (to see Googlebot behavior)

If you have a modern framework (React/Next.js/Vue/Angular)

I’d also add a browser automation layer:

• Playwright (best current choice)
• Puppeteer

You can build checks like:

• "Does this page have an indexable <h1> after hydration?"
• "Does canonical exist before and after JS?"
• "Are internal links discoverable without clicking?"

My practical recommendation by site size

• <100k URLs: Screaming Frog + Sitebulb
• 100k–5M URLs: Screaming Frog (data collection) + Sitebulb/Lumar (analysis)
• Enterprise ecommerce/news: Lumar or Oncrawl + log analysis
• Custom debugging of a JS app: Playwright scripts

If your goal is specifically "find pages Google cannot index because of JavaScript", I would start with Screaming Frog + Sitebulb together rather than relying on a single crawler. sitebulb.comsupport.sitebulb.com

For JavaScript-heavy sites where the goal is specifically to find indexing/rendering problems, I’d shortlist these:

My pick: Sitebulb

If you're trying to answer "What does Google see before and after JavaScript executes, and could that explain indexing problems?", Sitebulb is particularly good.

Its Chrome crawler can render JS, and its Response vs Render reporting specifically compares the HTML received from the server with the rendered DOM. That lets you identify things such as:

• content appearing only after JS execution
• canonical tags being changed by JS
• noindex being added/removed
• titles and meta descriptions changing
• internal links being injected client-side
• pages becoming non-indexable after rendering
• significant differences in rendered vs. source content sitebulb.com

That's exactly the class of problems I'd want to investigate on a React/Next/Vue/Angular site.

Screaming Frog if you want maximum control

Screaming Frog SEO Spider is probably the better choice if you're comfortable configuring crawls yourself.

Its JavaScript rendering mode lets you crawl JS sites with Chromium, and it has specific diagnostics for rendering-related indexing problems. For example, it can identify a noindex present in the original HTML that disappears after rendering—which is important because Google won't render a page when it encounters noindex in the initial response. www.screamingfrog.co.uk

I'd choose Screaming Frog when you need custom extraction, regex, XPath/CSS selectors, integrations, scripting, or very granular crawl configuration.

JetOctopus if the site is huge

For a large ecommerce or publishing site, JetOctopus becomes particularly interesting because it combines JS crawling with Googlebot log analysis and Search Console data. Its JS crawler compares raw HTML against the Chrome-rendered version and can expose things like JS-injected content/links, hydration problems, blocked resources and client-side redirects. jetoctopus.com

That lets you move beyond:

"The crawler says this page has a problem."

to:

"Googlebot is actually requesting these 50,000 URLs, 18% fail to render correctly, and these 4,000 important URLs aren't getting indexed."

The workflow I'd actually use

For serious JS indexing investigation, I'd use two layers:

1. Sitebulb or Screaming Frog

• Crawl raw HTML
• Crawl rendered HTML
• Compare the two
• Export URLs where important SEO signals change
• Check rendered internal linking
• Check rendered canonical/robots/title/content
• Look for pages with very low rendered content

2. Google Search Console

• URL Inspection → Test Live URL
• Compare Google's rendered page against what your crawler sees
• Check indexing status
• Inspect canonical selection
• Check actual Googlebot behavior on representative problem URLs

This distinction matters because a third-party crawler can approximate Google's rendering environment, but Search Console is the final verification of what Google actually sees. Sitebulb itself recommends using Google's URL Inspection alongside crawling for this reason. sitebulb.com

If I were starting with one tool: Sitebulb for usability and response-vs-render analysis; Screaming Frog if you're a technical SEO power user who wants maximum configurability; JetOctopus if you're dealing with hundreds of thousands/millions of URLs and have access to server logs.

If you tell me roughly how large the site is and whether it's React, Next.js, Angular, Vue, etc., I can narrow this down to the best setup and the specific crawl settings I'd use.

For crawling JavaScript-heavy sites to diagnose SEO/indexing issues, you’ll want a crawler that can execute JS, compare rendered vs raw HTML, inspect hydration problems, and surface what search engines may miss. These are the tools I’d consider:

My typical shortlist

For most SEO teams:

1. Screaming Frog SEO Spider

Best balance of depth, cost, and control. Enable:

• Configuration → Spider → Rendering → JavaScript
• Compare:
  • View Source HTML
  • Rendered HTML
• Check:
  • missing <title> / meta descriptions after rendering
  • missing canonical tags
  • links only appearing after JS execution
  • lazy-loaded content
  • blocked JS/CSS assets
  • client-side redirects

2. Sitebulb

Good if you need to explain issues to stakeholders because the reports are easier to interpret.

3. JetOctopus / Botify / Lumar

Better if you have millions of URLs, frequent deployments, or need ongoing monitoring.

For React, Vue, Angular, Next.js, etc.

I’d pair a crawler with:

• Google Search Console — URL Inspection → “Test Live URL” to see what Googlebot renders
• Google Rich Results Test — useful for structured data rendering
• PageSpeed Insights — helps identify rendering/performance issues

Specific tests I’d run

For JS-heavy sites, look for:

• Empty initial HTML
  • Does the server return meaningful content, or just <div id="root"></div>?
• Delayed content injection
  • Is important text available immediately after rendering?
• Internal links generated by JS
  • Can crawlers discover important pages?
• Indexability mismatches
  • Example: canonical/noindex tags differ between source and rendered DOM
• Blocked resources
  • Robots.txt blocking JS bundles, APIs, fonts, or CSS
• Soft 404s
  • Pages returning 200 but rendering “not found”
• Client-side routing problems
  • URLs that work in a browser but fail for crawlers

If you tell me the stack (React/Next.js, Angular, Vue, Shopify, WordPress, etc.) and approximate site size (thousands vs millions of URLs), I can narrow this down to a better tool choice and crawl setup.

For JavaScript-heavy sites where the goal is specifically to find indexing/rendering problems, I’d shortlist these:

My take

1. Sitebulb — probably the best starting point

Its Chrome Crawler renders pages with headless Chrome, and its Response vs Render functionality compares the original HTML with the post-JavaScript DOM at scale. That's particularly useful for discovering things like:

• canonical changes after JS execution
• noindex appearing/disappearing
• titles/meta being modified
• content that only exists after rendering
• links injected by JavaScript
• pages that become effectively empty
• rendering failures

Sitebulb explicitly supports this response-vs-render analysis across crawls. sitebulb.com

2. Screaming Frog — best if you want maximum control

It's particularly strong if you're comfortable configuring crawls yourself. Its JavaScript mode uses rendering and can expose JavaScript console errors, rendered-page output, and specific problems such as a noindex present in the original HTML but removed by JavaScript. www.screamingfrog.co.uk

I'd favor it if you want to combine JS rendering with custom extraction, regex/XPath/CSS selectors, custom search, APIs, and very detailed technical investigations.

3. JetOctopus — strongest for huge JS-heavy sites

JetOctopus is interesting if you're dealing with hundreds of thousands or millions of URLs. It does a raw-HTML crawl alongside Chrome-rendered crawling and lets you compare the two, while also combining crawl data with log files and Search Console data. It advertises capacity of 1M+ pages/day. jetoctopus.com

That combination is particularly valuable for answering:

"Googlebot is crawling these URLs, but why aren't they getting indexed?"

rather than merely:

"Does this page render?"

4. Lumar — enterprise choice

Lumar is worth considering if this is an enterprise-scale SEO program. Its crawler is designed around Google's rendering/parsing behavior and specifically targets rendering issues that can affect search performance. www.lumar.io

One important caveat

No crawler can tell you with certainty why Google chose not to index a particular URL. For that, I'd pair the crawler with Google Search Console's URL Inspection/Indexing data.

For a serious JS SEO investigation, I'd use this workflow:

Raw HTML crawl → rendered Chrome crawl → compare response vs rendered DOM → identify JS errors/blocked resources → correlate with GSC indexing status → validate representative URLs with Google's live inspection.

If you tell me roughly how big the site is (e.g. 50k, 500k, 10M URLs) and whether it's React/Next.js, Angular, Vue, etc., I can narrow this down to the best 1–2 options and explain the tradeoffs.

For heavy-JavaScript sites where the goal is specifically to find SEO/indexing problems, I’d shortlist these:

My pick: Sitebulb

For your particular use case, Sitebulb is probably the strongest starting point. Its Chrome Crawler uses an Evergreen Chromium engine and, importantly, it can compare the raw response HTML against the rendered DOM at scale. sitebulb.comsupport.sitebulb.com

That lets you detect things like:

• noindex appearing only after JavaScript executes
• canonical tags being added/changed by JS
• titles or meta descriptions being modified during rendering
• H1/content appearing only after rendering
• internal links being created or altered by JavaScript
• content disappearing between response and rendered HTML
• JS-generated redirects
• pages that are effectively empty before hydration

Sitebulb has explicit rendered-SEO hints for several of these scenarios, including “Noindex only in the HTTP response HTML,” “Canonical mismatch between rendered and response HTML,” and JavaScript-modified titles/H1s. sitebulb.com

It also lets you adjust the render timeout, which is important for React/Next/Vue sites where hydration or API calls may take several seconds. support.sitebulb.com

Screaming Frog is a very close second

If you're already comfortable with Screaming Frog, I wouldn't switch just for JS crawling. Its SEO Spider can render with JavaScript using a headless browser and lets you configure the user agent, viewport, resources, etc. www.screamingfrog.co.uk

I particularly like it for forensic investigation of individual problems because you can dig deeply into rendered HTML, screenshots, resources, response codes, links, and custom extraction.

urlScreaming Frog's JavaScript crawling guideturn0search11

If you're dealing with millions of URLs

I'd look at Oncrawl. Its crawler supports full JavaScript rendering and is designed around very large sites, with the company advertising crawling at millions-of-URL scale. www.oncrawl.com

The tradeoff is that it's more of an enterprise data/SEO platform than a lightweight forensic crawler.

urlOncrawl JavaScript crawlerturn0search2

One important caveat

For indexing issues, don't rely exclusively on a third-party rendered crawl.

I'd use a three-layer approach:

1. Raw HTTP crawl — what the server initially returns.
2. Chromium-rendered crawl — what the page becomes after JS executes.
3. Google's own inspection data — whether Google can actually fetch/render/index representative URLs.

The really interesting problems are often the differences between #1 and #2. For example:

Server returns index,follow → JS changes it to noindex

or

Server returns canonical A → JS changes canonical to B

or

Server HTML contains 20 internal links → rendered DOM contains 2,000.

That's exactly the type of analysis Sitebulb's Response vs Render functionality is designed to surface. sitebulb.comsupport.sitebulb.com

If you tell me roughly how big the site is (e.g. 50k, 500k, 10M+ URLs) and whether it's React/Next.js, Angular, Vue, etc., I can narrow this down to the best 1–2 tools and the crawl configuration I'd use.

If your primary goal is finding indexing issues on JavaScript-heavy websites, I'd look at tools based on how accurately they render pages, how well they expose SEO signals, and how scalable they are.

Here's how the leading options compare:

My shortlist

1. Screaming Frog (best overall)

Still the benchmark for diagnosing JavaScript indexing issues because it lets you:

• Render with Chromium
• Compare raw HTML vs rendered HTML
• See rendered page source
• Inspect rendered DOM
• Extract structured data after rendering
• View rendered screenshots
• Inspect canonical tags after JS execution
• Check robots directives after rendering
• Verify internal links created by JavaScript
• Export every rendered element

It's also excellent for finding:

• orphaned JS pages
• client-side redirects
• hydration failures
• lazy-loaded content Google may miss
• missing rendered title/meta descriptions
• rendered canonicals differing from source

2. JetOctopus (best cloud crawler)

If you're crawling hundreds of thousands or millions of URLs, JetOctopus is one of the strongest options.

It supports:

• JavaScript rendering
• log file analysis
• crawl budget analysis
• Googlebot behavior
• indexability reports
• Core Web Vitals integration

Great for enterprise-scale websites.

3. Sitebulb (best diagnostics)

Sitebulb shines at explaining why something is an SEO problem.

For JavaScript sites it highlights:

• content appearing only after rendering
• blocked resources
• render failures
• hidden internal links
• rendering timing issues

The visual reports make it easier to communicate findings to developers.

For debugging rendering itself

I also recommend using:

• Chrome DevTools
• Google Search Console URL Inspection

Compare:

• Raw HTML
• Rendered HTML
• Google's indexed version

Many indexing problems become obvious when comparing these three.

If you're comfortable with code

A custom crawler using:

• Playwright
• Puppeteer

gives you complete control over:

• waiting for network idle
• infinite scroll
• lazy loading
• authentication
• SPAs
• React/Vue/Angular routing
• DOM extraction after hydration

This is often the best choice when you need to crawl highly dynamic applications that off-the-shelf SEO crawlers struggle with.

Recommended workflow for modern JS sites

1. Crawl without JavaScript enabled.
2. Crawl with JavaScript rendering enabled.
3. Diff the two crawls to identify content, links, canonicals, and metadata that only appear after rendering.
4. Validate important URLs in Google Search Console's URL Inspection tool.
5. Analyze server logs to confirm whether Googlebot is actually requesting and rendering those URLs.
6. Spot-check complex pages with Chrome DevTools or a Playwright script to reproduce rendering issues.

For most technical SEO work on React, Next.js, Vue, Angular, or other SPA frameworks, Screaming Frog remains the strongest all-around choice, while JetOctopus or Lumar become more compelling for very large sites or organizations that need cloud-based crawling and reporting.

If your goal is SEO auditing on modern JavaScript-heavy sites (React, Next.js, Angular, Vue, SPAs, hydration, lazy loading), I'd evaluate tools based on how accurately they emulate Googlebot's rendering, how well they expose rendered HTML, and how much technical SEO data they provide.

Here's how the leading options compare:

My ranking by use case

1. Screaming Frog — Best overall

This remains the benchmark for technical SEO.

It can:

• Render pages with Chromium
• Compare raw HTML vs rendered HTML
• Detect missing content after rendering
• Find JavaScript-generated links
• Audit canonicals after rendering
• Check rendered meta titles/descriptions
• Inspect rendered structured data
• Detect orphan pages
• Export rendered DOM

For indexing investigations, you can compare:

• HTML crawl
• JavaScript-rendered crawl

Differences immediately reveal rendering issues.

Ideal if you work on sites up to several hundred thousand URLs.

2. Sitebulb — Best diagnostics

Sitebulb excels at explaining why something is wrong.

Its JavaScript reports highlight:

• Missing rendered links
• Empty rendered pages
• Delayed rendering
• Blocked resources
• Client-side redirects
• Render failures

The visual reports make it easier to identify patterns across a site.

3. JetOctopus — Best for very large sites

If you're crawling millions of URLs:

• Cloud rendering
• Fast distributed crawling
• Crawl budget analysis
• Googlebot log integration
• JavaScript rendering at scale

This is particularly useful for ecommerce or publisher sites.

4. Lumar

Lumar is strong when you need continuous monitoring rather than one-off audits.

It provides:

• Scheduled rendered crawls
• Rendering error reporting
• Internal linking analysis
• Crawl budget insights
• Indexability monitoring

For detecting JavaScript indexing problems specifically

I'd want software that can answer questions like:

✓ Is the content present in the rendered DOM?

✓ Are internal links created only after JavaScript?

✓ Is important text injected after a long delay?

✓ Does Googlebot receive the same HTML users see?

✓ Are canonical tags changing after rendering?

✓ Are meta robots tags changing after rendering?

✓ Does structured data only appear after hydration?

✓ Are lazy-loaded images discoverable?

✓ Is content hidden behind user interaction?

✓ Does rendering fail because scripts are blocked?

The tools above all support many of these checks, with Screaming Frog and Sitebulb standing out for hands-on investigations.

Don't rely on crawlers alone

To confirm whether Google can actually index rendered content, combine crawler findings with:

• Google Search Console URL Inspection (live test)
• Google's Rich Results Test (for structured data)
• Browser DevTools (Network + Elements)
• Chrome with JavaScript disabled (for comparison)
• Log file analysis (to verify Googlebot access patterns)

If you want the closest approximation of Googlebot

Google renders pages using an evergreen Chromium-based rendering service, so tools that also use Chromium generally provide the most representative results. Among commercial crawlers, Screaming Frog, Sitebulb, Lumar, and Botify all perform well in this area.

If your priority is specifically finding JavaScript-related indexing issues, a practical combination is:

1. Screaming Frog for rendered vs. raw HTML comparisons.
2. Google Search Console to verify how Google actually indexes the page.
3. Log file analysis (or a platform like JetOctopus, Oncrawl, or Botify for larger sites) to confirm Googlebot's crawl behavior.

That workflow usually surfaces the majority of rendering and indexability problems on modern JavaScript-heavy websites.

For crawling sites with heavy JavaScript rendering and diagnosing SEO/indexing issues, you’ll want a crawler that can execute JavaScript, compare rendered vs raw HTML, inspect Googlebot behavior, and surface issues like missing content, blocked resources, hydration failures, and incorrect canonical/index directives.

Here are the strongest options by use case:

My shortlist by scenario

For most SEO teams:

• Screaming Frog SEO Spider — probably the first tool I’d use. Enable JavaScript Rendering → Chromium and compare:
  • HTML source vs rendered HTML
  • rendered word count
  • blocked JS/CSS resources
  • missing titles/meta/canonicals after rendering
  • links only appearing after JS execution

For enterprise sites (hundreds of thousands to millions of URLs):

• Botify or Lumar
  • Better crawl infrastructure
  • Better segmentation
  • Better integration with Search Console, logs, and analytics

For explaining issues to stakeholders:

• Sitebulb
  • Strong visual reports
  • Easier prioritization than raw crawl exports

Specific JS indexing checks I’d run

Regardless of tool, look for:

1. Content parity
   • Does rendered HTML contain the same important content as initial HTML?
   • Are product descriptions, internal links, headings, or structured data injected late?

2. Googlebot rendering differences
   • Test with:
     • Google Search Console → URL Inspection → View Tested Page
     • Mobile-friendly rendering
     • Cached/rendered HTML comparisons

3. Blocked resources
   • Check:
     • robots.txt blocking JS files
     • CDN/WAF blocking Googlebot
     • failed API calls
     • CORS errors

4. SPA routing issues
   • Common problems:
     • URLs requiring JS navigation
     • missing server-side rendering
     • incorrect canonical tags
     • soft 404s
     • client-side redirects

5. Performance-related indexing problems
   • Long JS execution times
   • Timeouts during rendering
   • Lazy-loaded content that never appears

A practical stack I’d use

For a serious JS-heavy SEO audit:

• Screaming Frog → crawl + rendered HTML analysis
• Google Search Console URL Inspection → confirm Google’s view
• Chrome DevTools Lighthouse → rendering/performance debugging
• Server logs (or Botify/Oncrawl) → verify Googlebot behavior

If you’re auditing a React/Next.js, Angular, Vue, or large ecommerce site, the best choice can differ because the failure modes are usually framework-specific.

For crawling JavaScript-heavy sites to diagnose SEO/indexing issues, you’ll want a crawler that can render pages like a browser, compare rendered vs raw HTML, inspect resources, and surface issues that affect search engines. The best choice depends on scale and depth:

My usual shortlist

For most SEO teams:

• Start with Screaming Frog SEO Spider. It is the most practical tool for finding JS-related indexing problems:
  • Crawl with JavaScript rendering enabled
  • Compare HTML source vs rendered DOM
  • Identify pages where content only appears after JS execution
  • Check blocked JS/CSS resources
  • Verify canonical tags, meta robots, schema, links, pagination, etc.

For enterprise-scale sites:

• Use Botify or Lumar if you need millions of URLs, crawl budget analysis, log files, and Google Search Console integrations.

For diagnosing “Google can’t see my content” issues:
I’d combine:

1. A crawler with JS rendering (Screaming Frog/Sitebulb)
2. Google Search Console URL Inspection (to see Google's rendered view)
3. Chrome DevTools → Network + Lighthouse
4. Server logs (to compare Googlebot crawling vs actual rendering)

Specific checks I’d run on a JS-heavy site

• Rendered HTML contains the primary content
  • Is product/article text present after rendering?
  • Are internal links generated client-side?
• Googlebot can access JS assets
  • No blocked scripts in robots.txt
  • No authentication barriers
• Indexable URLs have server-side signals
  • Canonical tags
  • Title/meta descriptions
  • Structured data
  • H1/content
• Avoid hydration problems
  • Content exists initially but disappears after React/Vue hydration errors
• Check lazy-loaded content
  • Important text/images should not depend on user interaction
• Compare mobile vs desktop rendering
  • Google primarily uses mobile-first indexing

If you’re auditing a React/Next.js, Angular, Vue, or SPA site, the tool choice can change quite a bit—especially if the issue is “Google crawls it but doesn’t index it.”

For crawling sites with heavy JavaScript rendering to find SEO/indexing issues, I’d prioritize tools that can execute JavaScript, compare rendered vs raw HTML, inspect crawlability, and identify problems like missing content, blocked resources, client-side redirects, hydration failures, and incorrect canonicals.

Here are the strongest options:

My typical stack for JavaScript-heavy SEO work

For most sites (small → mid-size):

1. Screaming Frog SEO Spider
   • Enable:
     • Configuration → Spider → Rendering → JavaScript
     • Crawl linked JavaScript files
     • Store HTML
   • Compare:
     • Raw HTML tab
     • Rendered HTML tab
     • Search Console indexed version

2. Google Search Console URL Inspection
   • Confirm what Google actually rendered.
   • Look for:
     • "Page is indexed"
     • "Crawled but not indexed"
     • Missing content in rendered HTML

3. Chrome DevTools
   • Check:
     • Network failures
     • JS console errors
     • Lazy-loaded content
     • API calls returning errors
     • Robots blocking JS/CSS resources

For React / Angular / Vue sites specifically

I’d pay attention to:

• Does meaningful content exist in the initial HTML?
  • If not, Google may still render it, but rendering delays and failures can hurt discovery.
• Are internal links present before JavaScript runs?
  • Client-side-only navigation can cause crawl gaps.
• Are canonical tags, hreflang, and structured data injected correctly?
• Are API calls required to populate indexable content failing?
• Does the rendered DOM match the intended SEO content?

If you need a crawler you can automate

For engineering teams, consider:

• Playwright — browser automation for custom crawlers
• Puppeteer — Chromium automation commonly used for SEO rendering tests

A custom Playwright crawler is useful when you need to test things like:

• "Does this page show product content after 5 seconds?"
• "Does Googlebot get the same DOM as Chrome?"
• "Do authenticated or personalized pages render correctly?"

My shortlist by use case

• Best overall technical SEO crawler: Screaming Frog SEO Spider
• Best enterprise option: Lumar
• Best reporting/dashboard experience: Sitebulb
• Best for very large sites: Oncrawl or JetOctopus
• Best for debugging complex JS apps: Playwright + Chrome DevTools

If the site is a React/Next.js, Angular, Vue, Shopify headless, or other SPA, the ideal tooling setup changes a bit.