ChatGPT answers, unedited

For a staging site you control, good options are:

• Playwright — authenticate once, save storageState, then crawl with a real Chromium browser so JS renders normally. Playwright
• OWASP ZAP + Ajax/Client Spider — supports browser-based authentication and client scripts, specifically suited to authenticated modern JS apps. ZAP ZAP
• OWASP PTK Agent — can wrap Playwright/Puppeteer/Selenium/Cypress journeys and combine authenticated browser execution with security analysis. OWASP Foundation

For safety, use a dedicated staging account, read-only permissions where possible, throttle the crawler, restrict it to the staging hostname, and disable destructive workflows.

For a staging site that intentionally blocks bots, you usually want a crawler that can do three things:

1. Authenticate (login flow, cookies, headers, SSO)
2. Render JavaScript (real Chromium/browser execution)
3. Run safely inside your network or with allowlisted access (rather than weakening bot protections)

Good options:

Screaming Frog Semrush SiteOne Crawler KrawlifyFor a staging environment, I’d typically choose one of these patterns:

Option 1: Run a browser-based crawler with a test account (most flexible)

Use Playwright:

• Launch Chromium
• Log in with a dedicated staging user
• Save the authenticated storage state
• Crawl routes
• Capture:

• rendered DOM
• screenshots
• console errors
• failed network requests
• accessibility/performance metrics

Example flow:

create staging-audit-user
        ↓
Playwright login once
        ↓
save cookies/session
        ↓
crawl authenticated URLs
        ↓
generate audit artifacts

This avoids pretending to be a search bot and keeps the staging controls intact.

Option 2: Allowlist your crawler instead of disabling protection

For a WAF/bot-management setup:

• Add a staging-only rule:

• allow your CI runner IP
• allow your crawler service IP
• require a secret header
• keep rate limits low
• Keep normal bot blocking enabled for everyone else

Example:

if environment == staging
and request header X-Audit-Key == secret
then allow
else apply normal bot rules

This is safer than globally turning off bot protection.

Option 3: Use your existing logged-in browser session

Useful when staging uses SSO/MFA and automated login is awkward:

• Open Chrome
• Authenticate normally
• Attach crawler to the browser session
• Crawl with existing cookies

Tools like Krawlify are designed around this model. Krawlify

Things to avoid

• ❌ Spoofing Googlebot/Bingbot user agents to bypass protection
• ❌ Sharing employee accounts with crawlers
• ❌ Putting staging credentials into third-party SaaS without approval
• ❌ Disabling WAF rules globally

A practical stack I’d use

For a modern React/Next/Vue app:

• Playwright + Chromium for crawling
• axe-core for accessibility checks
• Lighthouse CI for performance
• CI pipeline runner with staging-only credentials
• Export:

• screenshots
• HAR files
• JSON findings
• HTML report

If you tell me what blocks the crawler (Cloudflare, Akamai, basic auth, SSO, robots.txt, custom middleware, etc.) and what you’re auditing (SEO, accessibility, security, regression testing), I can narrow the setup.

Yes. For a staging site that intentionally blocks unauthenticated bots, I’d favor real-browser crawling with a dedicated test account rather than trying to defeat the bot protection.

Good options

• Playwright — best if your goal is a controlled JS-rendered crawl/audit. It runs a real Chromium/Firefox/WebKit browser, and you can authenticate once and reuse the resulting cookies/local storage via storageState. Playwright

• Authenticate through the normal login UI.
• Save the authenticated browser state.
• Crawl URLs with JS enabled.
• Restrict navigation to your staging hostname and avoid destructive actions.
• This is particularly good for auditing SPA routes, lazy-loaded content, hydration, and authenticated UI.
• OWASP ZAP + browser-based authentication — best if you want the crawl combined with security-oriented passive/active analysis. ZAP supports browser-based authentication and can use its AJAX/Client Spider with an authenticated user. ZAP ZAP

• Create a ZAP context for staging.
• Configure a dedicated test user.
• Use Browser Based Authentication for ordinary modern login flows.
• Use Client Script Authentication if the login flow involves unusual JS, SSO, MFA/TOTP, etc. ZAP
• Run the AJAX/Client Spider as that user.
• Playwright → ZAP proxy — probably the most flexible setup if you need both accurate browser rendering and security inspection. Have Playwright drive the authenticated browser while routing traffic through ZAP. This gives you browser-level JS execution plus ZAP's request/response analysis.

For a staging environment, I'd set it up like this

                 staging.example.com
                         │
                  dedicated test user
                         │
                    Playwright
                  Chromium + JS
                         │
                    HTTP proxy
                         ▼
                        ZAP
                  ┌──────┴──────┐
                  │             │
              crawler       security
              / routes       analysis

Keep the account deliberately low-privilege and preferably make staging data disposable. Also put the authentication state in a secret/temporary location: Playwright explicitly warns that its saved browser state can contain cookies and headers capable of impersonating the test account. Playwright

One important distinction

If your "bot blocking" is something like a WAF rule based on User-Agent, don't build the audit around spoofing or bypassing the production-style protection. Since this is your staging environment, the cleaner approach is to create an explicit scanner/test-account allow path at the WAF or staging gateway, then let the crawler authenticate normally.

For a modern authenticated SPA, my first choice would be Playwright for discovery/rendering + ZAP through the Playwright proxy for security auditing. ZAP itself specifically supports headless browser authentication for automation. github.com

If you tell me whether the login is OIDC/SAML, username/password, or something like Auth0/Okta, I can give you a concrete safe setup for the crawler.

Yes. For a staging site that blocks unauthenticated bots and relies on client-side JavaScript, I’d favor browser-based crawlers over traditional HTTP crawlers.

Best options

• Playwright — probably the most flexible choice if you want a controlled audit pipeline. It runs a real browser with JavaScript enabled by default, supports HTTP credentials, and can save/reuse authenticated cookies, local storage, IndexedDB, and other browser state. Playwright Playwright

• Log in once with a dedicated staging account.
• Save the resulting storageState.
• Crawl pages using that state rather than putting credentials into every request.
• Intercept requests to prevent writes or calls to external production services.
• Set a distinctive User-Agent so your staging WAF can explicitly allow the crawler.
• OWASP ZAP — better if the goal is a security-oriented audit rather than merely checking links/content. Its Browser-Based Authentication runs authentication inside a browser and is specifically intended for modern applications and the Ajax Spider. It can also handle multi-step login flows and TOTP. ZAP ZAP

• Create a ZAP Context restricted to the staging hostname.
• Configure browser-based authentication.
• Use the Ajax Spider for JS-heavy routes.
• Keep the scan limited to staging and authenticated test accounts.
• Playwright + your own crawler — my preference if you're doing a safe functional/SEO/accessibility/rendering audit. You get precise control over what URLs are visited, which HTTP methods are allowed, rate limits, authentication, screenshots, DOM extraction, and network isolation.

A safe architecture

I'd set it up roughly like this:

Dedicated staging account
        ↓
Browser login
        ↓
Save authenticated browser state
        ↓
Playwright browser context
        ↓
URL allowlist ──→ staging.example.com/*
        ↓
JS rendering + crawling
        ↓
Block/abort:
  - POST/PUT/PATCH/DELETE
  - payments
  - emails
  - webhooks
  - external production APIs
        ↓
Collect:
  - final URL
  - status
  - rendered HTML
  - console errors
  - failed requests
  - screenshots
  - accessibility/SEO signals

The particularly important part is not simply bypassing the bot block. Have the staging WAF recognize your crawler explicitly—e.g. a dedicated source IP, service credential, or narrowly scoped User-Agent—and keep the crawler restricted to the staging hostname.

For authentication, Playwright's recommended pattern is to authenticate once and persist the browser state, while keeping the resulting auth file out of source control because it can contain credentials capable of impersonating the test account. Playwright

If your staging login is HTTP Basic Auth rather than an application login, Playwright also supports origin-scoped HTTP credentials directly. Playwright

My pick: Playwright for a controlled site audit; ZAP + Playwright/browser authentication if you also want vulnerability-oriented crawling.

Yes. For a staging site that intentionally blocks unauthenticated bots, I’d use a real browser crawler with a dedicated staging account, rather than trying to disguise a bot.

Good options

• Playwright — best if you want a programmable, repeatable audit. It runs Chromium/Firefox/WebKit, executes JavaScript, and can reuse authenticated cookies/local storage/IndexedDB through storageState. Playwright Playwright

• Log in once with a test account.
• Save the browser state.
• Crawl URLs using that state.
• Add route/domain restrictions so the crawler cannot leave staging.
• Particularly good for checking SPA routes, lazy-loaded content, authenticated navigation, screenshots, and rendered DOM.
• OWASP ZAP + its Client Spider — probably the best fit if this is a security/technical audit rather than just an SEO crawl. ZAP's Client Spider is specifically recommended for modern web apps and operates through a browser/DOM, while supporting ZAP's authentication mechanisms. ZAP

• Configure a ZAP Context for the staging hostname.
• Create a dedicated low-privilege test user.
• Use Browser Based Authentication if login requires JavaScript, MFA/TOTP, or multiple form steps. ZAP ZAP
• Run the Client Spider as that user.
• Keep the context strictly in scope.
• ZAP AJAX Spider — still useful for JS-heavy applications, although ZAP now recommends Client Spider for modern apps. ZAP

A safe staging setup

I'd structure it like this:

staging.example.com
        │
        ▼
dedicated audit account
        │
        ▼
Playwright / ZAP browser
        │
        ├── JS rendering
        ├── authenticated session
        ├── robots/bot restrictions remain irrelevant
        └── hostname/path allowlist

The important part is authentication rather than bot evasion. If the staging WAF blocks automation based on User-Agent, IP, or browser characteristics, explicitly allowlist the crawler's staging IP or service account instead of attempting to bypass the control.

For Playwright, the particularly useful pattern is to authenticate once and reuse the resulting state; Playwright explicitly warns that the saved state can contain cookies and headers capable of impersonating the test account, so keep it outside source control. Playwright

For ZAP, you can configure a Context with authentication, session management, a logged-in verification indicator, and a specific user; the resulting user can then be selected by the spider. ZAP ZAP

My pick: use Playwright for a controlled rendered-site/SEO/content audit, and ZAP Client Spider for a security audit. If you tell me whether your staging auth is basic auth, SSO/OIDC, username/password, or something like Cloudflare Access, I can give you the safest concrete setup.

Yes. For a staging audit where you need real authentication + JavaScript rendering, I’d favor browser-based crawlers rather than trying to spoof a bot user-agent.

Good options

• OWASP ZAP — probably the best fit for a security-oriented audit. Its browser-based authentication can handle modern login flows, including multi-step forms and TOTP, and its current Client Spider uses a real browser/DOM to explore JavaScript-heavy applications. ZAP specifically recommends Client Spider over its older AJAX Spider for modern apps. ZAP GitHub
• Playwright — best if you want a controlled/custom crawler. You can authenticate once, save the browser storageState, then crawl with an authenticated browser context. It supports HTTP auth as well as normal application login flows. Playwright Playwright
• Sitebulb — a good SEO/technical-audit choice if you want a GUI rather than building the crawler. It supports HTTP authentication and form-based authentication specifically for staging environments. Sitebulb Support
• Oncrawl — another SEO-oriented option with HTTP authentication support, although its staging guidance is more restrictive around robots directives. Oncrawl Help

What I'd use

For a safe staging audit, I'd set up:

Playwright → authenticated test account → Chromium → crawl only staging → capture rendered DOM/network requests

or, if you're looking for security issues:

ZAP → authenticated browser context → Client Spider → passive scan → optionally active scan

ZAP is particularly useful because authentication can be associated with a defined user/session, and its browser authentication state can be used by browser-driven exploration. ZAP ZAP

Make the bot blocking explicit

Instead of weakening your staging bot protection globally, give the crawler a narrowly scoped exception:

• Allow only the crawler's fixed source IP(s), or
• Require a dedicated staging header/token, or
• Allow a dedicated test account/session.
• Restrict access to the staging hostname and expected paths.
• Disable destructive actions for the crawler account.
• Rate-limit it fairly aggressively.
• Keep production completely outside the crawler's allowlist.

For example, a dedicated crawler-audit account with read-only permissions is preferable to using an administrator's session. Also keep Playwright/ZAP authentication state out of source control; Playwright explicitly warns that saved browser state can contain cookies and headers capable of impersonating the account. Playwright

If you tell me whether your staging auth is Basic Auth, SSO/OIDC, a username/password form, or something like Okta/Auth0, I can recommend the cleanest setup and give you a minimal Playwright or ZAP configuration.

Make the bot blocking explicit

Yes. For a staging site that blocks unauthenticated bots, I’d use a real browser crawler with a dedicated staging account, rather than trying to bypass the bot protection.

Good options

• Playwright — probably the best fit if your goal is primarily SEO/UX/rendering auditing. It runs a real Chromium/Firefox/WebKit browser, executes JavaScript, and can save/reuse an authenticated browser state containing cookies, local storage, etc. Playwright Playwright

• Log in once with a staging account.
• Save the storageState.
• Crawl URLs using that state.
• Restrict the crawler to your staging hostname and avoid destructive actions.
• OWASP ZAP + browser/AJAX Spider — better if "audit" includes security testing as well as crawling. ZAP supports browser-based authentication and client-script authentication, and its AJAX Spider launches a browser to explore JavaScript-heavy applications. ZAP ZAP

• For modern SPAs, ZAP's Browser Based Authentication + AJAX Spider is a strong combination.
• If the login flow is complicated, record a ZAP client/Selenium authentication script. ZAP ZAP
• ZAP's Automation Framework can make the whole process repeatable in CI. ZAP ZAP
• Playwright crawler + your existing auditing tooling — often the cleanest architecture if you're doing a large site audit. Have Playwright handle authentication/rendering and URL discovery, then feed the rendered pages/URLs into your SEO/accessibility/performance checks.

For your staging setup

I'd configure it roughly like:

Staging
   │
   ├── dedicated audit account
   │
   └── Playwright
        ├── authenticate
        ├── persist session
        ├── crawl only *.staging.example.com
        ├── execute JavaScript
        ├── capture redirects/statuses
        ├── capture rendered HTML
        └── collect links/resources

If the bot blocker is your own staging WAF/CDN rule, an even safer approach is to create a narrowly scoped exception for the crawler's identity (for example, a dedicated service account/IP or staging-only authentication header), rather than defeating the protection. If you're using ZAP, it can also apply an authentication header to requests when that's how your staging application authenticates. ZAP

One important precaution: don't put the saved Playwright authentication state in Git. It can contain cookies and headers capable of impersonating the test account. Playwright

My pick: use Playwright for a safe authenticated JS-rendered site audit; use ZAP + Playwright/browser authentication if you also want security findings.

Yes. For a staging site that blocks unauthenticated bots, I’d favor real-browser crawlers with a dedicated test account, rather than trying to disguise a bot.

Best options

• OWASP ZAP + Client Spider — probably the best fit for a security audit. ZAP’s current recommendation for modern JavaScript-heavy applications is the Client Spider, which actually drives a browser. It supports browser-based authentication, and ZAP can re-authenticate when a session expires. ZAP ZAP
• Playwright — best if you want a highly controlled, repeatable crawl. You can log in once, save the authenticated browser state, and reuse it for subsequent crawls. It renders the site in Chromium/Firefox/WebKit like a normal browser. Playwright
• OWASP PTK — useful if the audit is primarily security-focused. It operates from an authenticated browser context and can inspect browser-generated traffic, JavaScript, DOM behavior, and runtime state; it also integrates with Playwright/Puppeteer/Selenium/Cypress and ZAP. OWASP Foundation
• ZAP Browser-Based Authentication + Client Spider — particularly good when your login involves several forms, JavaScript, redirects, or TOTP. ZAP explicitly supports these flows and automatically authenticates browsers used by its modern spiders. ZAP

A safe staging setup

I'd structure it like this:

Dedicated staging hostname
        │
        ├── allow only your audit runner's IP/network
        │
        └── dedicated low-privilege audit account
                    │
             real browser session
                    │
          ┌─────────┴─────────┐
          │                   │
      Playwright          ZAP Client Spider
          │                   │
     crawl/render        passive + active audit

For Playwright, authenticate interactively or through your test login flow, save storageState, and launch subsequent browser contexts with that state. Playwright specifically warns that the saved state contains credentials/cookies and should not be committed to source control. Playwright Playwright

For ZAP, configure a Context containing only the staging hostname, create a dedicated user, select Browser Based Authentication, and provide a verification URL that clearly distinguishes logged-in from logged-out state. Then use the Client Spider. ZAP can also use a recorded/client authentication script when the login is too complicated for automatic browser authentication. ZAP ZAP

If the bot block is at the WAF

Don't try to defeat the WAF with spoofed user agents or bot-evasion tricks. Instead, make an explicit staging exception such as:

• allowlist the crawler's fixed egress IP;
• allow the staging hostname only from your audit network/VPN;
• give the crawler a dedicated authentication account;
• restrict the account's privileges;
• disable destructive operations for that account;
• put hard limits on crawl rate/concurrency;
• keep the crawler strictly scoped to the staging hostname.

That gives you the important distinction between "authenticated browser traffic that the staging environment intentionally permits" and "bypassing bot protection."

One current ZAP detail worth knowing: as of September 2026, ZAP is moving away from the older AJAX Spider toward the Client Spider; existing AJAX Spider automation remains supported, but Client Spider is the recommended choice for new modern-JS crawls. ZAP

My pick: use Playwright for deterministic authenticated crawling, then put ZAP Client Spider/PTK behind that workflow when the goal is a security audit rather than merely discovering URLs.

Yes. For a staging site that deliberately blocks unauthenticated bots, I’d favor real-browser crawlers rather than traditional HTTP-only crawlers.

Best options

• Playwright — probably the best fit if you want control and repeatability. It runs a real Chromium/Firefox/WebKit browser, executes JavaScript, and can save/reuse authenticated browser state (cookies, local storage, IndexedDB, etc.). Playwright Playwright
• OWASP ZAP + Ajax Spider — good if the goal is specifically a security audit. ZAP supports browser-based authentication, including multi-step login flows and TOTP, and its Ajax Spider is designed for modern JavaScript applications. ZAP
• Burp Suite + browser-driven crawling — useful when you want authenticated crawling combined with interception, passive analysis, and targeted security testing. I'd use it when the audit is more security-focused than SEO/accessibility-focused.

A safe staging setup

I'd structure it like this:

1. Create a dedicated low-privilege staging account specifically for crawling.
2. Authenticate once in Playwright and save its browser state.
3. Start a fresh browser context using that state.
4. Crawl only your staging hostname, with a strict URL allowlist.
5. Keep concurrency/rate limits low so the crawler doesn't accidentally behave like a DoS tool.
6. Disable destructive actions where possible—don't follow logout/delete/checkout/admin mutation endpoints merely because they're linked.
7. Block outbound requests to production and unrelated third-party systems.
8. Give the crawler a distinctive User-Agent so your staging WAF/logs can identify it.
9. If the bot protection is yours, preferably allowlist the crawler's staging IP/service identity rather than trying to evade the bot protection.
10. Keep the authentication-state file out of source control; Playwright explicitly warns that it can contain credentials capable of impersonating the test account. Playwright

For example, Playwright can authenticate through the UI and persist the resulting state, then subsequent browser contexts start already logged in. Playwright

My pick: use Playwright for the crawl/rendering layer, and ZAP alongside it if this is a security audit. ZAP's browser-based authentication is particularly useful when the login flow itself is JavaScript-heavy. ZAP ZAP

If you tell me whether you're auditing SEO, accessibility, security, or all three, I can give you a concrete crawler architecture and configuration for your staging environment.

For a staging environment you own/control, the safest approach is usually not to bypass bot protection, but to give your audit crawler an explicit authenticated path (allowlisted identity, test account, or staging-only access rule). The main options depend on what kind of protection you have.

1. SEO/site audit crawlers with authentication + JS rendering

Good for technical SEO, links, metadata, rendering issues:

• Screaming Frog — supports:

• JavaScript rendering via Chromium
• form-based authentication (log in through an embedded browser)
• custom cookies/headers
• crawling behind staging restrictions Screaming Frog Screaming Frog
• Sitebulb — supports:

• HTTP authentication
• custom headers
• cookies for authenticated staging crawls Sitebulb Support
• Oncrawl — supports:

• HTTP auth
• IP allowlisting workflows
• staging/pre-production crawling configurations Oncrawl Help

Typical setup:

• Create a staging-only audit user
• Authenticate with cookies/session
• Enable JS rendering
• Lower crawl concurrency so you don't stress staging

2. Browser automation crawlers (best when the site behaves like a real user)

For React/Vue/Angular apps, authenticated dashboards, complex flows:

• Playwright
• Puppeteer
• Crawlee (built around browser crawlers and supports Playwright-based crawling) Crawlee

A common pattern:

1. Launch headless Chromium
2. Log in with a test account
3. Save the session storage/cookies
4. Crawl URLs while reusing that session
5. Capture: - rendered HTML
6. screenshots
7. console errors
8. failed network requests
9. accessibility/performance data

This is usually the closest match to "what a real authenticated user sees."

3. Managed browser crawling services

Useful if you don't want to maintain browser infrastructure:

• Browserless
• Browserbase
• Apify

These can run Playwright/Puppeteer-style sessions, but for a private staging site you generally want:

• private network access (VPN/VPC)
• allowlisted IPs
• test credentials
• audit logs

4. Recommended staging-friendly architecture

A robust pattern:

Staging site
    |
    +-- allowlist crawler identity
    |
    +-- require authentication
    |
    +-- create audit user
    |
Crawler
    |
    +-- login
    +-- store session cookie
    +-- render JS
    +-- crawl slowly
    +-- export report

Avoid:

• disabling bot protection globally
• using production credentials
• using CAPTCHA-solving/bypass services against your own staging controls unless you are specifically testing that defense

Quick chooser

If you tell me what is blocking bots (Cloudflare, basic auth, SSO, robots.txt, WAF rule, IP allowlist, etc.) and whether you need SEO auditing vs functional testing, I can suggest the cleanest setup.

For a staging environment that blocks bots, you generally want a crawler that supports authenticated sessions + real browser rendering, rather than trying to “spoof” a bot. The safest pattern is to give the audit tool a controlled way in (test credentials, allowlisted IP, signed header, or staging-only auth bypass) and let it run with a headless browser.

Options to look at:

For a locked-down staging site, I’d usually use one of these patterns:

Option 1: Browser session crawl (most realistic)

1. Create a dedicated staging audit user.
2. Log in through a real browser.
3. Export or reuse the session cookies.
4. Crawl with a tool that can attach those cookies.
5. Enable JavaScript rendering.

This catches issues like:

• client-side routing failures
• missing API calls
• hydration problems
• lazy-loaded content
• authenticated pages rendering incorrectly

Option 2: Add a staging-only crawler allowlist

Instead of weakening bot protection globally:

• Allow only your crawler runner’s IP range.
• Require a special header, e.g. X-Staging-Audit: true.
• Keep it restricted to staging.
• Log all crawler traffic.

Avoid allowing based only on User-Agent; it is easy to misconfigure and not a strong security boundary.

Option 3: Use signed bot authentication

Some platforms support cryptographic bot verification instead of simple allowlists. For example, Web Bot Auth-style approaches use signed headers so the site can verify an approved crawler. squirrelscan

If you want a DIY stack

A common setup:

• Playwright → browser automation
• Chromium → JS execution
• Storage state file → saved authenticated session
• Lighthouse → performance/accessibility checks
• Crawler queue → URL discovery and limits

Example flow:

CI runner
   |
   |-- Playwright login
   |
   |-- save auth state
   |
   |-- crawl URLs
   |
   |-- render JS
   |
   |-- collect screenshots + HTML + network errors

Staging safety checklist

• Use a non-production account with minimal permissions.
• Rate-limit the crawler.
• Disable destructive actions in the test account.
• Block indexing (noindex, robots rules) but don’t rely on those for security.
• Capture network failures (401/403/500), not just page HTML.
• Keep crawl scope limited (start URLs + depth/page limits).

If your staging block is Cloudflare/WAF, HTTP basic auth, SSO, or an application login, the best crawler choice changes slightly.

Yes. If this is your staging environment, the safest pattern is to give the crawler a dedicated low-privilege test identity and authenticate normally, rather than trying to defeat the bot protection.

Good options:

• Playwright — probably the best fit if you need real browser rendering. It executes Chromium/Firefox/WebKit, can log in through the UI, and can persist authenticated cookies/local storage/IndexedDB as a storageState file for subsequent crawling. Playwright
• Sitebulb — useful if the goal is primarily an SEO/technical crawl. Its current documentation specifically covers crawling authenticated staging sites and supports several authentication approaches. Sitebulb Support
• Screaming Frog SEO Spider — good for conventional site auditing; its JavaScript rendering can execute client-side JS. For authenticated staging, configure authentication at the crawler level rather than weakening the staging gate.
• Browser-based custom crawler — Playwright plus a small URL-discovery loop gives you the most control over exactly what gets requested, concurrency, headers, authentication and actions.
• Burp Suite + browser — better if "audit" means security testing rather than SEO/accessibility. You can browse authenticated pages with a real browser and inspect the resulting requests. OWASP specifically recommends examining authenticated and unauthenticated sessions when auditing dynamically generated JavaScript and sensitive endpoints. OWASP Web Security Testing Guide

What I'd use

For a staging audit, I'd use:

Playwright → dedicated test account → authenticated browser state → allowlisted staging hostname → low concurrency.

For example, authenticate once and save the state:

await page.goto('https://staging.example.com/login');
await page.getByLabel('Email').fill(process.env.AUDIT_USER!);
await page.getByLabel('Password').fill(process.env.AUDIT_PASSWORD!);
await page.getByRole('button', { name: /sign in/i }).click();

await page.waitForURL('https://staging.example.com/');
await page.context().storageState({
  path: 'playwright/.auth/staging.json'
});

Then run the crawler using that state:

const context = await browser.newContext({
  storageState: 'playwright/.auth/staging.json'
});

Playwright explicitly warns that this state can contain cookies and headers capable of impersonating the account, so keep it out of source control and preferably use a short-lived/disposable audit account. Playwright

Make it safe

I'd also put these guardrails around the crawl:

• Use a dedicated read-only account, not an admin account.
• Give the crawler a strict staging-host allowlist so links can't take it to production.
• Cap concurrency and request rate.
• Don't follow logout, delete, purchase, mutation, or other destructive endpoints automatically.
• Prefer a staging dataset containing synthetic/non-sensitive data.
• Keep the authentication state and credentials in secrets storage.
• If the bot blocker is yours, allowlist the crawler's IP/user-agent at the staging layer rather than attempting to evade its detection.
• Capture both the rendered DOM and network requests; JS rendering can reveal API endpoints and dynamically generated content that an HTML-only crawler misses. OWASP recommends specifically examining JavaScript and dynamically generated responses during web application testing. owasp.org

If you tell me whether this is primarily an SEO crawl, accessibility audit, security assessment, or visual/regression audit, I can recommend a concrete crawler stack and configuration for that use case.

Yes. For a staging site that intentionally blocks generic bots, I’d use a real browser-based crawler with a dedicated staging account, rather than trying to disguise a bot.

Good options

• playwright.dev — Best if you want a programmable crawler/audit harness. It runs Chromium/Firefox/WebKit, executes the site's JavaScript, and can persist authenticated browser state (cookies, local storage, IndexedDB, etc.) via storageState. Playwright GitHub

• Log in once with a test account.
• Save the authenticated state.
• Crawl URLs using a normal browser context.
• Capture rendered DOM, network requests, console errors, screenshots, accessibility data, etc.
• Particularly good if you want the audit integrated into CI.
• zaproxy.org — Best if "audit" includes security testing as well as crawling. Its Client Spider launches a real browser and handles JavaScript-heavy applications; as of July 2026, ZAP recommends Client Spider over AJAX Spider for modern apps. GitHub

• Supports browser-based authentication, including complex login flows and TOTP.
• Can maintain authenticated sessions while spidering.
• You can combine its traditional spider with Client Spider for broader endpoint coverage. ZAP ZAP
• Playwright + ZAP together — Probably my preferred setup for a serious staging audit. Use Playwright to establish the authenticated browser session and exercise difficult UI flows, while ZAP observes/proxies the traffic and performs passive/active security analysis.

For a bot-blocked staging environment

I'd make the staging controls explicitly recognize your audit runner rather than bypassing the bot protection:

staging
  ├── dedicated audit account
  ├── crawler runner IP allowlisted
  ├── rate limit kept conservative
  ├── production/destructive actions disabled
  └── audit URLs scoped to staging hostname

For authentication, don't put a real user's password or reusable session cookie into the crawler configuration. Playwright specifically warns that persisted auth state can contain credentials/cookies capable of impersonating the account and recommends keeping it out of source control. Playwright

If your blocker is something like a WAF/bot-management layer, the cleanest approach is usually a staging-only exception keyed to the crawler's IP/service identity, rather than modifying browser fingerprints or attempting to evade the detection.

Which I'd choose

If you tell me what is blocking the staging crawler (Cloudflare, Akamai, AWS WAF, robots.txt, application middleware, etc.) and what authentication you use (SSO/OIDC, basic auth, form login, etc.), I can give you a concrete safe setup.

Yes. For a staging site that intentionally blocks unauthenticated bots, I’d favor browser-based crawlers that can carry a real authenticated session rather than trying to disguise a conventional bot.

Good options

• zaproxy.org + Client Spider — probably the best fit for a security/technical audit. ZAP’s Client Spider launches a browser, executes JavaScript, has DOM access, and supports its authentication mechanisms. It is currently the recommended modern-app spider over the older AJAX Spider. ZAP ZAP
• playwright.dev — best if you want a programmable crawler/audit harness. Log in once, save the browser storageState, then crawl using an authenticated Chromium context. It handles cookies, local storage, IndexedDB, and modern browser behavior. Playwright Playwright
• screamingfrog.co.uk — convenient for an SEO/content/technical crawl. Its JavaScript rendering uses a browser, and it supports logging into form-authenticated staging sites through its built-in Chrome browser. Its staging guidance specifically notes that this can also work where bot protection such as reCAPTCHA is present. Screaming Frog Screaming Frog

How I'd set up a safe staging audit

1. Create a dedicated low-privilege audit account. Don't use a production account or an administrator.
2. Authenticate through the site's normal login flow, preferably interactively. With Playwright, save the resulting session as storageState; don't commit that file because it can contain credentials/session material. Playwright
3. Allowlist the crawler at the staging edge/WAF rather than weakening bot protection globally. A narrowly scoped IP/rule for your audit runner is preferable to removing the protection.
4. Constrain the crawl to staging. Explicitly scope the hostname and exclude production, logout endpoints, destructive actions, account deletion, checkout/payment, etc.
5. Use a read-only account and conservative crawl rate. For an initial audit, I'd disable active/destructive security tests and start with passive crawling/rendering.
6. Render with a real browser. This catches SPA routes, dynamically injected links, client-side redirects, lazy content, and authenticated API calls that a plain HTTP crawler misses.
7. Verify authentication continuously. A crawler can silently fall back to the login page after a session expires, producing a misleading "successful" crawl.

For a security audit, my first choice would be ZAP Client Spider + browser-based authentication; ZAP can also run the authenticated crawl through its automation framework. ZAP ZAP

For a custom site/UX/SEO audit, I'd use Playwright, because you have much finer control over what gets visited and what constitutes a safe action.

The key distinction is: don't bypass the staging bot control by spoofing/evading it. Have the staging environment explicitly recognize your authorized audit runner, then let the crawler authenticate normally.

For a staging site that blocks bots, you usually want a crawler that can run with your credentials/session, execute JavaScript in a real browser, and avoid bypassing your protections in production-like ways. The safest pattern is to explicitly allow your audit runner (IP, service account, header, or test user) rather than weakening bot controls. squirrelscan

Here are the main options:

1. Headless browser crawlers with authenticated sessions (best for JS apps)

These run Chromium/Firefox, log in, execute JS, and audit the rendered DOM.

• Screaming Frog SEO Spider

• Supports JavaScript rendering through Chromium.
• Can use custom headers, cookies, and authentication workflows.
• Good for technical SEO, links, metadata, structured data, and rendered-page checks. theStacc
• Sitebulb

• Browser-based rendering and audit reports.
• Useful when you want visual reports and team workflows. theStacc
• Oncrawl

• Enterprise crawler with JavaScript rendering and large-scale crawl configuration. Oncrawl

2. Playwright/Puppeteer custom crawler (most control)

For staging environments, this is often the cleanest approach.

Typical flow:

1. Create a dedicated staging audit account.
2. Launch a browser.
3. Authenticate: - form login
4. SSO test account
5. pre-generated session cookie
6. OAuth test flow
7. Crawl allowed routes.
8. Capture: - rendered HTML
9. screenshots
10. console errors
11. failed network requests
12. accessibility/SEO findings

Example stack:

• Microsoft Playwright
• Puppeteer

This works well in CI because you can inject secrets from your CI secret store and never share credentials with a crawler vendor.

3. Authenticated audit platforms

Useful if you want scheduled scans and dashboards.

• Semrush Site Audit can enable JavaScript rendering and crawl configuration options. Semrush
• Some accessibility auditors can operate inside an authenticated browser session rather than crawling from the outside. Complaudax

Recommended staging setup

A common safe architecture:

staging.example.com
        |
        |
  allow audit runner
        |
        v
Playwright/Screaming Frog
        |
        |
 authenticated test user
        |
        v
JS-rendered audit

Add controls like:

• dedicated crawler-audit@example test user
• read-only permissions
• rate limits
• audit-only IP allowlist
• request header such as:

X-Staging-Audit: true

Then your bot protection can allow only that path.

What to configure in your bot protection

Instead of disabling bot blocking:

• Allowlist the crawler's IP range (if stable).
• Allow a signed header/token from your CI runner.
• Exempt only staging paths: - /audit/*
• /sitemap-test/*
• /protected-test/*
• Keep production bot rules unchanged.
• Log crawler traffic separately.

For a modern React/Next/Vue app, I’d pick:

• CI/security-minded: Playwright + Lighthouse + axe-core.
• SEO/technical audit: Screaming Frog with JS rendering. theStacc
• Enterprise scheduled audits: Oncrawl or similar platforms with browser rendering. Oncrawl

If you tell me your stack (Cloudflare/Akamai/WAF, SSO type, React/Next/etc., and whether this is SEO, accessibility, security, or QA auditing), I can suggest the exact crawler/auth pattern.

Yes. For a staging environment that requires authentication and has client-side rendering, I’d use a crawler that can maintain an authenticated browser session rather than trying to defeat the bot protection.

Good options

• screamingfrog.co.uk — probably the most flexible desktop choice. It supports basic/digest auth and, importantly, form-based authentication through its built-in Chrome browser. You can log in normally, have it retain the resulting cookies, and then crawl authenticated pages. It also supports JavaScript rendering. Screaming Frog Screaming Frog
• sitebulb.com — particularly nice for SEO/technical audits. Its Chrome Crawler renders JavaScript, and its staging authentication supports HTTP auth, form authentication/cookies, etc. It also has a Response-vs-Render report, which is useful for identifying content/links that only appear after JS execution. Sitebulb Support Sitebulb Support Sitebulb Support
• Playwright/Puppeteer — best if you need a custom security/audit workflow rather than an off-the-shelf SEO crawler. You can explicitly automate login, preserve a session, control navigation, capture screenshots/network requests, and restrict the crawl to a known URL set.

For your particular staging setup

I'd configure it like this:

1. Create a dedicated staging audit account, rather than using an employee's credentials.
2. Give that account read-only access wherever possible.
3. Authenticate through the application's normal login flow and retain the resulting session cookie/token.
4. Use a real Chromium-based renderer for the crawl.
5. Allowlist the crawler's IP/user-agent at the staging WAF/bot-protection layer rather than attempting to disguise the crawler.
6. Keep the crawl constrained to staging and set a URL/depth/rate limit.
7. Disable actions that could mutate data—POSTs, form submissions, deletes, checkout/payment flows, etc.
8. If the staging environment is intentionally noindex/robots-blocked, configure the crawler to ignore those directives only within the staging audit. Sitebulb explicitly documents this workflow. Sitebulb

For example, with Screaming Frog, the combination of Forms Based authentication + JavaScript rendering is a strong fit when the staging site has an actual login page. Its authentication mechanism supplies the crawler with the authenticated browser cookies, including scenarios where bot protection relies on the authenticated session. Screaming Frog

One important distinction: if your "bot blocking" is a WAF returning a 403/JS challenge before the login page is reachable, don't try to work around that with spoofed headers or user-agent tricks. The clean solution is to create a narrow staging-only allowlist for your audit runner.

If you tell me whether the staging auth is Basic Auth, SSO/OIDC, a normal username/password form, or Cloudflare/WAF, I can recommend the safest setup and exact crawler configuration.

Yes. For a staging site that intentionally blocks ordinary bots, I’d favor real-browser crawlers that can maintain an authenticated session rather than trying to defeat the bot protection.

Good options

• OWASP ZAP + Client Spider — probably the best fit for a security audit. ZAP can authenticate using a real browser, retain the session, and crawl JavaScript-heavy applications. As of July 2026, ZAP recommends its Client Spider over the older AJAX Spider for modern apps. ZAP GitHub

zaproxy.org

• Playwright — excellent if you want a controlled, custom crawler. You can log in once, save the authenticated browser state, then launch crawls using that state while Chromium executes the site's JavaScript normally. Playwright

playwright.dev

• ZAP + Playwright/browser-driven exploration — a particularly useful combination: use a real browser to authenticate and render the application, while ZAP proxies the traffic so you get passive security analysis and an application map. ZAP explicitly supports browser-based authentication and client-side session handling. ZAP ZAP

For your "blocks bots" situation

I'd configure staging roughly like this:

1. Create a dedicated low-privilege audit account.
2. Run the crawler from an approved staging IP/network, if your bot protection supports allowlisting.
3. Authenticate through the actual login UI rather than injecting production credentials or bypassing controls.
4. Persist the resulting browser cookies/storage state.
5. Crawl with JavaScript enabled.
6. Start with passive-only analysis so the crawler doesn't mutate application state.
7. Explicitly exclude logout, destructive actions, payments, account deletion, admin mutations, etc.
8. Only enable active security testing against staging after confirming the test account and environment can tolerate it.

ZAP's browser authentication can handle multi-step login flows, and its client-script authentication can cover more complicated flows such as custom login sequences. ZAP ZAP

If the goal is primarily SEO/rendering auditing, I'd use Playwright. If it's primarily security auditing, I'd start with ZAP Client Spider. If you tell me whether the staging auth is SSO/OIDC, basic login, or something like MFA, I can give you a concrete setup for the appropriate option.

Yes. For a staging site that blocks ordinary bots but permits an authenticated session, I’d use one of these approaches:

• Sitebulb — probably the easiest GUI option. Its Chrome Crawler uses headless Chromium for JS rendering, and it supports form-based authentication, HTTP auth, cookies, and custom HTTP headers. It also has crawl limits and speed controls, which are useful for keeping staging load low. Sitebulb Support Sitebulb Support
• Screaming Frog SEO Spider — another strong desktop option. It supports staging authentication via cookies/custom HTTP headers, and its JS rendering mode can use a browser renderer. Screaming Frog specifically recommends reducing crawl speed on staging if the environment is fragile. Screaming Frog
• Playwright — best if you want a highly controlled/custom audit. You can launch Chromium, authenticate through the actual login flow, persist the authenticated browser state, then crawl URLs while executing JS. This is particularly good when authentication involves SSO, MFA-compatible flows, custom headers, or complicated app state.
• Puppeteer — similar to Playwright, particularly convenient if your tooling is already Node/Chrome based. It's more of a framework for building your own crawler than an out-of-the-box SEO auditor.

What I'd do for a safe staging audit

I'd avoid trying to defeat the bot protection itself. Instead, have the staging environment explicitly permit your crawler:

1. Create a dedicated staging audit account with read-only/minimal permissions.
2. Allowlist the crawler's IP, service identity, or a dedicated authentication header if possible.
3. Authenticate normally and preserve the session cookie/token.
4. Use Chromium/Chrome rendering so client-side routes and content are actually executed.
5. Restrict the crawl to the staging hostname and relevant URL paths.
6. Start with perhaps 10–50 URLs, verify behavior, then increase gradually.
7. Put a conservative request/Chrome-instance limit on it. Sitebulb, for example, provides URL limits, crawl-depth limits, cookies, session isolation, and crawl-speed controls. Sitebulb Support Sitebulb Support
8. Exclude destructive endpoints (POST actions, logout, account deletion, admin mutations, etc.). Ideally the crawler should only make GET requests.
9. Keep staging's noindex/robots protections intact for external crawlers; configure the authorized audit crawler separately rather than making staging publicly crawlable.

For SEO/technical auditing specifically, I'd lean Sitebulb or Screaming Frog. For an application/security-style audit where you need to control browser state and navigation precisely, I'd choose Playwright.

One particularly useful setup is Playwright for authentication + a crawler/audit layer for URL discovery, because you can save an authenticated browser state and reuse it without putting the actual password into every crawl request.

Yes. For a staging site that intentionally blocks unauthenticated bots, I’d use a real browser crawler with a dedicated staging account, rather than trying to defeat the bot protection.

Good options

• Playwright — best if you want maximum control or an automated audit pipeline. It runs a real Chromium/Firefox/WebKit browser, supports JavaScript naturally, and can save/reuse authenticated browser state (cookies, local storage, etc.). Playwright

• Log in once with a dedicated test account.
• Save storageState.
• Crawl URLs using that state.
• Capture rendered DOM, links, HTTP responses, console errors, screenshots, and accessibility data.
• Particularly good if you want to build your own audit checks.
• Screaming Frog SEO Spider — probably the easiest off-the-shelf choice for a conventional site audit. It supports forms-based authentication, authenticated cookies, and JavaScript rendering via Chromium. It can also run custom JavaScript/extraction against rendered pages. Screaming Frog Screaming Frog

• Configure Configuration → Authentication → Forms Based.
• Log into staging through its browser.
• Enable JavaScript rendering.
• Crawl with the authenticated session.
• It can export an authentication profile for scheduled/CLI crawls. Screaming Frog
• Sitebulb — another good GUI crawler, particularly if you're looking for an SEO/technical audit rather than building a custom security-testing harness. Its current documentation specifically covers authentication for staging environments. Sitebulb Support

For a bot-blocked staging environment

I'd structure it like this:

Internet
   │
   ▼
Staging WAF / bot protection
   │
   │  allowlist crawler IP
   │  OR allow dedicated test identity
   ▼
Staging application
   │
   ▼
Authenticated browser session
   │
   ├── rendered DOM
   ├── links/resources
   ├── JS console errors
   ├── network requests
   ├── screenshots
   └── audit assertions

Don't weaken the bot protection globally. Instead, give your crawler a narrowly scoped exception: ideally an allowlisted source IP, dedicated staging hostname, or dedicated test account/session. This lets you test the application in approximately the same protected configuration while avoiding the crawler fighting the WAF.

A particularly safe Playwright setup

Create a read-only audit account and authenticate it once:

audit account
    ↓
login
    ↓
save authenticated browser state
    ↓
start new browser context with that state
    ↓
crawl staging URLs

Playwright explicitly supports this pattern through storageState; its documentation also warns that the resulting state can contain credentials/cookies capable of impersonating the account, so keep the auth-state file out of source control. Playwright

I'd also put guardrails around the crawler:

• Use a dedicated account with no administrative/write privileges.
• Block or exclude logout, delete, purchase, mutation, and other state-changing URLs.
• Cap concurrency and request rate; staging systems are often less resilient than production. Screaming Frog similarly recommends reducing crawl speed if staging starts returning errors/timeouts. Screaming Frog
• Set a crawl scope such as staging.example.com/**.
• Disable links that can trigger destructive actions.
• Keep authentication secrets and session state in a secret store/ephemeral workspace.
• If the audit is security-oriented, separate crawling from active vulnerability testing so the crawler doesn't accidentally mutate application state.

One important distinction: JavaScript rendering ≠ authentication bypass. If the staging WAF recognizes your crawler as a bot, the clean solution is to explicitly authorize the crawler (IP/session/account), then let the browser render the application normally. That gives you a much more representative audit.

If your goal is specifically SEO/technical crawling, I'd pick Screaming Frog. If you want custom security/accessibility/performance assertions or CI integration, I'd pick Playwright.

audit account
    ↓
login
    ↓
save authenticated browser state
    ↓
start new browser context with that state
    ↓
crawl staging URLs

Playwright explicitly supports this pattern through storageState; its documentation also warns that the resulting state can contain credentials/cookies capable of impersonating the account, so keep the auth-state file out of source control.

For a staging site that intentionally blocks bots, you generally want a crawler that can act like an authenticated browser, not just a user-agent switch. The key capabilities to look for are:

• Authentication support

• Basic auth credentials (Authorization header)
• Login flow automation (username/password form, SSO where possible)
• Reuse of an existing browser session/cookies
• Custom headers/tokens
• IP allowlisting or WAF bypass for the crawler runner
• Real browser rendering

• Chromium/Playwright/Puppeteer-based crawling
• JavaScript execution
• SPA hydration support (React/Vue/Angular/Next.js)
• Ability to capture the post-render DOM, screenshots, console errors, and failed network requests
• Staging-safe controls

• Crawl rate limits
• URL allowlists
• No indexing/no external sharing
• Ability to run inside your VPC or CI environment if the staging site is sensitive

Some options:

Enterprise/SEO audit crawlers

• Screaming Frog — good for technical audits; supports JavaScript rendering through a headless Chromium engine and can be configured for authenticated crawling using cookies, custom headers, or forms. theStacc
• Semrush Site Audit — supports JavaScript rendering and has options for crawling restricted/pre-production sites, including credential-based access for password-protected areas. Semrush
• Lumar — aimed at larger sites; commonly used for scheduled technical audits and complex crawl setups. theStacc

Developer-controlled approaches

For staging, these are often the cleanest:

• Playwright

• Log in once
• Save browser storage state (cookies/local storage)
• Crawl URLs as an authenticated user
• Inspect rendered DOM
• Puppeteer

• Similar approach using Chrome DevTools Protocol
• Useful for custom audit scripts

A typical flow:

1. Launch headless Chromium.
2. Authenticate against staging.
3. Save the authenticated session.
4. Visit each route.
5. Wait for network idle or app-specific readiness.
6. Collect: - rendered HTML
7. screenshots
8. console errors
9. failed requests
10. accessibility/SEO/security checks

Accessibility-focused authenticated auditing

If the goal is accessibility rather than SEO crawling, browser-session tools can be useful because they audit pages you can already access while logged in. For example, some tools run audits inside an authenticated browser session rather than trying to crawl from the outside. Complaudax

Recommended staging setup

For most teams, I would use one of these patterns:

CI-friendly

• Playwright crawler
• Dedicated staging test account
• Allowlisted runner IP
• Export reports as artifacts

SEO/technical audit

• Screaming Frog or Semrush with: - JS rendering enabled
• staging credentials
• restricted crawl scope

Security-sensitive staging

• Self-host the crawler inside the same network
• Avoid giving third-party SaaS access to staging credentials

A few details would change the recommendation:

• Is the bot block a robots.txt/WAF rule, basic auth, SSO, or an application login?
• Is the site a SPA (React/Vue/etc.) or server-rendered?
• Are you auditing SEO, accessibility, security, or general QA?

Yes. For a staging site that intentionally blocks unauthenticated bots, I’d use a real browser-based crawler with a dedicated test account, rather than trying to bypass the bot protection at the HTTP level.

Good options

• OWASP ZAP + browser-based authentication — probably the best fit for a security audit. ZAP's modern AJAX Spider / Client Spider can execute JavaScript, and its browser-based authentication runs the login flow inside the browser. ZAP specifically recommends browser-based authentication for modern JS-heavy applications. ZAP ZAP
• Playwright — excellent if your goal is primarily authenticated crawling, rendering, screenshots, link discovery, and custom audit logic. You can authenticate once and reuse the resulting browser state (storageState) across crawls. Playwright Playwright
• Playwright + ZAP — a particularly strong combination: use Playwright to establish the authenticated browser session and exercise SPA routes, while ZAP provides proxying, passive analysis, and DAST. OWASP also documents browser-based integrations between its security tooling and ZAP. OWASP
• ZAP Client Script Authentication — useful when login involves SSO, MFA/TOTP, multiple screens, pop-ups, or other flows that automatic credential filling can't handle. ZAP can record a browser authentication sequence and replay it for the authenticated spiders. ZAP

A safe staging setup

I'd configure it roughly like this:

1. Create a dedicated low-privilege audit account.
2. Allow that account/browser profile through the staging bot/WAF rule, rather than weakening the protection globally.
3. Authenticate in a real Chromium browser.
4. Crawl only your staging hostname and explicitly permitted API/static domains.
5. Exclude destructive operations such as DELETE, account deletion, payments, email sending, and state-changing admin actions.
6. Explicitly prevent the crawler from following logout links. ZAP has a logout-avoidance facility for its traditional and AJAX spiders. ZAP
7. Rate-limit the crawler so the staging environment isn't accidentally load-tested.
8. Keep authentication state/cookies out of source control. Playwright warns that its saved auth state can contain credentials capable of impersonating the test user. Playwright

If I were choosing

For a security audit:
ZAP Browser-Based Authentication → AJAX/Client Spider → passive scan → carefully scoped active scan.

For a comprehensive authenticated site crawl:
Playwright → dedicated authenticated context → JS rendering → URL/DOM/API discovery → optionally proxy through ZAP.

The important distinction is that "authenticated" and "JavaScript-capable" should happen in the same browser context. Simply giving a conventional HTTP crawler a session cookie often misses SPA content that only appears after client-side rendering. ZAP's recent authentication work explicitly addresses this problem. ZAP

If you tell me whether the staging login is username/password, SSO/OIDC, or MFA, I can recommend the exact ZAP/Playwright setup and crawler configuration.

Yes. For a staging site that intentionally blocks generic bots, I’d favor real-browser crawlers that authenticate as a dedicated test user rather than trying to disguise a bot.

Best options

• zaproxy.org — Client Spider + Browser-Based Authentication. This is probably the best fit for a security audit. ZAP’s current recommendation for modern JS applications is its Client Spider, which launches browsers, executes JavaScript, accesses the DOM, and can crawl authenticated content. ZAP GitHub

• Configure a staging-only context and dedicated low-privilege account.
• Use Browser Based Authentication so the browser itself performs the login; it supports multi-step login flows and TOTP. ZAP
• If your login flow is unusually complicated, ZAP also supports Client Script Authentication, which can replay a recorded browser authentication flow. ZAP
• For coverage, run the traditional Spider alongside Client Spider; ZAP specifically recommends combining them. GitHub
• playwright.dev — excellent if you want a custom crawler or audit harness rather than a security scanner. It drives Chromium/Firefox/WebKit as a real browser and supports saving/reusing authenticated browser state. Playwright

• Log in once with a dedicated account.
• Persist the authenticated browser state.
• Crawl routes and interact with the application using the browser.
• Be careful with the saved state: Playwright notes that it can contain cookies/headers capable of impersonating the test account and shouldn't be committed to source control. Playwright

For a safe staging audit

I'd set it up roughly like this:

staging → dedicated test account → browser authentication → strict allowlist → JS-rendering crawler → passive audit

Keep the crawler constrained to the staging hostname, use a non-production account, and explicitly exclude destructive endpoints such as logout/account deletion/payment/admin actions. ZAP's Client Spider has scope controls, crawl-depth/state/time limits, and logout avoidance that are useful for this. ZAP

If your immediate goal is security auditing rather than just SEO/link discovery, I'd choose ZAP Client Spider + Browser-Based Authentication first. ZAP can passively inspect the traffic generated during the authenticated browser crawl without modifying responses; reserve active scanning for an explicitly authorized staging target because active scanning sends attack traffic. ZAP

If you tell me whether your auth is OIDC/OAuth, SAML, username/password + MFA, or a custom login, I can recommend the cleanest ZAP/Playwright setup for it.

For a staging environment, the safest pattern is usually not to “bypass” bot protection, but to give your audit crawler an explicit authenticated path (allowlisted account, token, or signed bot identity) and then run it with browser rendering enabled. Sitebulb Support squirrelscan

Common crawler options:

For JS rendering, look for these crawler capabilities:

• Headless Chromium/Playwright rendering — executes React/Vue/Angular/etc. and audits the post-render DOM. squirrelscan Cloudflare Docs
• Wait conditions — wait for network idle, specific selectors, or app hydration before capturing HTML.
• Authenticated browser context — the browser session and cookies persist while pages are crawled.
• Custom viewport/device settings — useful for responsive audits.
• HAR/network capture — helps diagnose missing API calls or blocked assets. GitHub

Tools commonly used for this kind of staging audit:

• Screaming Frog — supports JavaScript rendering and authenticated crawling workflows.
• Sitebulb — supports HTTP auth, headers, cookies, and staging crawl configurations. Sitebulb Support
• Playwright — good when you need a custom authenticated browser journey. Cloudflare Docs
• Puppeteer — similar approach using Chrome automation. Cloudflare Docs

A practical staging setup I’d recommend:

1. Create a dedicated audit-only user (least privilege).
2. Allowlist the crawler’s identity at the staging WAF/CDN layer.
3. Give the crawler either: - a short-lived token/header, or
4. a pre-authenticated browser session.
5. Enable JS rendering.
6. Limit crawl rate and URL scope to avoid stressing staging.
7. Capture screenshots + rendered HTML + network errors for comparison.

If your blocker is something specific (Cloudflare, Akamai, SSO, basic auth, robots.txt, or an internal WAF), the best authentication option changes.

Yes. For a staging site that intentionally blocks ordinary bots, I’d use a real browser with a dedicated test account, rather than trying to disguise a crawler as a search bot.

Best options

My first choice is ZAP's Client Spider. As of July 2026, ZAP officially recommends it for modern JavaScript-heavy applications; it launches a browser, interacts with the DOM, and supports ZAP's authentication mechanisms. www.zaproxy.org

For authentication, ZAP supports browser-based authentication, including multi-step forms and TOTP, and can automatically authenticate browsers launched for the spider. www.zaproxy.org

If you want maximum control: Playwright

Playwright is particularly good if your audit needs to follow a specific workflow:

1. Create a dedicated staging account with minimal privileges.
2. Run Playwright against staging.
3. Log in normally through the UI.
4. Save the authenticated browser state.
5. Start new browser contexts using that state.
6. Crawl your allowed routes and inspect the rendered DOM/network traffic.

Playwright explicitly supports reusing authenticated storageState, which can contain cookies and local storage. It warns that these files can effectively contain credentials, so keep them out of source control. playwright.dev

Making it safe

Since this is staging, I'd configure the crawler to be deliberately constrained:

• Allowlist the staging hostname, not the production domain.
• Use a dedicated test account, preferably with the least privileges needed.
• Give the crawler a finite URL/depth/time budget.
• Disable destructive actions such as account deletion, payments, email sending, and data mutation where possible.
• Prefer GET/navigation discovery initially; separately test state-changing endpoints.
• Rate-limit requests so the staging environment isn't accidentally load-tested.
• Keep authentication cookies/tokens in ephemeral storage.
• Verify that redirects cannot take the crawler from staging to production.
• If your bot protection has an allowlist, allow the crawler's fixed egress IP/user-agent rather than weakening the site's bot controls globally.

For a security audit, ZAP is especially useful because it can combine the browser crawl with authenticated HTTP analysis and active/passive scanning. ZAP supports several authentication mechanisms and session-verification strategies rather than requiring you to simply stuff a cookie into requests. www.zaproxy.org

Practical recommendation: use Playwright for deterministic authenticated crawling + ZAP Client Spider for security-oriented discovery/scanning. That gives you a real JS-capable browser without having to circumvent the staging bot protection, while keeping the audit tightly scoped.

Yes. For a staging site that intentionally blocks generic bots, I’d favor real-browser crawlers that can carry a test account/session rather than trying to disguise a conventional bot.

Good options

Playwright is probably the cleanest choice if your goal is specifically "crawl the authenticated staging site and audit what actually renders." It can authenticate once and save browser state, then reuse that state for subsequent contexts. Its saved state can include cookies, local storage, IndexedDB, and relevant WebAuthn state. playwright.dev

For a security-oriented audit, OWASP ZAP is particularly attractive. Its browser-based authentication is designed for modern applications, including authentication flows involving multiple forms and TOTP, and authenticated users can be associated with a ZAP context. www.zaproxy.org

One important current detail: ZAP now recommends its Client Spider for modern JavaScript applications rather than the older AJAX Spider, while still recommending the traditional spider alongside it because the two can discover different endpoints. github.com

For your "safe staging" requirement

I'd configure the crawler roughly like this:

1. Create a dedicated low-privilege audit account.
2. Authenticate through the real login flow once.
3. Persist the browser's auth state rather than embedding credentials in crawler requests.
4. Restrict crawling to your staging hostname/subtree.
5. Disable or intercept destructive actions (POST, PUT, DELETE, account changes, checkout, etc.) unless they're explicitly part of the audit.
6. Rate-limit concurrency.
7. Cap URL count/depth and prevent infinite calendar/filter/query-string combinations.
8. Capture rendered HTML, console errors, failed network requests, screenshots, and discovered URLs.
9. Keep the authentication-state file out of source control; Playwright explicitly warns that it can contain credentials capable of impersonating the test account. playwright.dev

If you're doing both accessibility/SEO/rendering auditing and security testing, a particularly effective setup is Playwright/Crawlee for deterministic rendered crawling + ZAP proxied through that browser for security analysis. That lets the browser handle whatever anti-bot/auth/JS behavior staging requires while ZAP observes the resulting traffic.

I would not recommend spoofing Googlebot or trying to evade your staging bot protection. If the protection is yours, the safer approach is to allowlist the crawler's staging identity/IP or give the audit account a narrowly scoped bypass.

Yes. For a staging audit, I’d favor browser-based crawlers that let you establish an authenticated session and then reuse it, rather than trying to defeat the bot protection itself.

Good options

Playwright is probably the cleanest starting point if the goal is specifically crawl/render/audit. It can authenticate once, save the browser's authenticated state, and launch subsequent browser contexts with that state. Its stored state can include cookies, local storage, IndexedDB, and relevant WebAuthn state. playwright.dev

For example, the architecture can be:

staging
   │
   ├── dedicated audit account
   │
   ▼
Playwright login
   │
   ├── save authenticated storageState
   │
   ▼
authenticated browser context
   │
   ├── crawl URLs
   ├── execute JavaScript
   ├── wait for network/DOM
   ├── collect links/API calls
   └── capture screenshots/accessibility/HTTP data

If you want security scanning too

OWASP ZAP is particularly attractive. Its current authentication support includes browser-based authentication and client-side scripts, and authenticated sessions can be maintained while using its browser-driven spiders. www.zaproxy.org

One important current detail: ZAP now recommends its Client Spider for modern JavaScript applications rather than the older Ajax Spider, while recommending that you combine it with the traditional spider for better endpoint coverage. github.com

So I'd use:

ZAP
 ├─ Traditional Spider → server-discoverable URLs
 ├─ Client Spider      → JS/client-side routes
 ├─ authenticated user → protected application
 └─ passive/security analysis

Making the staging bot block safe

Rather than configuring the crawler to impersonate or evade a production bot defense, make a staging-only exception:

• Give the crawler a dedicated account.
• Allowlist its fixed egress IP, if possible.
• Add a staging-only header/API key or mTLS identity.
• Restrict the exception to the staging hostname.
• Rate-limit the crawler.
• Disable destructive operations or use a read-only account.
• Keep authentication state in a secret store; Playwright explicitly warns that its saved state can contain credentials/cookies capable of impersonating the account. playwright.devgithub.com

If your bot protection is something like Cloudflare/Akamai/etc., I would not recommend trying to bypass the challenge programmatically. An explicit staging allowlist is both safer and considerably more reliable.

My default recommendation: Playwright + a staging-specific crawler identity for functional/SEO/accessibility rendering, or ZAP + Playwright/browser authentication if the audit also includes security testing.

Yes. For a staging site that blocks unauthenticated bots, you want a real-browser crawler with an authenticated session, rather than a traditional HTTP-only crawler.

Best options

• OWASP ZAP Client Spider — my first choice for a security audit. As of July 2026, ZAP recommends its Client Spider for modern JavaScript-heavy applications. It launches a real browser, executes JavaScript, follows client-side navigation, and can run as a configured authenticated user. ZAP ZAP

• Configure a staging-only context and test account.
• Use Browser-Based Authentication if login is a normal web flow.
• For complicated authentication, use ZAP's Client Script Authentication, which can replay a recorded browser login flow. ZAP ZAP
• Combine Client Spider with the traditional Spider; ZAP says they discover different sets of endpoints. ZAP
• Playwright — excellent if you want a crawler you control programmatically. It runs an actual Chromium/Firefox/WebKit browser and supports persisting authenticated browser state with storageState, so you can authenticate once and crawl subsequent pages as that user. Playwright

• Particularly good for SPAs, complex login flows, SSO, and workflows where you need explicit control over what gets clicked.
• You can put tight limits around URLs, depth, concurrency, and actions to make the audit safer.
• OWASP PTK / PTK Agent — worth considering if the goal is more than crawling. It operates in the browser context and is designed for authenticated applications and SPAs, including client-side security analysis; its automation layer supports Playwright, Puppeteer, Selenium, and Cypress. OWASP Foundation

A safe staging setup

I'd structure it like this:

1. Create a dedicated low-privilege audit account.
2. Give the crawler credentials only for staging; don't reuse production credentials.
3. Scope the crawler to the staging hostname and explicitly exclude production domains.
4. Authenticate using the browser itself rather than trying to fake cookies/headers.
5. Start with passive crawling and endpoint discovery.
6. Put limits on concurrency, crawl depth, request rate, and total requests.
7. Disable destructive actions where possible—logout, account deletion, purchases, emails, mutations, etc.
8. Only then enable active security testing if you specifically want DAST findings.

ZAP is particularly convenient here because it can maintain authentication state and re-authenticate when the session expires. ZAP Its passive scanning is considered safe because it analyzes proxied requests/responses without modifying them; active scanning, by contrast, sends attack requests and should only be used against targets you have authorization to test. GitHub

If your immediate goal is "crawl the whole authenticated JS site without getting blocked," I'd use Playwright or ZAP Client Spider. If the goal is "crawl + security audit," I'd start with ZAP Client Spider + Browser-Based/Client-Script Authentication.

Yes. For a staging site that deliberately blocks ordinary bots, I’d favor real-browser crawlers that authenticate as a dedicated test user rather than trying to defeat the bot protection.

Good options

1. OWASP ZAP + Client Spider — best for a security audit
   • ZAP’s Client Spider is currently its recommended crawler for modern JavaScript-heavy applications. It drives a browser, has DOM access, and supports ZAP’s authentication mechanisms. www.zaproxy.orgwww.zaproxy.org
   • Its Browser Based Authentication can actually perform the login flow in the browser, including multi-step forms and TOTP. www.zaproxy.orgplaywright.dev
   • You can put the staging host in a ZAP Context, define a dedicated user, and restrict the spider to that scope. ZAP can re-authenticate the user when necessary. www.zaproxy.org
   • For comprehensive coverage, run the traditional ZAP Spider in addition to Client Spider; ZAP specifically recommends combining them because they discover different things. www.zaproxy.orgwww.zaproxy.org

2. Playwright — best for a controlled authenticated crawl
   • Playwright runs an actual browser, so JavaScript executes normally.
   • You can log in once and save the authenticated browser state—cookies, local storage, IndexedDB, etc.—then reuse it for subsequent crawls. playwright.dev
   • This is particularly nice if your staging authentication involves SSO or complicated UI flows that generic HTTP crawlers struggle with.

3. Playwright → ZAP proxy — my preferred setup for security testing
   • Have Playwright handle the login and JavaScript navigation, while routing browser traffic through ZAP.
   • That gives you a realistic browser crawler plus ZAP's request/response history and passive security analysis.
   • Keep the authenticated account low-privilege and staging-only.

Safe staging configuration

I'd set it up roughly like this:

Dedicated staging account
        ↓
Playwright / ZAP browser authentication
        ↓
Real Chromium browser + JavaScript
        ↓
ZAP proxy
        ↓
Staging environment only

For safety, configure:

• Allowlist: only the staging hostname/subdomains.
• Dedicated account: no production access, no real customer data.
• Read-only where possible: prevent crawler clicks from deleting records, sending email, changing billing, etc.
• Rate limit: modest concurrency and request rate.
• Disable destructive URLs/actions: logout, delete, purchase, password changes, admin mutations, etc.
• Short-lived credentials/session state: Playwright explicitly warns that saved auth state can contain cookies/headers capable of impersonating the account. playwright.dev
• Keep auth files out of Git/CI artifacts.

One important distinction: don't weaken the staging bot protection globally just to accommodate the crawler. Instead, allow the scanner's browser/user/IP through a narrowly scoped staging rule if your bot/WAF layer permits it. That preserves the security boundary you're trying to test.

If your goal is specifically "crawl every authenticated JS route and then run a passive security audit", I'd choose ZAP Client Spider + Browser Based Authentication, or Playwright behind ZAP if the login flow is unusually complicated.

Yes. For a staging site that deliberately blocks generic bots, I’d use a real browser crawler with a dedicated staging account, rather than trying to disguise a bot.

Good options

My first choice: ZAP Client Spider + browser-based authentication. ZAP's current guidance recommends Client Spider for modern web apps; it launches a real browser, interacts with JavaScript-driven UI, and maintains a client-side map of discovered application paths. github.comwww.zaproxy.org

For a custom crawl, Playwright is particularly convenient. You can authenticate once, save the browser's authenticated state, and reuse it for subsequent contexts. The saved state can include cookies, local storage, and IndexedDB, which is useful for SPAs. playwright.dev

A safe staging setup

I'd structure it like this:

1. Create a dedicated low-privilege audit account.
   • Don't use a developer's personal session.
   • Give it only the permissions you actually need to audit.
   • If possible, create separate accounts for different roles.

2. Authenticate through the normal UI.
   • This handles SSO, MFA/TOTP, redirects, CSRF tokens, etc.
   • ZAP supports browser-based authentication and client scripts specifically for modern authentication flows. www.zaproxy.org

3. Persist the authenticated browser state.
   • With Playwright, use storageState.
   • With ZAP, configure the authentication/session context.
   • Treat the resulting cookies/tokens like credentials; Playwright explicitly warns that auth-state files can allow someone to impersonate the account. playwright.dev

4. Allowlist the crawler at the staging edge.
   • Prefer an IP/service-account allowlist or an explicit staging-only crawler exception.
   • Don't weaken bot protection globally or attempt to evade production defenses.

5. Start with passive crawling.
   • ZAP's spiders discover resources while passive scanning observes the traffic; this is considerably safer than immediately launching active attack rules. www.zaproxy.org
   • Then selectively enable active security testing against staging.

6. Put hard boundaries around the crawl.
   • Restrict the target hostname.
   • Exclude logout, delete, purchase, email-send, admin mutation, and other state-changing endpoints.
   • Cap concurrency/request rate.
   • Use a crawl depth/page limit.
   • Ideally reset the staging database between runs.

One particularly useful combination

For a modern SPA, I'd actually use both:

Playwright → authenticated browser exploration → ZAP proxy → ZAP Client Spider/passive analysis

That gives you a real JS-capable browser while ZAP observes the resulting HTTP traffic. ZAP also supports browser-based session handling for client-side authentication, including authenticated requests made through its Ajax/client spiders. www.zaproxy.orgwww.zaproxy.org

If you're already using ZAP, I'd choose Client Spider rather than AJAX Spider for a new setup; ZAP changed its recommendation to Client Spider in July 2026. github.com

ZAP authentication documentation
Playwright authentication documentation

Yes. For an authorized staging audit, the best options are crawlers that can reuse a real authenticated browser session and execute JavaScript rather than trying to defeat the bot protection.

Good options

Screaming Frog is probably the easiest if your goal is a conventional site audit. It supports form-based authentication through its built-in Chrome browser, can retain the resulting cookies, and has JavaScript rendering. It also supports custom HTTP headers/cookies. www.screamingfrog.co.uk

Playwright is the most flexible if the staging login is complicated—SSO, MFA-assisted test accounts, SPA authentication, etc. You can log in once, save the authenticated browser state, and reuse it for subsequent crawls. Keep that state file out of source control because it can contain credentials/cookies capable of impersonating the account. playwright.dev

OWASP ZAP is a particularly good choice if "audit" means security testing. Its client-script authentication can drive a browser login and its Client/AJAX Spider can subsequently crawl using the authenticated context. www.zaproxy.org

For your bot-blocking situation

I would not try to disguise the crawler as an unrelated bot. Instead, make the staging environment explicitly trust your audit traffic:

1. Create a dedicated, low-privilege staging account.
2. Allowlist the crawler's IP at the staging/WAF layer.
3. Give that account read-only permissions where possible.
4. Authenticate normally and retain the resulting session/cookies.
5. Enable JS rendering.
6. Start with a small URL/template set.
7. Throttle concurrency/rate—staging systems are often less resilient than production. Screaming Frog specifically recommends reducing crawl speed if staging starts returning errors/timeouts. www.screamingfrog.co.uk
8. Make sure the crawler cannot accidentally trigger destructive authenticated actions. This matters particularly with authenticated crawls because a crawler may follow links such as logout, delete, or state-changing endpoints. www.screamingfrog.co.ukwww.screamingfrog.co.uk

If the bot protection is something like Cloudflare/Akamai, an even cleaner architecture is to add a narrowly scoped WAF rule such as "staging + audit IP + dedicated auth account → permit" rather than weakening bot protection globally.

For a pure site/SEO audit, I'd choose Screaming Frog. For a custom crawl of a modern authenticated SPA, Playwright. For a security assessment, ZAP.

Yes. For a staging site that intentionally blocks generic bots, I’d use real-browser crawlers with an authenticated session, rather than trying to disguise a conventional crawler.

Good options

OWASP specifically lists ZAP and Burp among its web-testing tools, and recommends browser automation approaches for applications whose functionality depends heavily on JavaScript. owasp.org

My preferred setup

Playwright → authenticated browser context → crawl → feed traffic into ZAP/Burp.

Playwright is particularly convenient because you can authenticate once and persist the resulting cookies/local storage as storageState, then launch subsequent browser contexts already authenticated. playwright.dev

Conceptually:

staging
   │
   ▼
Playwright
   ├── authenticate with dedicated test account
   ├── load pages in real Chromium
   ├── execute JS / SPA navigation
   ├── discover links, routes, API calls
   └── capture authenticated traffic
             │
             ▼
       ZAP / Burp
       ├── passive analysis
       ├── authenticated crawling
       └── controlled active scanning

Making it safe

Since this is staging, I'd put a few guardrails around the crawler:

• Dedicated low-privilege test account, not your personal/admin account.
• Separate credentials per crawler worker if you run parallel crawls.
• Restrict the crawler to the staging hostname and explicitly allowed API domains.
• Disable or mock email, payments, webhooks, SMS, destructive operations, etc.
• Start with passive crawling, then enable active scanning deliberately.
• Rate-limit requests so your staging environment doesn't become its own DoS test.
• Keep authentication state out of Git. Playwright explicitly warns that persisted auth state can contain cookies/headers capable of impersonating the test account. playwright.dev
• If the staging bot block is yours, preferably create a crawler-specific allow rule based on something controlled (source IP, staging-only credential, mTLS, or a dedicated crawler header), rather than attempting to evade the bot defense.

For authorization auditing, it's also useful to create separate authenticated contexts for different roles and compare what each can access; OWASP specifically recommends testing horizontal and vertical authorization boundaries. playwright.devplaywright.devowasp.org

If the goal is primarily SEO/content completeness, I'd lean Playwright. If it's primarily security auditing, I'd use Playwright + ZAP/Burp, with Playwright handling the difficult login/JS portions and the security proxy doing the analysis.

Yes. For a staging site that intentionally blocks ordinary bots, I’d favor browser-based crawlers that can authenticate as a dedicated test user and execute JavaScript, rather than trying to disguise a bot.

Good options

For a staging audit, Playwright is probably the safest and most controllable choice. It can authenticate once and reuse the resulting browser state, including cookies/local storage, for subsequent pages. playwright.dev

Recommended setup

Use a dedicated staging account with minimal permissions:

1. Create a crawl-only test account.
2. Authenticate it through the normal login flow.
3. Save its browser storageState.
4. Launch isolated browser contexts using that state.
5. Crawl only your staging hostname.
6. Limit concurrency and request rate.
7. Disable destructive actions (POST, PUT, DELETE) or explicitly exclude those routes.
8. Capture rendered DOM, HTTP responses, console errors, failed resources, screenshots, and optionally accessibility data.
9. Keep the authentication-state file out of source control—the Playwright state can contain cookies and headers capable of impersonating the account. playwright.dev

For example, conceptually:

staging.example.com
       │
       ▼
  Playwright
       │
       ├── authenticate as staging-crawler
       │
       ├── save authenticated browser state
       │
       ├── visit URL
       │      └── execute JS
       │
       ├── extract links from rendered DOM
       │
       └── enqueue only same-origin URLs

This is preferable to weakening the staging bot protection globally. If the blocker is at a WAF/reverse-proxy layer, you can instead create a narrow allow rule for the crawler's fixed egress IP, while retaining authentication and rate limits.

One important distinction

If your goal is SEO/rendering auditing, Sitebulb or Screaming Frog can be considerably easier than building the crawler yourself. Sitebulb specifically documents crawling authentication-protected staging environments, including HTTP Basic/Digest/NTLM authentication. support.sitebulb.com Oncrawl likewise supports authenticated pre-production crawling. help.oncrawl.com

If your goal is security testing, I'd lean toward Playwright plus an intercepting proxy rather than an SEO crawler. OWASP's testing guidance explicitly recommends comparing authenticated and unauthenticated behavior and examining dynamically generated JavaScript responses. help.oncrawl.comowasp.org

My default choice: Playwright + dedicated low-privilege account + IP allowlisting at the staging perimeter + strict same-origin/rate limits. That gives you real JS execution and authentication without making the staging environment generally accessible to bots.

Yes. For a staging audit, I’d favor a real browser crawler with an explicit test account/session, rather than trying to disguise a bot as a normal browser.

Good options

Playwright is probably the cleanest foundation. It supports authenticating once and saving the resulting browser state—cookies, local storage, IndexedDB, etc.—then starting subsequent browser contexts already authenticated. playwright.dev

Playwright authentication documentation

For your staging setup

I'd configure the crawler roughly like this:

1. Create a dedicated staging audit account, preferably with read-only permissions.
2. Authenticate it normally through the login flow.
3. Save its browser storageState.
4. Launch isolated browser contexts using that state.
5. Crawl with Chromium enabled so client-side routing and JS-rendered content execute.
6. Restrict requests to the staging hostname and approved API domains.
7. Disable destructive interactions—POST/PUT/DELETE forms, account changes, purchases, etc.
8. Rate-limit the crawler and cap concurrency.
9. Capture HTTP responses, console errors, JS exceptions, screenshots, and rendered DOM.
10. Delete/expire the authentication state after the audit.

Playwright specifically warns that saved authentication state can contain cookies and headers capable of impersonating the account, so keep the state file out of source control and treat it like a credential. playwright.dev

For an actual security audit, ZAP plus a real browser session is particularly useful because you can combine authenticated crawling with the OWASP testing methodology. OWASP's WSTG covers authentication, authorization, session management, configuration, and client-side testing, including staging/UAT deployments. owasp.org

One important distinction: don't defeat the bot protection itself if it's there to protect staging. Instead, make an explicit staging exception—e.g. allowlist the crawler's IP, use a dedicated audit token/header, or configure the WAF/bot-control policy to permit your audit account. That gives you a much safer and more reproducible audit than spoofing fingerprints or trying to evade detection.

If you tell me whether you're using Cloudflare/Akamai/AWS WAF/etc. and whether the app uses cookie, OAuth/OIDC, or SSO authentication, I can recommend the exact crawler architecture.

Yes. For a staging site that blocks generic bots but permits an authenticated browser, I’d favor browser-based crawlers rather than trying to disguise a bot.

Good options

My first choice for a site audit is Playwright + ZAP together. Playwright can establish a legitimate test-user session and render the application; ZAP can passively observe the resulting traffic. ZAP specifically recommends its Client Spider for modern JavaScript-heavy applications and supports browser-based authentication. www.zaproxy.org

A safe staging setup

1. Create a dedicated staging test account with the minimum permissions needed.
2. Authenticate through the normal UI rather than bypassing the site's bot controls.
3. Save the browser's authenticated state and reuse it for crawling. Playwright supports storageState for cookies/local storage and related browser state. playwright.dev
4. Restrict the crawler to the staging hostname.
5. Exclude destructive endpoints such as logout, account deletion, password changes, checkout/payment, and mutation-heavy APIs.
6. Start with passive crawling, then explicitly opt into active security tests separately. ZAP notes that passive scanning doesn't modify requests/responses, whereas active scanning sends attack traffic and can put the target at risk. www.zaproxy.org
7. Rate-limit concurrency so your staging environment doesn't get hammered.

If the bot block is yours

The cleanest approach is actually to allowlist the crawler in staging—for example, a dedicated crawler identity/IP or a staging-only authentication mechanism—rather than attempting to make the crawler look like a human.

For a modern SPA, a particularly useful architecture is:

Playwright → authenticated browser → ZAP proxy → Client Spider/passive scanner

ZAP can also use browser-based authentication or a recorded client-side authentication script, which is useful when login involves MFA, several redirects, or client-side state. www.zaproxy.org

If you tell me whether you're auditing SEO/accessibility, broken links/content, or security, I can recommend the exact crawler configuration and safe crawl boundaries.

For a staging site that blocks bots, you usually want a crawler that supports authenticated sessions + real browser rendering, rather than trying to make a public SEO crawler behave like a normal visitor. Good options:

For a protected staging environment, the usual setup is:

1. Run the crawler inside the same network boundary
   • Put the crawler on a VPN, bastion host, CI runner, or internal machine.
   • Avoid opening staging publicly just for an audit.

2. Authenticate like a real user
   • Prefer:
     • saved browser session cookies
     • OAuth/SAML login flow automation
     • test-user credentials
     • injected auth headers (if your app supports them)
   • Avoid embedding production credentials.

3. Enable browser rendering
   • Turn on Chromium/headless browser mode.
   • Wait for:
     • network idle
     • API calls completing
     • lazy-loaded content
     • client-side routing
   • Capture rendered HTML/screenshots where possible. JS rendering is important because HTML-only crawls can miss content generated after load. www.semrush.com

4. Whitelist the crawler instead of weakening security
   • Add a staging-only allow rule:
     • source IP allowlist
     • special user-agent/token
     • service account
     • bypass rule limited to /audit/* or staging hostname
   • Keep bot protection enabled everywhere else.

For modern apps:

• React/Vue/Angular/Next.js with SSO: Playwright is often the most reliable.
• SEO/technical audit: Screaming Frog or Sitebulb with JS rendering.
• Enterprise scheduled audits: Semrush/Oncrawl/Lumar-style platforms.

A common safe pattern is: create a dedicated “audit user” → authenticate once → export cookies/session → run crawler from an internal runner → destroy the session afterward. This gives you realistic rendering without exposing staging to the public.

If you own/control the staging environment, the cleanest approach is not to defeat the bot protection, but to give your crawler an explicit authenticated/testing path. For a JS-heavy application, I’d use one of these:

My first choice: ZAP

OWASP ZAP authentication documentation

ZAP's Browser Based Authentication can actually drive a browser through the login flow, and its AJAX Spider can then use that authenticated browser context to crawl JS-generated content. It supports manually specified selectors when automatic login detection isn't sufficient, and can handle TOTP as well. www.zaproxy.org

For particularly complicated authentication—OAuth/SAML redirects, custom MFA, multi-step login—I'd record a Client Script/Zest authentication flow and have ZAP reuse it for the Client/AJAX Spider. www.zaproxy.org

If you primarily want a crawler: Playwright

Playwright authentication docs

Playwright is better when you want precise control over what gets visited:

1. Launch a real Chromium browser.
2. Log in using a dedicated staging account.
3. Save the authenticated browser state.
4. Start new contexts from that state.
5. Crawl links/routes and wait for JS/network activity.
6. Restrict navigation to your staging hostname.
7. Record requests, responses, console errors, screenshots, etc.

Playwright specifically supports persisting authenticated browser state so subsequent runs don't have to repeat the login. Its documentation also warns that the state file can contain cookies/headers capable of impersonating the account, so keep it out of source control. playwright.dev

Handling the bot block safely

Since this is staging, I'd configure the bot/WAF layer to recognize your scanner rather than trying to make the scanner look human:

• allowlist the scanner's fixed IP(s), if practical;
• create a staging-only rule that permits your test account;
• disable CAPTCHA/bot challenges for that account or staging hostname;
• give the crawler a dedicated low-privilege account;
• rate-limit the crawler rather than bypassing all protections;
• keep the staging hostname isolated from production credentials/data.

That gives you a much more reproducible audit and avoids building a bot-detection evasion mechanism.

A particularly good architecture is:
Playwright → authenticated JS exploration → ZAP proxy → passive/active security analysis.

ZAP can also handle client-side session management through browser authentication, which is useful when the authentication state lives in the browser rather than being a simple cookie/header. www.zaproxy.org

If you tell me whether your staging login is plain username/password, OAuth/OIDC, SAML, or MFA, I can suggest the safest concrete setup.

Yes. For a staging audit where you own/control the environment, I’d favor crawlers that can maintain an authenticated browser session and execute Chromium-based JavaScript.

Good options

Screaming Frog is probably the quickest fit. It supports form-based authentication through its built-in Chrome browser, can retain the resulting cookies, and has a JavaScript rendering mode for React/Vue/Angular-style sites. www.screamingfrog.co.uk

For a staging environment, I'd configure it this way

1. Use a dedicated staging account, not a real user's credentials.
2. Authenticate through the crawler's browser so it obtains the normal session cookies.
3. Enable JavaScript rendering.
4. If robots.txt blocks staging, configure the crawler to ignore it for this local audit rather than changing production behavior. Screaming Frog explicitly supports this workflow. www.screamingfrog.co.uk
5. If your bot protection is IP-based, allowlist the crawler's source IP rather than disabling the WAF globally.
6. Keep crawl concurrency low initially. Staging systems can be substantially less resilient than production; Screaming Frog recommends reducing crawl speed if you encounter timeouts or server errors. www.screamingfrog.co.uk
7. Prevent destructive actions. An authenticated JS crawler can encounter links/buttons that create records, log users out, submit forms, etc. This is particularly important with form-based authentication. www.seerinteractive.com

If you want the safest/most controllable approach

I'd use Playwright in a dedicated staging container:

staging
   │
   ├── dedicated audit account
   ├── IP allowlist
   ├── read-only/test permissions
   └── Playwright/Chromium
          │
          ├── login
          ├── persist session
          ├── visit URLs
          ├── wait for JS
          ├── capture rendered DOM
          └── record status/errors/screenshots

That gives you explicit control over which URLs are visited, which actions are permitted, authentication state, concurrency, timeouts, and network access. For a security-conscious staging audit, I'd choose this over trying to make a generic bot-protection bypass work.

One useful distinction: don't try to defeat the bot protection itself. If the staging WAF is blocking your crawler, have the staging environment recognize your crawler via an IP allowlist, dedicated authentication, or an audit-only environment. That gives you the same audit coverage without weakening a security control.

If your staging environment blocks bots, the safest approach is to use an authenticated, JavaScript-capable crawler that can run with explicit permission and a controlled identity (rather than trying to bypass the block). Common options:

A typical safe staging setup looks like:

1. Create a dedicated audit account
   • Read-only permissions if possible.
   • Do not use a personal admin account.
   • Add an identifiable user agent, e.g. StagingAuditBot/1.0.

2. Allowlist the crawler
   • Permit only your crawler IP(s) or service account.
   • Keep the block in place for unknown bots.

3. Authenticate normally
   • Log in through the staging login flow.
   • Store a session cookie or use the crawler’s built-in auth mechanism.
   • Avoid hardcoding production credentials.

4. Enable browser rendering
   • Use Chromium-based rendering so SPAs, lazy-loaded content, client-side routes, and JS-generated links are discovered.

5. Throttle requests
   • Use conservative crawl rates.
   • Avoid accidentally stressing staging services or dependent APIs.

For a developer-controlled audit, a Playwright flow is often the most flexible:

Open staging URL
→ authenticate with test account
→ save browser storage state
→ crawl URLs using authenticated context
→ capture HTML after JS execution
→ collect errors, links, metadata, accessibility/security findings

For a non-code SEO/technical audit, Screaming Frog or Sitebulb are usually the quickest path. For complex SSO, dashboards, or app-like sites, Playwright is usually the better fit.

If your staging block is specifically a robots.txt rule, WAF/bot protection, basic auth, SSO, or IP allowlist, the best setup differs slightly.

If your staging environment blocks bots, you’ll want an auditing crawler that can authenticate like a real browser and execute JavaScript rather than a simple HTTP fetcher. Common options:

For a staging environment, the safest pattern is usually:

1. Create a dedicated staging audit account
   • Read-only permissions if possible.
   • Avoid using a personal/admin account.
   • Add an identifying user-agent, e.g. StagingAuditBot/1.0.

2. Authenticate before crawling
   • Log in through the UI and export cookies/session storage, or
   • Automate login and reuse a saved browser context.

Example with Playwright:

   import { chromium } from "playwright";

   const browser = await chromium.launch();
   const context = await browser.newContext();

   const page = await context.newPage();

   await page.goto("https://staging.example.com/login");
   await page.fill("#email", "audit@example.com");
   await page.fill("#password", process.env.PASSWORD);
   await page.click("button[type=submit]");

   await page.waitForURL("**/dashboard");

   // Save authenticated state for the crawler
   await context.storageState({ path: "staging-auth.json" });

   await browser.close();

Then crawl with the saved session:

   const context = await browser.newContext({
     storageState: "staging-auth.json"
   });

3. Enable JS rendering
   • Make sure the crawler uses a Chromium engine.
   • Verify it waits for network idle or key DOM elements.
   • Watch for SPA routes that require client-side navigation.

4. Handle bot protection explicitly

Instead of trying to bypass protections, configure staging to allow your audit:

• Allowlist the crawler IP (if static).
• Add an authenticated staging-only bypass.
• Allow a specific audit user-agent plus valid session.
• Disable CAPTCHA/WAF challenges only in staging.
• Keep production protections unchanged.

5. Capture the things that matter

For a JS-heavy app, audit:

• Rendered HTML vs raw HTML differences
• Missing metadata after hydration
• Broken internal links
• Canonicals/robots directives
• HTTP status codes
• Console errors
• Network failures
• Accessibility issues

A common setup is:

• Screaming Frog/Sitebulb for a one-off SEO crawl
• Playwright for repeatable authenticated audits in CI
• BrowserStack when you also need browser/device coverage

If your staging blocker is a specific layer (Cloudflare, basic auth, SSO, VPN/IP allowlist, robots.txt, or a custom bot detector), the best approach changes a bit.

If your staging environment blocks bots, the safest approach is usually to use a crawler that can authenticate like a real user, execute JavaScript in a browser context, and identify itself clearly so you are not accidentally bypassing controls meant to protect the environment.

Common options:

For a staging audit, a typical safe setup looks like:

1. Create a dedicated audit account
   • Least privilege (read-only if possible)
   • No production credentials
   • No access to sensitive data

2. Allowlist the crawler identity
   • Prefer IP allowlisting or a staging-only rule
   • Avoid simply disabling bot protection globally
   • Keep the bypass limited to the audit window

3. Use authenticated browser sessions
   • Log in manually once and export cookies, or
   • Automate login with Playwright/Puppeteer

4. Enable JS rendering
   • Use Chromium rendering
   • Set appropriate wait conditions (network idle, DOM selectors, API completion)
   • Watch for SPA routes that normal crawlers may miss

5. Keep the crawl controlled
   • Limit concurrency
   • Set a clear user agent such as an internal audit crawler identifier
   • Avoid crawling admin paths, private APIs, or test data endpoints

Example Playwright pattern:

import { chromium } from "playwright";

const browser = await chromium.launch();
const context = await browser.newContext();

const page = await context.newPage();

await page.goto("https://staging.example.com/login");

await page.fill("#username", process.env.USERNAME);
await page.fill("#password", process.env.PASSWORD);
await page.click("button[type=submit]");

await page.waitForLoadState("networkidle");

await page.goto("https://staging.example.com");

console.log(await page.title());

await browser.close();

For a staging site protected by tools like WAFs, bot managers, or SSO, the cleanest pattern is usually:

• temporary allowlist + authenticated crawler account + JS-enabled crawler

rather than trying to make the crawler look like an untrusted bot that slips through defenses.

If you tell me what is blocking bots (Cloudflare, Akamai, AWS WAF, robots.txt, basic auth, SSO, custom middleware, etc.) and what audit you need (SEO, accessibility, security, performance), I can narrow down the best setup.

For a staging site that intentionally blocks bots, you’ll want a crawler that supports authenticated sessions and real browser rendering (JavaScript execution) rather than a simple HTTP crawler. The safest approach is usually to use a dedicated audit tool with staging-only credentials, allowlisting, and logging.

Options to consider:

For a staging environment, a common setup is:

1. Create a dedicated audit user
   • Read-only account
   • No production data access
   • Expiring credentials if possible

2. Authenticate once, reuse the session
   • Log in manually in a browser
   • Export cookies/storage state
   • Give the crawler the authenticated session

3. Allowlist the crawler
   • Prefer IP allowlisting or a staging-only bypass rule
   • Avoid globally disabling bot protection

4. Use a real browser crawler
   • Enable JavaScript rendering
   • Wait for network idle or app-specific selectors
   • Audit the rendered DOM, not just server HTML

5. Keep the audit contained
   • Limit crawl paths
   • Set concurrency/rate limits
   • Avoid triggering destructive actions (forms, account flows, APIs)

For a modern app behind SSO/WAF (Cloudflare Access, Okta, Auth0, etc.), I’d usually lean toward Playwright because it can perform the same browser flow a human uses, save authenticated state, and then crawl pages safely. For a marketing site or CMS staging environment, Screaming Frog with JS rendering is often simpler.

If you share what’s blocking the crawler (Cloudflare/WAF, HTTP auth, SSO, app login, robots.txt, IP restriction, etc.) and what stack the site uses (Next.js, React, WordPress, etc.), I can suggest the cleanest setup.

If your goal is to audit a staging site that requires authentication and blocks bots, the main capabilities to look for are:

• Authenticated crawling (cookies, headers, or login flow)
• Full JavaScript rendering (headless browser)
• Respect for your authentication/session (rather than bypassing security)
• Ability to throttle requests so you don't overload staging
• Exportable crawl data for SEO or QA

Some common options are:

Authentication approaches

Most enterprise crawlers support one or more of:

• Session cookies exported from your browser
• Basic authentication
• Bearer/API tokens via headers
• Login form automation
• Single Sign-On (depending on the tool)
• Client certificates (less common)

For example, with a Playwright script you can:

1. Log into staging normally.
2. Save the authenticated browser state.
3. Crawl pages using that saved session.
4. Render all JavaScript exactly like a real user.

This is often the most reliable approach for SPAs built with frameworks like React, Vue, Angular, or Next.js.

If your staging blocks bots

The recommended approach is to allowlist your crawler rather than trying to evade detection. Options include:

• Allowlisting the crawler's IP address.
• Providing a dedicated QA account.
• Allowlisting a specific user agent if appropriate.
• Running the crawler from inside your VPN or internal network.
• Using exported authentication cookies.

These approaches let you audit the site without weakening your staging security or attempting to circumvent protections.

For modern JavaScript applications

If you're auditing a React, Next.js, Nuxt, Angular, or Vue application, make sure the crawler can:

• Wait for network idle
• Execute deferred JavaScript
• Scroll or trigger lazy loading if needed
• Respect client-side routing
• Capture rendered HTML, not just the initial response

Typical workflow

A common setup for technical SEO audits is:

1. Log into staging.
2. Export the authenticated session (or use browser profile integration).
3. Configure the crawler to render JavaScript with Chromium.
4. Limit crawl speed (for example, 1–3 concurrent requests).
5. Restrict the crawl to your staging domain.
6. Export findings (status codes, titles, canonicals, structured data, internal links, rendered HTML, etc.).

This gives you an audit that's close to what a logged-in user experiences while staying within the access controls you've configured.

If your staging site uses a specific authentication method (such as SSO with Okta, Azure AD, GitHub, or Google, or a login form with MFA), I can suggest the most suitable crawler configuration for that setup.

If your staging environment is protected and you want to audit it without exposing it publicly, there are several legitimate approaches depending on how your authentication works and what kind of audit you're running (SEO, accessibility, performance, broken links, etc.).

Here's a comparison of common crawler options that support authentication and JavaScript rendering.

1. Screaming Frog (most common)

Excellent when your staging site requires:

• username/password login
• SSO
• cookies
• HTTP Basic Auth
• custom headers
• JavaScript

You can:

• enable JavaScript rendering
• log in manually using the embedded Chromium browser
• reuse authenticated cookies for the crawl
• inject authorization headers
• crawl behind VPN

This is the standard SEO workflow for staging environments.

2. Sitebulb

Very similar capabilities:

• renders JavaScript
• supports login flows
• handles cookies
• visual crawl reports

Often easier to configure than custom scripts.

3. Playwright (best for complex auth)

If your staging environment uses:

• Okta
• Azure AD
• Auth0
• OAuth
• MFA
• custom login flows

Playwright is usually the easiest solution.

Example workflow:

1. Launch Chromium
2. Login normally
3. Save storage state
4. Crawl pages while authenticated

const browser = await chromium.launch();

const page = await browser.newPage();

await page.goto("https://staging.example.com");

await page.fill("#email", "...");

await page.fill("#password", "...");

await page.click("button[type=submit]");

await page.context().storageState({
  path: "auth.json"
});

Subsequent crawls simply load:

storageState: "auth.json"

This is very reliable for modern authentication systems.

4. Puppeteer

Works similarly.

Useful if you're already using Puppeteer in your tooling.

5. Lighthouse

For performance audits:

lighthouse https://staging.example.com \
  --preset=desktop

You can also:

• attach cookies
• launch Chrome already authenticated
• connect to an existing browser session

6. Headless Chrome + existing profile

Sometimes the easiest solution.

Launch Chrome with your normal profile:

chrome --remote-debugging-port=9222

Then connect Playwright or Lighthouse to that browser.

Everything stays authenticated.

If the staging environment blocks bots

Common protections include:

• Cloudflare
• Akamai
• Imperva
• bot detection middleware
• IP allowlists

The recommended approach is not to try to bypass those protections. Instead, configure the staging environment to trust your audit tooling. Options include:

• allowlisting the crawler's IP address
• creating a dedicated test account
• exempting a specific authenticated user or role from bot checks
• using HTTP Basic Auth specifically for staging
• adding a custom request header (for example, X-Staging-Audit) that your staging infrastructure recognizes
• running the crawler from within your corporate network or VPN if access controls are network-based

This keeps the audit aligned with your security controls while avoiding false positives from bot mitigation.

For SEO audits specifically

A practical setup is:

• Crawler: Screaming Frog
• Rendering: Chromium JavaScript rendering
• Authentication: Manual login or imported cookies
• Environment: VPN or IP allowlist
• Reports: Canonicals, hreflang, metadata, redirects, structured data, internal links, orphan pages, and rendered HTML

If your site is built with React, Next.js, Angular, Vue, or another JavaScript-heavy framework, enabling full JavaScript rendering is important to ensure the crawler evaluates the rendered DOM rather than just the initial HTML.

If you can share what authentication mechanism your staging site uses (HTTP Basic Auth, SSO via Okta/Azure AD/Auth0, Cloudflare Access, VPN, cookie-based login, etc.) and what you're trying to audit (SEO, accessibility, performance, or all of the above), I can recommend the simplest setup for that specific environment.

If you need to audit a staging site that both requires authentication and renders content with JavaScript, the main distinction is between:

• Browser-based crawlers that can log in and execute JS (best for modern SPAs and authenticated apps).
• Headless browser automation that you script yourself for maximum control.
• SEO audit tools that support authenticated crawling.

Here are the common options:

1. Screaming Frog (often the easiest)

For most staging SEO audits:

• Render pages using Chromium.
• Authenticate via:
  • login form
  • Basic Auth
  • imported cookies
  • custom headers
• Respect robots if desired (or ignore for staging).
• Export links, titles, canonicals, structured data, etc.

This works well for React, Vue, Angular, Next.js, Nuxt, and similar frameworks.

Typical workflow:

1. Log into staging in the embedded browser.
2. Save session/cookies.
3. Start crawl from the authenticated state.
4. Enable JS rendering.

2. Sitebulb

Very similar capabilities:

• JS rendering
• authenticated crawling
• cookie/session support
• visual reports

Many teams prefer it for easier reporting.

3. Playwright (most flexible)

If your staging environment has:

• SSO
• MFA
• custom login
• bot protection
• unusual navigation

Playwright is usually the most reliable.

Example flow:

launch browser

go to login page

fill username/password

submit

wait for dashboard

crawl internal links

render each page

extract SEO data

Because you're driving a real browser, JS behaves almost exactly like a user session.

4. Puppeteer

Similar to Playwright but Chrome-focused.

Useful if your existing tooling already uses Puppeteer.

5. Cookie-based authentication

Many crawlers let you:

• log in once
• export cookies
• import cookies into the crawler

This avoids repeatedly automating login.

6. Basic Authentication

If staging uses HTTP Basic Auth:

https://username:password@example.com

or configure credentials in the crawler.

Most SEO tools support this directly.

7. Bearer tokens / custom headers

If your application authenticates using headers like:

Authorization: Bearer <token>

many browser automation tools let you inject these headers for every request.

8. Cloud/browser automation

If you need distributed or scheduled crawls:

• Browserless (Playwright/Puppeteer)
• Playwright running in CI (GitHub Actions, Azure DevOps, etc.)

This works well for nightly staging audits.

If your staging blocks bots

Many staging environments use protections such as Cloudflare, Akamai, or custom WAF rules that challenge automated traffic.

Rather than trying to evade those protections, the safest approach is to allowlist your auditing tool or environment. Common approaches include:

• allowlisting the crawler's IP addresses
• creating a dedicated test account
• issuing a long-lived session cookie for audits
• bypassing bot checks only for authenticated staging users
• exposing an internal-only audit endpoint or hostname

These approaches let you test the site as intended without weakening production defenses.

Recommended setup

For most engineering teams:

• One-off SEO audit: Screaming Frog with Chromium rendering and an authenticated session.
• Automated regression audits: Playwright script that logs in, crawls the site, and checks SEO and rendering on each deployment.
• Large enterprise staging: Playwright plus an IP allowlist or dedicated staging authentication path.

This combination gives you accurate rendering of JavaScript-heavy pages while staying within your staging environment's intended access controls.

If your goal is to audit a staging site that requires authentication and renders content client-side, there are several crawler approaches that support login flows and JavaScript execution. The best choice depends on whether you're focused on SEO, accessibility, broken links, performance, or general QA.

Here's a comparison:

1. Screaming Frog (often the easiest)

It supports:

• Login forms
• Session cookies
• HTTP authentication
• Custom request headers
• JavaScript rendering using Chromium
• SPA frameworks like React, Vue, Angular, and Next.js

Typical workflow:

1. Enable JavaScript rendering.
2. Log in via browser or configure authentication.
3. Import cookies or use the built-in login.
4. Crawl the staging site.

This is usually enough for SEO validation of staging environments.

2. Playwright

For maximum flexibility, Playwright is often the strongest option.

It can:

• Log into SSO providers
• Handle MFA (when feasible)
• Execute arbitrary JavaScript
• Wait for hydration
• Scroll lazy-loaded pages
• Capture rendered HTML
• Take screenshots
• Export accessibility trees

Example flow:

launch browser
go to login page
submit credentials
wait for dashboard
save authenticated session
crawl URLs
wait for network idle
extract rendered DOM

You can also reuse the saved authenticated state for future crawls.

3. Crawlee

If you need to crawl hundreds or thousands of authenticated pages, Crawlee builds on Playwright or Puppeteer and adds:

• Automatic queue management
• Session persistence
• Retry logic
• Robots handling (configurable)
• Parallel crawling

This is a good fit for enterprise-scale staging audits.

4. Lighthouse CI

If your concern is rendering quality rather than crawl completeness:

• Log in with Playwright or Puppeteer.
• Run Lighthouse against authenticated pages.
• Collect:
  • Performance
  • Accessibility
  • Best Practices
  • SEO

This works well in CI/CD.

Authentication methods that work well

Depending on your staging setup, crawlers can authenticate using:

• Username/password login forms
• Session cookies
• Bearer tokens
• API keys (for API endpoints)
• HTTP Basic Authentication
• Custom request headers
• Client certificates (less common)
• SSO flows (via browser automation)

Session cookies are often the simplest approach if you can export them from a logged-in browser and supply them to the crawler.

Handling bot protection

If your staging environment intentionally blocks automated traffic (for example, via a web application firewall or bot mitigation service), the recommended approach is to configure an allowlist or a bypass specifically for your authorized audit tools rather than trying to evade those protections. Common approaches include:

• Allowlisting the crawler's IP addresses.
• Creating a staging-only bypass cookie or header.
• Exempting authenticated users or service accounts.
• Using HTTP Basic Authentication in front of staging.
• Restricting access via VPN or identity-aware access, then authenticating normally.

These approaches let you perform legitimate testing while preserving the bot protections for everyone else.

Recommended setups

• SEO audit: Screaming Frog with JavaScript rendering and authenticated cookies.
• Large authenticated crawl: Crawlee with Playwright.
• Performance testing: Playwright + Lighthouse CI.
• Custom QA or regression testing: Playwright with a saved authenticated session.

If you can share what protects your staging environment (for example, Cloudflare, Akamai, Imperva, AWS WAF, or another solution) and what kind of audit you want to run (SEO, accessibility, performance, broken links, etc.), I can recommend the most suitable authentication and crawling configuration.