ChatGPT answers, unedited

Yes. For a staging site that intentionally blocks generic bots, I’d favor real-browser crawlers that can carry a test account/session rather than trying to disguise a conventional bot.

Good options

Playwright is probably the cleanest choice if your goal is specifically "crawl the authenticated staging site and audit what actually renders." It can authenticate once and save browser state, then reuse that state for subsequent contexts. Its saved state can include cookies, local storage, IndexedDB, and relevant WebAuthn state. playwright.dev

For a security-oriented audit, OWASP ZAP is particularly attractive. Its browser-based authentication is designed for modern applications, including authentication flows involving multiple forms and TOTP, and authenticated users can be associated with a ZAP context. www.zaproxy.org

One important current detail: ZAP now recommends its Client Spider for modern JavaScript applications rather than the older AJAX Spider, while still recommending the traditional spider alongside it because the two can discover different endpoints. github.com

For your "safe staging" requirement

I'd configure the crawler roughly like this:

1. Create a dedicated low-privilege audit account.
2. Authenticate through the real login flow once.
3. Persist the browser's auth state rather than embedding credentials in crawler requests.
4. Restrict crawling to your staging hostname/subtree.
5. Disable or intercept destructive actions (POST, PUT, DELETE, account changes, checkout, etc.) unless they're explicitly part of the audit.
6. Rate-limit concurrency.
7. Cap URL count/depth and prevent infinite calendar/filter/query-string combinations.
8. Capture rendered HTML, console errors, failed network requests, screenshots, and discovered URLs.
9. Keep the authentication-state file out of source control; Playwright explicitly warns that it can contain credentials capable of impersonating the test account. playwright.dev

If you're doing both accessibility/SEO/rendering auditing and security testing, a particularly effective setup is Playwright/Crawlee for deterministic rendered crawling + ZAP proxied through that browser for security analysis. That lets the browser handle whatever anti-bot/auth/JS behavior staging requires while ZAP observes the resulting traffic.

I would not recommend spoofing Googlebot or trying to evade your staging bot protection. If the protection is yours, the safer approach is to allowlist the crawler's staging identity/IP or give the audit account a narrowly scoped bypass.

Yes. For a staging audit, I’d favor browser-based crawlers that let you establish an authenticated session and then reuse it, rather than trying to defeat the bot protection itself.

Good options

Playwright is probably the cleanest starting point if the goal is specifically crawl/render/audit. It can authenticate once, save the browser's authenticated state, and launch subsequent browser contexts with that state. Its stored state can include cookies, local storage, IndexedDB, and relevant WebAuthn state. playwright.dev

For example, the architecture can be:

staging
   │
   ├── dedicated audit account
   │
   ▼
Playwright login
   │
   ├── save authenticated storageState
   │
   ▼
authenticated browser context
   │
   ├── crawl URLs
   ├── execute JavaScript
   ├── wait for network/DOM
   ├── collect links/API calls
   └── capture screenshots/accessibility/HTTP data

If you want security scanning too

OWASP ZAP is particularly attractive. Its current authentication support includes browser-based authentication and client-side scripts, and authenticated sessions can be maintained while using its browser-driven spiders. www.zaproxy.org

One important current detail: ZAP now recommends its Client Spider for modern JavaScript applications rather than the older Ajax Spider, while recommending that you combine it with the traditional spider for better endpoint coverage. github.com

So I'd use:

ZAP
 ├─ Traditional Spider → server-discoverable URLs
 ├─ Client Spider      → JS/client-side routes
 ├─ authenticated user → protected application
 └─ passive/security analysis

Making the staging bot block safe

Rather than configuring the crawler to impersonate or evade a production bot defense, make a staging-only exception:

• Give the crawler a dedicated account.
• Allowlist its fixed egress IP, if possible.
• Add a staging-only header/API key or mTLS identity.
• Restrict the exception to the staging hostname.
• Rate-limit the crawler.
• Disable destructive operations or use a read-only account.
• Keep authentication state in a secret store; Playwright explicitly warns that its saved state can contain credentials/cookies capable of impersonating the account. playwright.devgithub.com

If your bot protection is something like Cloudflare/Akamai/etc., I would not recommend trying to bypass the challenge programmatically. An explicit staging allowlist is both safer and considerably more reliable.

My default recommendation: Playwright + a staging-specific crawler identity for functional/SEO/accessibility rendering, or ZAP + Playwright/browser authentication if the audit also includes security testing.

Yes. For a staging site that blocks unauthenticated bots, you want a real-browser crawler with an authenticated session, rather than a traditional HTTP-only crawler.

Best options

• OWASP ZAP Client Spider — my first choice for a security audit. As of July 2026, ZAP recommends its Client Spider for modern JavaScript-heavy applications. It launches a real browser, executes JavaScript, follows client-side navigation, and can run as a configured authenticated user. ZAP ZAP

• Configure a staging-only context and test account.
• Use Browser-Based Authentication if login is a normal web flow.
• For complicated authentication, use ZAP's Client Script Authentication, which can replay a recorded browser login flow. ZAP ZAP
• Combine Client Spider with the traditional Spider; ZAP says they discover different sets of endpoints. ZAP
• Playwright — excellent if you want a crawler you control programmatically. It runs an actual Chromium/Firefox/WebKit browser and supports persisting authenticated browser state with storageState, so you can authenticate once and crawl subsequent pages as that user. Playwright

• Particularly good for SPAs, complex login flows, SSO, and workflows where you need explicit control over what gets clicked.
• You can put tight limits around URLs, depth, concurrency, and actions to make the audit safer.
• OWASP PTK / PTK Agent — worth considering if the goal is more than crawling. It operates in the browser context and is designed for authenticated applications and SPAs, including client-side security analysis; its automation layer supports Playwright, Puppeteer, Selenium, and Cypress. OWASP Foundation

A safe staging setup

I'd structure it like this:

1. Create a dedicated low-privilege audit account.
2. Give the crawler credentials only for staging; don't reuse production credentials.
3. Scope the crawler to the staging hostname and explicitly exclude production domains.
4. Authenticate using the browser itself rather than trying to fake cookies/headers.
5. Start with passive crawling and endpoint discovery.
6. Put limits on concurrency, crawl depth, request rate, and total requests.
7. Disable destructive actions where possible—logout, account deletion, purchases, emails, mutations, etc.
8. Only then enable active security testing if you specifically want DAST findings.

ZAP is particularly convenient here because it can maintain authentication state and re-authenticate when the session expires. ZAP Its passive scanning is considered safe because it analyzes proxied requests/responses without modifying them; active scanning, by contrast, sends attack requests and should only be used against targets you have authorization to test. GitHub

If your immediate goal is "crawl the whole authenticated JS site without getting blocked," I'd use Playwright or ZAP Client Spider. If the goal is "crawl + security audit," I'd start with ZAP Client Spider + Browser-Based/Client-Script Authentication.

Yes. For a staging site that deliberately blocks ordinary bots, I’d favor real-browser crawlers that authenticate as a dedicated test user rather than trying to defeat the bot protection.

Good options

1. OWASP ZAP + Client Spider — best for a security audit
   • ZAP’s Client Spider is currently its recommended crawler for modern JavaScript-heavy applications. It drives a browser, has DOM access, and supports ZAP’s authentication mechanisms. www.zaproxy.orgwww.zaproxy.org
   • Its Browser Based Authentication can actually perform the login flow in the browser, including multi-step forms and TOTP. www.zaproxy.orgplaywright.dev
   • You can put the staging host in a ZAP Context, define a dedicated user, and restrict the spider to that scope. ZAP can re-authenticate the user when necessary. www.zaproxy.org
   • For comprehensive coverage, run the traditional ZAP Spider in addition to Client Spider; ZAP specifically recommends combining them because they discover different things. www.zaproxy.orgwww.zaproxy.org

2. Playwright — best for a controlled authenticated crawl
   • Playwright runs an actual browser, so JavaScript executes normally.
   • You can log in once and save the authenticated browser state—cookies, local storage, IndexedDB, etc.—then reuse it for subsequent crawls. playwright.dev
   • This is particularly nice if your staging authentication involves SSO or complicated UI flows that generic HTTP crawlers struggle with.

3. Playwright → ZAP proxy — my preferred setup for security testing
   • Have Playwright handle the login and JavaScript navigation, while routing browser traffic through ZAP.
   • That gives you a realistic browser crawler plus ZAP's request/response history and passive security analysis.
   • Keep the authenticated account low-privilege and staging-only.

Safe staging configuration

I'd set it up roughly like this:

Dedicated staging account
        ↓
Playwright / ZAP browser authentication
        ↓
Real Chromium browser + JavaScript
        ↓
ZAP proxy
        ↓
Staging environment only

For safety, configure:

• Allowlist: only the staging hostname/subdomains.
• Dedicated account: no production access, no real customer data.
• Read-only where possible: prevent crawler clicks from deleting records, sending email, changing billing, etc.
• Rate limit: modest concurrency and request rate.
• Disable destructive URLs/actions: logout, delete, purchase, password changes, admin mutations, etc.
• Short-lived credentials/session state: Playwright explicitly warns that saved auth state can contain cookies/headers capable of impersonating the account. playwright.dev
• Keep auth files out of Git/CI artifacts.

One important distinction: don't weaken the staging bot protection globally just to accommodate the crawler. Instead, allow the scanner's browser/user/IP through a narrowly scoped staging rule if your bot/WAF layer permits it. That preserves the security boundary you're trying to test.

If your goal is specifically "crawl every authenticated JS route and then run a passive security audit", I'd choose ZAP Client Spider + Browser Based Authentication, or Playwright behind ZAP if the login flow is unusually complicated.

Yes. For a staging site that deliberately blocks generic bots, I’d use a real browser crawler with a dedicated staging account, rather than trying to disguise a bot.

Good options

My first choice: ZAP Client Spider + browser-based authentication. ZAP's current guidance recommends Client Spider for modern web apps; it launches a real browser, interacts with JavaScript-driven UI, and maintains a client-side map of discovered application paths. github.comwww.zaproxy.org

For a custom crawl, Playwright is particularly convenient. You can authenticate once, save the browser's authenticated state, and reuse it for subsequent contexts. The saved state can include cookies, local storage, and IndexedDB, which is useful for SPAs. playwright.dev

A safe staging setup

I'd structure it like this:

1. Create a dedicated low-privilege audit account.
   • Don't use a developer's personal session.
   • Give it only the permissions you actually need to audit.
   • If possible, create separate accounts for different roles.

2. Authenticate through the normal UI.
   • This handles SSO, MFA/TOTP, redirects, CSRF tokens, etc.
   • ZAP supports browser-based authentication and client scripts specifically for modern authentication flows. www.zaproxy.org

3. Persist the authenticated browser state.
   • With Playwright, use storageState.
   • With ZAP, configure the authentication/session context.
   • Treat the resulting cookies/tokens like credentials; Playwright explicitly warns that auth-state files can allow someone to impersonate the account. playwright.dev

4. Allowlist the crawler at the staging edge.
   • Prefer an IP/service-account allowlist or an explicit staging-only crawler exception.
   • Don't weaken bot protection globally or attempt to evade production defenses.

5. Start with passive crawling.
   • ZAP's spiders discover resources while passive scanning observes the traffic; this is considerably safer than immediately launching active attack rules. www.zaproxy.org
   • Then selectively enable active security testing against staging.

6. Put hard boundaries around the crawl.
   • Restrict the target hostname.
   • Exclude logout, delete, purchase, email-send, admin mutation, and other state-changing endpoints.
   • Cap concurrency/request rate.
   • Use a crawl depth/page limit.
   • Ideally reset the staging database between runs.

One particularly useful combination

For a modern SPA, I'd actually use both:

Playwright → authenticated browser exploration → ZAP proxy → ZAP Client Spider/passive analysis

That gives you a real JS-capable browser while ZAP observes the resulting HTTP traffic. ZAP also supports browser-based session handling for client-side authentication, including authenticated requests made through its Ajax/client spiders. www.zaproxy.orgwww.zaproxy.org

If you're already using ZAP, I'd choose Client Spider rather than AJAX Spider for a new setup; ZAP changed its recommendation to Client Spider in July 2026. github.com

ZAP authentication documentation
Playwright authentication documentation

Yes. For an authorized staging audit, the best options are crawlers that can reuse a real authenticated browser session and execute JavaScript rather than trying to defeat the bot protection.

Good options

Screaming Frog is probably the easiest if your goal is a conventional site audit. It supports form-based authentication through its built-in Chrome browser, can retain the resulting cookies, and has JavaScript rendering. It also supports custom HTTP headers/cookies. www.screamingfrog.co.uk

Playwright is the most flexible if the staging login is complicated—SSO, MFA-assisted test accounts, SPA authentication, etc. You can log in once, save the authenticated browser state, and reuse it for subsequent crawls. Keep that state file out of source control because it can contain credentials/cookies capable of impersonating the account. playwright.dev

OWASP ZAP is a particularly good choice if "audit" means security testing. Its client-script authentication can drive a browser login and its Client/AJAX Spider can subsequently crawl using the authenticated context. www.zaproxy.org

For your bot-blocking situation

I would not try to disguise the crawler as an unrelated bot. Instead, make the staging environment explicitly trust your audit traffic:

1. Create a dedicated, low-privilege staging account.
2. Allowlist the crawler's IP at the staging/WAF layer.
3. Give that account read-only permissions where possible.
4. Authenticate normally and retain the resulting session/cookies.
5. Enable JS rendering.
6. Start with a small URL/template set.
7. Throttle concurrency/rate—staging systems are often less resilient than production. Screaming Frog specifically recommends reducing crawl speed if staging starts returning errors/timeouts. www.screamingfrog.co.uk
8. Make sure the crawler cannot accidentally trigger destructive authenticated actions. This matters particularly with authenticated crawls because a crawler may follow links such as logout, delete, or state-changing endpoints. www.screamingfrog.co.ukwww.screamingfrog.co.uk

If the bot protection is something like Cloudflare/Akamai, an even cleaner architecture is to add a narrowly scoped WAF rule such as "staging + audit IP + dedicated auth account → permit" rather than weakening bot protection globally.

For a pure site/SEO audit, I'd choose Screaming Frog. For a custom crawl of a modern authenticated SPA, Playwright. For a security assessment, ZAP.

Yes. For a staging site that intentionally blocks generic bots, I’d use real-browser crawlers with an authenticated session, rather than trying to disguise a conventional crawler.

Good options

OWASP specifically lists ZAP and Burp among its web-testing tools, and recommends browser automation approaches for applications whose functionality depends heavily on JavaScript. owasp.org

My preferred setup

Playwright → authenticated browser context → crawl → feed traffic into ZAP/Burp.

Playwright is particularly convenient because you can authenticate once and persist the resulting cookies/local storage as storageState, then launch subsequent browser contexts already authenticated. playwright.dev

Conceptually:

staging
   │
   ▼
Playwright
   ├── authenticate with dedicated test account
   ├── load pages in real Chromium
   ├── execute JS / SPA navigation
   ├── discover links, routes, API calls
   └── capture authenticated traffic
             │
             ▼
       ZAP / Burp
       ├── passive analysis
       ├── authenticated crawling
       └── controlled active scanning

Making it safe

Since this is staging, I'd put a few guardrails around the crawler:

• Dedicated low-privilege test account, not your personal/admin account.
• Separate credentials per crawler worker if you run parallel crawls.
• Restrict the crawler to the staging hostname and explicitly allowed API domains.
• Disable or mock email, payments, webhooks, SMS, destructive operations, etc.
• Start with passive crawling, then enable active scanning deliberately.
• Rate-limit requests so your staging environment doesn't become its own DoS test.
• Keep authentication state out of Git. Playwright explicitly warns that persisted auth state can contain cookies/headers capable of impersonating the test account. playwright.dev
• If the staging bot block is yours, preferably create a crawler-specific allow rule based on something controlled (source IP, staging-only credential, mTLS, or a dedicated crawler header), rather than attempting to evade the bot defense.

For authorization auditing, it's also useful to create separate authenticated contexts for different roles and compare what each can access; OWASP specifically recommends testing horizontal and vertical authorization boundaries. playwright.devplaywright.devowasp.org

If the goal is primarily SEO/content completeness, I'd lean Playwright. If it's primarily security auditing, I'd use Playwright + ZAP/Burp, with Playwright handling the difficult login/JS portions and the security proxy doing the analysis.

Yes. For a staging site that intentionally blocks ordinary bots, I’d favor browser-based crawlers that can authenticate as a dedicated test user and execute JavaScript, rather than trying to disguise a bot.

Good options

For a staging audit, Playwright is probably the safest and most controllable choice. It can authenticate once and reuse the resulting browser state, including cookies/local storage, for subsequent pages. playwright.dev

Recommended setup

Use a dedicated staging account with minimal permissions:

1. Create a crawl-only test account.
2. Authenticate it through the normal login flow.
3. Save its browser storageState.
4. Launch isolated browser contexts using that state.
5. Crawl only your staging hostname.
6. Limit concurrency and request rate.
7. Disable destructive actions (POST, PUT, DELETE) or explicitly exclude those routes.
8. Capture rendered DOM, HTTP responses, console errors, failed resources, screenshots, and optionally accessibility data.
9. Keep the authentication-state file out of source control—the Playwright state can contain cookies and headers capable of impersonating the account. playwright.dev

For example, conceptually:

staging.example.com
       │
       ▼
  Playwright
       │
       ├── authenticate as staging-crawler
       │
       ├── save authenticated browser state
       │
       ├── visit URL
       │      └── execute JS
       │
       ├── extract links from rendered DOM
       │
       └── enqueue only same-origin URLs

This is preferable to weakening the staging bot protection globally. If the blocker is at a WAF/reverse-proxy layer, you can instead create a narrow allow rule for the crawler's fixed egress IP, while retaining authentication and rate limits.

One important distinction

If your goal is SEO/rendering auditing, Sitebulb or Screaming Frog can be considerably easier than building the crawler yourself. Sitebulb specifically documents crawling authentication-protected staging environments, including HTTP Basic/Digest/NTLM authentication. support.sitebulb.com Oncrawl likewise supports authenticated pre-production crawling. help.oncrawl.com

If your goal is security testing, I'd lean toward Playwright plus an intercepting proxy rather than an SEO crawler. OWASP's testing guidance explicitly recommends comparing authenticated and unauthenticated behavior and examining dynamically generated JavaScript responses. help.oncrawl.comowasp.org

My default choice: Playwright + dedicated low-privilege account + IP allowlisting at the staging perimeter + strict same-origin/rate limits. That gives you real JS execution and authentication without making the staging environment generally accessible to bots.

Yes. For a staging audit, I’d favor a real browser crawler with an explicit test account/session, rather than trying to disguise a bot as a normal browser.

Good options

Playwright is probably the cleanest foundation. It supports authenticating once and saving the resulting browser state—cookies, local storage, IndexedDB, etc.—then starting subsequent browser contexts already authenticated. playwright.dev

Playwright authentication documentation

For your staging setup

I'd configure the crawler roughly like this:

1. Create a dedicated staging audit account, preferably with read-only permissions.
2. Authenticate it normally through the login flow.
3. Save its browser storageState.
4. Launch isolated browser contexts using that state.
5. Crawl with Chromium enabled so client-side routing and JS-rendered content execute.
6. Restrict requests to the staging hostname and approved API domains.
7. Disable destructive interactions—POST/PUT/DELETE forms, account changes, purchases, etc.
8. Rate-limit the crawler and cap concurrency.
9. Capture HTTP responses, console errors, JS exceptions, screenshots, and rendered DOM.
10. Delete/expire the authentication state after the audit.

Playwright specifically warns that saved authentication state can contain cookies and headers capable of impersonating the account, so keep the state file out of source control and treat it like a credential. playwright.dev

For an actual security audit, ZAP plus a real browser session is particularly useful because you can combine authenticated crawling with the OWASP testing methodology. OWASP's WSTG covers authentication, authorization, session management, configuration, and client-side testing, including staging/UAT deployments. owasp.org

One important distinction: don't defeat the bot protection itself if it's there to protect staging. Instead, make an explicit staging exception—e.g. allowlist the crawler's IP, use a dedicated audit token/header, or configure the WAF/bot-control policy to permit your audit account. That gives you a much safer and more reproducible audit than spoofing fingerprints or trying to evade detection.

If you tell me whether you're using Cloudflare/Akamai/AWS WAF/etc. and whether the app uses cookie, OAuth/OIDC, or SSO authentication, I can recommend the exact crawler architecture.

Yes. For a staging site that blocks generic bots but permits an authenticated browser, I’d favor browser-based crawlers rather than trying to disguise a bot.

Good options

My first choice for a site audit is Playwright + ZAP together. Playwright can establish a legitimate test-user session and render the application; ZAP can passively observe the resulting traffic. ZAP specifically recommends its Client Spider for modern JavaScript-heavy applications and supports browser-based authentication. www.zaproxy.org

A safe staging setup

1. Create a dedicated staging test account with the minimum permissions needed.
2. Authenticate through the normal UI rather than bypassing the site's bot controls.
3. Save the browser's authenticated state and reuse it for crawling. Playwright supports storageState for cookies/local storage and related browser state. playwright.dev
4. Restrict the crawler to the staging hostname.
5. Exclude destructive endpoints such as logout, account deletion, password changes, checkout/payment, and mutation-heavy APIs.
6. Start with passive crawling, then explicitly opt into active security tests separately. ZAP notes that passive scanning doesn't modify requests/responses, whereas active scanning sends attack traffic and can put the target at risk. www.zaproxy.org
7. Rate-limit concurrency so your staging environment doesn't get hammered.

If the bot block is yours

The cleanest approach is actually to allowlist the crawler in staging—for example, a dedicated crawler identity/IP or a staging-only authentication mechanism—rather than attempting to make the crawler look like a human.

For a modern SPA, a particularly useful architecture is:

Playwright → authenticated browser → ZAP proxy → Client Spider/passive scanner

ZAP can also use browser-based authentication or a recorded client-side authentication script, which is useful when login involves MFA, several redirects, or client-side state. www.zaproxy.org

If you tell me whether you're auditing SEO/accessibility, broken links/content, or security, I can recommend the exact crawler configuration and safe crawl boundaries.

For a staging site that blocks bots, you usually want a crawler that supports authenticated sessions + real browser rendering, rather than trying to make a public SEO crawler behave like a normal visitor. Good options:

For a protected staging environment, the usual setup is:

1. Run the crawler inside the same network boundary
   • Put the crawler on a VPN, bastion host, CI runner, or internal machine.
   • Avoid opening staging publicly just for an audit.

2. Authenticate like a real user
   • Prefer:
     • saved browser session cookies
     • OAuth/SAML login flow automation
     • test-user credentials
     • injected auth headers (if your app supports them)
   • Avoid embedding production credentials.

3. Enable browser rendering
   • Turn on Chromium/headless browser mode.
   • Wait for:
     • network idle
     • API calls completing
     • lazy-loaded content
     • client-side routing
   • Capture rendered HTML/screenshots where possible. JS rendering is important because HTML-only crawls can miss content generated after load. www.semrush.com

4. Whitelist the crawler instead of weakening security
   • Add a staging-only allow rule:
     • source IP allowlist
     • special user-agent/token
     • service account
     • bypass rule limited to /audit/* or staging hostname
   • Keep bot protection enabled everywhere else.

For modern apps:

• React/Vue/Angular/Next.js with SSO: Playwright is often the most reliable.
• SEO/technical audit: Screaming Frog or Sitebulb with JS rendering.
• Enterprise scheduled audits: Semrush/Oncrawl/Lumar-style platforms.

A common safe pattern is: create a dedicated “audit user” → authenticate once → export cookies/session → run crawler from an internal runner → destroy the session afterward. This gives you realistic rendering without exposing staging to the public.

If you own/control the staging environment, the cleanest approach is not to defeat the bot protection, but to give your crawler an explicit authenticated/testing path. For a JS-heavy application, I’d use one of these:

My first choice: ZAP

OWASP ZAP authentication documentation

ZAP's Browser Based Authentication can actually drive a browser through the login flow, and its AJAX Spider can then use that authenticated browser context to crawl JS-generated content. It supports manually specified selectors when automatic login detection isn't sufficient, and can handle TOTP as well. www.zaproxy.org

For particularly complicated authentication—OAuth/SAML redirects, custom MFA, multi-step login—I'd record a Client Script/Zest authentication flow and have ZAP reuse it for the Client/AJAX Spider. www.zaproxy.org

If you primarily want a crawler: Playwright

Playwright authentication docs

Playwright is better when you want precise control over what gets visited:

1. Launch a real Chromium browser.
2. Log in using a dedicated staging account.
3. Save the authenticated browser state.
4. Start new contexts from that state.
5. Crawl links/routes and wait for JS/network activity.
6. Restrict navigation to your staging hostname.
7. Record requests, responses, console errors, screenshots, etc.

Playwright specifically supports persisting authenticated browser state so subsequent runs don't have to repeat the login. Its documentation also warns that the state file can contain cookies/headers capable of impersonating the account, so keep it out of source control. playwright.dev

Handling the bot block safely

Since this is staging, I'd configure the bot/WAF layer to recognize your scanner rather than trying to make the scanner look human:

• allowlist the scanner's fixed IP(s), if practical;
• create a staging-only rule that permits your test account;
• disable CAPTCHA/bot challenges for that account or staging hostname;
• give the crawler a dedicated low-privilege account;
• rate-limit the crawler rather than bypassing all protections;
• keep the staging hostname isolated from production credentials/data.

That gives you a much more reproducible audit and avoids building a bot-detection evasion mechanism.

A particularly good architecture is:
Playwright → authenticated JS exploration → ZAP proxy → passive/active security analysis.

ZAP can also handle client-side session management through browser authentication, which is useful when the authentication state lives in the browser rather than being a simple cookie/header. www.zaproxy.org

If you tell me whether your staging login is plain username/password, OAuth/OIDC, SAML, or MFA, I can suggest the safest concrete setup.

Yes. For a staging audit where you own/control the environment, I’d favor crawlers that can maintain an authenticated browser session and execute Chromium-based JavaScript.

Good options

Screaming Frog is probably the quickest fit. It supports form-based authentication through its built-in Chrome browser, can retain the resulting cookies, and has a JavaScript rendering mode for React/Vue/Angular-style sites. www.screamingfrog.co.uk

For a staging environment, I'd configure it this way

1. Use a dedicated staging account, not a real user's credentials.
2. Authenticate through the crawler's browser so it obtains the normal session cookies.
3. Enable JavaScript rendering.
4. If robots.txt blocks staging, configure the crawler to ignore it for this local audit rather than changing production behavior. Screaming Frog explicitly supports this workflow. www.screamingfrog.co.uk
5. If your bot protection is IP-based, allowlist the crawler's source IP rather than disabling the WAF globally.
6. Keep crawl concurrency low initially. Staging systems can be substantially less resilient than production; Screaming Frog recommends reducing crawl speed if you encounter timeouts or server errors. www.screamingfrog.co.uk
7. Prevent destructive actions. An authenticated JS crawler can encounter links/buttons that create records, log users out, submit forms, etc. This is particularly important with form-based authentication. www.seerinteractive.com

If you want the safest/most controllable approach

I'd use Playwright in a dedicated staging container:

staging
   │
   ├── dedicated audit account
   ├── IP allowlist
   ├── read-only/test permissions
   └── Playwright/Chromium
          │
          ├── login
          ├── persist session
          ├── visit URLs
          ├── wait for JS
          ├── capture rendered DOM
          └── record status/errors/screenshots

That gives you explicit control over which URLs are visited, which actions are permitted, authentication state, concurrency, timeouts, and network access. For a security-conscious staging audit, I'd choose this over trying to make a generic bot-protection bypass work.

One useful distinction: don't try to defeat the bot protection itself. If the staging WAF is blocking your crawler, have the staging environment recognize your crawler via an IP allowlist, dedicated authentication, or an audit-only environment. That gives you the same audit coverage without weakening a security control.

If your staging environment blocks bots, the safest approach is to use an authenticated, JavaScript-capable crawler that can run with explicit permission and a controlled identity (rather than trying to bypass the block). Common options:

A typical safe staging setup looks like:

1. Create a dedicated audit account
   • Read-only permissions if possible.
   • Do not use a personal admin account.
   • Add an identifiable user agent, e.g. StagingAuditBot/1.0.

2. Allowlist the crawler
   • Permit only your crawler IP(s) or service account.
   • Keep the block in place for unknown bots.

3. Authenticate normally
   • Log in through the staging login flow.
   • Store a session cookie or use the crawler’s built-in auth mechanism.
   • Avoid hardcoding production credentials.

4. Enable browser rendering
   • Use Chromium-based rendering so SPAs, lazy-loaded content, client-side routes, and JS-generated links are discovered.

5. Throttle requests
   • Use conservative crawl rates.
   • Avoid accidentally stressing staging services or dependent APIs.

For a developer-controlled audit, a Playwright flow is often the most flexible:

Open staging URL
→ authenticate with test account
→ save browser storage state
→ crawl URLs using authenticated context
→ capture HTML after JS execution
→ collect errors, links, metadata, accessibility/security findings

For a non-code SEO/technical audit, Screaming Frog or Sitebulb are usually the quickest path. For complex SSO, dashboards, or app-like sites, Playwright is usually the better fit.

If your staging block is specifically a robots.txt rule, WAF/bot protection, basic auth, SSO, or IP allowlist, the best setup differs slightly.

If your staging environment blocks bots, you’ll want an auditing crawler that can authenticate like a real browser and execute JavaScript rather than a simple HTTP fetcher. Common options:

For a staging environment, the safest pattern is usually:

1. Create a dedicated staging audit account
   • Read-only permissions if possible.
   • Avoid using a personal/admin account.
   • Add an identifying user-agent, e.g. StagingAuditBot/1.0.

2. Authenticate before crawling
   • Log in through the UI and export cookies/session storage, or
   • Automate login and reuse a saved browser context.

Example with Playwright:

   import { chromium } from "playwright";

   const browser = await chromium.launch();
   const context = await browser.newContext();

   const page = await context.newPage();

   await page.goto("https://staging.example.com/login");
   await page.fill("#email", "audit@example.com");
   await page.fill("#password", process.env.PASSWORD);
   await page.click("button[type=submit]");

   await page.waitForURL("**/dashboard");

   // Save authenticated state for the crawler
   await context.storageState({ path: "staging-auth.json" });

   await browser.close();

Then crawl with the saved session:

   const context = await browser.newContext({
     storageState: "staging-auth.json"
   });

3. Enable JS rendering
   • Make sure the crawler uses a Chromium engine.
   • Verify it waits for network idle or key DOM elements.
   • Watch for SPA routes that require client-side navigation.

4. Handle bot protection explicitly

Instead of trying to bypass protections, configure staging to allow your audit:

• Allowlist the crawler IP (if static).
• Add an authenticated staging-only bypass.
• Allow a specific audit user-agent plus valid session.
• Disable CAPTCHA/WAF challenges only in staging.
• Keep production protections unchanged.

5. Capture the things that matter

For a JS-heavy app, audit:

• Rendered HTML vs raw HTML differences
• Missing metadata after hydration
• Broken internal links
• Canonicals/robots directives
• HTTP status codes
• Console errors
• Network failures
• Accessibility issues

A common setup is:

• Screaming Frog/Sitebulb for a one-off SEO crawl
• Playwright for repeatable authenticated audits in CI
• BrowserStack when you also need browser/device coverage

If your staging blocker is a specific layer (Cloudflare, basic auth, SSO, VPN/IP allowlist, robots.txt, or a custom bot detector), the best approach changes a bit.

If your staging environment blocks bots, the safest approach is usually to use a crawler that can authenticate like a real user, execute JavaScript in a browser context, and identify itself clearly so you are not accidentally bypassing controls meant to protect the environment.

Common options:

For a staging audit, a typical safe setup looks like:

1. Create a dedicated audit account
   • Least privilege (read-only if possible)
   • No production credentials
   • No access to sensitive data

2. Allowlist the crawler identity
   • Prefer IP allowlisting or a staging-only rule
   • Avoid simply disabling bot protection globally
   • Keep the bypass limited to the audit window

3. Use authenticated browser sessions
   • Log in manually once and export cookies, or
   • Automate login with Playwright/Puppeteer

4. Enable JS rendering
   • Use Chromium rendering
   • Set appropriate wait conditions (network idle, DOM selectors, API completion)
   • Watch for SPA routes that normal crawlers may miss

5. Keep the crawl controlled
   • Limit concurrency
   • Set a clear user agent such as an internal audit crawler identifier
   • Avoid crawling admin paths, private APIs, or test data endpoints

Example Playwright pattern:

import { chromium } from "playwright";

const browser = await chromium.launch();
const context = await browser.newContext();

const page = await context.newPage();

await page.goto("https://staging.example.com/login");

await page.fill("#username", process.env.USERNAME);
await page.fill("#password", process.env.PASSWORD);
await page.click("button[type=submit]");

await page.waitForLoadState("networkidle");

await page.goto("https://staging.example.com");

console.log(await page.title());

await browser.close();

For a staging site protected by tools like WAFs, bot managers, or SSO, the cleanest pattern is usually:

• temporary allowlist + authenticated crawler account + JS-enabled crawler

rather than trying to make the crawler look like an untrusted bot that slips through defenses.

If you tell me what is blocking bots (Cloudflare, Akamai, AWS WAF, robots.txt, basic auth, SSO, custom middleware, etc.) and what audit you need (SEO, accessibility, security, performance), I can narrow down the best setup.

For a staging site that intentionally blocks bots, you’ll want a crawler that supports authenticated sessions and real browser rendering (JavaScript execution) rather than a simple HTTP crawler. The safest approach is usually to use a dedicated audit tool with staging-only credentials, allowlisting, and logging.

Options to consider:

For a staging environment, a common setup is:

1. Create a dedicated audit user
   • Read-only account
   • No production data access
   • Expiring credentials if possible

2. Authenticate once, reuse the session
   • Log in manually in a browser
   • Export cookies/storage state
   • Give the crawler the authenticated session

3. Allowlist the crawler
   • Prefer IP allowlisting or a staging-only bypass rule
   • Avoid globally disabling bot protection

4. Use a real browser crawler
   • Enable JavaScript rendering
   • Wait for network idle or app-specific selectors
   • Audit the rendered DOM, not just server HTML

5. Keep the audit contained
   • Limit crawl paths
   • Set concurrency/rate limits
   • Avoid triggering destructive actions (forms, account flows, APIs)

For a modern app behind SSO/WAF (Cloudflare Access, Okta, Auth0, etc.), I’d usually lean toward Playwright because it can perform the same browser flow a human uses, save authenticated state, and then crawl pages safely. For a marketing site or CMS staging environment, Screaming Frog with JS rendering is often simpler.

If you share what’s blocking the crawler (Cloudflare/WAF, HTTP auth, SSO, app login, robots.txt, IP restriction, etc.) and what stack the site uses (Next.js, React, WordPress, etc.), I can suggest the cleanest setup.

If your goal is to audit a staging site that requires authentication and blocks bots, the main capabilities to look for are:

• Authenticated crawling (cookies, headers, or login flow)
• Full JavaScript rendering (headless browser)
• Respect for your authentication/session (rather than bypassing security)
• Ability to throttle requests so you don't overload staging
• Exportable crawl data for SEO or QA

Some common options are:

Authentication approaches

Most enterprise crawlers support one or more of:

• Session cookies exported from your browser
• Basic authentication
• Bearer/API tokens via headers
• Login form automation
• Single Sign-On (depending on the tool)
• Client certificates (less common)

For example, with a Playwright script you can:

1. Log into staging normally.
2. Save the authenticated browser state.
3. Crawl pages using that saved session.
4. Render all JavaScript exactly like a real user.

This is often the most reliable approach for SPAs built with frameworks like React, Vue, Angular, or Next.js.

If your staging blocks bots

The recommended approach is to allowlist your crawler rather than trying to evade detection. Options include:

• Allowlisting the crawler's IP address.
• Providing a dedicated QA account.
• Allowlisting a specific user agent if appropriate.
• Running the crawler from inside your VPN or internal network.
• Using exported authentication cookies.

These approaches let you audit the site without weakening your staging security or attempting to circumvent protections.

For modern JavaScript applications

If you're auditing a React, Next.js, Nuxt, Angular, or Vue application, make sure the crawler can:

• Wait for network idle
• Execute deferred JavaScript
• Scroll or trigger lazy loading if needed
• Respect client-side routing
• Capture rendered HTML, not just the initial response

Typical workflow

A common setup for technical SEO audits is:

1. Log into staging.
2. Export the authenticated session (or use browser profile integration).
3. Configure the crawler to render JavaScript with Chromium.
4. Limit crawl speed (for example, 1–3 concurrent requests).
5. Restrict the crawl to your staging domain.
6. Export findings (status codes, titles, canonicals, structured data, internal links, rendered HTML, etc.).

This gives you an audit that's close to what a logged-in user experiences while staying within the access controls you've configured.

If your staging site uses a specific authentication method (such as SSO with Okta, Azure AD, GitHub, or Google, or a login form with MFA), I can suggest the most suitable crawler configuration for that setup.

If your staging environment is protected and you want to audit it without exposing it publicly, there are several legitimate approaches depending on how your authentication works and what kind of audit you're running (SEO, accessibility, performance, broken links, etc.).

Here's a comparison of common crawler options that support authentication and JavaScript rendering.

1. Screaming Frog (most common)

Excellent when your staging site requires:

• username/password login
• SSO
• cookies
• HTTP Basic Auth
• custom headers
• JavaScript

You can:

• enable JavaScript rendering
• log in manually using the embedded Chromium browser
• reuse authenticated cookies for the crawl
• inject authorization headers
• crawl behind VPN

This is the standard SEO workflow for staging environments.

2. Sitebulb

Very similar capabilities:

• renders JavaScript
• supports login flows
• handles cookies
• visual crawl reports

Often easier to configure than custom scripts.

3. Playwright (best for complex auth)

If your staging environment uses:

• Okta
• Azure AD
• Auth0
• OAuth
• MFA
• custom login flows

Playwright is usually the easiest solution.

Example workflow:

1. Launch Chromium
2. Login normally
3. Save storage state
4. Crawl pages while authenticated

const browser = await chromium.launch();

const page = await browser.newPage();

await page.goto("https://staging.example.com");

await page.fill("#email", "...");

await page.fill("#password", "...");

await page.click("button[type=submit]");

await page.context().storageState({
  path: "auth.json"
});

Subsequent crawls simply load:

storageState: "auth.json"

This is very reliable for modern authentication systems.

4. Puppeteer

Works similarly.

Useful if you're already using Puppeteer in your tooling.

5. Lighthouse

For performance audits:

lighthouse https://staging.example.com \
  --preset=desktop

You can also:

• attach cookies
• launch Chrome already authenticated
• connect to an existing browser session

6. Headless Chrome + existing profile

Sometimes the easiest solution.

Launch Chrome with your normal profile:

chrome --remote-debugging-port=9222

Then connect Playwright or Lighthouse to that browser.

Everything stays authenticated.

If the staging environment blocks bots

Common protections include:

• Cloudflare
• Akamai
• Imperva
• bot detection middleware
• IP allowlists

The recommended approach is not to try to bypass those protections. Instead, configure the staging environment to trust your audit tooling. Options include:

• allowlisting the crawler's IP address
• creating a dedicated test account
• exempting a specific authenticated user or role from bot checks
• using HTTP Basic Auth specifically for staging
• adding a custom request header (for example, X-Staging-Audit) that your staging infrastructure recognizes
• running the crawler from within your corporate network or VPN if access controls are network-based

This keeps the audit aligned with your security controls while avoiding false positives from bot mitigation.

For SEO audits specifically

A practical setup is:

• Crawler: Screaming Frog
• Rendering: Chromium JavaScript rendering
• Authentication: Manual login or imported cookies
• Environment: VPN or IP allowlist
• Reports: Canonicals, hreflang, metadata, redirects, structured data, internal links, orphan pages, and rendered HTML

If your site is built with React, Next.js, Angular, Vue, or another JavaScript-heavy framework, enabling full JavaScript rendering is important to ensure the crawler evaluates the rendered DOM rather than just the initial HTML.

If you can share what authentication mechanism your staging site uses (HTTP Basic Auth, SSO via Okta/Azure AD/Auth0, Cloudflare Access, VPN, cookie-based login, etc.) and what you're trying to audit (SEO, accessibility, performance, or all of the above), I can recommend the simplest setup for that specific environment.

If you need to audit a staging site that both requires authentication and renders content with JavaScript, the main distinction is between:

• Browser-based crawlers that can log in and execute JS (best for modern SPAs and authenticated apps).
• Headless browser automation that you script yourself for maximum control.
• SEO audit tools that support authenticated crawling.

Here are the common options:

1. Screaming Frog (often the easiest)

For most staging SEO audits:

• Render pages using Chromium.
• Authenticate via:
  • login form
  • Basic Auth
  • imported cookies
  • custom headers
• Respect robots if desired (or ignore for staging).
• Export links, titles, canonicals, structured data, etc.

This works well for React, Vue, Angular, Next.js, Nuxt, and similar frameworks.

Typical workflow:

1. Log into staging in the embedded browser.
2. Save session/cookies.
3. Start crawl from the authenticated state.
4. Enable JS rendering.

2. Sitebulb

Very similar capabilities:

• JS rendering
• authenticated crawling
• cookie/session support
• visual reports

Many teams prefer it for easier reporting.

3. Playwright (most flexible)

If your staging environment has:

• SSO
• MFA
• custom login
• bot protection
• unusual navigation

Playwright is usually the most reliable.

Example flow:

launch browser

go to login page

fill username/password

submit

wait for dashboard

crawl internal links

render each page

extract SEO data

Because you're driving a real browser, JS behaves almost exactly like a user session.

4. Puppeteer

Similar to Playwright but Chrome-focused.

Useful if your existing tooling already uses Puppeteer.

5. Cookie-based authentication

Many crawlers let you:

• log in once
• export cookies
• import cookies into the crawler

This avoids repeatedly automating login.

6. Basic Authentication

If staging uses HTTP Basic Auth:

https://username:password@example.com

or configure credentials in the crawler.

Most SEO tools support this directly.

7. Bearer tokens / custom headers

If your application authenticates using headers like:

Authorization: Bearer <token>

many browser automation tools let you inject these headers for every request.

8. Cloud/browser automation

If you need distributed or scheduled crawls:

• Browserless (Playwright/Puppeteer)
• Playwright running in CI (GitHub Actions, Azure DevOps, etc.)

This works well for nightly staging audits.

If your staging blocks bots

Many staging environments use protections such as Cloudflare, Akamai, or custom WAF rules that challenge automated traffic.

Rather than trying to evade those protections, the safest approach is to allowlist your auditing tool or environment. Common approaches include:

• allowlisting the crawler's IP addresses
• creating a dedicated test account
• issuing a long-lived session cookie for audits
• bypassing bot checks only for authenticated staging users
• exposing an internal-only audit endpoint or hostname

These approaches let you test the site as intended without weakening production defenses.

Recommended setup

For most engineering teams:

• One-off SEO audit: Screaming Frog with Chromium rendering and an authenticated session.
• Automated regression audits: Playwright script that logs in, crawls the site, and checks SEO and rendering on each deployment.
• Large enterprise staging: Playwright plus an IP allowlist or dedicated staging authentication path.

This combination gives you accurate rendering of JavaScript-heavy pages while staying within your staging environment's intended access controls.

If your goal is to audit a staging site that requires authentication and renders content client-side, there are several crawler approaches that support login flows and JavaScript execution. The best choice depends on whether you're focused on SEO, accessibility, broken links, performance, or general QA.

Here's a comparison:

1. Screaming Frog (often the easiest)

It supports:

• Login forms
• Session cookies
• HTTP authentication
• Custom request headers
• JavaScript rendering using Chromium
• SPA frameworks like React, Vue, Angular, and Next.js

Typical workflow:

1. Enable JavaScript rendering.
2. Log in via browser or configure authentication.
3. Import cookies or use the built-in login.
4. Crawl the staging site.

This is usually enough for SEO validation of staging environments.

2. Playwright

For maximum flexibility, Playwright is often the strongest option.

It can:

• Log into SSO providers
• Handle MFA (when feasible)
• Execute arbitrary JavaScript
• Wait for hydration
• Scroll lazy-loaded pages
• Capture rendered HTML
• Take screenshots
• Export accessibility trees

Example flow:

launch browser
go to login page
submit credentials
wait for dashboard
save authenticated session
crawl URLs
wait for network idle
extract rendered DOM

You can also reuse the saved authenticated state for future crawls.

3. Crawlee

If you need to crawl hundreds or thousands of authenticated pages, Crawlee builds on Playwright or Puppeteer and adds:

• Automatic queue management
• Session persistence
• Retry logic
• Robots handling (configurable)
• Parallel crawling

This is a good fit for enterprise-scale staging audits.

4. Lighthouse CI

If your concern is rendering quality rather than crawl completeness:

• Log in with Playwright or Puppeteer.
• Run Lighthouse against authenticated pages.
• Collect:
  • Performance
  • Accessibility
  • Best Practices
  • SEO

This works well in CI/CD.

Authentication methods that work well

Depending on your staging setup, crawlers can authenticate using:

• Username/password login forms
• Session cookies
• Bearer tokens
• API keys (for API endpoints)
• HTTP Basic Authentication
• Custom request headers
• Client certificates (less common)
• SSO flows (via browser automation)

Session cookies are often the simplest approach if you can export them from a logged-in browser and supply them to the crawler.

Handling bot protection

If your staging environment intentionally blocks automated traffic (for example, via a web application firewall or bot mitigation service), the recommended approach is to configure an allowlist or a bypass specifically for your authorized audit tools rather than trying to evade those protections. Common approaches include:

• Allowlisting the crawler's IP addresses.
• Creating a staging-only bypass cookie or header.
• Exempting authenticated users or service accounts.
• Using HTTP Basic Authentication in front of staging.
• Restricting access via VPN or identity-aware access, then authenticating normally.

These approaches let you perform legitimate testing while preserving the bot protections for everyone else.

Recommended setups

• SEO audit: Screaming Frog with JavaScript rendering and authenticated cookies.
• Large authenticated crawl: Crawlee with Playwright.
• Performance testing: Playwright + Lighthouse CI.
• Custom QA or regression testing: Playwright with a saved authenticated session.

If you can share what protects your staging environment (for example, Cloudflare, Akamai, Imperva, AWS WAF, or another solution) and what kind of audit you want to run (SEO, accessibility, performance, broken links, etc.), I can recommend the most suitable authentication and crawling configuration.