ChatGPT answers, unedited

Yes—but there's an important distinction.

Most AI crawler analytics tools show which pages AI bots visit, but very few reconstruct complete crawl sessions with entry and exit paths. AI crawlers don't behave like human visitors, so "sessions" are harder to infer. Most products stop at page-level analytics. A handful are starting to visualize crawl sequences. Cloudflare Docs Ahrefs Help Center

If your goal is "show me the paths GPTBot, ClaudeBot, PerplexityBot, etc. take through my site", here's how the landscape looks:

What you're asking for is more like a Sankey graph

Ideally you'd see something like:

GPTBot

robots.txt
      │
      ▼
sitemap.xml
      │
      ▼
/blog/
      │
 ┌────┴─────┐
 ▼          ▼
/pricing   /docs
 │            │
 ▼            ▼
/api      /about
 │
 ▼
EXIT

or

ClaudeBot

Homepage
   │
   ▼
Features
   │
   ▼
Pricing
   │
   ▼
Case Study
   │
   ▼
Exit

That's still uncommon in commercial dashboards.

If you really want entry → path → exit

The best approach is often to analyze server logs instead of relying on analytics.

You can reconstruct sessions by:

• grouping requests by crawler IP + User-Agent
• ordering requests by timestamp
• breaking sessions after ~5–10 minutes of inactivity
• generating: - entry pages
• transition matrix
• exit pages
• Sankey diagrams
• crawl depth
• loops
• abandoned branches

This produces exactly the visualization you're describing.

I'd build it differently

If I were building this today, I'd create a dashboard showing:

• Entry pages by AI crawler
• Top crawl flows (Sankey diagram)
• Exit pages
• Average crawl depth
• Pages that every crawler reaches
• Pages only one crawler reaches
• Dead-end pages where crawlers stop
• Comparison between GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.

That kind of report is much more actionable than simple "top pages crawled."

If your site runs behind Cloudflare, NGINX, Apache, or another server where you have access to logs, this is very feasible to build. The raw data already contains almost everything needed—you mainly need the session reconstruction and visualization layer.

Yes—but there's an important distinction.

Most AI crawler analytics tools show which pages AI bots visit, but very few reconstruct complete crawl sessions with entry and exit paths. AI crawlers don't behave like human visitors, so "sessions" are harder to infer. Most products stop at page-level analytics. A handful are starting to visualize crawl sequences. Cloudflare Docs Ahrefs Help Center

If your goal is "show me the paths GPTBot, ClaudeBot, PerplexityBot, etc. take through my site", here's how the landscape looks:

What you're asking for is more like a Sankey graph

Ideally you'd see something like:

GPTBot

robots.txt
      │
      ▼
sitemap.xml
      │
      ▼
/blog/
      │
 ┌────┴─────┐
 ▼          ▼
/pricing   /docs
 │            │
 ▼            ▼
/api      /about
 │
 ▼
EXIT

or

ClaudeBot

Homepage
   │
   ▼
Features
   │
   ▼
Pricing
   │
   ▼
Case Study
   │
   ▼
Exit

That's still uncommon in commercial dashboards.

If you really want entry → path → exit

The best approach is often to analyze server logs instead of relying on analytics.

You can reconstruct sessions by:

• grouping requests by crawler IP + User-Agent
• ordering requests by timestamp
• breaking sessions after ~5–10 minutes of inactivity
• generating: - entry pages
• transition matrix
• exit pages
• Sankey diagrams
• crawl depth
• loops
• abandoned branches

This produces exactly the visualization you're describing.

I'd build it differently

If I were building this today, I'd create a dashboard showing:

• Entry pages by AI crawler
• Top crawl flows (Sankey diagram)
• Exit pages
• Average crawl depth
• Pages that every crawler reaches
• Pages only one crawler reaches
• Dead-end pages where crawlers stop
• Comparison between GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.

That kind of report is much more actionable than simple "top pages crawled."

If your site runs behind Cloudflare, NGINX, Apache, or another server where you have access to logs, this is very feasible to build. The raw data already contains almost everything needed—you mainly need the session reconstruction and visualization layer.

Yes—but there's an important distinction.

Most AI crawler analytics tools show which pages AI bots visit, but very few reconstruct complete crawl sessions with entry and exit paths. AI crawlers don't behave like human visitors, so "sessions" are harder to infer. Most products stop at page-level analytics. A handful are starting to visualize crawl sequences. developers.cloudflare.comahrefs.com

If your goal is "show me the paths GPTBot, ClaudeBot, PerplexityBot, etc. take through my site", here's how the landscape looks:

What you're asking for is more like a Sankey graph

Ideally you'd see something like:

GPTBot

robots.txt
      │
      ▼
sitemap.xml
      │
      ▼
/blog/
      │
 ┌────┴─────┐
 ▼          ▼
/pricing   /docs
 │            │
 ▼            ▼
/api      /about
 │
 ▼
EXIT

or

ClaudeBot

Homepage
   │
   ▼
Features
   │
   ▼
Pricing
   │
   ▼
Case Study
   │
   ▼
Exit

That's still uncommon in commercial dashboards.

If you really want entry → path → exit

The best approach is often to analyze server logs instead of relying on analytics.

You can reconstruct sessions by:

• grouping requests by crawler IP + User-Agent
• ordering requests by timestamp
• breaking sessions after ~5–10 minutes of inactivity
• generating:
  • entry pages
  • transition matrix
  • exit pages
  • Sankey diagrams
  • crawl depth
  • loops
  • abandoned branches

This produces exactly the visualization you're describing.

I'd build it differently

If I were building this today, I'd create a dashboard showing:

• Entry pages by AI crawler
• Top crawl flows (Sankey diagram)
• Exit pages
• Average crawl depth
• Pages that every crawler reaches
• Pages only one crawler reaches
• Dead-end pages where crawlers stop
• Comparison between GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.

That kind of report is much more actionable than simple "top pages crawled."

If your site runs behind Cloudflare, NGINX, Apache, or another server where you have access to logs, this is very feasible to build. The raw data already contains almost everything needed—you mainly need the session reconstruction and visualization layer.

Yes—but there's an important distinction.

Most AI crawler analytics tools show which pages AI bots visit, but very few reconstruct complete crawl sessions with entry and exit paths. AI crawlers don't behave like human visitors, so "sessions" are harder to infer. Most products stop at page-level analytics. A handful are starting to visualize crawl sequences. Cloudflare Docs Ahrefs Help Center

If your goal is "show me the paths GPTBot, ClaudeBot, PerplexityBot, etc. take through my site", here's how the landscape looks:

What you're asking for is more like a Sankey graph

Ideally you'd see something like:

GPTBot

robots.txt
      │
      ▼
sitemap.xml
      │
      ▼
/blog/
      │
 ┌────┴─────┐
 ▼          ▼
/pricing   /docs
 │            │
 ▼            ▼
/api      /about
 │
 ▼
EXIT

or

ClaudeBot

Homepage
   │
   ▼
Features
   │
   ▼
Pricing
   │
   ▼
Case Study
   │
   ▼
Exit

That's still uncommon in commercial dashboards.

If you really want entry → path → exit

The best approach is often to analyze server logs instead of relying on analytics.

You can reconstruct sessions by:

• grouping requests by crawler IP + User-Agent
• ordering requests by timestamp
• breaking sessions after ~5–10 minutes of inactivity
• generating: - entry pages
• transition matrix
• exit pages
• Sankey diagrams
• crawl depth
• loops
• abandoned branches

This produces exactly the visualization you're describing.

I'd build it differently

If I were building this today, I'd create a dashboard showing:

• Entry pages by AI crawler
• Top crawl flows (Sankey diagram)
• Exit pages
• Average crawl depth
• Pages that every crawler reaches
• Pages only one crawler reaches
• Dead-end pages where crawlers stop
• Comparison between GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.

That kind of report is much more actionable than simple "top pages crawled."

If your site runs behind Cloudflare, NGINX, Apache, or another server where you have access to logs, this is very feasible to build. The raw data already contains almost everything needed—you mainly need the session reconstruction and visualization layer.

Yes—but there's an important distinction.

Most AI crawler analytics tools show which pages AI bots visit, but very few reconstruct complete crawl sessions with entry and exit paths. AI crawlers don't behave like human visitors, so "sessions" are harder to infer. Most products stop at page-level analytics. A handful are starting to visualize crawl sequences. developers.cloudflare.comahrefs.com

If your goal is "show me the paths GPTBot, ClaudeBot, PerplexityBot, etc. take through my site", here's how the landscape looks:

What you're asking for is more like a Sankey graph

Ideally you'd see something like:

GPTBot

robots.txt
      │
      ▼
sitemap.xml
      │
      ▼
/blog/
      │
 ┌────┴─────┐
 ▼          ▼
/pricing   /docs
 │            │
 ▼            ▼
/api      /about
 │
 ▼
EXIT

or

ClaudeBot

Homepage
   │
   ▼
Features
   │
   ▼
Pricing
   │
   ▼
Case Study
   │
   ▼
Exit

That's still uncommon in commercial dashboards.

If you really want entry → path → exit

The best approach is often to analyze server logs instead of relying on analytics.

You can reconstruct sessions by:

• grouping requests by crawler IP + User-Agent
• ordering requests by timestamp
• breaking sessions after ~5–10 minutes of inactivity
• generating:
  • entry pages
  • transition matrix
  • exit pages
  • Sankey diagrams
  • crawl depth
  • loops
  • abandoned branches

This produces exactly the visualization you're describing.

I'd build it differently

If I were building this today, I'd create a dashboard showing:

• Entry pages by AI crawler
• Top crawl flows (Sankey diagram)
• Exit pages
• Average crawl depth
• Pages that every crawler reaches
• Pages only one crawler reaches
• Dead-end pages where crawlers stop
• Comparison between GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.

That kind of report is much more actionable than simple "top pages crawled."

If your site runs behind Cloudflare, NGINX, Apache, or another server where you have access to logs, this is very feasible to build. The raw data already contains almost everything needed—you mainly need the session reconstruction and visualization layer.

Yes—but there's an important distinction.

Most AI crawler analytics tools show which pages AI bots visit, but very few reconstruct complete crawl sessions with entry and exit paths. AI crawlers don't behave like human visitors, so "sessions" are harder to infer. Most products stop at page-level analytics. A handful are starting to visualize crawl sequences. developers.cloudflare.comahrefs.com

If your goal is "show me the paths GPTBot, ClaudeBot, PerplexityBot, etc. take through my site", here's how the landscape looks:

What you're asking for is more like a Sankey graph

Ideally you'd see something like:

GPTBot

robots.txt
      │
      ▼
sitemap.xml
      │
      ▼
/blog/
      │
 ┌────┴─────┐
 ▼          ▼
/pricing   /docs
 │            │
 ▼            ▼
/api      /about
 │
 ▼
EXIT

or

ClaudeBot

Homepage
   │
   ▼
Features
   │
   ▼
Pricing
   │
   ▼
Case Study
   │
   ▼
Exit

That's still uncommon in commercial dashboards.

If you really want entry → path → exit

The best approach is often to analyze server logs instead of relying on analytics.

You can reconstruct sessions by:

• grouping requests by crawler IP + User-Agent
• ordering requests by timestamp
• breaking sessions after ~5–10 minutes of inactivity
• generating:
  • entry pages
  • transition matrix
  • exit pages
  • Sankey diagrams
  • crawl depth
  • loops
  • abandoned branches

This produces exactly the visualization you're describing.

I'd build it differently

If I were building this today, I'd create a dashboard showing:

• Entry pages by AI crawler
• Top crawl flows (Sankey diagram)
• Exit pages
• Average crawl depth
• Pages that every crawler reaches
• Pages only one crawler reaches
• Dead-end pages where crawlers stop
• Comparison between GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.

That kind of report is much more actionable than simple "top pages crawled."

If your site runs behind Cloudflare, NGINX, Apache, or another server where you have access to logs, this is very feasible to build. The raw data already contains almost everything needed—you mainly need the session reconstruction and visualization layer.

Yes—but there's an important distinction.

Most AI crawler analytics tools show which pages AI bots visit, but very few reconstruct complete crawl sessions with entry and exit paths. AI crawlers don't behave like human visitors, so "sessions" are harder to infer. Most products stop at page-level analytics. A handful are starting to visualize crawl sequences. developers.cloudflare.comahrefs.com

If your goal is "show me the paths GPTBot, ClaudeBot, PerplexityBot, etc. take through my site", here's how the landscape looks:

What you're asking for is more like a Sankey graph

Ideally you'd see something like:

GPTBot

robots.txt
      │
      ▼
sitemap.xml
      │
      ▼
/blog/
      │
 ┌────┴─────┐
 ▼          ▼
/pricing   /docs
 │            │
 ▼            ▼
/api      /about
 │
 ▼
EXIT

or

ClaudeBot

Homepage
   │
   ▼
Features
   │
   ▼
Pricing
   │
   ▼
Case Study
   │
   ▼
Exit

That's still uncommon in commercial dashboards.

If you really want entry → path → exit

The best approach is often to analyze server logs instead of relying on analytics.

You can reconstruct sessions by:

• grouping requests by crawler IP + User-Agent
• ordering requests by timestamp
• breaking sessions after ~5–10 minutes of inactivity
• generating:
  • entry pages
  • transition matrix
  • exit pages
  • Sankey diagrams
  • crawl depth
  • loops
  • abandoned branches

This produces exactly the visualization you're describing.

I'd build it differently

If I were building this today, I'd create a dashboard showing:

• Entry pages by AI crawler
• Top crawl flows (Sankey diagram)
• Exit pages
• Average crawl depth
• Pages that every crawler reaches
• Pages only one crawler reaches
• Dead-end pages where crawlers stop
• Comparison between GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.

That kind of report is much more actionable than simple "top pages crawled."

If your site runs behind Cloudflare, NGINX, Apache, or another server where you have access to logs, this is very feasible to build. The raw data already contains almost everything needed—you mainly need the session reconstruction and visualization layer.

Yes. There are now several tools specifically for this, although “show me the exact path an AI crawler took through my site” is a more specialized requirement than ordinary bot analytics.

Best options

1. Cloudflare AI Crawl Control — probably the best starting point

Cloudflare AI Crawl Control

If your site is behind Cloudflare, this is the one I'd look at first. It can identify AI crawlers such as GPTBot, ClaudeBot and Bytespider, show the paths they request, and let you filter by crawler, operator, hostname and path. Its Metrics view also groups paths into patterns such as /blog/* and /docs/*. developers.cloudflare.com

It can also expose referral/destination information on paid plans, so you can distinguish things like:

AI crawler → /article-a → /article-b → /pricing → exit

However, Cloudflare's native UI isn't really a Sankey/path-flow visualization of individual crawler journeys. You'd likely export/query the data and build that visualization yourself. Cloudflare provides a GraphQL Analytics API for exactly that kind of custom analysis. developers.cloudflare.com

2. Ahrefs Bot Analytics

Ahrefs Bot Analytics

This is worth considering if you're already using Ahrefs. It captures bot activity server-side and lets you isolate AI bots and see which pages they crawl most. help.ahrefs.com

It's more oriented toward “which bots are crawling which pages?” than an actual journey map.

3. Webflow LLM Bot Insights

If your site happens to be on Webflow, its newer LLM Bot Insights feature uses server logs to show which LLM bots visited and which pages they requested, organized by bot/provider/purpose/path. help.webflow.com

If what you specifically want is a visual crawler map

I'd build the data around something like:

                 ┌── /about
                 │
GPTBot ── /blog/post-1 ── /blog/post-2 ── /pricing ── EXIT
                 │
                 └── /docs/api ── /docs/api/auth ── EXIT

ClaudeBot ── / ── /products ── /product-x ── EXIT

And calculate:

• Entry pages — first URL requested by each crawler/session
• Exit pages — last URL requested before a crawl session ends
• Transitions — A → B → C
• Path frequency — most common crawler routes
• Crawler-specific paths — GPTBot vs ClaudeBot vs Google-Extended, etc.
• Depth — how many pages a crawler typically traverses
• Dwell/time between requests — where meaningful
• Dead ends — pages where crawlers frequently stop
• Content clusters — sections disproportionately crawled
• Robots.txt compliance — whether crawlers are going where you've allowed them

One important caveat: server logs don't necessarily tell you a crawler's true “session.” You have to infer a crawl sequence from requests, since crawlers don't behave like a human browser session with a persistent cookie. So an entry/exit visualization is usually an inferred path rather than a literal session replay.

If you tell me what your site runs on (Cloudflare, AWS, nginx, WordPress, Webflow, etc.), I can recommend the simplest setup for getting an actual AI-crawler path/entry/exit visualization, including tools that don't require building it from scratch.

Yes. There are tools that can show AI crawler activity through your site, including which bots arrived, which URLs they requested, and (with the right data) the sequence of pages they followed. The key distinction is that most tools track requests, not a true “path” like a human analytics session, because many crawlers do not execute JavaScript or maintain sessions.

Options:

1. AI crawler-specific analytics

• Cloudflare AI Crawl Control — if your site uses Cloudflare, this is one of the strongest options. It can show AI crawler requests by crawler, operator, hostname, path, trends, and status codes. developers.cloudflare.com
• LogLens AI — analyzes server/CDN logs and presents AI bot activity (GPTBot, ClaudeBot, Perplexity, etc.) with page-level detail. loglens.ai
• SearchSeal — focuses on which AI crawlers reached your site and what pages they accessed. searchseal.com
• Ooky AI Sessions — specifically aims at grouping crawler requests into sessions and showing page sequences (closer to the “crawler path” idea). ooky.ai

2. General log analytics that can visualize paths

If you want a Sankey diagram / flow map like:

GPTBot
  ↓
/blog/article-1
  ↓
/pricing
  ↓
/contact

you usually build it from server logs using:

• Matomo Analytics (self-hosted analytics; can analyze visits and paths if you capture bot sessions)
• Grafana + log sources like Nginx/Apache/Cloudflare logs
• Elastic Observability (powerful but more technical)

3. WordPress-specific options

If your site is WordPress:

• LLM Bot Tracker plugin records AI crawler visits, URLs, bots, and trends inside WordPress. wordpress.org

What I would look for in a tool

For “AI crawler path visualization,” prioritize:

✅ Bot identity verification (avoid fake GPTBot user agents)
✅ URL-by-URL crawl sequences
✅ Entry URL and last URL
✅ Time between requests
✅ HTTP status codes (200/404/403/5xx)
✅ robots.txt and sitemap comparison
✅ Exportable raw logs

A typical setup is:

Cloudflare/CDN logs → AI crawler analyzer → graph dashboard

That gives you something like a “Google Analytics User Flow report,” but for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.

If you tell me your stack (WordPress, Shopify, Cloudflare, AWS, Nginx, etc.), I can narrow it to the best fit.

Yes. AI-crawler path visualization is becoming a real analytics category, although most tools today show crawler activity by URL rather than a polished “Sankey diagram of the crawl journey.”

Best options

1. Cloudflare AI Crawl Control — probably the best starting point

Cloudflare AI Crawl Control

If your site is behind Cloudflare, this is the one I'd try first. It can show:

• Which AI crawlers are hitting you — GPTBot, ClaudeBot, Google, Microsoft, etc.
• Which paths they request
• Activity by crawler/operator
• Path-pattern analysis such as /blog/*, /docs/*
• Requests over time
• Referrer/destination information
• CSV export for building your own crawl-path visualization

Cloudflare also exposes the underlying data through its GraphQL Analytics API, so you can build a custom visualization if the native UI isn't enough. developers.cloudflare.comdevelopers.cloudflare.com

Important limitation: Cloudflare's native analytics aren't really a “crawler journey” map. They tell you what paths were requested, but not necessarily a visual sequence like:

/ → /pricing → /docs → /api → EXIT

2. Ahrefs Bot Analytics

Ahrefs Bot Analytics

Ahrefs now has Bot Analytics, including AI assistants, and reports which bots visit and which pages they crawl. It's useful if you want something closer to SEO analytics rather than infrastructure logs. help.ahrefs.com

Again, though, it's primarily page/crawler reporting, not a visual path-flow explorer.

3. Webflow LLM Bot Insights

If you're on Webflow, Webflow now has LLM bot insights, which specifically identifies LLM bots and the pages they request. It organizes activity by bot, provider, purpose, and page path. help.webflow.com

If you specifically want “crawl paths”

What you're describing is slightly more sophisticated:

For each AI crawler, show the sequence of URLs it requested, identify the entry page, visualize transitions between pages, and identify the final/exit page.

For example:

                    ┌─ /about
                    │
GPTBot ── / ────────┼─ /products ── /pricing ── EXIT
                    │
                    └─ /blog ── /article-1 ── /article-2 ── EXIT

You can absolutely build this from server/CDN request logs. The key data fields are:

• timestamp
• crawler identity / user agent
• requested URL
• referrer
• response status
• session/request grouping
• hostname
• crawler IP/fingerprint where appropriate

Cloudflare's API gives you much of the underlying crawler/path data, so it would be a particularly good foundation for this. developers.cloudflare.comdevelopers.cloudflare.com

One caveat: “entry” and “exit” are harder to define for crawlers than for humans. A crawler doesn't necessarily maintain a browser-like session, and it may fetch URLs concurrently or revisit a page hours later. So I'd visualize request sequences and transitions rather than claiming every sequence represents a literal crawl session.

If you're thinking about building a tool/product around this, there's still a pretty interesting gap here: “Google Analytics-style user-flow visualization, but specifically for GPTBot/ClaudeBot/Gemini/etc.” Cloudflare has the raw ingredients, but its current UI is more analytics/dashboard-oriented than a true crawl-path explorer. developers.cloudflare.com

Yes. The closest off-the-shelf option right now is Cloudflare AI Crawl Control, but there’s an important distinction: most tools show which pages AI crawlers request, rather than a true “crawler journey” visualization like:

/ → /pricing → /docs → /blog → exit

Best options

1. Cloudflare AI Crawl Control — best overall

• Identifies AI crawlers such as GPTBot, ClaudeBot, Bytespider, etc.
• Shows most popular paths requested by each crawler.
• Lets you filter by crawler, operator, hostname, path, and time.
• Groups URLs into patterns such as /blog/* and /docs/*.
• Its GraphQL API exposes the underlying data, so you can build your own path-flow/Sankey visualization. developers.cloudflare.comdevelopers.cloudflare.com

Cloudflare AI Crawl Control

2. Ahrefs Bot Analytics — easier SEO-oriented view
Ahrefs now has Bot Analytics that tracks which bots visit and which pages they crawl, including AI assistants. It's useful for understanding crawler coverage, although it isn't really designed as a page-to-page journey map. help.ahrefs.com

3. Webflow LLM Bot Insights — if your site is on Webflow
Webflow has a dedicated LLM-bot view showing bots, providers, purposes, and requested page paths. It's more of a reporting interface than a crawl-path visualization. help.webflow.com

What you're describing is a little more specialized

If you specifically want something like this:

                    ┌─ /about
                    │
GPTBot ──> / ───────┼─> /products ──> /pricing ──> EXIT
                    │
                    └─> /docs ──> /api ──> /docs/auth ──> EXIT

ClaudeBot ──> /blog ──> /blog/article-1 ──> /blog/article-2 ──> EXIT

I don't see a mainstream analytics product that does this particularly well out of the box. The raw data exists, though. Cloudflare's API exposes crawler, URL/path, referrer and request information, which means you could turn it into a Sankey diagram / network graph / crawl-path explorer. developers.cloudflare.comdevelopers.cloudflare.com

One caveat: a crawler doesn't necessarily have a conventional browser "session," so entry → next page → exit has to be inferred by grouping requests into crawler sessions/time windows. HTTP requests don't inherently tell you that request #2 was caused by request #1.

If you're thinking about building or buying this for AI SEO / GEO, I'd actually recommend a slightly different visualization: AI crawler → pages crawled → pages cited/referred traffic → human landing pages. That can tell you not just where the bots go, but whether their crawling appears to result in ChatGPT/Claude/Perplexity traffic. Cloudflare already exposes some of the referral-side data needed for this. developers.cloudflare.com

If you tell me what your site runs on (Cloudflare, WordPress, Shopify, custom, etc.), I can point you to the best existing tool—or outline how to build the crawler-path visualization from your logs.

Yes. The important distinction is that AI crawlers generally don't behave like human visitors, so normal Google Analytics-style session-path tools often won't see them. Server/CDN request logs are the better source.

Best options right now

1. Cloudflare AI Crawl Control — probably the closest fit
Cloudflare AI Crawl Control

If your site is behind Cloudflare, its AI Crawl Control can identify crawlers such as GPTBot, ClaudeBot and Bytespider, then break activity down by crawler, URL/path, operator, hostname and time. It also has path/pattern analysis and CSV export. developers.cloudflare.com

The limitation: it isn't really a visual "journey map" showing:

/ → /blog → /article → /pricing → exit

It is more of a powerful request/path analytics system.

2. Botify — better for actual site-structure visualization

Botify SiteCrawler

Botify has visualizations of site structure, inlinks/outlinks, segments and crawler behavior. support.botify.com

It's more SEO/crawler-oriented than specifically AI-agent-oriented, though.

3. Build it from server logs — best if you specifically want "AI crawler journeys"

This is actually what I'd recommend if your goal is exactly:

"Show me how each AI agent traverses my site, where it enters, what sequence of pages it follows, and where it stops."

You can capture:

• AI crawler / user agent
• timestamp
• URL
• referrer
• HTTP status
• response size
• IP/ASN where appropriate
• crawl session/window
• previous URL → current URL
• first URL = entry point
• last URL before inactivity = exit point

Then visualize it as a Sankey/flow diagram or directed graph, e.g.:

                    ┌─ /pricing ─────── EXIT
                    │
/                    ├─ /docs/api ───── /docs/auth ── EXIT
└─ /blog ────────────┤
                     ├─ /about ──────── EXIT
                     │
                     └─ /products ──── /product-x ── EXIT

Cloudflare exposes its AI Crawl Control data through a GraphQL Analytics API, so you could also build this without maintaining your own raw-log pipeline. developers.cloudflare.com

One important wrinkle

I'd distinguish three kinds of AI traffic:

Those have very different meanings. A crawler repeatedly going / → blog → article → article tells you something quite different from an AI search agent entering /pricing and fetching three related pages.

If you tell me what your stack is (Cloudflare, AWS, Vercel, Nginx, WordPress, Shopify, etc.), I can point you to the closest existing product—or sketch the architecture for a dashboard that gives you an actual AI crawler path graph with entry/exit pages.

Yes. What you’re describing is basically AI-crawler path analytics: seeing which AI bots enter your site, the sequence of URLs they request, where they stop, and which pages are most/least crawled.

A few options are getting close:

• Cloudflare AI Crawl Control — probably the strongest starting point if you use Cloudflare. It identifies AI crawlers, lets you filter by crawler/operator/path, and its newer path-pattern views show which areas of your site they target. It also provides metrics and CSV exports. developers.cloudflare.com
• Ahrefs Bot Analytics — tracks bots by category, including AI assistants, and shows which pages they crawl. It’s currently free during beta. help.ahrefs.com
• Scrunch — more focused on connecting AI-crawler activity with AI referrals/conversions. It can tell you which bots crawl which pages and connect that to downstream traffic. www.techradar.com
• Server/CDN-log based solution — if you want the actual "entry → page A → page B → page C → exit" visualization, this is the most flexible approach. You can reconstruct sessions from crawler request logs and render a Sankey/flow diagram or site graph.

The important distinction

Most existing tools can show:

GPTBot → /pricing
GPTBot → /docs/api
ClaudeBot → /blog/article-1
PerplexityBot → /guides/

But considerably fewer give you a true:

GPTBot
/ → /products → /products/widget → /pricing → EXIT

with the ability to aggregate thousands of crawl sessions into a visual path map.

Cloudflare is particularly interesting here because its current analytics exposes most popular paths, path patterns, crawler filters, and downloadable data, which gives you much of the raw material for building exactly that view. developers.cloudflare.com

If you're asking because you want to build/buy something specifically that looks like a user-flow visualization, but for AI crawlers, I can also identify the closest existing products and show you what each one can/can't visualize.

Yes. There are now tools specifically aimed at AI crawler visibility, though the exact “show me the path a crawler took through my site like a user journey map” feature is still emerging. Most tools focus on bot identification + URLs visited + crawl sequences rather than a polished Google Analytics–style flow visualization. ooky.ai

Some options:

For the specific visualization you described:

“Show AI crawler paths through my site, entry points, exits, and the sequence of pages visited”

you’d ideally want:

1. Raw request capture
   • timestamp
   • crawler identity (GPTBot, ClaudeBot, Perplexity, etc.)
   • URL requested
   • referrer (if provided)
   • status code
   • session grouping

2. A path visualization layer
   • Sankey diagram: / → /pricing → /docs → exit
   • node graph: pages as nodes, crawler movement as edges
   • entry/exit reports:
     • “GPTBot entered 63% of the time through blog articles”
     • “ClaudeBot exits after product pages”
     • “AI crawlers never reach your documentation section”

3. Data source
   • Cloudflare logs/workers
   • Nginx/Apache logs
   • CDN logs
   • reverse proxy logs

A custom version is also very feasible: ingest bot logs into something like ClickHouse/Postgres, then visualize with Grafana, Apache Superset, or a graph library. That would probably give you the closest thing to a “Google Analytics User Flow for AI agents.”

One important distinction: AI crawlers usually do not execute JavaScript, so tools like traditional analytics platforms often miss them. Server/CDN log analysis is usually the right foundation. www.surva.ailoglens.ai

If you tell me your stack (WordPress, Shopify, Cloudflare, Next.js, etc.) and whether you want existing SaaS or self-hosted, I can narrow it down.

Yes. There are now tools specifically aimed at AI-crawler observability, although the exact “show me the path an AI bot took through my site, including entry → intermediate pages → exit” visualization is still a bit of a gap.

Best options

1. Cloudflare AI Crawl Control — probably the best starting point

If your site is behind Cloudflare, its current AI Crawl Control gives you:

• Which AI crawlers are visiting
• Requests by crawler/operator
• Most-requested URLs and URL patterns
• Crawl trends
• Allowed vs. unsuccessful requests
• Referral/destination patterns
• Bandwidth consumed
• Filtering by crawler and path

Cloudflare added path-pattern grouping such as /blog/*, /docs/*, etc., specifically to show which areas AI crawlers target. developers.cloudflare.com

urlCloudflare AI Crawl Control documentationturn0search9

But: it isn't really a visual journey map showing GPTBot → /article-a → /article-b → /article-c → exit.

2. Botify — stronger for actual crawl-path analysis

Botify is much closer to the SEO/crawl-analysis side of what you're describing. Its server-log analysis can show how search engines explore your site, how deep they go, crawl frequency, errors, and which pages aren't crawled. support.botify.com

The catch is that Botify is primarily an enterprise SEO/log-analysis platform, rather than an AI-agent journey visualization product.

urlBotify SiteCrawler documentationturn0search0

The interesting option: build it from server/CDN logs

If your goal is specifically:

“Show me a graph of how GPTBot/ClaudeBot/PerplexityBot actually traversed my site.”

you can get considerably closer by analyzing server/CDN logs, rather than Google Analytics.

For each crawler request, you'd capture something like:

crawler       timestamp       URL                 referrer
GPTBot        12:01:03        /                  -
GPTBot        12:01:04        /guides/seo        /
GPTBot        12:01:05        /guides/seo/tools   /guides/seo
GPTBot        12:01:06        /pricing            /guides/seo/tools

Then turn that into a directed graph:

                    ┌──────────────┐
                    │   /         │
                    │  ENTRY      │
                    └──────┬───────┘
                           │
                           ▼
                    ┌──────────────┐
                    │ /guides/seo  │
                    └──────┬───────┘
                           │
              ┌────────────┴──────────┐
              ▼                       ▼
       /guides/tools              /pricing
              │                       │
              ▼                       ▼
          /article-x               EXIT

You could then answer much more interesting questions:

• Where do AI crawlers enter?
• Which pages do they visit next?
• What pages are never reached?
• What's the average crawl depth?
• Which pages act as hubs?
• Where do crawlers stop?
• Which crawler has the deepest paths?
• GPTBot vs ClaudeBot vs Google-Extended vs PerplexityBot
• Do different AI crawlers follow different internal-link structures?
• Which pages get crawled but never subsequently generate AI referral traffic?

One important caveat: don't use client-side analytics alone. AI crawlers frequently make server-side HTTP requests without executing your analytics JavaScript, so the server/CDN layer is where this data needs to be captured.

If you're thinking about building this

I actually think there's a pretty compelling product gap here.

Most existing AI crawler dashboards answer:

“Who crawled me and which URLs did they request?”

What you're describing answers:

“How did the crawler navigate my information architecture?”

Those are quite different.

A good UI would have three views:

1. Crawler Journey Map — nodes = pages, edges = transitions, animated/filterable by bot.
2. Entry/Exit Analysis — Sankey-style visualization of entry pages → paths → exit pages.
3. Crawler Comparison — e.g. GPTBot vs ClaudeBot vs Perplexity showing different site traversal patterns.

And I'd use raw CDN/server logs as the source of truth, with Cloudflare being a particularly convenient data source if the site is already there.

If you tell me what your site runs on (Cloudflare, Nginx, Vercel, AWS, WordPress, etc.), I can recommend the closest existing product—or sketch the architecture for building exactly this visualization.

Yes. There are now tools that can get fairly close to what you're describing, but there’s an important distinction:

• Crawler analytics: Which AI bots visited which URLs, when, and how often.
• Path visualization: The actual sequence of URLs an individual crawler/session followed — e.g. / → /pricing → /docs → /api → exit.
• Entry/exit analysis: Which pages AI crawlers tend to start from and where they stop.
• AI-agent browsing: Browser-based AI agents can look more like human sessions and may require different instrumentation than traditional crawlers. arxiv.org

Tools worth looking at

Cloudflare AI Crawl Control
Probably the best starting point if you're already on Cloudflare. It identifies AI crawlers and shows which ones are accessing your site and how frequently. Cloudflare sees requests at the edge, including cached requests that might never reach your origin. www.cloudflare.com

Ahrefs Bot Analytics
Interesting if your goal is SEO/GEO rather than infrastructure. It tracks bot categories and the pages crawled, including AI assistants. help.ahrefs.com

Surva AI Crawler Analytics
More specifically focused on AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with reporting on which pages they actually access. www.surva.ai

But for the visualization you have in mind...

If you mean something like this:

                    ┌─ /about ── /team
                    │
GPTBot ── / ── /products ── /pricing ── X
                    │
                    └─ /docs ── /api ── /reference ── X

ClaudeBot ── /docs ── /guides ── /tutorial ── X

Perplexity ── / ── /blog ── /article/123 ── /article/456 ── X

That's a much less commonly available visualization. Most products give you crawl counts by bot and URL, rather than reconstructing and visually displaying the crawl graph/session.

You can build this fairly cleanly from CDN/server request logs:

crawler identity + timestamp + URL + referrer + session/request grouping

Then visualize it as a directed graph, with:

• 🟢 entry pages
• 🔴 exit pages
• node size = number of crawler visits
• edge thickness = number of transitions
• color = GPTBot / ClaudeBot / PerplexityBot/etc.
• filters for date, crawler, section, status code
• "replay this crawl" to see the sequence
• separate view for training crawlers vs search/retrieval crawlers vs AI agents

One caveat: you can't always know the true path from ordinary HTTP logs. Crawlers don't necessarily provide a reliable referrer, and a bot can request URLs concurrently or discover URLs through your sitemap rather than following links. So a good visualization should distinguish observed transitions from inferred site traversal.

Also, GA4's normal Path Exploration isn't really the answer here: its path graph is designed around analytics events/user journeys, not server-side AI crawler requests. support.google.com

If you're thinking about building this as a product/internal dashboard, I can also lay out exactly what the data pipeline and visualization would look like—and which existing tools you'd need versus what you'd have to build yourself.

Yes. What you’re describing is AI crawler path analysis: seeing an AI bot enter at page A, move through pages B → C → D, and then stop/exit.

The closest options I’d look at are:

1. Cloudflare AI Crawl Control — closest fit

urlCloudflare AI Crawl Controlturn0search0

If your site is behind Cloudflare, this is probably the first thing I'd try. It can show:

• Which AI crawlers are visiting — GPTBot, ClaudeBot, Google, Bytespider, etc.
• Most popular paths requested by AI crawlers
• Requests by crawler/operator
• Path/pattern analysis such as /blog/* or /docs/*
• AI referral sources and which pages receive those referrals
• Filters by crawler, hostname and path
• CSV export and API access developers.cloudflare.com

But: it doesn't appear to give you a true visual journey graph like:

/ → /article-a → /article-b → /pricing → EXIT

That's an important distinction.

2. Ahrefs Bot Analytics

urlAhrefs Bot Analyticsturn0search2

Ahrefs now has Bot Analytics that tracks which bots visit, how frequently, and which pages they crawl, including AI assistants. help.ahrefs.com

It's useful for answering "what pages are AI crawlers interested in?", but again, it's more analytics/reporting than a visual path explorer.

3. Build the path visualization from server/CDN logs

If your actual goal is the visualization you described, this may be the better solution.

You capture AI crawler requests from your server/CDN logs:

timestamp
crawler / user-agent
IP or request fingerprint
URL
referrer
status
response time

Then reconstruct sessions approximately:

GPTBot
  │
  ├── /                ← entry
  │
  ├── /guides/
  │
  ├── /guides/seo
  │
  ├── /pricing
  │
  └── EXIT             ← last request

You could then visualize aggregate behavior as a Sankey/flow diagram:

                    ┌── /blog/a ── /blog/b ── EXIT
                    │
AI crawler ── / ────┼── /docs ─── /docs/api ── EXIT
                    │
                    └── /pricing ────────────── EXIT

That would let you answer much more interesting questions:

• Where do AI crawlers enter?
• What pages do they crawl next?
• Which pages cause them to stop?
• What are the most common crawl sequences?
• Which pages are never reached?
• How does GPTBot's behavior differ from ClaudeBot?
• Do crawlers follow internal links or mostly hit URLs independently?
• Which pages are disproportionately important to AI crawlers?

One caveat

Traditional analytics like Google Analytics aren't ideal for this because many crawlers don't execute JavaScript. Server/CDN-level logs are much more reliable for this use case. www.reddit.com

If you're specifically looking for a tool that gives you a visual "AI crawler journey map" rather than just bot counts and top pages, I don't see a mature off-the-shelf product that does this particularly well yet. Cloudflare gets closest on the data side, but the actual path graph is something you'd likely build on top of the request data.

If you tell me what your site runs on (Cloudflare, Vercel, AWS, nginx, WordPress, etc.), I can recommend the simplest setup for getting an actual AI crawler path visualization.

Yes. This is an emerging category of analytics as AI crawlers become a larger share of web traffic. There isn't yet a "Google Analytics for AI bots" that's universally adopted, but several tools can already visualize crawler behavior, entry pages, and crawl paths.

Here are the best options, depending on what you want.

If your goal is AI crawler intelligence

The most interesting approach is analyzing raw server logs instead of JavaScript analytics.

From logs you can identify:

• GPTBot
• ChatGPT-User
• ClaudeBot
• Claude-User
• PerplexityBot
• Google-Extended
• Bytespider
• Applebot
• Amazonbot
• Common Crawl
• and hundreds more.

Then you can reconstruct sessions like:

ClaudeBot

/home
   ↓
/blog/ai-search
   ↓
/pricing
   ↓
/docs/api
   ↓
Exit

or

GPTBot

/
/blog
/blog/post-1
/blog/post-2
/blog/post-3
Exit

This is very similar to human clickstream analysis, except based on request timestamps rather than browser events.

Tools that actually visualize paths

If you want something that looks like a Sankey diagram or flow map:

• Elastic + Kibana
• Grafana
• Splunk
• Datadog Log Analytics
• Mixpanel (after importing logs)

These can generate diagrams such as:

Homepage
   │
   ├──► Blog
   │       │
   │       ├──► Pricing
   │       │
   │       └──► API Docs
   │
   └──► About

or show:

• top entry pages
• most common next page
• average crawl depth
• where bots abandon
• orphan pages never reached
• pages every crawler visits

If you're running Cloudflare

Cloudflare has become one of the better options because it can distinguish verified AI crawlers and show traffic by bot type. While it doesn't provide full clickstream visualization out of the box, you can export request logs and build those visualizations in Grafana or Kibana.

A gap in the market

There's still a notable opportunity for a dedicated product that focuses specifically on AI crawler journeys. Imagine a dashboard with:

• Interactive crawl maps
• Sankey diagrams of AI crawler navigation
• Heatmaps of entry and exit pages
• Comparisons of GPTBot vs. ClaudeBot vs. PerplexityBot
• Crawl-depth analysis
• "Pages frequently cited by AI" correlations
• Recommendations for improving AI discoverability

As AI search grows, tooling like this is likely to become more common.

If you're interested in building something yourself, it's quite feasible: ingest web server logs (or Cloudflare logs), group requests into crawler sessions using IP, user agent, and time windows, then visualize the transitions as a directed graph or Sankey diagram. That approach gives you much richer insight than traditional pageview analytics because it shows how AI agents actually traverse your content.

Yes, but the answer depends on what you mean by “AI crawler paths.” There are a few different levels:

1. AI crawler activity (which bots hit which pages)
2. A visual path map (page A → page B → page C)
3. Entry/exit analysis like human analytics
4. AI referral traffic (ChatGPT/Perplexity/etc. sending users)

For true AI crawler path visualization, the market is still immature, but these are the closest options:

Cloudflare’s AI crawler analytics are among the closer fits because they can break down AI crawler activity by crawler/operator and show things like popular paths, requested URLs, and referral patterns. developers.cloudflare.com

For the exact visualization you described — “GPTBot entered at /pricing, crawled /docs/api, then exited after /contact” — you usually need to build it from server logs:

1. Capture:
   • timestamp
   • user agent (e.g., GPTBot, ClaudeBot)
   • IP/request fingerprint
   • URL requested
   • referrer
   • response code

2. Sessionize requests by crawler.

3. Visualize with:
   • Sankey diagrams (entry → pages → exit)
   • directed graphs
   • flow maps

A typical stack would be:

• Log source: Cloudflare logs, Nginx/Apache logs, AWS CloudFront logs
• Storage: BigQuery, ClickHouse, Elasticsearch
• Visualization: Grafana, Apache Superset, Metabase, or a custom D3.js graph

One caveat: AI crawlers often do not behave like humans. Many fetch isolated pages from sitemaps or links rather than “browsing” a site in a linear session, so an entry/exit funnel may look very different from a user journey.

If your goal is SEO/GEO optimization (“what AI systems are reading and what content they favor”), or security/cost control (“who is scraping and how much”), the ideal tool choice is different.

Yes. This is an emerging category of SEO/analytics tooling sometimes called LLM analytics, AI bot analytics, or AI crawler observability. The market is still immature, but there are several tools that can show some combination of:

• Which AI crawlers visit your site (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.)
• Landing (entry) pages
• Crawl paths (where available)
• Exit points
• Crawl frequency and depth
• Pages that AI systems are likely using as sources
• Referrers from AI chat products (when users click through)

It's worth distinguishing two different things:

1. AI crawlers (GPTBot, ClaudeBot, etc.) that index content.
2. Human visitors coming from AI products (ChatGPT, Perplexity, Gemini, Claude, etc.).

These require different data sources.

Some of the more established options include:

• Matomo — Can analyze server logs and bot traffic if configured.
• Plausible Analytics — Good at identifying AI referral traffic from chat products.
• Cloudflare Analytics — If your site uses Cloudflare, you can inspect bot traffic and filter verified AI crawlers.
• Microsoft Clarity — Useful for human visitors arriving from AI referrals, but not AI bots themselves.
• ELK Stack (Elasticsearch + Logstash + Kibana) — Excellent if you want to build your own crawl-path dashboards from raw server logs.

For AI-focused observability, newer vendors are building specifically for this use case, including products that identify AI bots, estimate LLM visibility, and report crawl behavior. Their feature sets change quickly, but they generally focus on:

• AI crawler identification
• Crawl frequency
• Pages indexed by AI systems
• AI referral monitoring
• Content optimization for LLMs

Can they show actual crawler paths?

If the data comes from server logs, yes. Every request has:

• timestamp
• crawler user agent
• IP
• URL requested

That allows reconstruction of sessions like:

GPTBot

/
/blog/
/blog/ai/
/pricing/
/docs/api/
/docs/pricing
(exit)

From this you can generate:

• Sankey diagrams
• Flow diagrams
• Entry/exit funnels
• Crawl trees
• Heat maps

Many commercial dashboards don't yet expose these visualizations, but they're straightforward to build from log data.

If you want the best visibility

The most complete approach is:

1. Enable detailed web server logging (Nginx, Apache, IIS, etc.).
2. Identify verified AI crawler user agents.
3. Import logs into a visualization platform such as Kibana or Grafana.
4. Build:
   • Entry page ranking
   • Exit page ranking
   • Average crawl depth
   • Most common crawl paths
   • Time between visits
   • Crawl graphs

This often provides more insight than current commercial AI SEO tools because you're working from the raw request stream.

If your goal is specifically to understand how ChatGPT, Claude, Gemini, and Perplexity bots traverse your site, a custom dashboard built from server logs is currently the most powerful option.

If you're looking for something closer to Google Analytics for AI crawlers, that category is rapidly developing, but no single product yet offers the same level of mature flow visualization that GA4 provides for human users. A combination of server logs plus a visualization platform remains the most flexible solution.

Yes. There are several types of software that can help visualize AI crawler activity through your website, including where crawlers enter, what they fetch, and where they leave. The exact capabilities vary because many analytics tools were built for human visitors rather than bots.

1. Bot/crawler analytics platforms

These focus on identifying automated traffic and showing paths.

• Cloudflare Bot Management
  • Detects bots and crawlers using traffic analysis.
  • Can show bot traffic trends, blocked/allowed requests, and bot categories.
  • Useful if your site is behind Cloudflare.

• DataDome Bot Protection
  • Provides bot analytics, including crawler behavior patterns.
  • More enterprise-focused.

• Akamai Bot Manager
  • Enterprise bot visibility and traffic analysis.

These are strongest for "who is crawling and what are they doing?"

2. Log analysis tools (best for true crawler paths)

If you want actual paths like:

GPTBot → /pricing → /docs → /api → /exit

you usually need server log analysis.

Options:

• Matomo Analytics
  • Privacy-focused analytics.
  • Can ingest logs with additional tooling and visualize journeys.

• GoAccess
  • Open-source web log analyzer.
  • Can show:
    • top crawlers
    • requested URLs
    • referrers
    • time patterns
    • request sequences

• Elastic Observability with Elasticsearch + Kibana
  • Best if you want custom crawler journey maps.
  • You can build dashboards showing:
    • crawler → landing page → page sequence → last page
    • crawl frequency
    • content categories crawled

3. SEO crawler visualization tools (for simulated crawls)

These don't track external AI bots, but they visualize crawler paths through your site architecture:

• Screaming Frog SEO Spider
  • Crawl your own site.
  • Visualizes crawl paths, internal linking, orphan pages, depth.

• Sitebulb
  • Strong visual site auditing and crawl maps.

4. AI-crawler-specific monitoring

If your goal is specifically ChatGPT, Claude, Perplexity, Google-Extended, etc. crawlers, look for tools that analyze server logs and user agents.

Common AI crawler identifiers include:

• GPTBot
• ChatGPT-User
• ClaudeBot
• anthropic-ai
• PerplexityBot
• Google-Extended

A typical setup:

Server logs
     ↓
Crawler identification
     ↓
Session reconstruction
     ↓
Path visualization
     ↓
Dashboard

A custom dashboard in Kibana, Grafana, or a log analytics platform is currently the most flexible way to get a Sankey diagram / flow map of AI crawler journeys.

If I were building this today

For a site owner wanting a visual "AI crawler map," I'd use:

1. Cloudflare → capture bot traffic
2. Log export → store requests
3. Elasticsearch + Kibana → create crawler journey dashboards
4. Visualizations:
   • entry pages
   • page-to-page transitions
   • exit pages
   • crawl depth
   • AI bot comparison

A key question: are you trying to visualize AI bots crawling your site today, or are you trying to understand how AI search engines/users might traverse your content? The best tool choice differs.

Yes. What you’re looking for exists, but it is usually built from server logs/CDN bot analytics rather than traditional web analytics (Google Analytics often won’t see crawlers because they don’t run normal browser JavaScript).

Tools fall into a few categories:

1. AI crawler dashboards (closest match)

These show:

• which AI bots visited (GPTBot, ClaudeBot, PerplexityBot, etc.)
• URLs requested
• crawl frequency
• bandwidth consumed
• entry URLs and common paths

Cloudflare AI Crawl Control is probably the closest off-the-shelf option. It provides AI crawler metrics, crawler breakdowns, popular paths, referral information, and filtering by crawler/operator/path. developers.cloudflare.com

2. Log visualization platforms (best for “path maps”)

If you want something like a Sankey diagram / flow visualization:

GPTBot
  ↓
/blog/article-a
  ↓
/pricing
  ↓
/docs/api
  ↓
exit

you typically pipe web server logs into:

• Grafana Labs + Loki (custom dashboards)
• Elastic Observability (Kibana dashboards)
• Splunk

You can group requests by:

• user agent (AI crawler identity)
• session/crawl ID
• URL sequence
• timestamp windows
• referrer
• exit URL

3. Bot intelligence / AI visibility tools

Some newer SEO/GEO tools focus more on whether AI systems are discovering your content rather than raw navigation paths. They can be useful if your goal is “are AI assistants finding my important pages?” rather than “draw the crawler journey.”

A few caveats:

• True crawler paths are imperfect. A bot may fetch pages in parallel, revisit URLs, or not behave like a human session.
• Entry/exit tracking requires stitching requests together, usually using IP + user agent + time window (or CDN-provided bot identifiers).
• AI agents that browse like humans can be harder to distinguish from normal visitors than traditional crawlers. arxiv.org

If you want the closest thing to a “Google Analytics User Flow report, but for AI crawlers”, I’d usually set up:

Cloudflare AI Crawl Control → export logs → Grafana/Kibana → crawler path visualization

That gives you both the identification layer and the visual journey layer. developers.cloudflare.com

I can also suggest a stack depending on whether your site is on WordPress, Shopify, Cloudflare, AWS, Nginx/Apache, or a custom app.