
Do you know what AI crawlers are actually doing on your website? Everyone's talking about AI search visibility and mostly measuring 'prompt visibility'. Google Analytics excludes known bots and doesn't show how much it excluded. Search Console reports Google's crawlers, not OpenAI, Anthropic, Perplexity or Bing. Here's what we can see in your server logs: → What GPTBot requests. → What ClaudeBot requests. → What Claude-User requests. → What ChatGPT-User requests. → What OAI-SearchBot requests. → What PerplexityBot requests. → Claude-Agent browser traffic. → What Perplexity-User requests. → What Claude-SearchBot requests. → AI crawlers hitting 404s and 5xx. → Disallowed paths AI crawlers hit. → AI crawlers getting 403s and 429s. → AI crawlers being served redirects. → Each AI bot's last robots.txt fetch. → AI crawlers seen for the first time. → OAI-AdsBot checking ChatGPT ad pages. → When each AI crawler request happened. → AI bot names coming from unlisted IPs. → How often AI bots refetch the same page. → AI bot speed and bandwidth, where logged. → Disallowed paths user-triggered bots hit. → Training vs search vs user-triggered bots. And that's just the AI side. → Crawl volume by URL pattern. → The busiest unverified bots. → Fake Googlebot and Bingbot. → Possible credentials in URLs. → Scanners probing .env and .git. → Orphan pages bots still request. → Bingbot getting 403s, 429s or 5xx. → URLs flipping between 200 and 404. → Redirected URLs bots keep requesting. → Sitemap URLs no verified bot fetched. → A/B-test cookies in crawler-UA checks. → Estimated human hits missing from GA4. → Crawl wasted on filters and parameters. → Crawled URLs missing from your sitemap. → Every Googlebot hit, not GSC's examples. → Possible robots.txt and sitemap caching. → Sensitive URLs returning 200 to scanners. Still use Google Analytics and Search Console, your server logs supplement them. But they don't show the full cross-crawler, request-level picture. A crawler visiting a page isn't proof of an AI citation. Salience.com









