AI Crawler Directives
Robots.txt AI Bot Inspector
Verify whether GPTBot, ClaudeBot, PerplexityBot, or Google-Extended are blocked by your robots.txt configuration.
Standard AI User-Agents
User-agent: GPTBot
OpenAI's crawler for training and real-time retrieval in ChatGPT.
User-agent: ClaudeBot
Anthropic's crawler used for Claude knowledge and citations.
User-agent: PerplexityBot
Perplexity AI's indexing bot used for generating answer engine links.
User-agent: Google-Extended
Google's standalone token for Gemini training and AI integration.