Free tool
AI crawler checker: can AI bots read your site?
Enter a domain to see, for 21 AI crawlers, whether your robots.txt lets them reach your homepage, which product each one feeds, and what to change. We also check for llms.txt and the sitemaps your robots.txt declares.
Which AI crawlers matter
Search & answers
These decide whether AI search engines can find, show and cite your pages.
- OAI-SearchBotOpenAI · ChatGPT search
- Surfaces sites in ChatGPT search answers. Blocking it keeps your pages out of those answers.
- Claude-SearchBotAnthropic · Claude search results
- Crawls to improve the search results Claude shows users. Blocking it can reduce your visibility in those answers.
- PerplexityBotPerplexity · Perplexity search
- Indexes sites so Perplexity can surface and link them in answers. Perplexity says it is not used to train foundation models.
- GooglebotGoogle · Google Search, AI Overviews, AI Mode
- Google Search's crawler. AI Overviews and AI Mode draw on the Search index, so blocking it removes you from both.
- BingbotMicrosoft · Bing and Microsoft Copilot
- Bing's crawler. Microsoft Copilot grounds web answers in the Bing index, so blocking it keeps you out of Copilot.
- ApplebotApple · Siri, Spotlight, Safari
- Apple's crawler for search features in Siri, Spotlight and Safari.
- DuckAssistBotDuckDuckGo · DuckAssist answers
- Fetches pages in real time to source DuckAssist answers. DuckDuckGo says the data is not used to train AI models.
On-demand fetch
These fetch a page live when someone asks an assistant about it.
- ChatGPT-UserOpenAI · ChatGPT on-demand visits
- Opens a page when a ChatGPT user or custom GPT asks for it. Not used for automatic crawling.
- Claude-UserAnthropic · Claude on-demand visits
- Fetches a page when a Claude user asks a question that needs it.
- Perplexity-UserPerplexity · Perplexity on-demand visits
- Visits a page when a Perplexity user asks about it. Perplexity says this fetcher generally ignores robots.txt.
- meta-externalfetcherMeta · Meta AI on-demand visits
- Fetches a link on behalf of a Meta AI user. Meta says it may bypass robots.txt because a person asked for it.
- MistralAI-UserMistral AI · Le Chat on-demand visits
- Fetches a page when a Le Chat user asks for it. Not used for automatic crawling.
Model training
These collect data for model training. Blocking them does not hide you from AI search.
- GPTBotOpenAI · OpenAI model training
- Crawls public pages that may be used to train OpenAI's generative models. It does not decide ChatGPT search visibility.
- ClaudeBotAnthropic · Claude model training
- Collects public web content that may be used to train Anthropic's Claude models.
- Google-ExtendedGoogle · Gemini training and grounding
- A robots.txt token, not a separate crawler: it controls whether Google may use your pages for Gemini models. It does not affect Google Search.
- Applebot-ExtendedApple · Apple model training
- A robots.txt token, not a separate crawler: it controls whether pages Applebot crawled may train Apple's generative models.
- meta-externalagentMeta · Meta AI training
- Meta's crawler for training AI models and improving Meta products.
- CCBotCommon Crawl · Open web dataset
- Builds Common Crawl's open web archive, one of the most common sources of LLM training data.
- BytespiderByteDance · ByteDance AI
- ByteDance's crawler, widely reported to collect data for its AI models.
Other
Specialised crawlers outside search and training.
- GoogleOtherGoogle · Google research crawls
- Generic crawler Google product teams use for one-off crawls and internal research, outside Search.
- Google-CloudVertexBotGoogle · Vertex AI Agents
- Crawls sites at the site owner's request to build Vertex AI Agents. Not used for Google Search.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search relies on OAI-SearchBot, and ChatGPT-User opens pages when a user asks. You can block GPTBot to opt out of training and still allow OAI-SearchBot so ChatGPT can show and cite you.
How do I allow AI crawlers in robots.txt?
Give each bot its own group, for example "User-agent: OAI-SearchBot" followed by "Allow: /", and remove "Disallow: /" from any group that names it. A bot without its own group follows your "User-agent: *" group, so a site-wide "Disallow: /" there blocks every AI crawler you have not listed.
What exactly does this checker test?
It fetches https://yourdomain/robots.txt and /llms.txt, then works out for each crawler whether your homepage path / is allowed or blocked, and which group decided it. It only reads robots.txt: a firewall or CDN rule that blocks AI bots (Cloudflare offers one) can still stop them even when robots.txt allows them.
What is llms.txt and do I need one?
llms.txt is a proposed convention: a Markdown file at /llms.txt that summarises your site and links its key pages for language models. It is not an official standard and no major AI engine has said it relies on it, but it is cheap to add.
Does Google-Extended control AI Overviews?
No. Google-Extended only controls whether Google may use your content for Gemini models; Google says it does not affect inclusion in Google Search. AI Overviews and AI Mode are part of Search, so they follow Googlebot.
Crawlable is step one. Are you in the answer?
AnswerTrack asks ChatGPT, Google AI Overview, Google AI Mode, Gemini, Perplexity and Copilot your buyers' questions every day and shows whether you are mentioned, who is recommended instead, and which sources get cited.