CrawllexaAI Crawler Checker
Paste your robots.txt to see which search engines and AI crawlers can reach your pages, from Googlebot to GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. The checker follows the matching rules Google documents, then helps you write a policy that fits your goals.
Runs in your browser. Nothing you paste is uploaded or stored.
Check your robots.txt
Opens your robots.txt in a new tab so you can copy it. Browsers can't fetch another site's files for you.
A path such as /blog/, or a full URL.
Generate a crawler policy
Choose a starting point, adjust individual crawlers, and copy the result. The output is a complete robots.txt.
One path per line. To keep an exception open, start the line with Allow:. A crawler with its own group ignores the rules for everyone else, so these are added to each group.
How to use the checker
- Open your robots.txt. It always lives at the root of your domain, for example
https://www.example.com/robots.txt. - Copy the whole file and paste it into the checker.
- Enter the page or path you care about. Check your most important templates too, such as a product page or a blog post, not only the home page.
- Read each result. It shows whether the crawler may fetch that path, and the exact line that decided it.
The crawlers it checks
The checker covers crawlers whose behavior is documented by the companies that run them. They fall into four groups, and the difference matters: blocking a training crawler has no effect on search, while blocking a search crawler removes you from that product's answers.
Search engines
| Crawler | What it does | If you block it | robots.txt |
|---|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode. Source | Google can't crawl the page for Search, AI Overviews or AI Mode. The URL can still be indexed from links, without a description. | Follows robots.txt |
| Bingbot Microsoft | Crawls for Bing, whose index also powers Microsoft Copilot answers. Source | The page can't be crawled for Bing or Copilot answers built on the Bing index. | Follows robots.txt |
| Applebot Apple | Crawls for search features in Spotlight, Siri and Safari. Its data may also train Apple's models unless Applebot-Extended is blocked. Source | The page can't appear in Apple's search features in Spotlight, Siri and Safari. | Follows robots.txt |
AI search and answers
| Crawler | What it does | If you block it | robots.txt |
|---|---|---|---|
| OAI-SearchBot OpenAI | Crawls to show websites in ChatGPT search answers. Source | The page isn't shown in ChatGPT search answers, though it can still appear as a navigational link. | Follows robots.txt |
| PerplexityBot Perplexity | Crawls to surface and link websites in Perplexity. Not used for AI training. Source | The page drops out of Perplexity's search results. | Follows robots.txt |
| Claude-SearchBot Anthropic | Indexes content to improve Claude's search results. Source | The page isn't indexed for Claude's search, which may reduce your visibility in its answers. | Follows robots.txt |
AI training
| Crawler | What it does | If you block it | robots.txt |
|---|---|---|---|
| GPTBot OpenAI | Crawls content that may be used to train OpenAI's AI models. Source | Your content isn't used to train OpenAI's models. ChatGPT search isn't affected. | Follows robots.txt |
| ClaudeBot Anthropic | Collects web content that may be used to train Anthropic's AI models. Source | Your future content is excluded from Anthropic's training data. | Follows robots.txt |
| Google-Extended | A control token, not a separate crawler. Covers training Gemini models and grounding answers in the Gemini app. Source | No effect on Google Search, AI Overviews or AI Mode. Your pages are left out of Gemini training and Gemini app grounding. | Follows robots.txt |
| Applebot-Extended Apple | A control token for whether Applebot's data may train Apple's foundation models. Source | Your content isn't used to train Apple's models. Applebot can still crawl it for search features. | Follows robots.txt |
| CCBot Common Crawl | Builds Common Crawl's free public archive of the web, which anyone can download, including to train AI models. Source | Your pages stop being added to Common Crawl's archive. | Follows robots.txt |
Fetches on a user's request
| Crawler | What it does | If you block it | robots.txt |
|---|---|---|---|
| ChatGPT-User OpenAI | Fetches a page when a ChatGPT user's request needs it. Source | Blocking may not work: OpenAI says robots.txt may not apply to requests a user starts. | May not follow robots.txt |
| Perplexity-User Perplexity | Fetches pages to answer a Perplexity user's question. Source | Blocking may not work: Perplexity says this fetcher generally ignores robots.txt. | Generally ignores robots.txt |
| Claude-User Anthropic | Fetches pages when a Claude user asks a question that needs them. Source | Claude can't retrieve your pages when its users ask for them. | Follows robots.txt |
How the checker reads robots.txt
It follows Google's robots.txt specification, which is based on the Robots Exclusion Protocol standard:
- A crawler uses only one group. If a group names the crawler, it follows that group and ignores the
*group entirely. If several groups name it, their rules are combined. - The longest matching rule wins.
Allow: /blog/beatsDisallow: /for a blog post, because its path is longer. - Ties go to the less restrictive rule. If an allow and a disallow rule are the same length, the page is allowed.
- Wildcards work.
*matches any sequence of characters, and$marks the end of the URL. - Unsupported lines are ignored. Google doesn't use
crawl-delay, andnoindexinside robots.txt has never been supported.
One consequence surprises people: the moment you add a group for a crawler, say to allow OAI-SearchBot, that crawler stops following your * rules. If those rules kept crawlers out of /cart/ or internal search, repeat them in the new group. The policy generator does this for you.
What robots.txt can't tell you
- Whether your server lets crawlers in. Firewalls, bot protection and CDNs can block crawlers before robots.txt is read. Check your logs for 403 and 429 responses to these user agents.
- Whether a page stays out of search. robots.txt controls crawling, not indexing. A blocked URL can still be indexed from links. Use
noindexto keep a page out of results. - Whether AI Overviews can use your content. That's controlled by Googlebot access, snippet controls and the generative AI setting in Search Console, not by a separate AI crawler.
- What user-triggered fetchers will do. Some of them don't follow robots.txt, because a person asked for the page.
For the full picture, see our guide to AEO and GEO, which covers how AI answers choose their sources, and our technical SEO guide, which covers crawling and indexing in depth.
AI crawler checker: frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI uses separate crawlers: GPTBot collects content that may be used for training, while OAI-SearchBot crawls for ChatGPT search. Blocking GPTBot keeps your content out of training without affecting whether ChatGPT search can show your pages. Blocking OAI-SearchBot is what removes you from ChatGPT search answers.
Can I block Google's AI Overviews with robots.txt?
Not without also dropping out of regular search results. AI Overviews and AI Mode use Googlebot, and Google-Extended doesn't affect them. To opt out, use the Search generative AI features setting in Search Console, which applies to the whole site, or the nosnippet, data-nosnippet and max-snippet controls for specific content.
The checker says a crawler is allowed. Why isn't it visiting my site?
This tool only reads robots.txt. Firewalls, bot-protection services and CDNs can block crawlers before robots.txt is ever read. Check your server or CDN logs for the crawler's user agent and for 403 or 429 responses.
Is the robots.txt I paste uploaded anywhere?
No. The checker and the policy generator run entirely in your browser. Nothing you paste is uploaded or stored.
What's the difference between search, training and user-triggered crawlers?
Search crawlers index pages so they can appear in search results and AI answers. Training crawlers collect content that may be used to train AI models. User-triggered fetchers retrieve a page because a person asked an assistant about it, and some of them don't follow robots.txt because a person started the request.
Want to know how AI answers see your brand?
Our AI visibility work covers crawler access, content structure and how often you're cited.