AI Crawler Access: robots.txt Settings for ChatGPT, Gemini and Claude

AI products use separate bots for search and for training, and you can control each one on its own. To be visible in AI answers, let the search bots in, and decide on the training bots as a business choice.

Most guides about being cited by AI tools skip the first step: a bot has to be able to fetch the page. This guide lists the main crawlers, shows a starter robots.txt and covers the firewall and server checks that catch most problems. The names and behavior below come from each company's own documentation, which changes, so use the links to confirm before you edit your file.

What is the difference between AI search bots and AI training bots?

A search bot fetches pages so that a product can find and cite them when a user asks a question. A training bot collects content to help build future models. Blocking a training bot does not stop a product from citing you through its search bot, and the reverse is also true. That is why the controls are separate.

Which bots should you allow?

For answer visibility, allow the search and retrieval bots: Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. Those are the ones that find pages to cite. Whether to allow training bots such as GPTBot, ClaudeBot and Google-Extended is up to you: some businesses welcome them as a path to being known by future models, and others keep them out for licensing reasons. Either choice can sit alongside full search visibility.

A starter robots.txt

This example allows the search bots, sets your sitemap and leaves training bots as a decision you make explicitly. Adjust the training section to match your policy.

# Search and retrieval bots: allowed
User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training bots: change to "Disallow: /" if your policy is to opt out
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Check your firewall, CDN and hosting rules

This is the step that catches the most sites. Many security and bot-protection tools block unknown or automated visitors by default, and they do it before a bot ever reads robots.txt. If your robots.txt welcomes a bot but your CDN or web application firewall challenges it, the bot sees an error and moves on.

• Look in your CDN or firewall dashboard for bot-management settings and any rule that blocks or challenges AI crawlers.
• Allow the search bots by their published user agent, and where the company publishes IP ranges, by those too.
• Ask your host whether it adds default blocks on top of your own settings.

How do you verify that bots can reach your pages?

1. Read your server logs. Search for the bot names (for example OAI-SearchBot) and check that the responses are successful.
2. Test as the bot. Request a page with that user agent from the command line and check that you receive the full page and a 200 status code.
3. Use Search Console. The URL Inspection tool shows how Google renders your page.
4. Look at the rendered text. If a bot only receives a shell page with content loaded later by scripts, it may see nothing.

curl -A "OAI-SearchBot" -I https://www.example.com/blog/your-post

What should you do on staging and private pages?

Keep staging sites and private pages out of search with a password or a noindex directive, not with robots.txt alone: a blocked URL can still be listed if other pages link to it. Use robots.txt for crawl control and noindex or authentication for pages that should not appear anywhere.

How often should you review these settings?

Quarterly is a good rhythm. AI companies add and rename bots, CDN vendors change default rules, and your own site changes. Put a quick check in your calendar: confirm the bot names against the vendor pages, test one URL with each search bot and skim your logs for blocked requests.

Make sure the content is in the HTML

Many crawlers do not run scripts the way a browser does. Pages that render their text on the server, or that send the full text in the initial HTML, are the safest. A quick check: view the page source and search for a sentence from your article. If it is there, bots can read it.

Do you need an llms.txt file?

Google's guide to its AI features says special AI files are not needed, and a large study cited by Advanced Web Ranking found that most sites with an llms.txt file received no requests for it in the period measured. Adding one is harmless, but time on crawl access, clear content and mentions pays off more. Our AI search optimization guide shows where to spend it.

Once bots can reach your pages, the next step is seeing what they say about you. Follow our method for tracking your brand in AI answers.