Preparing your site for AI search comes down to three layers: deciding in robots.txt which AI crawlers may access what, describing your content with structured data so machines can interpret it, and, optionally, offering models a curated summary of your site through llms.txt. Only the first two are established practice. llms.txt is still a proposal, and not a file every engine reads.
Answer engines such as ChatGPT, Gemini, Perplexity and Google AI Overviews rely on several crawlers with different jobs. Some collect data to train models, some build a search index, and some fetch your page live at the moment a user asks a question. Which of them you let in, and which you turn away, has a direct effect on whether you show up. Here is our field checklist.
What are AI crawlers actually doing on your site?
A classic search crawler indexes and ranks your pages. On the AI side there is no single bot; providers split their crawlers by purpose. Knowing the split lets you ask a better question than "should we block AI?"
- Training crawlers collect content to train large language models. Blocking them limits the use of your content in future model versions, but on its own it does not decide your visibility in search answers.
- Search and indexing crawlers feed the answer engine's own search index. Block these and the engine will struggle to cite you in up-to-date answers.
- User-triggered agents fetch a page on demand, when someone pastes a link into a chat or asks something that needs fresh information.
In short, opting out of training is not the same as opting out of search. Many brands choose the first and keep the second open, because the whole point of an SEO, SEM and GEO strategy is to be the source those answers cite.
Managing GPTBot, ClaudeBot and other AI crawlers with robots.txt
robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may fetch. These are the user-agent names you will meet most often:
- GPTBot (OpenAI): crawls content for model training.
- OAI-SearchBot (OpenAI): used to surface sites in ChatGPT search results. OpenAI states it is not used to train its models.
- ChatGPT-User (OpenAI): fetches a page when a ChatGPT user needs it. Because these requests are user-initiated, OpenAI notes that robots.txt rules may not apply to them the way they apply to automatic crawling.
- ClaudeBot (Anthropic): crawls content for model training. Anthropic also uses Claude-SearchBot for search and Claude-User for user-initiated requests.
- PerplexityBot (Perplexity): used to surface sites in Perplexity search results. User-initiated requests appear as Perplexity-User.
- Google-Extended (Google): not a separate crawler but a control token in robots.txt. It decides whether content Googlebot has already crawled may be used to train Gemini models and to ground their answers. It does not affect your ranking in Google Search or your inclusion in AI Overviews, which follow the regular Googlebot rules.
A site that wants to stay out of training but remain visible in AI search would, for example, set Disallow: / for GPTBot, ClaudeBot and Google-Extended, and Allow: / for OAI-SearchBot, Claude-SearchBot and PerplexityBot. Admin areas, carts and account pages should stay closed to every bot.
Two caveats. First, robots.txt is a request, not access control. Well-behaved crawlers follow it; others may not. Anything that truly needs protecting belongs behind authentication. Second, crawler names and behaviour change over time. Check each provider's current documentation before you write rules, and look at your server logs to see which bots really show up.
What is llms.txt, and does it work?
llms.txt is a file format proposed in 2024 by Jeremy Howard of Answer.AI. The idea is simple: put a short, Markdown-formatted guide at the root of your site that a language model can read quickly. The proposed structure usually looks like this:
- A heading with the name of the site or project.
- A one- or two-sentence summary.
- Lists of links to key pages with short descriptions: services, products, documentation, contact.
- Optionally, an llms-full.txt that gathers the detailed content into one file.
To be clear, llms.txt is a proposal today, not a standard. There is no shared agreement around it the way there is for robots.txt or sitemap.xml, and none of the major search or answer engines has publicly committed to using it for ranking or citation. Some developer tools and documentation platforms have adopted it, and some agents can read it, but "we added llms.txt, so AI will find us now" is not a realistic expectation.
That said, it is cheap. A well-written llms.txt forces your team to state clearly what the site is about, and if the proposal gains traction you are already prepared. We treat it as a small improvement added after the fundamentals are in place, never as a substitute for them.
AI can only describe you as well as it understands you. Clarity first, files second.
Structured data: the language machines understand
For an answer engine to describe you correctly, your content has to be interpretable, not just readable. Structured data written with the Schema.org vocabulary, ideally as JSON-LD, does that job. The types we prioritise:
- Organization: company name, logo, contact details and social profiles. It defines who your brand is in one place.
- Article or BlogPosting: headline, published and modified dates, publisher. Important as a freshness signal.
- BreadcrumbList: shows where a page sits within the site.
- FAQPage: marks up question-and-answer sections explicitly, close to the format answer engines like to quote.
- Product, Service or LocalBusiness: describes what you sell and where you operate.
Markup must match what is visible on the page. Putting information into JSON-LD that the page itself does not show goes against search engine guidelines and erodes trust. Treating this as part of your software foundation rather than a one-off plugin setting keeps the markup from quietly breaking every time a template changes.
The technical baseline: can crawlers actually read your pages?
Content only helps if the crawler can see it:
- Server-rendered content: many AI crawlers read raw HTML and may not execute JavaScript. Main copy, headings and links should be present in the initial HTML response.
- An up-to-date sitemap.xml: list your important pages with last-modified dates, and reference the sitemap from robots.txt.
- A clean heading hierarchy: a sensible H2 and H3 structure on every page, with paragraphs that answer questions directly.
- Speed and stability: slow or failing pages are a problem for every crawler.
- Firewall settings: bot protection at the CDN or WAF layer can quietly block crawlers you have allowed in robots.txt.
- Log monitoring: regularly check which pages AI crawlers request in your server logs. That is how you confirm your rules work.
These steps are the technical side of GEO. We cover the content side, writing with the clarity answer engines want to quote, in What Is GEO?
Frequently asked questions
Will adding llms.txt improve my rankings?
You should not expect it to. llms.txt is a proposal, and the major search and answer engines have not announced using it as a ranking signal. Treat it as a low-cost addition, not a replacement for the fundamentals.
If I block GPTBot, will I disappear from ChatGPT?
Not directly. GPTBot crawls for training; appearing in ChatGPT search depends on allowing OAI-SearchBot. You can manage the two separately in robots.txt.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls the use of your content in Gemini models. AI Overviews are part of Google Search and follow Googlebot rules, and blocking Googlebot would remove you from search altogether.
Which structured data type should I start with?
Start with Organization, then add Article and BreadcrumbList to your content pages. If your pages have question-and-answer sections, FAQPage markup makes sense too.
Where to begin?
Open your current robots.txt and your server logs first, and see which bots arrive and what you allow. Then make a deliberate decision about training and search crawlers, add Organization markup, and confirm your key pages can be read without JavaScript. llms.txt comes at the end of that list.
In our SEO, SEM and GEO work we run this technical audit alongside strategy and content. For the wider picture, read SEO, SEM and GEO in the Age of AI Search. If you want to review how ready your site is for AI search together, get in touch with us.