A few years ago, the list of bots that mattered to a typical website was short: Googlebot, Bingbot and a handful of SEO tools. Today, server logs also show crawlers from AI companies collecting pages for model training, for live answers in chat assistants, or both. Website owners are asked to decide what to allow — often without a clear picture of what each crawler does.

This article explains the main AI crawlers, what robots.txt can and cannot control, what llms.txt is, and how to make a decision that fits your business.

The main AI crawlers

Each company publishes the name its crawler uses, so it can be addressed in robots.txt. The most common ones:

  • GPTBot — OpenAI's crawler for collecting content that may be used to train its models.
  • ClaudeBot — Anthropic's crawler.
  • PerplexityBot — Perplexity's crawler for its answer engine.
  • Google-Extended — not a separate crawler but a control token. It tells Google whether content may be used to train and ground its Gemini models.

Some companies operate more than one user agent — for example, one for training and another that fetches a page when a user asks the assistant about it. The exact list changes, so check each company's documentation when you set your rules rather than relying on a list copied from somewhere else.

An important detail about Google

Blocking Google-Extended does not remove your site from Google's AI Overviews. AI Overviews are part of Google Search and follow the rules you set for Googlebot. The only way to keep pages out of them is to restrict them in Search itself, which also affects regular results. For most businesses that would be a much larger loss than any gain.

What robots.txt can and cannot do

robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may fetch. A rule for a specific bot looks like this:

User-agent: GPTBot

Disallow: /

Three limitations are worth knowing:

  • It is a request, not a lock. Reputable crawlers follow it; others may not. Content that must stay private needs authentication, not robots.txt.
  • It is not retroactive. Blocking a crawler today does not remove content it has already collected.
  • It controls fetching, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it. To keep a page out of the index, it must be crawlable and carry a noindex tag.

What is llms.txt?

llms.txt is a proposal published in 2024: a Markdown file at the root of a site that gives language models a short, clean summary of the site and links to its most useful pages. The idea is that a model reading a site would get a curated map instead of navigating menus and scripts.

It is important to be precise about its status. llms.txt is a proposal, not an adopted standard, and the major search engines have not committed to using it. Adding one is cheap and harmless, but it should not be expected to change rankings or AI visibility on its own. If you have limited time, a clear site structure, a valid sitemap and pages that answer questions directly will do more.

How to decide what to allow

There is no universal answer. The decision depends on how your business benefits from being read.

Usually allow

If you sell products or services and want customers to find you, visibility in AI assistants is a new route to the same customers. Blocking the crawlers that power those assistants removes you from their answers while protecting little — your product descriptions and service pages exist to be read.

Consider restricting

If your content is the product — paid research, original journalism, a database — training crawlers may take value without returning visitors. Many publishers block training crawlers while allowing those that fetch pages to answer a specific user question with a link.

Always restrict

Admin areas, internal search results, carts, accounts and staging copies should be closed to all crawlers. These rules matter regardless of AI.

Whatever you choose, write it down with the reason, and review it once or twice a year. The list of crawlers and their purposes keeps changing.

How to check your robots.txt

  1. Open it. Go to yourdomain.com/robots.txt. If it does not exist, all crawlers are allowed everywhere.
  2. Look for broad blocks. A Disallow: / under User-agent: * blocks everything — a common leftover from a staging site.
  3. Check resources. Blocking CSS and JavaScript folders can stop search engines from rendering pages correctly.
  4. Check the sitemap line. robots.txt should point to your XML sitemap with a Sitemap: line.
  5. Check rule order and spelling. Bot names must match exactly; a typo makes a rule silently ineffective.

In SEO Melody: Site Audit starts every crawl by reading robots.txt and the sitemap, so blocked sections and a missing sitemap show up in the findings. Index Coverage then lists pages that search engines are not allowed to index — blocked by robots.txt, marked noindex, pointing their canonical elsewhere or missing from the sitemap.

A note on being a good crawler

Crawlers are also judged by how they behave. A well-behaved crawler identifies itself, follows robots.txt and does not overload the server. Some sites set Crawl-delay to ask bots to wait between requests; not every search engine honours it, but it is a clear signal of the site's limits.

In SEO Melody: our own crawler follows the same rules it checks for. It respects robots.txt, including Crawl-delay, and limits how often it requests pages from a single host, so auditing a small site does not put load on its server.

Summary

AI crawlers are now a normal part of a website's traffic. robots.txt lets you decide which of them may read your pages, but it is a request rather than protection, and it does not control Google's AI Overviews. llms.txt is an optional extra, not a requirement. For most businesses, the sensible default is to stay readable, close what should never be public, and review the decision as the landscape changes.