robots.txt for AI: allow, block, or decide

One small public file already tells AI assistants whether they may read your store. Here is how to open yours, understand it, and make the call on purpose.

Your store already has an answer to the question of whether AI assistants may read your catalogue. It was written into a small text file, quite possibly by a plugin, a theme, or a developer you no longer work with, for a reason that had nothing to do with AI. You can read that answer in under a minute, and you should, because right now it is a decision that was made for you.

Open yours first

Type your store's web address into a browser, add /robots.txt on the end, and press enter. Something like yourstore.com/robots.txt. No login, no admin screen, no plugin.

What loads is a short list of plain text lines. It is public by design, the same way a shop's opening hours in the window are meant to be read by anyone walking past, and every site on the internet can be checked this way, including your competitors'.

What the file is doing

robots.txt is the sign at the entrance for automated visitors. It lists rules by visitor name, saying which parts of your site each named visitor may read and which parts it should leave alone. Search engines read it, and many companies publish separate names for programs used in AI training, search, and user-requested visits.

The whole thing runs on good manners rather than locks. Nothing in the file physically stops anyone, it simply states your wishes. Published training and search programs from OpenAI, Anthropic, Google, and Perplexity follow these rules. Some visits made at a person's request are different. OpenAI says its ChatGPT-User rules may not apply, and Perplexity says its Perplexity-User fetcher generally ignores the file.

How to read a rule

Each rule has two halves. The first names a visitor, and the second says what that visitor may or may not fetch.

A rule naming GPTBot followed by an instruction to stay away from everything means OpenAI's program is not reading your store. A rule that names no one in particular, using a general marker instead, applies to every visitor that does not have a rule of its own further down. Plenty of stores have exactly one general rule, keeping automated visitors out of admin pages and checkout, and nothing about AI at all. On those stores, AI visitors are welcome by default, which is usually what the owner wanted anyway.

Files often mix both kinds, a general rule for everyone plus specific rules naming particular visitors. When both could apply, the rule that names a visitor by name is the one that decides that visitor's access, and the general rule is set aside for them. This is exactly the sort of thing that is easy to misread while skimming a file you have never opened before, which is why checking the actual result beats reasoning it out by eye.

There are nine current controls worth checking. OpenAI publishes GPTBot for possible model training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for visits a person requests. Anthropic publishes ClaudeBot for possible model training, Claude-SearchBot for search, and Claude-User for visits a person requests. Perplexity publishes PerplexityBot for search and Perplexity-User for visits a person requests. Google-Extended is different. It is a control in robots.txt for Gemini training and grounding, not a separate visitor name that appears in your server records. Each company explains its controls in its own documentation: OpenAI, Anthropic, Perplexity, and Google.

The block nobody chose

The awkward case is not the deliberate block. It is the rule written broadly to stop unwanted traffic, or added by a security plugin during a rough week, that turns out to cover AI visitors as well.

The file cannot tell the difference between traffic you meant to turn away and traffic you never thought about. It does what the rule says. And because a blocked AI visitor changes nothing a shopper can see, there is no moment where anyone notices. The store looks fine, sells fine, and is quietly unreadable to one whole category of visitor.

What blocking actually means

The effect of a block depends on the control. Blocking a training program tells the company not to use that crawl for future model training. Blocking a search program can keep the page out of that company's search answers. A user-requested fetch may follow different rules, and blocking one company does not stop another company or an ordinary browser from reaching a public page.

Allowing a search or training program means that specific program may read the pages covered by the rule. It does not say whether the company will use them, and it does not make a training program part of a shopping answer. Whether a page is used is decided on that company's side. Allowing is a starting condition, not an outcome.

Neither choice is automatically the right one. Some owners block on purpose, because they would rather a shopper arrived at their site and read their words than have their product details summarised inside someone else's chat window, and that is a coherent position held for a real reason. Others want search and user-requested readers to have a clear route to their public pages, on the grounds that a tool cannot use facts it never receives.

This article is not here to pick for you. It is here so that the choice is actually a choice.

Your data stays yours either way

It is worth being clear about what this file is and is not, because the topic attracts a lot of noise.

Blocking AI visitors does not protect your prices from competitors, who can open your product pages in a browser like anyone else. Allowing AI visitors does not hand your catalogue over, in the sense of losing control of it, because a program is reading the same public page a shopper could already see.

The underlying facts about your products, what they are, what they cost, and whether they are in stock, belong to your store and stay there. What you actually own, and what is worth protecting, is the source those facts come from. The file is a switch for one kind of visitor, not a lock on your business.

Making the change

Related playbooks