When your bot protection turns AI visitors away
You have looked at your logs and found refusals going out to visitors with names like GPTBot or PerplexityBot. Something in front of your shop is turning them around. This is how to find out which setting did it, and then decide, on purpose, what you want to happen instead. The examples below use Cloudflare, because its settings are the ones named most often when shop owners report this.
1. Find out which setting did it
You can do this.
Do not guess, and do not start switching things off. Cloudflare Security Events can identify the service that acted on a recorded request. Depending on the plan and traffic level, the displayed events may be sampled.
In Cloudflare, open Security, then Analytics, then the Events tab. Each entry has a Service field, and that field names the product responsible. If it says Bot Fight Mode, that is your answer. If it says Managed rules, that is a different answer with a different fix.
One catch worth knowing before you go looking. On the free and Pro plans this event log keeps only twenty four hours. On Business it is three days. If the refusals you saw were last week, they are gone, and you will need to leave the page open or reproduce the visit.
2. What a refusal actually means
Cloudflare documents that when one of its security features blocks a request, the answer sent back is usually a 403, and that rate limiting sends a 429 instead. So the number itself is a clue about which family of setting you are in.
There is a third thing that is neither. A challenge is not a block. It is a puzzle handed to the visitor to prove it is a real browser. Vendor crawlers generally cannot complete a browser challenge, so the requested page may not be delivered. From your side that looks like a refusal too, and from the reader's side the page never arrives. The outcome is the same. The fix is not.
3. The four usual causes, in the order worth checking
Browser integrity check. This one is on by default, which is why it is first. Cloudflare describes it as challenging visitors that arrive without a name, or with an unusual one. A self-identifying reader sends a name, but that alone does not rule this setting out. Use the event's Service and action fields.
Bot fight mode, or Super Bot fight mode. The first is the free setting. Cloudflare describes it as issuing deliberately expensive puzzles to anything matching known automated patterns, and warns in its own documentation that it may challenge legitimate traffic. Read section 6 before you touch this one, because it behaves differently from everything else here.
The AI settings. Cloudflare has a control specifically for AI readers, and it has been renamed. The older one, called Block AI bots, is marked for retirement. The newer one is called Configure AI bot policies, and it is the more useful of the two because it splits AI visitors into three kinds: Search, Agent and Training. Each kind can be set to block everywhere, block only on pages carrying ads, or allow.
Rate limiting. This is the one that sends a 429 rather than a 403. A reader working through your catalogue quickly can trip it without doing anything wrong.
4. The default you never chose
Cloudflare has changed what happens by default, so a setting you never touched can be doing the deciding for you. Domains that join Cloudflare after that change have the Training and Agent kinds blocked by default on pages that carry ads, while Search stays allowed by default.
Two things about that are worth holding on to. Existing customers who block Training are affected as well: crawlers Cloudflare classifies as both Search and Training are blocked with them unless the customer opts out. So check the dashboard even on an older domain, and check it again after any plan change. And the split it uses is the same three-way split you should be thinking in anyway.
5. The decision is three decisions
Developer needed to carry it out, but the decision is yours and nobody else can make it.
This is where most people go wrong, because they treat AI readers as one thing and set one switch. The three kinds do genuinely different jobs.
Training readers collect pages that may go into building the models. A training-only control is separate from a search-only control. Cloudflare also classifies some crawlers as serving both purposes, so a Training block can take those crawlers with it.
Search readers gather pages so an assistant can show and link your shop when somebody asks. This is the group a shop usually wants, and it is the group a blanket block quietly removes.
Agent readers fetch a page because a person just asked for it in real time. Turning these away means that when a shopper points an assistant at your product page, nothing arrives.
If your shop wants to be findable, the honest setting for most stores is to allow Search, think hard about Agent, and please yourself about Training. What you should not do is leave all three answered by a default you never chose.
6. Letting a reader back in, and the one that will not let you
Developer needed.
For most of these settings the way back is ordinary. Cloudflare's per-reader control, called AI Crawl Control, is available on all plans and lets you set a named reader to Allow. Super Bot Fight Mode can be skipped with the documented rule action. Free Bot Fight Mode cannot be skipped by a WAF custom rule; an IP Access rule can instead take precedence.
Then there is the exception, and it catches people out constantly.
Bot fight mode, the free one, cannot be excepted. Cloudflare states plainly in its own documentation that you cannot skip it with a firewall rule. It does not run in the same place as the other rules, so an allow rule you add will simply have no effect on it. If that setting is what your event log named, your real choices are to turn it off, to allow the reader by its network address instead, or to move to a plan with the more configurable version.
That is worth knowing before you spend an afternoon writing a rule that was never going to work.
What this does not do
Finding the setting is not the same as fixing the shop. A reader that gets in still has to find your prices and your stock in a form it can use, and that is a different playbook.
None of this tells you whether an assistant ever tried to visit. Your event log shows what was refused, not what was never attempted.
And these names move. The AI control has been renamed at least once, and the defaults for new domains have changed. Everything above describes how the controls behave rather than what a screen is called this month. If a name in this article does not match what you see on your screen, believe your screen.
Last, this is Cloudflare specific. Other providers use different controls and evaluation orders. Start with that provider's event log and documentation.
Show the technical detail (for your developer)
Cloudflare returns 403 for most security-feature blocks and 429 for rate limiting rules. A managed challenge is identified by the cf-mitigated: challenge response header rather than by a distinct status code.
Evaluation order on the Ruleset Engine is DDoS L7, then custom rules, then rate limiting, then Managed Rules, then Super Bot Fight Mode. Browser Integrity Check, IP Access Rules, Zone Lockdown, User Agent Blocking and Security Level are evaluated outside that pipeline.
Bot Fight Mode does not run on the Ruleset Engine, so Skip, Bypass and Allow actions have no effect on it. IP Access Rules do take precedence over it.
For a WAF custom rule matching verified bots, use cf.client.bot on all plans and cf.verified_bot_category to match a specific category. The cf.bot_management.verified_bot and cf.bot_management.score fields require the Bot Management Enterprise add-on.
AI Crawl Control enforces a per-crawler block by creating or updating a WAF custom rule on the zone, and can respond 403 or 402. The 402 path is pay per crawl, which is in closed beta.