Menu
Playbook

Is your firewall hiding your store from AI tools?

The protection that keeps attacks off your store can also turn away the AI tools shoppers use, without telling you. Here is how to check yours today.

Benjamin Ong 6 min read
LAYER 1 · back field, diagonal security scene · 1300 × 560 LAYER 2 · filter plane and request stream LAYER 3 · storefront, verified paths, paused reader slot: article-firewall-gate · 650×280 · 3 layers · generation pending
A store firewall checks several automated visitors, allowing verified requests while one useful catalogue reader is stopped for review.

Nothing in your admin will ever tell you this happened. No email, no warning banner, no line on your orders screen. A guard standing in front of your store looked at a visitor, decided it looked like trouble, and turned it away at the door. Most days that guard is right and you are glad it is there. Some days the visitor it turned away was the reader an AI shopping tool sent to look at your products.

That is the whole problem in one sentence. The protection works, it just cannot always tell which automated visitors you actually want.
CONTEXT

The guard you probably never installed

A firewall is a filter that sits in front of your store and decides who gets through. Most owners never chose one on purpose. It arrived with the hosting plan, or with a security plugin somebody installed years ago, or with the service that speeds your site up and happens to screen traffic on the way past.

Its job is to spot obvious bad behaviour, a flood of requests from one place, an address already known for attacks, a script poking at your login page, and stop it before it reaches your store. Stores get probed constantly, and most owners never notice because this is being handled quietly on their behalf.

None of that is the problem. The problem is what happens to the visitors that are neither obviously good nor obviously bad.

CONTEXT

Reading twenty product pages in a minute looks like an attack

Alongside the firewall there is usually a second, narrower rule that simply counts. It watches how often one visitor asks for pages, and when that count climbs faster than a human browsing would, it assumes an attack and shuts the visitor out. In your settings it is often labelled "rate limiting".

Now picture an AI tool checking your catalogue. It may open your homepage, then a category, then several product pages faster than a person would browse them. That is normal for software. To a rule that only counts requests, it can still look like abuse. Cloudflare's own guidance lists both rapid requests and automated traffic as common reasons a request is blocked.

So the visitor can get cut off partway through. The failure may never appear in your store admin. The tool may move on having read a fraction of your catalogue, or none of it.

CONTEXT

A rule that only knows the old names

Automated visitors are not all the same, and protection services often keep lists of the ones they have verified. The service checks more than a claimed name, because names can be copied. For example, Cloudflare verifies known automated visitors using signed requests, published network addresses, or reverse lookups.

The readers that AI tools send are newer. Some of them did not exist when your protection was configured, and a rule written to allow "the ones we know" quietly turns away everything else. Nothing breaks. No error appears anywhere you would look. Your store simply becomes unreadable to a growing group of shopping tools, and the first evidence you get is no evidence at all.

LAYER 1 · three inspection zones · 1300 × 560 LAYER 2 · rule sheet, filter console, host trail slot: article-firewall-diagnosis · 650×280 · 2 layers · generation pending
Three inspection points show the public robots file, the protection service, and the hosting layer where an automated visitor may be blocked.
DECIDE

The trade-off is real, and it belongs to you

Being well protected and being widely readable pull against each other. A tighter setting keeps out bad traffic and makes it harder for anyone to copy your entire catalogue in one sitting, and it also turns away AI tools you might have wanted reading your products. A looser setting keeps you readable to everyone, including the visitors you would rather not have.

There is no setting that gives you all of one and none of the other, and anyone telling you otherwise is selling something. Plenty of stores look at that trade and deliberately keep the door narrow, because the catalogue is the business and protecting it comes first. Plenty of others open it wider on purpose. Both are defensible.

The only genuinely bad outcome is the one where nobody chose.A default from a plugin you installed in 2023 is currently making this decision for your store, and it has never once asked you about it.
CHECK

How to check what your store is doing right now

Three checks, easiest first.

  1. Open your robots.txt file.

    This is a plain text file at yourstorename.com/robots.txt that lists rules for automated visitors, telling each one what it may and may not read. Type that address into a browser and look for lines naming an AI tool followed by "Disallow". A line reading "Disallow: /" means that visitor is shut out of your entire store.

  2. Look inside your security plugin or protection service.

    Find its list of blocked or challenged visitors, and any setting named something like "bot protection" or "rate limiting". Defaults and available controls vary, so review the setting that is actually active on your store.

  3. Ask whoever hosts your store.

    Some blocking happens before a request ever reaches your own files, at the server or network level. Your robots.txt can look completely open while a rule you cannot see is turning visitors away underneath it. Your host or protection provider can read that out for you, because you genuinely cannot see it from your own admin screen.

Check your own store.

You can have this answered for you in about a minute. The free scan reads your store the way an AI tool would, checks your robots.txt, and tells you plainly whether something turned it away or whether it got a clear read. No plugin, no card, no signup. Run the free scan.

WHAT IT MEANS

Why a blocked check is not a failed check

When our scan cannot get a clear read on part of your store, because something stopped it partway, we do not guess and we do not mark the check as broken. We leave that check out of your score entirely and work the score out again from the checks we could actually finish.

That sounds like the soft option. It is the honest one. Broken means we saw a real problem on your store. Unreadable means we could not see well enough to say either way, and those are different situations with different fixes. A store that is genuinely unreadable to software and a store we simply were not allowed to finish reading should not carry the same label, so they do not.

If a check comes back that way, treat it as a lead rather than a let-off. Something stood between an automated reader and your pages, and it is worth knowing what.

DECIDE

Turning the accident into a decision

The goal is not to talk you into opening your store to everything. The goal is to make sure the setting you are running today is the one you would choose today.

If you check your robots.txt and your protection settings and conclude that keeping certain automated visitors out is right for your store, keep it, and now you know you meant it. If instead you find a rule turning away an AI tool nobody meant to block, that is worth fixing. Many protection tools support a narrow exception, but the name alone is not enough because it can be copied. Use the verification method the company publishes, such as its current network address list, and keep the exception limited to the pages that reader needs. Your host or your developer can make that change once you tell them exactly which rule to look at.

Start with robots.txt, since it takes thirty seconds and needs no login. Then your security settings. Then your host, for the part you cannot see.

Or let the scan do the first pass for you. It reads your store the way an AI tool does, and where it cannot tell for certain, it says so instead of guessing on your behalf. Run the free scan and find out which door your store is holding open.