Bots Now Outnumber Humans Online. The Real Question Is Whether They Can Use Your Catalog.

In June 2026, the make-up of internet traffic crossed a line that had been forecast for years. Public reporting around Cloudflare Radar traffic data showed automated agents and bots generating about 57.4% of global web requests, with humans down to about 42.6% — and Cloudflare CEO Matthew Prince noted publicly that he had expected the crossover in 2027, not mid-2026. Separately, HUMAN’s 2026 State of AI Traffic benchmark reported that AI-driven traffic nearly tripled in 2025 and that automated traffic is now growing several times faster than human traffic. The web quietly became something machines use more than people do, ahead of schedule.

The reflexive response has been to pull up the drawbridge. Coverage of Microsoft’s agentic-commerce updates reported that around 80% of websites are blocking these agents. That’s an understandable instinct — bot traffic carries cost and risk, and not all of it is welcome. But for anyone selling products, blocking is the easy half of a decision whose hard half almost nobody is discussing: when an agent you do want — a procurement assistant, a shopping agent sourcing on a buyer’s behalf — reaches your catalog, can it actually use what it finds? Most catalogs fail that test whether they block bots or not.

Access and usability are two different problems

There are really two questions hiding inside “what do we do about AI bots,” and they’re easy to conflate. The first is access: should an agent be allowed to reach your pages at all? That’s a security and policy call, and blocking indiscriminate scrapers can be the right one. The second is usability: once an agent you want is in, can it parse your products into something it can act on? You can get the first one right — allow the good agents, block the bad ones — and still lose, because the agents you allowed hit a catalog they can’t make sense of and route around you anyway.

This is the part the block-or-allow debate misses. When an agent can’t work with a brand’s products, it can’t discover them, recommend them, or create demand for them. Blocking turns you invisible to agents on purpose. An unusable catalog turns you invisible to them by accident. The second is the one quietly costing product businesses, because it feels like you’ve done the work — the agent got in — when the actual barrier is one level deeper, in the data.

For the visibility side of the problem, start with our guide to AI visibility for product catalogs. For the data-readiness side, use the AI-ready product data guide as the operating checklist.

Why “the agent got in” isn’t the finish line

Letting a good agent reach your catalog only matters if the catalog gives it something to act on. For a weak-standard, multi-supplier catalog, that’s exactly where it breaks down. The agent reaches a product that exists as two duplicate records under different supplier part numbers, with a spec in the wrong units and a key attribute blank. It can’t confirm the product matches the request, can’t trust the stock figure, can’t tell the duplicates apart — so it does the safe thing and recommends a competitor whose record is unambiguous.

You allowed the agent. You still lost the recommendation. The bottleneck was never access. It was a catalog that isn’t legible to software acting on its own.

The fix isn’t a crawler policy. It’s the data underneath: one canonical, trusted record per real-world product, with duplicates resolved, units reconciled, attributes complete, and confidence and provenance attached — kept current as supplier feeds change. That’s what turns “the agent reached us” into “the agent can use us.” A crawler rule decides who gets in. Catalog data quality decides whether getting in was worth anything.

This is why the canonical product record is becoming part of agentic commerce infrastructure. Agents don’t need five partial supplier versions of the same item. They need one trusted version they can parse, verify, compare, and act on.

What to actually do about it

Treat the two problems separately and don’t let the first distract you from the second.

1. 1
   Segment AI bot traffic by intent

   Separate known search crawlers, shopping agents, procurement agents, security scanners, partner systems, scrapers, and suspicious spoofed identities. A single allow-or-block rule is too blunt for a channel this commercially important.

2. 2
   Allow the agents that can create demand

   If an AI crawler or agent helps buyers discover, compare, or source products, treat access as a commercial channel decision rather than only a security decision.

3. 3
   Block the traffic that only creates cost or risk

   Indiscriminate scraping, abusive automation, and spoofed identities still deserve controls. Selective access is different from leaving the catalog open to everything.

4. 4
   Audit whether an allowed agent can use the catalog

   Test the records an agent would actually read: duplicate products, unit conflicts, blank attributes, stale stock, missing provenance, and identifiers that don’t resolve cleanly.

5. 5
   Maintain trusted product data continuously

   Resolve product identity, validate and enrich the canonical record, write trusted values back into ERP and PIM, and keep monitoring as supplier feeds change.

On access, make a deliberate choice: allow the agents that drive demand, block the ones that only cost you, and don’t reflexively wall off the channel that’s growing several times faster than human traffic. On usability — the one that actually determines outcomes — get your product data to a state an allowed agent can act on. That’s where Claro works: continuously resolving product identity, validating and enriching the canonical record, and writing trusted data back into your existing systems, so the agents you let in find a catalog they can read and recommend instead of one they skip.

The traffic shift isn’t a forecast anymore; it already happened. The catalogs that benefit from it are the ones that are both reachable and usable. Most are neither, or stop at the first. Getting the second right is the part still up for grabs.

We’ll show you, on your real data, what an allowed agent can parse — and where duplicates, unit conflicts, and missing attributes would make it route around you.

Sources and article inspiration

This article was shaped by the same traffic and agentic-commerce signals catalog teams are now reacting to:

Related Claro resources

FAQ

Claro keeps multi-supplier product and supplier data clean, matched, enriched, and validated across your existing ERP, PIM, and procurement systems. Your catalog should get smarter every day.