If your WAF (web application firewall) blocks non-Google bots, you are blocking your own customers.
That sentence used to be hyperbole. After Google I/O 2026, it is operational reality. On May 19, Sundar Pichai opened his keynote by framing the moment as "the agentic Gemini era". The next day, Parisa Tabriz, VP of Chrome, announced Gemini auto-browse in Chrome, an AI agent that fills forms, navigates pages, and completes purchases on the user's behalf, identified by user-agent strings that are not Googlebot.
Combined with Gemini Spark (the 24/7 personal agent rolling out to Ultra subscribers) and the broader set of agents running on browsers Google does not control, we now have a third class of visitor to your website. It's a third kind of visitor: not a human and not a crawler, but an agent acting for a human.
The defensive posture most sites have inherited from a decade of bot wars treats anything non-human as hostile. That posture is now hostile to your customers.
Three flavors of non-human traffic, and they're not interchangeable
The first thing to get straight is that "bot" is no longer a single category. The relevant taxonomy has three columns now.
Googlebot is the crawler that indexes the web for traditional Search. It identifies itself with a Googlebot user-agent string and obeys robots.txt. You have always had a clear relationship with it: allow it, disallow it, or partially restrict it.
Google-Extended is the user-agent token Google uses for AI training and AI Overview generation. Per Google's crawler documentation, you can disallow Google-Extended in robots.txt without affecting Googlebot. The catch: disallowing Google-Extended is the same as opting out of AI Overview citation. That's a tradeoff, not a default.
Agents acting on behalf of users is the new column. These are not Googlebot. They are not Google-Extended. They are Gemini, or Claude, or a custom agent built on the Model Context Protocol, executing keystrokes that a real human asked them to execute. They do not identify themselves with a unified user-agent string, since different agents use different conventions: some declare themselves clearly, some look like a regular Chrome session because they are running inside a regular Chrome session.
Crawling, training, and acting are three different intents with three different signatures, and one legacy WAF rule blocks them all because they don't have eyeballs.
How to identify agent traffic without breaking your customers
> The user is real. Only the keystrokes are automated. Blocking the keystrokes blocks the user.
Identification has to lean on behavior first, user-agent second. Here is what we tell clients to look for.
User-agent strings. Some agents declare themselves cleanly: Anthropic's user-agent, OpenAI's user-agent, Google's various Gemini fetchers. Maintain a maintained allowlist of declared agentic user-agents. Anthropic publishes theirs. OpenAI publishes theirs. Google's are documented at google.com/bot.html. When you see one, treat it as a legitimate visitor.
Behavioral signatures. Auto-browse agents have a distinctive pattern: they read the page structure quickly, fill forms in unnatural rhythm, and pause at known sensitive checkpoints. They don't generate mouse movements between fields. They don't have idle dwell time. That looks like a bot to a 2018-era fingerprint engine. It also looks like exactly what your customer asked the agent to do for them.
The IP doesn't help much. Chrome auto-browse runs inside the user's own browser, so the IP is the user's IP. Spark runs on Google Cloud VMs, so the IP is a known Google range, but the agent is still acting for a specific authenticated user. Neither IP class should be treated as suspicious by default.
Why blocking is dangerous in 2026
The framing has flipped. Five years ago, a blocked bot was a small win against scraping. Today, a blocked bot is often a failed checkout, a missed inquiry, or an aborted booking from a customer who delegated the task.
Consider the path. A user opens Chrome, asks Gemini to "find me a plumber and book an appointment for Saturday morning." Gemini queries Search, gets a few candidates, opens one of their websites, and tries to submit the contact form. Your WAF flags the form submission as bot traffic and serves a CAPTCHA. The agent, which has no human watching at that moment, moves to the next candidate.
You did not stop a scraper. You stopped a customer.
The pattern compounds: agentic booking, agentic calling, agentic checkout. Per Tabriz's Chrome announcement, auto-browse can already "search for similar items" across retail sites and add to cart. Information Agents will continuously monitor sites for changes their user cares about. Spark will plan multi-step workflows in the background. Every one of those touchpoints is a moment where defensive blocking translates directly to lost revenue.
A robots.txt strategy for the agentic era
There are two real decisions to make explicit in your robots.txt.
Decision one: do you want to be cited in AI Overviews? If yes, allow Google-Extended. Most small businesses should, since AI Overviews reach 2.5 billion monthly active users per the I/O 2026 keynote, and citations from Overviews drive disproportionately high-intent traffic. The cost is your content being used to ground AI summaries. The benefit is the citation itself. We have written about how citation is becoming the new position one and the math favors allowing.
Decision two: do you want agentic shoppers to be able to transact with you? If yes, allow the major declared agentic user-agents in robots.txt and at your WAF. Anthropic's, OpenAI's, Google's Gemini fetchers, and the agents emerging from the WebMCP standard. An agentic shopper behaves nothing like a scraper: it visits one or two pages, submits a form, and leaves, which means the traffic volume stays low while the conversion intent runs high.
Block neither, allow both, and you've covered the ground that matters.
Accessibility-first forms are agent-ready by default
Here is the convenient truth: the same things that help a screen reader help an agent. Semantic HTML, real form labels associated with their inputs, ARIA roles where they add meaning, a sensible tab order, and submit buttons that actually submit are exactly what both of them need.
If your contact form was built in 2015 with a The fix is the same fix accessibility consultants have been recommending for fifteen years: build forms with native HTML elements, use the The Chrome team's Modern Web Guidance, which they publish specifically for AI coding agents to follow when generating code, endorses the same patterns: native HTML elements like Most analytics tools count agent-driven sessions the same as human ones. That's a mistake. The behavior is different. Bounce rate looks unusual because the agent often loads one page, extracts what it needs, and moves on. Time on page looks low. Pages per session looks low. If you don't segment, those agent sessions drag your apparent engagement metrics down and obscure what humans are actually doing. Our recommendation: add a custom dimension to your analytics that tags sessions whose user-agent matches a known agentic signature. In GA4, that's a custom dimension on the session scope, populated from a parsed user-agent value. Then build a separate dashboard for the agent segment. You'll see what humans do, what agents do, and how your booking funnel performs for each. You may also discover something useful: agents often have higher purchase-completion rates than human sessions because they don't abandon. If your numbers ever show agentic checkout converting at three or four times your human rate, that's not noise, it's the next decade of traffic showing up early. Is your site invisible to AI search? Get a free AEO infrastructure audit and find out what your competitors are doing that you're not. 5 links · External AEO & AI Search WebMCP is the Chrome-proposed standard that lets your site declare actions agents can invoke. If you don't ship it, your competitors will AEO & AI Search Information Agents are background agents Google runs for users in summer 2026. If your content updates frequently, an agent might check you every hour AEO & AI Search Personal Intelligence connects Gmail, Photos, and Calendar to AI Mode, which means a user's own inbox is now a ranking signal Googlebot will never read, it is hostile to screen readers AND hostile to agents. A screen reader and an agent will both struggle, both fail, and convert at zero. element, match every input with an associated label, and make every interactive element keyboard-operable. The agentic web rewards exactly the accessibility work that human-first design also rewards, so there is no trade-off here. and the Popover API, WebAuthn over passwords, semantic structure. Those same patterns make a site legible to agents for the same reason they make it legible to people.Segment agent traffic in your analytics
Sources
Further Reading
Industry sources we cite.
Frequently asked.
Continue with.
WebMCP: the standard your site needs (that you've never heard of)
Information Agents: the always-on Googlebot you didn't ask for
Personal Intelligence: the ranking layer crawlers can't see