Produced byAbout BYLT Media
★ Produced by BYLT MEDIA · Paid Media Built for Growth ★
★ Community Edition ★Price: Free



Produced by
Paid Media · News

Google publishes guidance on allowing Google-Agent traffic

Google's new guide on Google-Agent says CDN, WAF and bot rules can block legitimate agent requests. Check your rules before user-triggered agents meet a challenge.

A checked Google-Agent ticket follows an arrow through a raised security barrier labelled “skip bot challenges” towards a website page.
Bot rules decide whether Google-Agent requests reach your site. Illustration: The SEM Dispatch.

If your CDN, web application firewall (WAF) or bot-detection service challenges Google-Agent, requests made on a user's behalf can be blocked on your site. On 17 November 2026 Google published a technical guide on allowing that traffic, and it names the rules most likely to get in the way. The people who need to read it are whoever owns bot rules for the sites you send paid and organic traffic to.

What changed

Google's crawling documentation changelog records the addition on 17 November. In its own words, Google "Published a technical guide on how to ensure that Google-Agent has access to your website." The stated reason is "To help publishers that want to enable Google-Agent for their website."

Documented now

  • Google describes Google-Agent as the name used by agents hosted on Google infrastructure to move around the web and perform actions upon user request.
  • Each request carries a User-Agent HTTP request header containing "Google-Agent" and comes from an IP address listed in user-triggered-agents.json.
  • Google says explicitly allowing this traffic in your CDN, WAF or bot-detection service helps prevent legitimate requests being blocked while preserving your existing security controls.
  • Some bot-detection services, CDNs and firewalls verify Google-Agent requests themselves. The guide says to refer to your provider for the exact steps.
  • If yours does not, the guide suggests four rules: confirm the User-Agent header contains Google-Agent, confirm the IP address is in user-triggered-agents.json, confirm any signed request (see below), and skip bot challenges for verified requests.

Experimental

Google says it is experimenting with the Web Bot Auth protocol using the https://agent.bot.goog identity. In the experiment, Google signs Google-Agent requests with the HTTP Message Signatures standard (RFC 9421). Participating requests include a Signature-Agent header set to g="https://agent.bot.goog", plus Signature and Signature-Input headers labelled g. The public key set is published at https://agent.bot.goog/.well-known/http-message-signatures-directory. The guide says to verify signatures only "if provided", so unsigned requests are still expected.

Background from the changelog

The Google-Agent user agent and the IP ranges for user-triggered agents were added to the documentation on 20 March 2026. That entry said the user agent was "rolling out over the next few weeks". On 4 May 2026 Google added Web Bot Auth documentation, saying some Google user agents were starting to use it during the experimental phase.

The guide's troubleshooting section lists the usual causes when allowed requests are still blocked. They are covered in the checks below.

What this means for specialists

The guide is written for people who control security rules, not for media buyers. The practical consequence lands on journeys you pay to send people into. The guide names on-site search, filtering, clicking into detail pages, add-to-cart and booking as actions an allow rule needs to cover, along with the underlying XHR, GraphQL or api.* endpoints. An agent acting for a user that gets stopped partway through one of those steps has nothing to complete.

The guide also says an allow rule can be overridden. It warns about broad cloud or data centre ASN blocks, managed bot rules and category-level AI agent block policies. A site that added an AI agent block policy in good faith may be blocking Google-Agent without anyone having decided that. Whether to allow it is a business decision, and the guide is aimed at publishers who want to.

Rate limits are a separate trap. Google notes that concurrent requests from multiple users share the same egress pool, so a per-IP limit can treat many users' agents as one noisy visitor.

This is an implication, not a measured effect. The sources give no data on how much Google-Agent traffic exists or how often it is blocked.

AdvertisementOpteoLearn more

What to do this week

Each of these is a short check for whoever owns the CDN, WAF or bot-detection service. Ask the client's dev team, or your own, to open the following.

  1. Provider settings. Check whether your CDN, WAF or bot-detection service verifies Google-Agent requests automatically. If it does, follow the provider's steps. If not, move to the next check.
  2. Manual allow rule. Look for a rule that confirms the User-Agent header contains Google-Agent and the IP address is in user-triggered-agents.json, and skips bot challenges for verified requests.
  3. Rule priority. Confirm the allow rule evaluates before broad cloud or data centre ASN blocks, managed bot rules and category-level AI agent block policies.
  4. Challenges. Check the rule skips interactive CAPTCHAs and JavaScript challenges as well as HTTP blocks, including standalone in-page CAPTCHA widgets triggered on search or form submission.
  5. Journey coverage. Walk through on-site search, filtering, detail pages, add-to-cart and booking, and check the XHR, GraphQL and api.* endpoints behind them are covered too, not just the homepage load.
  6. Rate limits and range updates. Check per-IP rate-limiting rules account for or exempt the verified Google-Agent ranges, and that IP-based rules stay synced with the latest user-triggered-agents.json.
  7. Proxy headers. Confirm load balancers and intermediate proxies preserve the original client IP and the User-Agent header, and, for signed requests, the Signature, Signature-Input and Signature-Agent headers.

What we do not know yet

  • Which products send Google-Agent traffic. The sources describe it only as agents hosted on Google infrastructure acting on a user's request. They do not name the products or surfaces.
  • Volume and reach. No figures are given for how much traffic to expect, or in which markets.
  • Rollout status. The 20 March 2026 changelog entry said the user agent was rolling out over the next few weeks. The sources do not say whether that is complete.
  • Web Bot Auth timing. Google calls it an experiment and gives no date for it ending, widening or becoming a requirement.
  • Visibility. The sources do not describe any report in Google's products that shows when Google-Agent requests were blocked on your site.
  • Opt-out controls. The sources do not document a Google-Agent-specific opt-out procedure, although the guide names category-level AI agent block policies as possible blockers.

Sources & Further Reading

Advertisement
Paid Media Built for Growth
Work with us →
Produced by
This article was filed by Teodor Yordanov of BYLT Media, paid media built for growth.
Visit

Readers & the desk

Ask the author

Have a question about this piece? Put it to the author. Selected questions are answered and published below.

Put a question to the author

10 to 500 characters0/500
No email needed. The desk publishes only selected questions.
Produced by
This article was researched, written and produced by BYLT Media, paid media built for growth.
Talk to the team