FEATUREDComputing Life · Share (鸭哥 research reports)· rssZH00:00 · 09·08
→Why Bots Are Finally Getting ID-Checked After 30 Years
Cloudflare launched BotBase for Operators on Aug 28, letting bot teams register identities and go through review. This is a sharp break: bots now make up 57.4% of web traffic, yet for 30 years the only gate was a voluntary robots.txt. The old equilibrium rested on three assumptions—search engines sent referral traffic back, false positives were cheap, and bot detection was easy. AI agents broke all three. LLM crawlers take content without sending visitors back (Anthropic's crawler generated one referral per 70,900 pages). Agents acting on behalf of paying users can't be blocked indiscriminately. Real browser environments defeat static fingerprinting. The only path left is requiring bots to declare identity and verify it cryptographically. A four-layer stack is forming: Web Bot Auth signing, purpose declaration, registration review, and platform defaults. The first three layers are voluntary; only the defaults have teeth. Cloudflare, serving 24.3% of all websites, controls the defaults, verification pipeline, directory, and payment channel. Blind spots remain: crawlers that refuse to register, private bilateral licensing deals, and API-based intermediaries all operate outside this system. The post notes Web Bot Auth has no formally adopted IETF document yet, and production formats already show intergenerational conflicts.
#Agent#Cloudflare#Anthropic#OpenAI
why featured
Featured · importance 78 · hook + knowledge + resonance
editor take
Cloudflare turned bot access from voluntary into enforced defaults with 24.3% web coverage, but the top three layers remain an honors system.
sharp
This piece connects signals from the past year: Web Bot Auth signing, BotBase registration, Content Signals purpose declaration, and Cloudflare's Sept 15 default block on AI crawlers for ad-containing pages. The four-layer stack sounds complete, but the top three are voluntary—OpenAI's main crawler GPTBot still hasn't adopted signing, Googlebot hasn't either, and there's no public evidence Anthropic's crawlers have. Only the bottom default layer has teeth, and Cloudflare serves 24.3% of all websites, 84.1% of sites with identifiable reverse proxies. It's effectively making defense decisions on behalf of site owners.
I'd discount the completeness claim. Web Bot Auth has zero formally adopted IETF documents—nine proposals, all individual submissions—yet AWS already defaults to allowing signed agent traffic in production. The format has already forked: Google follows the new spec, OpenAI uses the old string format, and Cloudflare's verification docs list the new dictionary format as a failure case. Build to one vendor's guide and you'll hit interop bugs.
The top Hacker News comment nailed it: still the honors system. What I'm actually watching is Cloudflare's payment pipeline—per-request billing ran in private beta for a year, then shifted to per-referral in July 2026. Identity registration, signature verification, blocking policy all sit with one platform. Site owners just accept the classification output. Same playbook as Google and Yahoo tying DMARC to inbox delivery in February 2024: edge gateway enforcement moves faster than standards bodies.
HKR breakdown
hook ✓knowledge ✓resonance ✓