Cloudflare adds real-time trust scoring for verified bots

Cloudflare has launched a system to monitor and report whether verified bots adhere to website preferences over time, addressing a gap between identity verification and ongoing behavior. The move follows incidents where known bots ignored site rules, including a 2025 case involving Perplexity’s crawler accessing restricted resources and circumventing blocks.

The new framework evaluates two core signals: whether bots respect robots.txt directives and how they respond to HTTP 429 (Too Many Requests) responses. These observations feed into a "trust band"—a dynamic range reflecting consistency in compliance rather than a binary pass/fail. Cloudflare emphasizes that trust is built through sustained patterns, not isolated events, and plans to publish these scores publicly on Radar once sufficient historical data is collected.

How the system works

Bot operators must first authenticate via Web Bot Auth, which establishes identity through signed requests catalogued in BotBase. The new system then tracks whether traffic aligns with published site preferences, such as crawl delays or path restrictions. Repeated violations—like ignoring robots.txt or failing to throttle after 429 responses—lower a bot’s trust score, while consistent compliance improves it.

Cloudflare clarifies that robots.txt sets minimum rules, not permissions. For example, a bot may still violate site preferences even if its actions aren’t explicitly forbidden in the file. The system treats HTTP 429 responses as direct feedback, expecting bots to adjust their request rates accordingly.

Implications for operators

The trust scores will eventually integrate with Cloudflare’s Bot Management tools, letting customers set thresholds for how verified bot traffic is handled. This could mean automatically blocking or challenging bots that fall below a certain trust level. For bot operators, the system provides a way to demonstrate accountability, while site owners gain visibility into whether allowed bots respect their rules.

The system is currently in internal testing, with plans to roll it out to all Cloudflare customers. Future expansions may include additional signals, such as adherence to crawl-delay directives or response to other HTTP status codes. Cloudflare’s goal is to make website preferences enforceable and ensure trust is earned through behavior, not just identity verification.

What to watch

Public trust scores on Radar could influence how site owners configure their bot policies, particularly for high-traffic crawlers. Bot operators may face pressure to improve compliance to maintain access to Cloudflare-protected sites. The system’s effectiveness will depend on how transparently Cloudflare communicates violations and whether it provides actionable feedback for remediation.

Cloudflare’s approach contrasts with static allowlists, which assume trust based on identity alone. By tying verification to observable behavior, the company aims to create a feedback loop where both site owners and bot operators can make informed decisions about automated traffic.