Cloudflare Agent Readiness Scanner

A diagnostic scanner operated by Cloudflare, allowing users to check whether their website is discoverable and usable by AI agents. Scans run on-demand and the information retrieved is shown only to the owner of the scanned domain.

What the scanner does

The scanner checks whether a website exposes standards that help automated agents discover, understand, and interact with it. These checks cover:

• Discoverability through robots.txt, sitemaps, link headers, and DNS for AI Discovery.
• Machine-readable content through Markdown content negotiation.
• Bot access controls, including AI directives, Content Signals, and Web Bot Auth.
• API, authentication, MCP, A2A, and agent skill discovery metadata.
• Agent-facing commerce protocol discovery where applicable.

The scanner produces a diagnostic report with a readiness level and guidance for the domain owner. It does not build a search index, train AI models, republish website content, test for security vulnerabilities, or perform transactions.

When requests are made

A scan runs when an authenticated Cloudflare customer visits the Agent Readiness experience in the Cloudflare dashboard. The customer can also explicitly request a rerun.

Agent Readiness is not a continuous crawler. Results are cached so that opening the dashboard again may return an existing result without making another request to the website. Fresh scans are rate-limited, have short per-request timeouts, and have a bounded overall duration.

Which websites are scanned

The scanner only accepts a domain that is configured as an active zone in the requesting customer's Cloudflare account. The customer must be authorized to access that zone.

How requests are identified

Requests to the scanned website use this User-Agent:

User-Agent: Mozilla/5.0 (compatible; Cloudflare-AgentReadiness/1.0)

HTTP requests made directly by the Agent Readiness Scanner's standards checks are signed using Web Bot Auth. The Browser Rendering navigation used for the WebMCP check is not currently signed. A signed request includes short-lived HTTP Message Signature headers similar to:

Signature-Agent: "https://web-bot-auth.chambers.cfdata.org"
Signature-Input: sig1=("@authority" "signature-agent");created=...;expires=...;keyid="...";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:...:

The scanner's Ed25519 public key is published in its signed HTTP Message Signatures directory. The corresponding private key is not published.

What a scan requests

A scan makes a small, bounded set of requests needed to test the supported standards. It does not recursively follow links or crawl the website.

• An initial HEAD request to confirm reachability, with a GET fallback when the server does not support or allow HEAD.
• GET requests to the homepage, /robots.txt, advertised or conventional sitemap locations, and relevant /.well-known/ discovery endpoints.
• A homepage request with Accept: text/markdown to test content negotiation.
• Limited protocol discovery requests. If the website advertises an MCP endpoint, the scanner can send MCP initialize and tools/list requests to inspect its advertised capabilities. It does not invoke any discovered tools.

DNS for AI Discovery checks use DNS-over-HTTPS resolvers. Those resolver queries are separate from requests made to the scanned website.

Information collected and returned

The scanner reads HTTP status codes, response headers, and the limited response content needed to evaluate each standard. It may read discovery documents and protocol capability metadata exposed by the website. It does not supply customer credentials to the scanned website or attempt to access authenticated content.

Scan results are private to the domain owner. Results are returned through authenticated, zone-scoped Cloudflare dashboard APIs and are available only to a Cloudflare account authorized to access the scanned zone. They are not published in a public index.

Questions or unexpected traffic

Website operators can identify scanner traffic using the User-Agent and verify its Web Bot Auth signature using the public-key directory above. Cloudflare customers who have questions or see unexpected scanner traffic can contact Cloudflare Support.