Fetches robots.txt files for Cloudflare AI Crawl Control to retrieve and evaluate crawler directives.
- Category
- Operator
- Cloudflare
- AccessIndicates who drives the bot's activity. Bots with "direct" access are controlled directly by their operator, while bots with "intermediary" access can be controlled by end users.
- Direct
- Signature Agent URLURL to the HTTP Message Signatures key directory the agent publishes under Web Bot Auth. Origins fetch this directory to verify signatures on the agent's HTTP requests.
- https://9ae614b6-fb5a-4549-8d3e-448628101e49.wba-bots.ai-audit.cfdata.org/.well-known/http-message-signatures-directory