Cloudflare Web Scanner is a well-behaved crawler that checks domain liveness, content type, robots.txt directives, and other web properties to map the shape of the web.
- Categoria
- Crawler del motore di ricerca
- Operatore
- Cloudflare
- Documentazione
- https://developers.cloudflare.com/ai-crawl-control
- Signature Agent URLURL to the HTTP Message Signatures key directory the agent publishes under Web Bot Auth. Origins fetch this directory to verify signatures on the agent's HTTP requests.
- https://d7c111cd-310f-45a1-bf94-1d7619c74a04.wba-bots.ai-audit.cfdata.org/.well-known/http-message-signatures-directory