The Cloudflare Crawler is a well-behaved crawler that retrieves web content. By default, it self-identifies as a bot, honors robots.txt directives, and cannot bypass CAPTCHAs or bot protection. Used by Cloudflare customers via the Browser Run /crawl endpoint.
- Category
- Operator
- Cloudflare
- AccessIndicates who drives the bot's activity. Bots with "direct" access are controlled directly by their operator, while bots with "intermediary" access can be controlled by end users.
- Intermediary
- Documentation
- https://cloudflare.com
- Signature Agent URLURL to the HTTP Message Signatures key directory the agent publishes under Web Bot Auth. Origins fetch this directory to verify signatures on the agent's HTTP requests.
- https://web-bot-auth-crawl.cloudflare-browser-rendering-085.workers.dev/.well-known/http-message-signatures-directory