Crawl websites and extract content to feed AI apps. Convert web data to Markdown or HTML, download files, and more.
- Category
- Operator
- Apify
- AccessIndicates who drives the bot's activity. Bots with "direct" access are controlled directly by their operator, while bots with "intermediary" access can be controlled by end users.
- Intermediary
- Documentation
- https://apify.com/apify/website-content-crawler
- Signature Agent URLURL to the HTTP Message Signatures key directory the agent publishes under Web Bot Auth. Origins fetch this directory to verify signatures on the agent's HTTP requests.
- https://api.apify.com/.well-known/http-message-signatures-directory