txtbot is a specialized crawler designed to access plain text files (e.g. robots.txt, security.txt, llms.txt, etc.).
- Category
- Operator
- webperf ltd
- AccessIndicates who drives the bot's activity. Bots with "direct" access are controlled directly by their operator, while bots with "intermediary" access can be controlled by end users.
- Direct
- Documentation
- https://txtbot.net/