Bot Information for Crawlson Verified

Crawlson is a search engine crawler for the crawlson.com search engine.

Category
Operator
Crawlson
AccessIndicates who drives the bot's activity. Bots with "direct" access are controlled directly by their operator, while bots with "intermediary" access can be controlled by end users.
Direct

User agent

Copy link
HTTP requests
Mozilla/5.0 (compatible; Crawlson/1.0; +https://www.crawlson.com/domain)
robots.txtLearn more...
User-Agent: Crawlson/
Disallow: /

HTTP traffic from Crawlson

Copy link

HTTP requests from the specified bot over the selected time period

Learn more...