Crawlson is a search engine crawler for the crawlson.com search engine.
- Category
- Search Engine Crawler
- Operator
- Crawlson
- Documentation
- https://www.crawlson.com
- HTTP requests
Mozilla/5.0 (compatible; Crawlson/1.0; +https://www.crawlson.com/domain)
- robots.txtLearn more...
User-Agent: Crawlson/ Disallow: /