Caliperbot crawls Conductor clients' and prospects' websites for HTML feature extraction to power Content Analytics features within our Searchlight web application.
- HTTP 요청
Caliperbot/1.0 (+http://www.conductor.com/caliperbot)
- robots.txt자세히 알아보기
User-Agent: Caliperbot/1.0 Disallow: /