SiteCrawler, part of the Botify Analytics suite, gives enterprise SEO teams the power to evaluate the structure and content of their websites just like a search engine
- Category
- Search Engine Optimization
- Operator
- Botify
- HTTP requests
Desktop: Mozilla/5.0 (compatible; botify; http://botify.com)
Desktop: Mozilla/5.0 (compatible; botify; http://botify.com)
- robots.txtLearn more...
User-Agent: botify Disallow: /