The Magpie Crawler indexes content for its soical media monitoring solution.
- Category
- AI Crawler
- Operator
- Brandwatch
- Documentation
- https://www.brandwatch.com/legal/magpie-crawler/
- HTTP requests
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
- robots.txtLearn more...
User-Agent: magpie-crawler Disallow: /