Amazon uses a crawler, also known as a spider or a bot to process and index the content of webpages. The Amazon crawler visits your site to determine its content in order to provide relevant ads.
- Category
- Advertising & Marketing
- Operator
- Amazon
- HTTP requests
Mozilla/5.0 (compatible;contxbot/1.0)
- robots.txtLearn more...
User-Agent: contxbot Disallow: /