Bot Information for Arquivo Web Crawler Verified

User agent

Copy link
HTTP requests
Arquivo-web-crawler (compatible; heritrix/3.4.0-20200304 +https://arquivo.pt/faq-crawling)
robots.txtLearn more...
User-Agent: Arquivo-web-crawler
Disallow: /

HTTP traffic from Arquivo Web Crawler

Copy link

HTTP requests from the specified bot over the selected time period

Learn more...