Crawler looking for broken links to help improve world wide web as a whole.
- HTTP 요청
Observer/1.0 (+https://obsrvr.net/about; crawler@obsrvr.net)
- robots.txt자세히 알아보기
User-Agent: obsrvr.net Disallow: /
Crawler looking for broken links to help improve world wide web as a whole.
Observer/1.0 (+https://obsrvr.net/about; crawler@obsrvr.net)
User-Agent: obsrvr.net Disallow: /