Crawler to extract the newest articles in the publisher's website (via RSS feed URL)
- 카테고리
- 피드 페쳐
- HTTP 요청
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; MidasNetworkBot/1.0; +https://midas-network.com/) Chrome/144.0.0.0 Safari/537.36
- robots.txt자세히 알아보기
User-Agent: MidasNetworkBot Disallow: /