Bot Information for Meta-WebIndexer Verified

The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI.

Operator
Meta
AccessIndicates who drives the bot's activity. Bots with "direct" access are controlled directly by their operator, while bots with "intermediary" access can be controlled by end users.
Direct

User agent

Copy link
HTTP requests
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
meta-webindexer/1.1
robots.txtLearn more...
User-Agent: meta-webindexer/1.1
Disallow: /

HTTP traffic from Meta-WebIndexer

Copy link

HTTP requests from the specified bot over the selected time period

Learn more...