Crawl and indexing
AI Crawler User-Agent Lookup
Paste user-agent strings or log lines to identify AI crawlers, and see what each one is for.
Runs in your browser; nothing is uploaded. · Free, no account · 2 min
Importing a saved run into your account is coming soon. Until then, your last runs stay in this browser only, and you can download your files.
How to use
- Paste user-agent strings or lines from your server log, one per line, or search the registry by name.
- Read the table: the crawler, its vendor, its purpose and what blocking it does.
- Note any warnings, such as an agent that may not follow robots.txt.
- Use the AI Crawler robots.txt Builder to allow or block the ones you choose.
What it checks and how it works
The registry lists the crawlers that OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, DuckDuckGo, Mistral, Common Crawl and ByteDance describe, each checked against the vendor's own documentation on 28 Oct 2026, with a link to that page. A user-agent header contains the crawler's token, so the tool matches the token as a case-insensitive part of the line.
The match shows what a user-agent string claims, not who sent the request: any client can send any string. Vendors publish IP ranges so you can verify a crawler; see their documentation. Google-Extended and Applebot-Extended are control tokens in robots.txt and never appear as a user agent in a request.
Sources
- RFC 9309: Robots Exclusion Protocol opened 29 Oct 2026
- OpenAI: Overview of OpenAI crawlers opened 28 Oct 2026
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? opened 28 Oct 2026
- Perplexity: Perplexity crawlers opened 28 Oct 2026
- Google: Google's common crawlers opened 28 Oct 2026
- Apple: About Applebot opened 28 Oct 2026
- Meta: Meta web crawlers opened 28 Oct 2026
- Amazon: Amazonbot opened 28 Oct 2026
- DuckDuckGo: DuckAssistBot opened 28 Oct 2026
- Mistral: Mistral crawlers opened 28 Oct 2026
- Common Crawl: CCBot opened 28 Oct 2026
- Google: AI features and your website opened 28 Oct 2026
- Bing Webmaster Blog: new options to control use of content in Bing Chat (22 Sep 2023) opened 28 Oct 2026
Related tools
AI Crawler robots.txt Builder
Choose which AI crawlers may read your site, and get a robots.txt that follows each vendor's own documentation.
robots.txt URL Tester
Paste your robots.txt and a list of URLs to see which are blocked for which crawler, and which line decides it.
llms.txt Validator
Paste your llms.txt and check its structure against the llmstxt.org format.
Meta Robots and X-Robots-Tag Generator
Build a meta robots tag or X-Robots-Tag header from Google's documented rules, and see conflicts before you publish.
Go further with Zaher: Read: how to check your AI visibility
Questions
About this tool
No. Anyone can send any user-agent string. To verify a crawler, check the request's IP address against the ranges the vendor publishes, as described in the vendor's documentation linked on this page.
Vendors describe training crawlers as collecting content that may be used to train models, and search crawlers as building an index that an assistant answers from. Blocking each has a different effect, shown in the table.
ByteDance publishes no documentation for it, so the tool cannot confirm its behavior from a vendor source and says so.
Get started today
Find out where your brand stands in AI. Free.
Run a free AI visibility analysis on your domain. No credit card, results in minutes. See what AI engines say about your brand today.
Brands already measuring their AI visibility with Zaher.
