Crawl and indexing
Meta Robots and X-Robots-Tag Generator
Build a meta robots tag or X-Robots-Tag header from Google's documented rules, and see conflicts before you publish.
Runs in your browser; nothing is uploaded. · Free, no account · 3 min
Indexing
Snippets and previews
Other
Server rule
Comma separated, such as pdf, docx. Used for the Apache and Nginx snippets.
Importing a saved run into your account is coming soon. Until then, your last runs stay in this browser only, and you can download your files.
How to use
- Choose who the rule applies to: all crawlers, Googlebot, Googlebot-News or another token.
- Tick the rules you need, such as noindex or nosnippet, and set any limits such as max-snippet.
- Copy the meta tag for pages, or the header and server rule for PDFs and other files.
- Read the notes: the tool flags rules that conflict or have no effect.
What it checks and how it works
The rules and their meanings are the ones in Google's documentation for the robots meta tag and the X-Robots-Tag header. When rules conflict, Google applies the more restrictive one, so nosnippet overrides max-snippet. indexifembedded only has an effect together with noindex. unavailable_after takes a date in a widely adopted format such as ISO 8601, and the tool writes it that way.
Google states that noarchive is no longer used for a cached link, so the tool does not offer it. A crawler can obey a noindex only if it can fetch the page, so do not also block the page in robots.txt.
Sources
Related tools
AI Crawler robots.txt Builder
Choose which AI crawlers may read your site, and get a robots.txt that follows each vendor's own documentation.
robots.txt URL Tester
Paste your robots.txt and a list of URLs to see which are blocked for which crawler, and which line decides it.
Canonical Tag Generator
Enter a page URL and get the canonical form of it, as a link tag and an HTTP header.
Sitemap Validator
Paste a sitemap.xml or a sitemap index and see every protocol problem, with the line of the URL it affects.
Questions
About this tool
Use the meta tag for HTML pages. Use the X-Robots-Tag header for files that have no head, such as PDFs, images and other downloads. They carry the same rules.
No. A crawler must fetch the page to see the noindex rule. If robots.txt blocks the URL, the rule is never read. Leave the page crawlable and use noindex.
It is the same as noindex together with nofollow. The tool tells you when your ticks add up to that.
Get started today
Find out where your brand stands in AI. Free.
Run a free AI visibility analysis on your domain. No credit card, results in minutes. See what AI engines say about your brand today.
Brands already measuring their AI visibility with Zaher.
