AI Bot Directives: Do They Follow noindex/nofollow Rules?
Go to the Best Of the SEO Community.
"I can't tell you how much they adhere to it, but I do suspect that many people who think they are blocking are not including all three."
robots.txt is a set of crawl directives that tells automated agents which parts of a site they're allowed to access. It's been the standard control mechanism for search engines and is the primary opt-out signal referenced by many AI services. What it doesn't do is enforce access. It communicates intent, and that intent is respected depending on the crawler and the context in which it operates.
What's Documented And What Isn't
The issue isn't just that controls exist, it's that their reliability varies widely depending on the AI service and the purpose of the crawl. Some systems publish clear opt-out mechanisms and follow them. Others don't document their behavior at all, or behave inconsistently in practice.
John Mueller separates documented behavior from assumptions by pointing out that not all AI services approach opt-out the same way:
"Of the AI services, I think OpenAI is the one (well, apart from Google, obviously—I work on some of the controls policies) that has the opt-out well defined and complies. You can test these things (I do), just beware that LLMs will explain things by guessing: if your URL is a.com/cheese and you ask what it's about, don't be surprised if it says cheese. The others either don't have documented controls or are flexible. One of them will fetch both robots.txt and your page at about the same time—and if your page comes back first, it wins. But all of this applies to training by crawling, which is the easy part of AI on the web."
He clarifies where control mechanisms actually exist today:
"Back to meta tags—as far as I know, only Bing has a control via meta-tag. Everything else is robots.txt. There are more control systems being discussed, but robots.txt is easy to understand, very strong and observable, and reasonably granular. With transparency—AI model creators tracking how they crawl and what they comply with—it's a strong choice. Your robots.txt knowledge will be useful."
Mueller follows with an important reminder about the limits of asking AI systems to explain their own behavior:
"They can't know about themselves (meaning large language models don't have direct visibility into their own crawling or compliance systems) unless it's explicitly in the system prompt or if it does live lookups (in which case, you can just read the docs yourself). They train from public data, and there won't be public data about it before it's live. Always read the docs directly—and ideally verify the behavior if you need to rely on it."
What This Means for Site Owners
There is no single switch for AI crawling. Some systems document their behavior and follow it. Others don't. robots.txt is still the strongest signal available, but it works through compliance, not enforcement.
Mueller closes with the real point:
"IMO this is one area where SEOs can be fantastic consultants to site owners — you know how crawling works, you know robots.txt, you can learn how LLMs work, and you can help make decisions on what makes sense for gaining awareness and visibility for companies and brands. The whole AI space is filled with hypemobiles, but you can choose to be a reasonable consultant and help clients make reasonable decisions (which could be to block all or parts, or leave it all open — at least make a decision). If you've been doing good work as an SEO for them, they'll trust you (hopefully) more than a rando Sam-From-The-Internet who claims AI is here to explode and fix everything."
The most practical move is still the same one SEOs have always made: understand how crawling works, verify what matters, and act intentionally.
What to Do Next
- Read the documentation. Don't rely on AI-generated explanations. Go to platform.openai.com, Google's crawler docs, and Bing's guidance directly.
- Block all three OpenAI bots if you want full opt-out. Most people miss at least one.
- Test your setup. Use server logs or tools like Screaming Frog to see what's actually being crawled.
- Make an explicit decision. Decide what you want indexed, what you want blocked, and what level of visibility you're willing to accept based on documented behavior.
The gap isn't in the tools. It's in the decision-making.
Channels
Join The Community
The SEO Community is a free organization, with private Slack group and events, for those interested in search and AI optimization to learn, discuss, and meet others in the industry.
Powered by MODX