AI Bot Directives: Do They Follow noindex/nofollow Rules?

What I've found so far is that when performing a function within ChatGPT, it will not work if your domain disallows via robots. But I've also seen results from those sites cited in a GPT Search context. So they're likely not generally respecting Robots, but are in acute user actions

What about on-page directives for noindex?

haven't tested that yet

fair

I asked GPT.....

 

No, *ChatGPT does not respect* noindex *rules* because it does not actively crawl or index web pages. Instead, when retrieving information via its web browsing tool, ChatGPT queries public search engines or directly visits URLs provided by the user. Here’s why:

 

1. *How ChatGPT Retrieves Information:*

• It uses a search engine to access publicly available information.

• It does not perform direct web crawling like a search engine crawler (e.g., Googlebot).

2. *Noindex Tags in Context:*

• Noindex tags are instructions to search engine crawlers to avoid indexing specific pages.

• ChatGPT bypasses this concept because it doesn’t parse or store information based on a site’s robots.txt or meta tags.

3. *Respect for Private Data:*

• ChatGPT adheres to ethical guidelines and OpenAI’s policies, which prohibit scraping or using data intended to be private or confidential.

 

If you’re concerned about sensitive or restricted content, the best approach is to avoid sharing such URLs here. Additionally, let me know if you’d like more detail on handling noindex or private data management!

Though how I am reading this, as long as you don't directly share the URL with GPT, it uses search engines so as long as you have the URL marked as noindex, then the search engines shouldn't return it

No, ChatGPT does not look at or respect robots.txt files because it doesn’t function as a web crawler. When using its web browsing tool, it interacts with public search engines or directly visits URLs provided by the user, but it does not independently crawl or index websites.

 

*Why ChatGPT Doesn’t Use robots.txt:*

 

1. *Not a Crawler:*

• The robots.txt file is designed to provide instructions to web crawlers like Googlebot or Bingbot, not tools like ChatGPT.

• Since ChatGPT doesn’t actively crawl the web, it doesn’t consult robots.txt for access permissions.

2. *Access to Public Content:*

• When browsing, it accesses content that is publicly available and retrievable by a browser or search engine.

3. *Ethical Data Use:*

• ChatGPT follows OpenAI’s policies to avoid accessing or using private, sensitive, or restricted content.

 

If you have specific concerns about privacy or ethical use of data, feel free to ask for clarification!

This is interesting - there are a lot of posts out there about adding

 

Allow: /privatePage/
Disallow: /privatePage/publicFile.jpg```
But if GPTs answer is accurate, then it doesn't matter - you can't block them from accessing your site via standard practices, you'd need to do it at a CDN level.

wondering why the tool I was using wouldn't function then

FWIW, I just tested a URL marked noindex and one that is blocked via robots.txt and GPT has zero issues looking at the page and giving me all the details about it

It even gave me a "helpful" guide on how to use the noindex directive on said URLs undefined

OpenAI has different bots for different use cases. Here is their documentation for how each works:

One for search, one for user queries and one for model training. They all use robots.txt tags.

 

<https://platform.openai.com/docs/bots/overview-of-openai-crawlers#:~:text=When%20users%20ask%20ChatGPT%20or,training%20generative%20AI%20foundation%20models|https://platform.openai.com/docs/bots/overview-of-openai-crawlers#:~:text=When%20users%20ask%20ChatGPT%20or,training%20generative%20AI%20foundation%20models>.

I can’t tell you how much they adhere to it, but I do suspect that many people who think they are blocking are not including all three.

Thank you @renee.bigelow! I’ll play around with this.

Of the AI services, I think openai is the one (well, apart from Google, obvs - I work on some of the controls polices) that has the opt-out well defined & complies. You can test these things (I do), just beware that LLMs will explain things by guessing: if your url is <http://a.com/cheese|a.com/cheese> and you ask what it's about, then don't be surprised if it says cheese. The others either don't have (documented) controls or are flexible (one of them will fetch both robots.txt and your page at about the same time, if your page comes back first, it wins). But all of this is for training by crawling, and that's the easy part of AI's on the web.

And back to meta tags, afaik only Bing has a control via meta-tag. Everything else is robots.txt. There are more control systems in discussion, but robots.txt is easy to understand, very strong & observable, and reasonably granular. I'm biases, but I think with transparency (AI model creators tracking how they crawl & what they comply with) it's a strong choice. Fun times. Your robots.txt knowledge will be useful. (JS sites is another fun angle here)

Thank you @softplus, appreciate the detailed response!

Now wondering which one fetches both… ????

They seem so trustworthy and innocent with their responses undefined

They can't know about themselves unless it's explicitly in the system prompt or if it does live lookups (in which case, you can just read the docs yourself). They train from public data, and there won't be public data about it before it's live. Always read the docs directly (and ideally verify the behavior, if you need to rely on it).

IMO this is one area where SEOs can be fantastic consultants to site owners - you know how crawling works, you know robots.txt, you can learn how LLMs work, and you can help make decisions on what makes sense for gaining awareness / visibility for companies & brands. The whole AI space is filled with hypemobiles, but you can choose to be a reasonable consultant and help clients to make reasonable decisions (which could be to block all or parts, or leave it all open -- at least make a decision). If you've been doing good work as an SEO for them, they'll trust you (hopefully) more than a rando Sam-From-The-Internet who claims AI is here to explode & fix everything.