Curious about llmstxt.org - Will it gain momentum?

The Best Of

Dec 27, 2024, 7:36 AM

Interesting idea: anyone think it will get more traction?

A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.

Forwarded thread from another channel:

John Mueller

Dec 27, 2024, 8:20 AM

This seems like a kludge. Just make a normal, crawlable site?

Dave Smart

Dec 27, 2024, 8:39 AM

Feels like a real good way to feed llms one thing and users another too i.m.h.o. I'd also worry about synchronicity too, update / change / add to your content, easy to forget the llm version (although you could I guess automate that to some extent)

Brent Rangen

Dec 27, 2024, 10:04 AM

Definitely not setting the internet ablaze. ????

MIchael Buckbee

Dec 30, 2024, 6:02 AM

Couple different things:

• We have quite sophisticated tools for taking HTML+Images and making it plain text of all sorts, so I'm not sure it helps anything there.

• It's built on the assumption that LLMs just want text, but the newest models are quite mutli-modal and support images, video, etc. which makes this feel outdated.

• One larger strategic change is that you want more `Brand Statement` content as part of your overall content strategy than preAI and that might not fit cleanly into how you present your marketing, so it might be helpful for that.

Emina Demiri-Watson

Jan 4, 2025, 2:47 AM

Cue consultants blaming it’s absence for lower results.

MIchael Buckbee

Jan 4, 2025, 9:47 AM

@emina well, I wonder how you even adopt it? Like do you re-render all your pages as markdown...that seems excessive.

Emina Demiri-Watson

Jan 5, 2025, 2:29 AM

@mike647 surely not! I just think it’s an overkill but then maybe Im not understanding something here.

John Mueller

Jan 5, 2025, 3:26 AM

Random data point - I crawled all our documentation and converted it into markdown to feed into NotebookLM. It's not super-hard (I used colab). Anyone training an AI model can do this if they're already crawling the web or ingesting HTML content. (That said, it was pretty cool to have all the content in a single folder :-))

Emina Demiri-Watson

Jan 5, 2025, 3:51 AM

@softplus plus now you can listen to the worlds longest podcast!

MIchael Buckbee

Jan 6, 2025, 7:11 AM

I can definitely see there being "Packages" of some kind of documentation like this built for ingestion into your AI model of choice.

Andrea Volpini

Jan 15, 2025, 1:38 PM

If you work with LLMs, markdown helps - this is why exists and service like are growing fast.

The best model to turn HTML to markdown in my personal experience is (ReaderLM-v2 is available from today)

The /llms.txt file – llms-txt

A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.

Tavily

Tavily is a search engine tailored for AI agents, delivering real-time, accurate results, intelligent query suggestions, and in-depth research capabilities.

Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown

Reader-LM-0.5B and Reader-LM-1.5B are two novel small language models inspired by Jina Reader, designed to convert raw, noisy HTML from the open web into clean markdown.

Written by

Jina AI

Reading time

13 mins read

Reader-LM: Small Language Models for Cleaning and Converting HTML to Markdown

Andrea Volpini

Feb 4, 2025, 11:24 AM

We did a quick update here -

Adding MarkItDown by Microsoft ???? #SEO #LLMs

Generate llms.txt

Provides brief background information and guidance for your website to large language models, by creating your /llms.text file.

Est. reading time

3 minutes

AI-Powered SEO • WordLift: Generate llms.txt

Andrea Volpini

Feb 5, 2025, 3:04 AM

(and a blog post with all the links to the different solutions - there is also a WordPress plugin called AEOmatic that could be interesting)

Mastering llms.txt: Your Essential Guide to GenAI-Optimized Website Indexing

Enhance your website's SEO by adopting the llms.txt standard, ensuring improved content accessibility and richer semantic connectivity for AI search engines.

Written by

Andrea Volpini

Est. reading time

6 minutes

WordLift Blog: Mastering llms.txt: Your Essential Guide to GenAI-Optimized Website Indexing

AEOmatic | WordPress LLMs.txt Generator Plugin

Optimize your WordPress website for AI tools with AEOmatic. Our plugin automates Answer Engine Optimization (AEO), making your content discoverable by ChatGPT, Gemini, and other AI tools. The next evolution of SEO is here.

MIchael Buckbee

Feb 5, 2025, 8:45 AM

Has any AI search provider said that they're ingesting LLMs.txt ?

John Mueller

Feb 5, 2025, 1:23 PM

It would be easy to see in server logs.

MIchael Buckbee

Feb 6, 2025, 6:32 AM

@softplus yeah - I'm working on some tracking of that `robots.txt` and `sitemap.xml` as I'm curious.

Anne Berlin

Feb 24, 2025, 1:24 PM

ive looked at the logfiles for one of the larger sites i manage, and llms.txt has only been requested 5 times in the last 30 days, and the only user agent is BuiltWith. At the same time, this site has been visited by both of Chat GPT's user agents almost a million times. this is not a content publisher, it's a drectory site, so YMMV.