Giles' blog

Adding /llms.txt

The /llms.txt file is an idea from Jeremy Howard. Rather than making LLMs parse websites with HTML designed to make it look pretty for humans, why not publish the same content separately as Markdown? It's generally not much extra effort, and could make your content more discoverable and useful for people using AIs.

I think its most useful for things like software documentation; Stripe and Anthropic seem to think so too, having both recently added it for theirs.

It's less obviously useful for a blog like this. But I write everything here in Markdown anyway, and just run it through markdown2 and some Jinja2 templates to generate the HTML, so I thought adding support would be a bit of fun; here it is.

One thing that isn't covered by the proposal, at least as far as I could see, is how LLMs should know that there is a special version of the site just for them. A link tag with type set to alternate seemed like a good idea for that; I already had one to help RSS readers find the feed URL:

<link rel="alternate" type="application/rss+xml" title="Giles Thomas" href="/feed/rss.xml" />

...so with a quick check of the docs to make sure I wasn't doing anything really stupid, I decided on this:

<link rel="alternate" type="text/markdown" title="LLM-friendly version" href="/llms.txt" />

There were a couple of other mildly interesting implementation details.

Obviously I didn't want to put all of the blog's content into a single file, so I made the top-level one have links to all of the posts (plus a link to the about page). These links needed to go to Markdown versions of the posts, of course, so what URLs should I use? The proposal says that:

We furthermore propose that pages on websites that have information that might be useful for LLMs to read provide a clean markdown version of those pages at the same URL as the original page, but with .md appended. (URLs without file names should append index.html.md instead.)

Now, that clashed a bit with the canonical URL patterns I use on this blog. You'll see that this page's URL ends with /2025/03/llmstxt, no trailing slash. It's actually an HTML file called index.html stored in a directory called llmstxt, but my static file setup handles all of that.

However, technically, from a URL point of view, that structure means that the page is claiming to be a file called llmstxt -- a directory is meant to end with a slash.

That means that the LLM-friendly version should be at the same URL with .md at the end -- so it is. My first implementation of that was to create another directory next to llmstxt called llmstxt.md and then to put the markdown into a file called index.html there, which was easy to do, but:

1. It meant that the file was served up with a content type of text/html which meant the browsers didn't render it properly, and
2. It was stupid and ugly.

So now the directory /2025/03/ contains a directory called llmstxt which contains a file called index.html for the human-readable post, but it also contains a file called llmstxt.md for the AI-friendly one.

Once I'd done that, I needed to link to them from the post pages as well, so I just added another alternate link:

<link rel="alternate" type="text/markdown" title="LLM-friendly version" href="{{ post.url }}.md" />

Once that was done, I had something that worked nicely! Total time taken, a couple of hours. I think it was worthwhile at that cost -- if I had a larger site or had the sources in a non-Markdown format, I might have been more hesitant.

Next, I need to do the PythonAnywhere documentation :-) [Update: that's done now!] Is anyone else adding /llms.txt to their sites?