According to DEV Community, llms.txt is a Markdown file placed at yourdomain.com/llms.txt. It tells language models who you are and which pages matter. Jeremy Howard proposed it in September 2024. The idea is that HTML pages are full of navigation, scripts and layout, so a short file gives models the essentials directly.
The format is simple. It needs one # heading with the site or project name, which is the only required part. Next comes one > blockquote with a short summary, ideally the sentence you would want an AI to repeat about you. After that come optional context paragraphs and ## sections, each holding a list of links written as a title, a URL and a one-line note. An optional ## Optional section can hold links a model may skip when short on space.
The article says the file must be served as plain text at the domain root. After deploying, open the address in a browser. If you see your homepage or a 404, your host is probably rewriting the route, which is common with single-page app setups that send every path to index.html.
The author lists five mistakes: linking to pages that are blank without JavaScript, writing a keyword list instead of sentences, leaving the file stale, using bare URLs with no notes, and blocking the bots that would read it. In the author's study of 286 developer portfolios, 85% had no llms.txt. Of 19 sites blocking AI crawlers, 17 had the block written by Cloudflare, not by the owner.
Why it matters
The article is open about the limits. No ranking benefit has been confirmed, and it says OpenAI, Anthropic and Google haven't said they use the file. It does not replace robots.txt, which sets what bots may fetch, or sitemap.xml, which lists every URL. It also won't help a site that shows nothing without JavaScript.
Even so, the cost is low. Writing the file pushes you to describe your site clearly, and that description can be reused elsewhere. The robots.txt point is worth noting: if AI crawlers are blocked there, they won't fetch llms.txt either, and the block may not be one you chose.
Source: DEV Community. This brief summarises their reporting; read the original for the full detail.