I published an llms.txt on this site in 2025, because the argument for it was clean and the cost was twenty minutes. A plain-Markdown summary of your site, at /llms.txt, written so a language model can read your facts without wading through nav bars and cookie banners.
I would still publish one. I have also stopped describing it as high-leverage, because the evidence since then does not support that, and most of the guides telling you to write one have not updated.
So this post is two things: an honest account of what llms.txt is and is not doing, and then the part that does move the needle, which is less exciting and more work.
The evidence, such as it is
Google has been explicit. John Mueller said in mid-2025 that no AI system was using the file and compared it to the meta keywords tag. Gary Illyes said Google does not support it and has no plans to. By 2026 Google's own AI optimization guidance carries a note saying the file is not needed for Google Search and is ignored by it.
The independent measurements are worse. Adoption sits around 10% of domains, and one widely-cited analysis found roughly 97% of published llms.txt files receive zero requests from AI crawlers. Studies looking for a correlation between having the file and getting cited have not found one.
It is not nothing. OpenAI maintains llms.txt files for its own SDK docs, Anthropic recommends the convention in its guidance on writing for agents, and agentic developer tooling does fetch it. Lighthouse added an experimental audit for it. But neither OpenAI nor Anthropic tells site owners to rely on it for visibility, and both point at robots.txt for anything to do with crawler behaviour.
The honest summary: it is a proposal that documentation sites and agent tooling use, not a ranking or citation mechanism. Publish it for the agents that read it, and do not expect traffic from it.
Writing one anyway
Twenty minutes, and it is a genuinely useful artifact to hand a coding agent or anyone asking what you work on.
Keep it short, factual and link-rich. H1 with your name or brand, a one-line summary in a blockquote that a model can quote verbatim, then facts grouped under H2s with bullet links.
# Your Name — What You Do
> A one-sentence summary: who you are, what you build, and the core
> technologies, written so a model can quote it verbatim.
## About
A short paragraph of plain facts. No marketing fluff — models reward clarity.
## Expertise
- Topic one — a few concrete specifics
- Topic two — frameworks and tools you use
## Selected work
- Project A — what it is: https://example.com/a
- Project B — what it is: https://example.com/b
## Links
- Website: https://www.yoursite.com
- Blog: https://www.yoursite.com/blog
- GitHub / LinkedIn / EmailState facts a model can repeat, which means names, technologies and URLs. Skip anything it cannot verify. Keep it under a few hundred lines and update it when your story changes, because a stale one is worse than none.
In Next.js, drop it in public/llms.txt and it is live at the root with no config. If you would rather generate it:
export function GET() {
const body = `# Your Name — What You Do
> ...
`;
return new Response(body, {
headers: { 'Content-Type': 'text/plain; charset=utf-8' },
});
}The crawler access question is the real one
This is where sites quietly lose, and unlike llms.txt it is load-bearing. If you are blocking the bots, nothing else you do matters, and plenty of sites block them by default without anyone having decided to.
import type { MetadataRoute } from 'next';
export default function robots(): MetadataRoute.Robots {
const base = 'https://www.yoursite.com';
return {
rules: [
{ userAgent: '*', allow: '/', disallow: ['/api/'] },
{
userAgent: [
'GPTBot', 'OAI-SearchBot', 'ChatGPT-User',
'ClaudeBot', 'anthropic-ai',
'PerplexityBot', 'Google-Extended',
],
allow: '/',
},
],
sitemap: `${base}/sitemap.xml`,
};
}Worth knowing that these user agents do different jobs. GPTBot trains, OAI-SearchBot and ChatGPT-User fetch for answers, Google-Extended governs Gemini training rather than Search indexing. Allowing all of them is a decision about your content, not a technical default, and it is reasonable to allow the answer-time crawlers and block the training ones.
Check your WAF too. Cloudflare and similar will happily challenge these agents regardless of what your robots.txt says, and a 403 from the edge is invisible in your own logs.
What earns a citation
The things that work are the same things that worked before any of this, which is unsatisfying and also why they keep working.
Answer the question in the first sentence or two, then explain. A model extracting a quotable line takes the first one that stands alone, so make the first one stand alone. Write headings the way people ask questions. Add an FAQ section with FAQPage schema, which remains the most citation-friendly structure there is.
Be specific. Numbers, versions and named tools get quoted; vague claims get skipped, because there is nothing in them to lift. This is the same reason the posts on this site with real figures in them get referenced and the general ones do not.
Keep the JSON-LD and semantic HTML solid. Person, Article, FAQPage, real headings, real lists. GEO is not a replacement for technical SEO, it is the same work with a different reader, and the Next.js technical SEO checklist covers the mechanics.
Checking whether any of it worked
Ask ChatGPT with browsing on, and Perplexity, a question your content answers, plus a few "what is [your brand]" queries, and see whether you come up.
Then go to your access logs, which is the part people skip. Filter for GPTBot, OAI-SearchBot and PerplexityBot and see what they fetched and what you returned them. A string of 403s is a more useful finding than any amount of prompting, and it is the thing you can actually fix.
Where I would spend the time
Publish the file. It costs twenty minutes and some agents do read it.
Then spend the rest of the afternoon on crawler access, because that is the one with a real failure mode, and on making individual pages answer their question in the first line. If you only do one of the three, do the access check.
Related: technical SEO for Next.js · build a RAG chatbot with Next.js and an LLM