From Robots.txt to LLMS.txt - The Evolution No One Asked For
Most of us in SEO know robots.txt. That quiet little file sitting in the root directory, politely telling search engines what they can and cannot crawl. It has been part of the web’s furniture for years. It was never glamorous, but it mattered.
It was the digital equivalent of saying, “You can look over here, but maybe leave that thank you page alone.”
Fast forward to now, and the conversation has changed. We are no longer only thinking about search engine bots crawling pages and ranking them in blue links. We are thinking about AI systems reading content, interpreting it, summarising it, remixing it, and feeding it back into the world through conversational answers.
That is a very different game.
Search engines indexed the web. Language models interpret it. And when your content is being interpreted, context matters. Attribution matters. Source control matters. Your brand narrative matters.
That is where LLMS.txt comes in. In simple terms, it is a proposed way to help large language models understand how they should interact with your website content. What they can use, what they should avoid, how they should cite you, and whether your content is suitable for retrieval, training, or reference.
Basically, robots.txt had a clever, slightly nerdier child, and that child now has opinions about content ethics.
What I Actually Did for Luckybeard
When I implemented Luckybeard’s LLMS.txt, the goal was not just to be early for the sake of being early, although that would be a very SEO thing to do. The real goal was brand control.
LuckyBeard is a creative consultancy built around brand experience architecture. The work blends research, innovation, strategy, creativity, and systems thinking. That means the thinking is not just content. It is intellectual property. It is methodology. It is how the brand shows up in the world.
So the file needed to do more than exist. It needed to give AI systems a clearer understanding of what LuckyBeard’s public content represents, how it should be used, and how it should be attributed.
The core thinking was simple:
- Guide AI systems toward the pages that best represent LuckyBeard’s thinking, work, positioning, and public proof points
- Exclude the pages that matter legally but do not really add value to AI understanding, such as privacy policies, cookie policies, and terms and conditions
- Clarify the preferred attribution so that if our content is used or referenced, the brand does not disappear into the fog like some anonymous background extra
- Allow retrieval and indexing, but not training, because there is a big difference between referencing our work and absorbing it as raw material
Or, to put it less formally, you can read it, you can reference it, and you can learn where the answer came from, but do not copy my homework and pretend you invented the concept.
The structure itself was not overly complicated, and that is part of the point. Good technical strategy does not always need to look like a NASA launch sequence. Sometimes it is about making clear decisions and documenting them properly.
For LuckyBeard, the important brand and knowledge pages were allowed for inclusion. These included pages like About, Work, Insights, and Get in Touch. These are the pages that explain who LuckyBeard is, what the business does, what it believes, how it thinks, and where the strongest public proof points live.
Those are the pages we would actually want AI systems to understand.
On the other hand, we excluded the less useful areas, such as terms and conditions, privacy policy, and cookie policy. Those pages matter, of course, but they are not exactly the strongest representation of the brand’s thinking, personality, capability, or strategic value. Let’s be honest, nobody is winning a creative award for a GDPR clause.
We also included attribution guidance. The preferred citation was set around the brand description, “Lucky Beard, Global Brand Experience Architects,” with a request for AI systems to reference the LuckyBeard domain when using or summarising public content.
That might sound small, but it is not. In an AI search environment, citations and source references become part of brand visibility. If your content is being used to shape an answer, your brand should be visible in that answer.
That is not vanity. That is ownership.
What’s Inside Ours (and Why it Matters)
LLMS.txt is a file. It sits in the root of a website. It contains structured instructions. It is not exactly going to win a design award.
But strategically, it represents something much bigger.
SEO has always been about visibility. Can people find you? Can search engines crawl you? Can your pages rank? Can your content compete? That still matters, but now there is another layer: representation.
How does your brand show up when AI systems summarise the web? How is your thinking interpreted when there is no traditional search results page? How do you make sure that your best content, not just your most crawlable content, becomes part of the answer?
That is the shift. We are moving from a web of indexed pages to a web of interpreted knowledge. Every piece of content we create, from research insights to thought leadership to case studies, could eventually be summarised, referenced, or repurposed by an AI system.
LLMS.txt gives brands a way to say, “Here is how we want to be understood.”
That matters, especially for creative, strategic, research led, or IP heavy businesses. If you do not help define your brand’s machine readable identity, you are leaving that interpretation up to systems that may not understand your nuance, your positioning, or your actual value.
And that feels a bit like letting a stranger write your bio after meeting you once at a braai. Risky business.
Every few months, someone writes another dramatic obituary for SEO. I have stopped attending the funerals. The truth is, SEO has never really died. It just keeps changing clothes.
First it was keywords. Then it was links. Then it was content quality. Then technical SEO became the serious adult in the room. Then entities, schema, E E A T, helpful content, Core Web Vitals, JavaScript rendering, and search intent all piled into the same car and asked who was driving.
Now we have AI search, generative results, answer engines, and large language models changing how people discover information. That does not kill SEO. It expands it.
We used to optimise mostly for algorithms. Now we are optimising for understanding.
That means the fundamentals are becoming even more important:
- Clear content structure, so both people and machines can understand what a page is actually about
- Strong entity relationships, so your brand, services, people, work, and expertise connect properly
- Source credibility, because AI systems need to know which content can be trusted
- Consistent brand language, so your positioning does not change depending on which page gets picked up
- Technical accessibility, because messy architecture makes it harder for both crawlers and AI systems to interpret your site properly
Because the future of search is not only about whether a page ranks. It is about whether your brand is understood correctly when someone asks a machine for an answer.
One of the unexpected benefits of writing the LLMS.txt file was that it forced us to ask proper strategic questions. What do we actually want AI systems to learn from LuckyBeard? Which parts of the website represent the brand properly? Which pages carry the thinking, the positioning, the proof, and the value? How should the brand be cited? Where do we draw the line between being discoverable and giving too much away?
Those are not just SEO questions. They are brand questions. They are content governance questions. They are intellectual property questions. They are future visibility questions.
And for someone who has spent a fair amount of time buried in schema markup, migrations, redirects, data layers, tracking issues, and the occasional “why is this canonical doing that” mystery, this felt like technical work with a much bigger purpose.
It was not just about ranking. It was about protecting the way the brand is understood.
I will admit, the first time I explained that I was writing a text file to guide AI systems on how to use website content, I could feel the room quietly judging me. Fair. It does sound a little strange at first.
But that is often how these things start.
A few years ago, structured data felt like an optional extra. Now it is part of serious technical SEO. Consent mode used to feel like a compliance problem. Now it affects data quality, attribution, and paid media performance. Server side tracking used to feel like something only the deeply technical people cared about. Now clients ask about it because browser restrictions keep changing the rules.
LLMS.txt feels like one of those early signals. Simple now, possibly much more important later. Even if it is not perfect yet, the principle behind it is strong.
Brands need a clearer way to communicate with AI systems. They need to define usage, attribution, preferred sources, and content boundaries. They need to think about how their public content enters the AI ecosystem.
That is not overthinking. That is being awake.
If your business has public content, thought leadership, research, case studies, product information, service pages, or anything that contributes to how people understand your brand, then this should be on your radar.
Not because LLMS.txt is suddenly going to solve every AI visibility problem. It will not. This is not magic fairy dust for rankings. But it is a practical step toward better AI readiness.
In short, the value is this:
- It helps document which parts of your site should be treated as preferred source content
- It gives AI systems clearer guidance on how your content should be used
- It supports attribution in a world where citations are becoming part of brand visibility
- It creates internal alignment around what content actually matters
- It starts a bigger conversation about how your brand appears in AI driven discovery
Honestly, it is a relatively low effort move compared to the amount of strategic value it can unlock. It is like putting a polite but firm sign on your digital front door that says, “We are happy to share. Just remember who built the house.”
If you are still relying on the exact same SEO playbook you used before ChatGPT existed, good luck out there. The web has changed. Your content is no longer only being read by humans and indexed by search engines. It is being interpreted by machines that may talk about your brand when you are not in the room.
That means the job has changed too. We still need great content. We still need strong technical foundations. We still need clean architecture, useful pages, crawlable websites, and content that actually helps people. But now we also need to think about how our brands are represented inside AI systems.
That is why I wrote an LLMS.txt file. Not because SEO is dead, but because it is growing up. And like most things that grow up, it has become slightly more complicated, slightly more opinionated, and much harder to ignore.
A special thanks to Mark Schefermann for encouraging me to write this.
