On this page
- What an llms.txt file is, and the problem it solves for AI models
- llms.txt vs llms-full.txt vs robots.txt vs sitemap
- How to write an llms.txt file, section by section
- A worked example: the llms.txt file we run
- Do AI tools actually use llms.txt? The honest take
- The method we use to write and QA an llms.txt file
- Frequently asked questions
- Where an llms.txt file fits in your AI visibility
Most websites hand AI tools the same tangled HTML they hand a browser: navigation, cookie banners, scripts, and the actual content buried somewhere in the middle. An llms.txt file fixes that. It is a short, linkable Markdown index that points AI crawlers and AI assistants straight at the pages that matter, in a format they can parse without wading through the rest.
This is a practitioner’s guide, not a glossary. By the end you will know what goes in an llms.txt file, how it differs from the files you already run, how to write one, and where the honest limits sit. We run one on our own website and build them for clients as part of GEO, so this is what we actually do, not what a spec sheet says we should.
What an llms.txt file is, and the problem it solves for AI models
Large language models work inside context windows. A context window is the fixed amount of text a model can hold at once, and it is smaller than most people assume. They cannot simply ingest your whole site, so when an AI tool reads a live page to understand your business, it burns that budget on markup, menus, and scripts before reaching anything useful. Complex HTML pages are expensive to read and easy to misread, which is where most websites lose the machine.
Why markdown files help AI crawlers
The llms.txt file solves this by doing the curation for the machine. Instead of leaving an AI system to guess which of your hundreds of URLs matter, you hand it one file that says: here is who we are, and here are the pages worth reading first. Converting complex HTML pages into clean Markdown files strips the noise, so AI systems spend their limited context windows on your key content. Markdown is close to a plain text file, cheap to parse and hard to misread, and because you curate it the model reads your priorities, not its guesses.
Jeremy Howard, co-founder of Answer.AI, proposed the format in September 2024, and the spec lives at llmstxt.org. It is a proposed standard, not an official one, which matters for how much weight you give it. It has not been ratified by a standards body such as the W3C or IETF, the way robots.txt was eventually formalised as RFC 9309, so treat it as a community convention rather than a settled rule. The file is deliberately simple: one Markdown file, one location, no server config, no schema validation, cheap enough to run while the industry works out whether it moves the needle.
An llms.txt file is you doing the machine’s reading for it, so it spends its limited attention on your content instead of your chrome.
llms.txt vs llms-full.txt vs robots.txt vs sitemap
The biggest source of confusion is assuming these files overlap. They do not. Each talks to a different reader and answers a different question, so running one does not replace another.
Robots.txt is the Robots Exclusion Protocol. It controls which crawlers may access which paths on your site. It is a permissions file, not a content file, and it is not Markdown. An XML sitemap is a machine-readable list of the indexable URLs on your site, built to help search engines discover and crawl pages. It lists locations. It does not curate or explain them, and neither file was designed to tell a large language model which content to read first the way a sitemap tells search engines which URLs to crawl.
The llms.txt file is different in kind. It is a curated Markdown index of your most important content, written for AI, with a one-line note on why each linked page matters. Its companion, the llms-full.txt file, goes further: instead of linking out, it concatenates the actual content of your key pages into one long Markdown document, so an AI tool can read your substantive content in a single fetch. That convention is looser than the core spec, best treated as a de facto pattern rather than a rule.
The reason this matters is that AI systems do not follow the fixed processing methods a crawler runs. They interpret what they read, so cleaner input shapes more accurate AI generated responses, for chatbots and AI agents alike.
| File | Reader | Format | Job |
|---|---|---|---|
robots.txt | Search and AI crawlers | Plain text directives | Controls which paths a crawler may access |
sitemap.xml | Search engines | XML | Lists indexable URLs for discovery |
llms.txt | Large language models and AI tools | Markdown index | Curates and links your key content for AI |
llms-full.txt | AI systems and AI tools | Concatenated Markdown | Ships the full content of key pages in one file |
The short version: robots.txt says what may be read, the sitemap says what exists, and the llms.txt file says what to read first. They are complementary, so running one is no reason to drop your sitemap. Search engines still lean on the sitemap and robots.txt exactly as before.
How to write an llms.txt file, section by section
The spec gives you a basic structure and little ceremony: a file a human could read in thirty seconds and a model could parse in one pass. You write it in any plain text editor, with no tooling to install. Here is the sequence we follow.
- Start with an H1 holding your site or project name. Exactly one H1. This is the only required element in the whole file.
- Add a blockquote summary. A single
>line that says what you do in one sentence. This is the line an AI tool is most likely to lift verbatim, so make it accurate and specific. - Write optional detail paragraphs. A few plain sentences of freeform context under the summary. Keep it factual, no marketing padding, because padding wastes the model's context.
- Build your
##sections. Group your links by theme (Services, Reference, Educational, Locations, Contact). Each section holds a Markdown list of[name](url): one-line notelinks. The note tells the model what the page is for. - Finish with an
## Optionalsection. This is a spec feature, not an afterthought: links here are the ones an AI tool can safely skip when its context window is tight. Put your llms-full.txt link and any nice-to-have pages here.
That is the entire structure. Here is a copyable skeleton to fill in. This is the minimal valid file we recommend starting with. As the worked example below shows, you do not need nineteen case studies or three files to have a complete implementation:
# Your Company Name
> One-sentence summary of what you do and who you serve.
Optional freeform detail. A sentence or two of plain context an AI
model should treat as background about the business.
## Services
- [Service one](https://yourdomain.com/service-one/): what it is, one line.
- [Service two](https://yourdomain.com/service-two/): what it is, one line.
## Reference
- [About](https://yourdomain.com/about/): who we are and our track record.
- [Case studies](https://yourdomain.com/case-studies/): verified client results.
## Optional
- [llms-full.txt](https://yourdomain.com/llms-full.txt): full content of key pages in one file.A few practical notes from running these. Use descriptive link notes, not keyword lists: the note is context for the model, not a place to stuff terms. Link to pages that genuinely answer a buyer question, not utility pages or the internal links in your footer. Use the same clean discipline you would in programming documentation: clear headings, tight lists, code blocks only where they earn their place. Both are machine readable files, plain Markdown files built for parsing, so treat yours as a table of contents, not a page for browsing. Refresh the file after any significant content change, the same cadence you would give a sitemap, so the index never points a model at a page you have moved or retired.
One AI search tactic, every Friday.
AI On Fire: a working note from the Firewire desk, sent to a few thousand Australian search marketers every Friday.
No spam. Unsubscribe anytime.
A worked example: the llms.txt file we run
Specs are easy to nod along to and hard to copy, so here is one we run on our own site. Our file at /llms.txt follows the spec exactly. It opens with a single H1, # Firewire Digital, then a blockquote summary:
# Firewire Digital
> Specialist search marketing agency for established Australian
> businesses. SEO, Google Ads, and Generative Engine Optimisation
> (GEO). A remote-first team rooted in Newcastle, NSW. Founded 2017.From there the file runs themed ## sections. ## Services links each service page with a one-line note. ## Reference links our /llm-info/ page, described in the file as the page AI assistants should treat as authoritative, alongside about and case studies. Then ## Recognition, ## Educational, ## Locations, and ## Contact, closing with an ## Optional section that links /llms-full.txt.
Our /llms-full.txt is a single long file, roughly 3,500 to 4,000 words, concatenating the content of our most important pages: about, every service, nineteen case studies, and the FAQs. One fetch, the whole story, no navigation. Our /llm-info/ page is the structured dossier we point AI assistants at, with an explicit section on how we want them to represent Firewire, and the llms.txt file names it as authoritative so a model knows which page to trust first.
You do not need nineteen case studies or three Markdown files to start. One clean llms.txt file at your root, following the skeleton above, is a complete and valid implementation.
Do AI tools actually use llms.txt? The honest take
Here is the part most guides skip. As of mid-2026, no major AI provider (OpenAI, Anthropic, or Google) has publicly confirmed that their AI crawlers or models consume llms.txt for ranking or citation. Anyone claiming a direct citation win from an llms.txt file is guessing, and that includes us. We cannot attribute a single AI citation to our own llms.txt file, and we will not pretend to.
Adoption is still early and niche. Independent trackers through 2025 counted anywhere from under a thousand to a few thousand domains with an llms.txt file, growing fast off a tiny base, and one analysis found the overwhelming majority of those files received zero AI-crawler requests. The early adopters are concentrated in developer-tool and documentation sites: Anthropic, Stripe and Vercel all publish one for their docs, the teams most comfortable maintaining a Markdown index. Scepticism exists too. Google’s John Mueller compared llms.txt to the old keywords meta tag on Reddit in April 2025, a site owner’s unverified claim about what their site is about, the kind of signal that was easy to game and that search engines eventually ignored. That comparison is worth taking seriously, though not everyone agrees. Others point out that llms.txt is different because it links to real pages whose content has to actually deliver, not a self-declared metadata field, a point Search Engine Land argued directly in a July 2025 piece.
So why do we run one, and build them for clients? Because it is low-cost, spec-clean, and future-facing. Writing a good llms.txt file forces you to articulate your key content clearly, useful no matter who reads it. A tight file hands AI models concise, expert-level information about what you do in one place, and it can improve how easily AI tools discover the content you most want quoted. If you want more visibility in generative AI search, it is a cheap hedge. If AI driven search starts consuming these files at scale, the sites already running a clean one are ready. If it does not, you have lost an hour on a tidy file. A sensible bet, not a growth hack, and we sell it as exactly that.
To be clear about where we can prove AI results: our client Ampcontrol is named as source #1 by ChatGPT for Australian battery energy storage, verified as at June 2026. That came from real GEO work, entity and schema engineering tracked through our own AI monitoring across the major engines, not from an llms.txt file. We keep the two separate on purpose, because conflating them is exactly the attribution theatre this guide argues against.
The method we use to write and QA an llms.txt file
The file is simple. Making sure it works is the part people skip. Here is the repeatable method we run, and it doubles as the AI check most implementations miss.
First, pull your key URLs. Take the XML sitemap and your top pages by traffic, and pick the handful that genuinely answer a buyer question rather than dumping the entire site into the index, because a bloated llms.txt file defeats its own purpose. Then draft the Markdown in the spec structure above, one clean note per link.
Now the test that separates a working file from a hopeful one. Paste the finished llms.txt file into a large language model and ask it a real buyer question, telling it to use only that file. If the model cannot find your key page, or answers wrong, the index has failed, so fix the notes or the links. This catches the two most common faults: a summary too vague to be quoted, and a key page left out entirely.
The one check this cannot replace is a human confirming the linked pages actually answer the question. The model tells you whether it can find and parse your content. It will not reliably tell you whether that content is any good. That judgement stays with a person, the same line we draw across all our AI work.
Frequently asked questions
What is an llms.txt file?
An llms.txt file is a Markdown file at your site root that gives AI tools a curated index of your most important content, each link carrying a one-line note. Proposed by Jeremy Howard in September 2024, it helps AI tools find your key content without wading through complex HTML and scripts.
Is llms.txt the same as a sitemap?
No. An XML sitemap is a machine-readable list of indexable URLs built to help search engines discover pages across your entire site. An llms.txt file is a curated Markdown index written for large language models that prioritises a few key pages, not every URL. Different reader, different job. Run both.
Do AI tools actually use llms.txt?
As of mid-2026, no major AI provider has publicly confirmed that their crawlers or models consume llms.txt for ranking or citation. Adoption is early, concentrated in developer and documentation sites, and some search figures remain sceptical. We run it because it is low-cost and future-facing, not because we can attribute a citation.
Where do I put my llms.txt file?
At the root of your domain, so it resolves at yourdomain.com/llms.txt. It is a single plain text file in Markdown format, the same root-level convention as robots.txt. No server configuration or special headers required. Any llms-full.txt file sits alongside it at the root.
What is llms-full.txt?
The llms-full.txt file concatenates the actual content of your key pages into one long Markdown file, rather than just linking to them like llms.txt does. It lets an AI tool read your substantive content in a single fetch. It is a looser, de facto convention, not part of the core proposed standard.
Do I need llms.txt if I already have robots.txt?
Yes, if you want the AI benefit, because they do different jobs. Robots.txt controls which crawlers may access which paths on your site. It does not curate content or tell a model what to read first. The llms.txt file is a Markdown index for AI tools. One governs access, the other attention.
Where an llms.txt file fits in your AI visibility
An llms.txt file is one clean, low-cost signal in a much larger picture. Real AI visibility comes from the entity, schema, and authority work that makes your brand legible to AI models in the first place, the foundation covered in our parent guide to AI search optimisation and our approach to AI SEO. The llms.txt file rides on top of that foundation. It is not a substitute for it.
If you want the file built and QA’d properly, and the foundation under it built so AI systems cite you rather than scrape you invisibly, that is the work our GEO service runs. We will set up your llms.txt file, and we will be honest about exactly what it can and cannot do.
