llms.txt and AI crawlers: make your site readable by machines on purpose
A generated llms.txt that maps your best pages, robots.txt rules that reflect a decision rather than a default, and HTML that AI crawlers can actually read. This site runs all three.
- Generated, never stale
- Crawler policy you chose
- 2–5 day setup
A map of your site, written for language models
llms.txt is a proposed convention: a Markdown file at the root of a site that tells a language model what the site is, in a sentence or two, and lists the pages worth reading with a one-line summary each. The structure is deliberately simple: a heading, a blockquote summary, then sections of links.
The idea is that a model with a limited context window should not have to crawl two hundred pages to understand a company. It reads one file, chooses the relevant pages, and fetches those. A companion file, llms-full.txt, can carry the full content for sites that want it.
Adoption by AI engines is partial and shifting, and we say so. We publish it anyway because the cost is a few lines in the build, it documents the site for agents of every kind, including the MCP servers and agents we build, and it forces a useful discipline: every page needs a title and a description that mean something.
# Teksmobile
> AI consulting, implementation and secure
> engineering since 2006. Teknowledge Software
> Limited (UK). Offices: United Kingdom,
> Australia, India, Sweden, South Korea.
## AI SEO & Marketing
- [AI SEO & Marketing Services](https://teksmobile.com/ai-seo/):
Technical SEO, generative engine optimisation
and marketing engineering from developers…
- [llms.txt & AI Crawler Setup](https://teksmobile.com/ai-seo/llms-txt/):
What llms.txt is, how we generate it…
## AI Glossary
- [llms.txt](https://teksmobile.com/glossary/llms-txt/):
A proposed root file that maps a site…Excerpt in the format of this site's own /llms.txt, generated at build time from page metadata.
What an AI crawler checks, in order
AI bots announce themselves. Decide what each may do.
Most AI companies publish the user agent strings of their crawlers and honour robots.txt. Some run separate bots for training data and for live retrieval when a user asks a question. The names and behaviours change, so we check vendor documentation at the time of setup rather than relying on a list.
| User agent | Operator | Typical purpose |
|---|---|---|
GPTBot | OpenAI | Crawling for model training and search features |
ClaudeBot | Anthropic | Crawling for model training |
PerplexityBot | Perplexity | Indexing for the answer engine |
Google-Extended | A control token for AI training use, separate from Googlebot | |
CCBot | Common Crawl | Open web archive used by many model trainers |
Purposes are summarised from vendor documentation at the time of writing and change. Verify before relying on them.
# Example: allow live retrieval, limit training
User-agent: *
Allow: /
Disallow: /api/
Disallow: /style-guide/
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xmlAn illustrative policy. Which bots to allow is a business decision we work through with you, not a default.
Three policies and what each costs you
| Policy | AI visibility | Content control | Fits when |
|---|---|---|---|
| Allow everything | Maximum | None | You sell services or products and want to be found and cited everywhere |
| Allow retrieval, block training | Good for live answers | Partial, where vendors separate the two | You want citations but not to feed training sets; most B2B sites |
| Block all AI crawlers | None in AI answers | Maximum, for compliant bots | Your content is the product: paywalled data, licensed research, publishing |
robots.txt is a request, not a wall. Compliant bots respect it; others do not. For content that must not be fetched, use authentication or rate limiting at the edge, which we set up on Cloudflare. For questions about personal data in AI systems, see AI data privacy.
A short engagement with a long shelf life
Review
Current robots.txt, log sample of AI bot activity, rendering test of key templates, existing llms.txt if any. Day 1–2.
Decide
A one-page policy: which bots, which paths, why. Signed off by whoever owns the content. Day 2–3.
Generate
llms.txt built from your metadata in the build, CMS or a scheduled job. Descriptions fixed where they are missing or vague. Day 3–5.
Verify
Fetch as each bot, confirm the files and rendering, add the check to the visibility monitor.
Being readable is step one
llms.txt, defined
The glossary entry, in two paragraphs.
Read itGEO programme
Once they can read you, give them something worth citing.
Start GEOStructured Data
The facts, in the form machines prefer.
Implement schemaAI Knowledge Base
The same retrieval mechanics, built for your own documents.
How retrieval worksFrequently asked questions
Is llms.txt an official standard?
No. It is a proposal published in 2024 that has been adopted by a number of documentation sites, developer tools and agencies. Support among AI engines is uneven and changes. It costs almost nothing to publish, it documents your site for any agent that reads it, and it does no harm if ignored.
Does llms.txt replace robots.txt or the sitemap?
No. robots.txt controls which crawlers may fetch which paths. The sitemap lists URLs for indexing. llms.txt is a curated, human-readable map of your most useful pages with one-line summaries, written for language models and agents. All three do different jobs.
Should we block AI crawlers?
It depends on what you sell. If your content is the product, blocking training crawlers can be sensible. If you want to be found and cited by AI assistants, blocking them makes you invisible there. Many vendors run separate crawlers for training and for live retrieval, so you can often allow one and not the other. We map the trade-offs with you before changing anything.
How do you generate llms.txt?
From the same metadata that builds the site: page titles, descriptions and sections, grouped and sorted. This site's llms.txt is produced that way at build time, so it is never out of date. For CMS sites we add a small endpoint or a scheduled export.
Can AI crawlers read our JavaScript-rendered pages?
Assume not. Most AI crawlers fetch raw HTML and do not execute JavaScript the way Googlebot does. If your main content only appears after a script runs, it is not available to them. Server-side rendering or static generation fixes this and helps classic search as well.