← All posts

What is X

llms.txt: what it is, how to write one, and if you need it

July 6, 2026 • 6 min read

A clear guide to llms.txt: what it is, when to use it, how to implement it, plus examples. See how it relates to SEO and AI answers without the hype.

Teams keep asking two things about llms.txt: do we need it, and what should it say? If you want a simple, useful policy for AI use of your content without slowing down real SEO work, this guide shows what llms.txt is, when it helps, and how to ship a clean version fast.

What llms.txt is and how it works

llms.txt is a plain-text file at the root of your domain that states how AI systems may crawl, use, or train on your content. It is an emerging convention, not a standard. Some AI crawlers will look for it. Others rely only on robots.txt, HTTP headers, or your terms. Treat it as a public, machine-friendly policy note, not an enforcement tool.

What it can express:

  • Scope of permitted use, like training, summarization, or embeddings
  • High level paths that are allowed or disallowed for AI use
  • Licensing or attribution preferences and a contact for data requests
  • Gentle rate guidance so bots do not overload your servers

What it cannot do: guarantee compliance. Respect is voluntary. If you must enforce access, use robots.txt rules per bot, server controls, authentication, and legal terms.

Expect mixed crawler behavior. Known agents like GPTBot, CCBot, ClaudeBot, PerplexityBot, and Google’s fetchers primarily honor robots.txt and sometimes page or header signals. Some will never fetch llms.txt. That is why the file should support, not replace, your other controls.

Do you actually need one?

You probably do if any of these apply:

  • You publish high-volume reference content like docs, datasets, or tutorials.
  • You sell or license content and want a simple, public statement of permitted use.
  • You have members-only areas that must stay out of AI training sets and answers.
  • You field frequent researcher or vendor requests about crawl and reuse.

You can likely wait if your site is a basic brochure with hours, pricing, and a small blog. In that case, focus on the work that moves traffic and revenue now: useful content, internal linking, fast pages, clean indexing, and stable technical health.

llms.txt does not boost rankings and will not force inclusion in AI answers. If your goal is visibility in search and AI overviews, invest in content that solves the query, strong information architecture, and structured data models understand. This is where a practical SEO automation platform helps. If you want a done-for-you blog writing service plus the compounding impact of a backlink building service, look for a system that plans topics around revenue pages, publishes on a reliable cadence, and secures contextual dofollow links from vetted sites. RankGoat automates topic and keyword planning, blog drafting and publishing, on-page SEO and schema, network dofollow backlinks, and daily link monitoring. It also handles multilingual localization and technical SEO cleanup so pages are crawlable and indexable. For AI results, its generative engine optimization work and on-page tweaks improve your odds of inclusion in AI answers. Indexing checks and fixes keep new URLs moving through Search Console, and free tools like a Domain Rating Checker and Sitemap Checker help with quick audits.

How to implement llms.txt the right way

Follow these steps to ship a file that is clear, discoverable, and low-maintenance.

  • Place it at the root. Host at https://yourdomain.com/llms.txt on your canonical hostname. Serve HTTP 200, text/plain, without redirects. If you use www and non-www, redirect one to the other so there is a single source.
  • Keep it short and readable. Simple key-value lines are easier for humans and bots. Avoid nested rules or exotic syntax. Update the date when you change policy.
  • Mirror intent in robots.txt. robots.txt is still the primary control. Add per-bot rules there if you need enforcement. Example snippet to block training bots from private paths: 
    User-agent: GPTBot
Disallow: /members/

User-agent: CCBot
Disallow: /members/

User-agent: *
Allow: /blog/
Disallow: /internal/
  • Use page and header signals where relevant. Meta robots and X-Robots-Tag headers handle indexing and caching. They do not control training, but they keep private or low-value pages out of search indexes and caches that AI systems may query.
  • Protect what must stay private. Authentication walls, paywalls with proper noindex on teaser pages, signed URLs for assets, and rate limits on APIs are stronger than any policy file.
  • Version and cache carefully. Store the file in source control. Set a modest cache TTL, like a day, so changes propagate but you are not hammered by repeated fetches.
  • Test and monitor. After publishing, hit the URL from the outside and confirm a 200 with the expected content. Check logs to see which agents request /llms.txt, how often they crawl, and whether behavior changes. If your team monitors self-hosted services, a platform like Foreseer can alert you to anomalies that may indicate abusive scraping or misconfigured access.

What to include in the file:

  • A short intro with policy date and scope
  • Allow and Disallow paths at a high level
  • License or a pointer to your site terms
  • Attribution preference
  • Suggested crawl delay
  • Contact email

Examples you can adapt

These samples are readable to humans and easy for bots to parse. Adjust paths, license language, and contact details to fit your site and legal terms.

Permissive policy for public docs:

# llms.txt for yourdomain.com
policy: training permitted for public, non-commercial models
policy: summarization permitted with attribution link
allow: /docs/
allow: /blog/
disallow: /account/
disallow: /checkout/
license: see https://yourdomain.com/terms
attribution: link to canonical URL when feasible
rate: crawl-delay 5s suggested
contact: data@yourdomain.com
updated: 2026-07-01

Restrictive policy for member content:

# llms.txt for yourdomain.com
policy: training not permitted
policy: summarization only for publicly accessible pages
allow: /blog/
allow: /press/
disallow: /members/
disallow: /courses/
disallow: /api/
rate: crawl-delay 10s suggested
contact: legal@yourdomain.com
updated: 2026-07-01

Brand-forward policy with clear attribution ask:

# llms.txt for yourdomain.com
policy: training permitted
policy: summarization permitted with visible source citation
allow: /guides/
allow: /resources/
disallow: /staging/
disallow: /tmp/
attribution: include brand name and canonical URL in citation
contact: partnerships@yourdomain.com
updated: 2026-07-01

These are signals, not locks. Keep them aligned with robots.txt and your terms. Review quarterly as the ecosystem evolves.

Key takeaways

  • llms.txt is a simple, public AI-use policy. It is advisory, not enforceable.
  • Use it when you publish at scale, license content, or get frequent data-use requests.
  • Keep syntax simple. State scope, allow and disallow paths, licensing, attribution, rate, and contact.
  • Rely on robots.txt, headers, authentication, and server controls to enforce access.
  • For visibility, prioritize strong content, architecture, links, and technical health. Use automation to scale that work, then add llms.txt to express your preferences.