Many Websites Implement llms.txt Incorrectly: What You Should Know

llms.txt has become a popular topic in technical SEO and AI search discussions, but there is a problem: many websites are adding the file without really understanding what it is supposed to do.

Some sites treat llms.txt like a second robots.txt. Others fill it with every URL on the website, use it as a sitemap replacement, or assume that simply publishing the file will improve their Google rankings.

That is not how the proposal is designed to work.

The current llms.txt proposal describes a Markdown-based file that gives AI agents a concise overview of a website and points them toward useful, more detailed resources. The specification was updated to version 2 in August 2026, so some older articles and implementation guides are already out of date. :contentReference[oaicite:0]{index=0}

More importantly, Google has clarified that llms.txt is not required for Google Search and does not provide a positive or negative ranking effect in Google Search. It can still be maintained for other services or systems that choose to use it. :contentReference[oaicite:1]{index=1}

So if you’re considering adding llms.txt to a website, the useful question isn’t “How do I make Google rank it?” It’s:

“How can I make my site’s information easier for AI agents to understand and retrieve?”

What Is llms.txt?

llms.txt is a proposed standard for providing AI agents with a concise, structured overview of a website.

The basic idea is fairly simple. Instead of forcing an AI agent to work through navigation menus, scripts, advertisements, repeated page elements, and large amounts of unrelated content, a site can provide a small Markdown file that explains what the site contains and links to the resources that matter.

The proposal was introduced by Jeremy Howard in September 2024 and has since evolved. The current version describes llms.txt as a human- and machine-readable Markdown document that can act as a curated entry point into a website’s information.

A very basic structure looks like this:

Example Website

> A short description explaining what this website is about.

Additional information that helps an AI agent understand the website.

Documentation

- [Getting Started](https://example.com/getting-started.md): Basic instructions for new users.
- [API Reference](https://example.com/api.md): Detailed API documentation.

The format is intentionally lightweight. According to the specification, the only required element is the H1 title. A short blockquote summary, additional information, and H2 sections containing lists of links can then be added.

What llms.txt Is Not

This is where many implementations go wrong.

llms.txt is not a replacement for your XML sitemap, it is not a directive system like robots.txt, and it is not a magic switch that tells Google to rank your content differently.

File	Main purpose
robots.txt	Provides crawler access instructions.
sitemap.xml	Lists URLs that search engines can discover and crawl.
llms.txt	Provides a curated, AI-friendly overview and links to useful resources.

The distinction matters because these files solve different problems.

The llms.txt proposal itself explicitly describes robots.txt and llms.txt as having different purposes. A sitemap generally describes the site’s indexable pages, while llms.txt is intended to provide a more selective route into information that an AI agent may need.

Google Does Not Need llms.txt for Search Rankings

This is probably the most important clarification for SEO professionals.

Google updated its Search documentation in June 2026 to clarify that llms.txt is not needed for Google Search. Google also states that maintaining one will not positively or negatively affect Search visibility or rankings.

That means you should not implement llms.txt because someone promised that it will:

• increase organic rankings;
• increase Domain Authority;
• make Google crawl a site faster;
• guarantee inclusion in AI Overviews;
• guarantee citations in ChatGPT or other AI systems;
• replace structured data or XML sitemaps.

None of those outcomes should be presented as guaranteed benefits of an llms.txt file.

There is still a potential use case, though. The proposal is aimed at AI agents and systems that choose to consume it. That makes it more relevant to the broader question of how websites expose information to AI systems than to traditional Google ranking optimization.

Why Many llms.txt Implementations Are Weak

Simply having the file is not the same thing as implementing it well.

A website can have a perfectly valid URL such as:

https://example.com/llms.txt

and still provide very little useful information.

Here are some of the common problems.

1. Treating llms.txt Like a Sitemap

One of the easiest mistakes is dumping hundreds or thousands of URLs into the file.

That defeats much of the point.

The proposal describes llms.txt as a concise overview that helps an agent find relevant information. A giant list of URLs doesn’t necessarily help an agent understand which resources matter or how they relate to the site.

Think of the difference this way:

Sitemap: “Here are the URLs on my website.”

llms.txt: “Here is what my website is about, and these are the resources you should look at to understand it.”

Those are very different jobs.

2. Adding Marketing Copy Instead of Useful Context

An llms.txt file is not a place to repeat your homepage sales pitch.

Statements such as “We are the world’s leading SEO platform” don’t provide much useful context unless they are relevant to understanding the site’s information.

A better summary explains what the website actually contains.

SerpBacklink

> SerpBacklink is an SEO resource covering backlinks, technical SEO,
> search optimization, AI tools, and digital marketing.

The goal is clarity, not persuasion.

3. Linking to Pages Without Explaining Them

The current specification allows links to be accompanied by short descriptions. Those descriptions are useful because they give the agent additional context before it decides whether to retrieve a resource.

Compare these:

- [SEO Guide](https://example.com/seo-guide)

with:

- [Technical SEO Guide](https://example.com/technical-seo-guide.md):
  Covers crawling, indexing, canonical URLs, redirects, and common technical issues.

The second version communicates much more information without becoming unnecessarily long.

4. Linking Only to HTML Pages When Better Machine-Readable Content Exists

This is one of the more interesting changes in version 2 of the proposal.

The current specification recommends providing clean Markdown versions of pages that agents may need. These can use either an appended .md extension or a replaced extension, depending on the implementation.

For example:

https://example.com/docs/api
https://example.com/docs/api.md

The proposal also recommends standard link relations such as:

<link rel="alternate" type="text/markdown" href="/docs/api.md">
<link rel="describedby" href="/llms.txt">

These relationships can also be communicated through HTTP headers.

This approach is considerably more useful than simply creating an llms.txt file and filling it with normal webpage URLs.

5. Creating an llms.txt File and Never Updating It

A stale llms.txt file can become misleading.

If your documentation changes, products are removed, URLs are redirected, or important resources are replaced, the file should reflect those changes.

For a site with frequently changing content, this is particularly important.

Think of the file as a maintained navigation layer, not a one-time SEO installation.

What Should an llms.txt File Contain?

There is no requirement to turn the file into a giant document.

The current proposal is intentionally compact. A useful implementation can contain:

• A clear H1 containing the website or project name.
• A short summary explaining what the site or project is.
• Important context that helps an agent interpret the information.
• Sections grouping related resources.
• Links to important, useful content.
• Short descriptions explaining what each linked resource contains.

The specification requires the H1, while the other elements are optional.

For a software documentation site, sections might look like:

- Getting Started

- API Reference

- Tutorials

- Examples

- Troubleshooting

- Optional

For an SEO website, the categories could be completely different:

- SEO Fundamentals

- Technical SEO

- Link Building

- AI Search

- SEO Tools

- Digital Marketing

The structure should reflect the information architecture of the actual site.

Should You Put Every Article in llms.txt?

Usually, there is no good reason to blindly include every article.

The better question is whether a resource helps an AI agent understand the site’s subject matter or answer questions about it.

For example, an SEO publication might prioritize:

• core guides;
• technical documentation;
• important definitions;
• original research;
• product documentation;
• frequently referenced resources;
• high-value tutorials.

A large archive of thin news posts may not deserve the same priority.

This is where editorial judgment becomes more useful than automation.

llms.txt and AI Search Are Not the Same Thing

It is tempting to describe llms.txt as an “AI SEO file,” but that description can create the wrong expectations.

AI systems differ in how they retrieve, index, process, and cite information. There is no universal rule saying that publishing llms.txt will make an AI system use your website.

The proposal itself describes llms.txt as something agents can view or search to discover useful information and then follow relevant links. That is different from guaranteeing that every AI system will consume the file.

So if your goal is better visibility in AI-generated answers, don’t reduce the entire strategy to one text file.

A stronger foundation includes:

• clear and accurate website content;
• strong information architecture;
• crawlable pages;
• descriptive titles and headings;
• useful internal linking;
• structured data where appropriate;
• clear authorship and source information;
• regularly maintained content;
• reliable external references;
• and, where useful, machine-readable resources such as Markdown documentation.

llms.txt can sit alongside those practices. It should not replace them.

How to Create a Better llms.txt File

If you want to implement it, start with the information architecture rather than the file itself.

Step 1: Define What an AI Agent Should Understand

Write down the essential information about the website.

For example:

• What does the site cover?
• Who is the intended audience?
• Which resources are authoritative?
• Which pages explain the most important concepts?
• Which documentation should an agent read first?

Keep this focused.

Step 2: Select the Important Resources

Don’t start by exporting your entire URL database.

Choose resources that genuinely help explain the website’s subject matter.

For each URL, ask:

“If an AI agent had only a limited amount of time and context, would this page be worth fetching?”

If the answer is no, the page probably doesn’t need to be prioritized.

Step 3: Use Descriptive Link Text

Avoid vague labels such as “Click here” or “Read more.”

Use titles that explain what the resource contains.

- [How Domain Authority Works](https://example.com/domain-authority.md):
  Explains Domain Authority, how it is calculated by third-party tools,
  and why it should not be treated as a Google ranking metric.

That description provides useful context before the linked page is even retrieved.

Step 4: Consider Markdown Versions of Important Pages

If your site contains documentation or technical resources, clean Markdown representations can make the information easier for agents to process.

The current proposal specifically recommends Markdown versions for pages containing information that agents may need.

This can be particularly useful for documentation-heavy websites.

Step 5: Keep It Maintained

Check the file whenever you restructure the site.

Look for:

• 404 URLs;
• redirect chains;
• removed pages;
• outdated descriptions;
• duplicate resources;
• incorrect categorization.

A small, accurate file is more useful than a large file full of outdated links.

What About WordPress?

WordPress users may encounter plugins that generate llms.txt automatically. The current llms.txt documentation lists integrations from several platforms and tools, including Yoast SEO, AIOSEO, and Wix.

Automation can be convenient, but it doesn’t automatically mean the resulting file is well-designed.

Before relying on a plugin-generated file, inspect the actual output.

Ask:

• Does the file contain a useful description?
• Are important resources prioritized?
• Are the links relevant?
• Are descriptions meaningful?
• Are outdated or low-value pages being included?
• Does the output follow the current specification?

This is especially important because the specification has changed over time. An implementation based on an older article or older plugin behavior may not reflect the current version of the proposal.

Common llms.txt Mistakes to Avoid

Mistake	Why It Is a Problem	Better Approach
Using it as a sitemap	Creates a large URL dump instead of useful context.	Curate important resources.
Expecting Google ranking benefits	Google says it is not required for Search and does not affect rankings.	Treat it as an optional AI-facing resource.
Adding promotional copy	Marketing language provides limited contextual value.	Explain the site’s purpose clearly.
Using vague link descriptions	Agents receive less context about linked resources.	Describe what each resource contains.
Ignoring Markdown resources	Important machine-readable content may be harder to discover.	Consider clean Markdown versions for suitable pages.
Never updating the file	Links and descriptions can become inaccurate.	Maintain it alongside site changes.
Assuming every AI system uses it	Different systems use different retrieval methods.	Treat adoption as dependent on the consuming system.

Does llms.txt Improve SEO?

If by “SEO” you mean Google Search rankings, there is an important distinction.

Google currently says llms.txt does not positively or negatively affect Search visibility or rankings.

So adding the file should not be presented as a ranking tactic.

If you’re interested in AI-facing discoverability, however, the file may serve a different purpose for systems that choose to use it. The proposal is specifically designed to help agents find and understand relevant information on a website.

That makes llms.txt better understood as an optional machine-readable information layer rather than a conventional Google SEO signal.

Should Every Website Have llms.txt?

Not necessarily.

A documentation-heavy software project has an obvious use case because agents may need to navigate APIs, tutorials, examples, and reference material.

A simple brochure website with five pages may gain much less from maintaining a dedicated file.

The proposal itself notes that the approach is particularly useful for software documentation, while also describing potential applications for businesses, personal websites, schools, and other information-rich sites.

In other words, implementation should follow the site’s needs.

How to Audit an Existing llms.txt File

If your website already has one, don’t assume that “file exists” means “implementation is complete.”

Run a quick audit.

• Check accessibility: Can the file be retrieved at its intended URL?
• Check structure: Does it follow the current Markdown-based format?
• Check relevance: Do the listed resources actually matter?
• Check descriptions: Does each important link explain what the resource contains?
• Check URLs: Do they return the expected content?
• Check freshness: Are removed or outdated pages still listed?
• Check machine-readable content: Are useful Markdown versions available where appropriate?
• Check expectations: Are you treating the file as an AI information layer rather than a Google ranking mechanism?

You can also test the file from the perspective of an AI agent: give the agent the llms.txt as its starting point and see whether it can correctly identify the site’s important resources. The llms.txt project itself recommends testing the file with an agent to evaluate how useful the structure is.

The Bigger SEO Lesson

The interesting part of llms.txt isn’t the filename.

It’s the idea behind it: make your website’s information easier for machines to understand without making it harder for humans to use.

That principle is useful even if a particular AI system never reads your llms.txt file.

Clear information architecture, concise explanations, strong internal linking, authoritative references, structured information, and clean documentation are already good practices for a useful website.

So don’t spend weeks obsessing over a tiny text file while the actual website is difficult to understand.

Fix the content first.

Then improve the machine-readable layer around it.

Frequently Asked Questions

Does Google use llms.txt for rankings?

No. Google clarified in June 2026 that llms.txt is not needed for Google Search and does not have a positive or negative effect on Search visibility or rankings.

Is llms.txt the same as robots.txt?

No. robots.txt provides crawler access instructions, while llms.txt is intended to provide AI agents with a concise overview and links to useful information.

Is llms.txt a replacement for sitemap.xml?

No. A sitemap and llms.txt serve different purposes. A sitemap helps expose URLs, while llms.txt provides a curated information layer for systems that choose to use it.

Should llms.txt contain every URL on a website?

Not by default. The proposal emphasizes a concise overview and curated links to useful resources. A large URL dump can undermine that purpose.

Does llms.txt guarantee citations in AI answers?

No. There is no universal guarantee that an AI system will retrieve or cite information from an llms.txt file. Different systems can use different retrieval methods.

Should I create Markdown versions of my pages?

For documentation and information that AI agents may need, the current llms.txt proposal recommends clean Markdown versions of relevant pages. This is particularly useful for documentation-heavy websites.

Does an llms.txt file need to be updated?

Yes, if you choose to maintain one. Links, descriptions, documentation, and site structure can change, so an outdated file can become less useful.

Final Takeaway

There is nothing wrong with adding llms.txt to a website, but it is easy to misunderstand what the file is for.

Don’t treat it as a secret Google ranking signal. Don’t turn it into a giant sitemap. And don’t assume that publishing /llms.txt automatically makes a website visible to every AI system.

Instead, use it as a concise map of your site’s most useful information for AI agents that support the format.

For websites with substantial documentation or knowledge resources, that can be genuinely useful. For smaller sites, it may be optional and relatively low priority.

The practical rule is simple: build a useful website first, then make that information easier for machines to understand.

That is a much more sustainable approach than chasing every new “AI SEO” tactic that appears.

References:

• llms.txt specification and proposal
• llms.txt specification changes
• Google Search documentation updates