robots.txt vs llms.txt: What Each File Actually Controls (And Why Most Advice Is Wrong)

What most guides get wrong

The dominant advice online treats llms.txt as "robots.txt for AI," implying identical authority. It is not. robots.txt relies on a widely honored convention with a long enforcement record. llms.txt is a proposal with no binding commitment from major AI providers. Adding it does not force any model to read your content a certain way. The second error is the assumption that blocking AI crawlers protects you.

In practice, blocking a crawler in robots.txt often removes you from that provider's AI answers entirely. For a personal injury firm or a wealth manager, disappearing from an assistant's response is not protection. It is lost demand.

The third error is scale advice. Generic guides tell everyone to add llms.txt. What I have found is that llms.txt tends to help large, complex sites with sprawling documentation far more than a tight ten-page practice site.

Context matters, and most guides skip it entirely.

What Does robots.txt Actually Control, And What Does llms.txt Control?

The cleanest way to hold these apart is to name the job each one does. robots.txt does access control. It lives at the root of your domain and tells automated agents which paths they may or may not crawl. It uses User-agent, Disallow, and Allow directives. When a compliant crawler arrives, it reads this file first and adjusts its behavior.

Google, Bing, and most reputable AI crawlers respect it. The important nuance: robots.txt governs crawling, not indexing and not usage of already-ingested data. A page blocked in robots.txt can still appear in results if it is linked elsewhere, because blocking crawl is not the same as blocking display. llms.txt does attention guidance. It is a proposed markdown file, also placed at your root, that gives language models a curated map of your most important content.

Think of it as a reading list with links and short descriptions, plus optional full-text sections. It does not block anything. It does not enforce anything.

It proposes: "If you are going to read my site, start here, and here is the plain-language context." Consider a healthcare provider with a large patient-education library. robots.txt might block /admin/ and /patient-portal/ from any crawler. That is access control protecting private and irrelevant paths. Meanwhile, an llms.txt file could point a model to the clinically reviewed condition pages, each with an author, a review date, and citations, so that when an assistant answers a question about a treatment, it draws from the reviewed content rather than an old blog post.

The distinction I keep returning to with clients: robots.txt decides reachability. llms.txt influences interpretation. You need both jobs handled. You do not choose between them any more than a law firm chooses between locking its file room and organizing its case index.

• robots.txt uses User-agent, Disallow, and Allow to govern crawl access.
• robots.txt controls crawling, not indexing or use of already-ingested content.
• llms.txt is a markdown reading list at your root that proposes priority content.
• llms.txt has no enforcement mechanism and blocks nothing.
• A blocked page can still surface if linked externally, so blocking is not hiding.
• In regulated sites, robots.txt protects private paths while llms.txt surfaces reviewed content.

Which File Is Actually Enforced, And Which Is Just a Polite Request?

This is the section most guides avoid because it undercuts the hype. Let me be direct about the enforcement gap. robots.txt carries real weight in practice. Reputable search and AI crawlers read it and adjust. It is not a legal contract, and bad actors ignore it, but the mainstream ecosystem treats it as a convention worth honoring.

When you Disallow a path, you can reasonably expect compliant crawlers to stay out. Two decades of observed behavior support that expectation. llms.txt has no such record. As of this writing, no major model provider has publicly committed to obeying llms.txt the way search engines obey robots.txt. It is a community proposal.

Some tools and smaller agents may read it. That is genuinely useful for developer-facing documentation, where an AI coding assistant benefits from a curated map. But if you publish llms.txt expecting an assistant to only cite the pages you listed, you will be disappointed.

Models read broadly. What I have found is that the honest framing changes how you invest. I do not tell a financial advisory client to spend a week hand-writing llms.txt content sections and then treat it as a control mechanism.

I tell them: robots.txt is where you make enforceable decisions about access, and llms.txt is where you offer a helpful, well-structured hint that costs little and may compound as adoption grows. There is a compliance angle here too. In regulated verticals, documented intent matters.

A clean robots.txt paired with a maintained llms.txt shows a governance body that you have a deliberate, reviewable policy for how automated agents interact with your content. That documentation has value even where enforcement is imperfect, because it demonstrates process. The practical rule I use: make your enforceable decisions in robots.txt, make your interpretive suggestions in llms.txt, and never confuse a suggestion for a guarantee.

• robots.txt is honored by reputable crawlers based on long-observed behavior.
• llms.txt has no binding commitment from major AI providers to date.
• llms.txt is genuinely useful for developer docs and structured knowledge bases.
• Do not expect llms.txt to restrict what a model reads or cites.
• In regulated fields, both files together form a reviewable governance artifact.
• Enforceable choices belong in robots.txt; interpretive hints belong in llms.txt.

The Access vs Attention Framework: How to Decide What Goes Where

Here is the first framework I give clients, because it removes the false versus and replaces it with a decision that actually maps to how the files work. I call it the Access vs Attention Framework. For any page or path, ask two separate questions: Question one, Access: should an automated agent be able to reach this at all? If the answer is no, that is a robots.txt job. Private portals, staging environments, thin filter pages, internal search results, duplicate parameter URLs.

These are access decisions. You Disallow them and move on. Question two, Attention: if an agent can reach this, does it deserve to be near the top of what a model reads to understand my expertise? If yes, that is an llms.txt job. Your cornerstone practice-area pages, your clinically reviewed guides, your regulatory disclosure explainers written in plain language.

You list them in llms.txt with a short, honest description each. The power of separating the questions is that it exposes contradictions. I have seen sites that Disallowed their most authoritative content in robots.txt (usually by accident, through an overly broad rule) while listing that same content in llms.txt.

The model was told "pay attention to this" and "you cannot reach this" simultaneously. Access always wins that conflict. The attention hint is worthless if the door is locked.

Run this framework as a table. One column for the URL pattern, one for the Access decision, one for the Attention decision. For a healthcare client, /patient-portal/ scores block on access and irrelevant on attention. /conditions/reviewed/* scores allow on access and high on attention. /tag/ archive pages might score allow on access but low on attention, so they stay out of llms.txt without being blocked.

What this framework prevents is the most expensive error: blocking the thing you most want cited. Do the access pass first, then the attention pass, and reconcile any conflicts by trusting the access layer. Attention only matters where access is granted.

• Ask two separate questions per page: reachability and priority.
• Reachability decisions belong in robots.txt as Allow or Disallow.
• Priority decisions belong in llms.txt as listed, described links.
• Access always overrides attention when the two conflict.
• Build a table: URL pattern, access decision, attention decision.
• Never list a Disallowed path in llms.txt; it signals a locked door with a welcome mat.

The Read-Path Audit: How to See What an AI Model Would Prioritize

The second framework I use is the Read-Path Audit. It answers a question most site owners never ask: if a language model followed my llms.txt as intended, what picture of my expertise would it build, and is that picture accurate and current? Here is the process I run. Step one, extract the read path. List every URL in your llms.txt in the order it appears, with its description. This is the model's proposed reading sequence.

Print it as a linear document. Step two, check freshness and accuracy on each entry. For a financial services client, this means confirming that a listed page reflects current regulations and current product terms, not a superseded version. I have found stale disclosures listed as priority content more than once. If a model reads your outdated overview first, it may anchor its understanding to the wrong facts.

In YMYL, that is a real risk, not a theoretical one. Step three, test the description against the page. Each llms.txt entry has a short description. Read it, then read the actual page. Do they match?

A description promising "a complete guide to estate planning in this jurisdiction" attached to a thin three-paragraph post is a credibility mismatch. Models and reviewers both notice mismatches. Step four, check for gaps. What are the three questions a prospective client asks most often? For a personal injury firm, that might be timelines, fees, and case value.

Are the pages that answer those questions in the read path? If your highest-intent answers are absent from llms.txt while a general "about us" page is featured, your read path is misaligned with demand. Step five, reconcile with robots.txt. Confirm none of the read-path URLs are blocked. This closes the loop with the Access vs Attention Framework.

The outcome of a Read-Path Audit is a short, honest priority list where every entry is reachable, current, accurately described, and mapped to real client questions. What I have found is that most sites do not have a bad llms.txt. They have an unmaintained one, which quietly points models at last year's version of the firm.

• Reconstruct the exact order a model would read from your llms.txt.
• Verify each listed page is current, especially regulatory and pricing content.
• Check that each description honestly matches the page it points to.
• Confirm your highest-intent client questions appear in the read path.
• Reconcile every read-path URL against robots.txt to catch blocks.
• Schedule the audit quarterly; an unmaintained llms.txt ages into inaccuracy.

Should You Block AI Crawlers in robots.txt? The Trade-Off Nobody States Clearly

This decision comes up constantly, and the advice online is usually absolutist in one direction or the other. The reality is a trade-off, and the right answer depends on your goals. Blocking gives you control and costs you presence. If you Disallow a provider's AI crawler in robots.txt, you reduce the chance that provider trains on or retrieves your content. For a firm with genuinely proprietary methodology or licensed content, that can be the correct call.

But the cost is real: if that provider powers an assistant your prospective clients use, blocking can remove you from those answers. For a demand-driven practice, that is lost visibility at the exact moment someone is researching. In regulated verticals, I frame it as a governance question, not a reflex.

A healthcare organization worried about outdated medical content being surfaced by an assistant has a legitimate reason to be cautious. But the better remedy is often keeping the current content reachable and well-structured so the accurate version is what gets read, rather than blocking everything and hoping the model has nothing. Here is the nuance most guides skip. Blocking a crawler in robots.txt does not retract content a model already ingested. If your pages were crawled before you added the block, that knowledge may persist in the model.

Blocking is forward-looking, not a delete button. So the fantasy of "block the AI crawler and disappear from AI" rarely matches reality. The approach I recommend: decide provider by provider.

You can Allow the assistants where visibility helps your demand goals and be selective about training crawlers where you have a specific rights concern. Document each decision and the reason. That documented, provider-specific policy is far stronger in a compliance review than a blanket block, and it keeps you present where presence matters.

The cost of inaction here is subtle. Firms that never make a deliberate choice usually end up with a default that neither protects their proprietary content nor secures their AI visibility. Making no decision is itself a decision, and rarely the right one.

• Blocking AI crawlers trades presence in AI answers for content control.
• Blocking is forward-looking; it does not remove already-ingested content.
• Decide provider by provider rather than applying a blanket block.
• In healthcare and finance, keeping current content reachable often beats blocking.
• Document each allow or block decision and its rationale for review.
• A default no-decision policy usually protects nothing and costs visibility.

How Do You Implement Both Files as One System?

Implementation is where the versus framing does the most damage, because it leads people to build one file and neglect the other. Treat them as one documented system instead. Start with robots.txt, at your domain root. Write your Access decisions. Disallow private paths, staging, internal search, and low-value parameter URLs.

Be conservative with broad Disallow rules; a single overreaching line can block a whole directory of authoritative content. Then explicitly decide your stance on named AI crawlers, provider by provider, and record why. Then build llms.txt, also at your root. Format it as markdown. Open with your organization name and a one-line description of what you do and for whom.

In a regulated field, name the jurisdiction and scope plainly. Then list your priority pages under clear headings, each with a short, honest description. For a legal site, group by practice area.

For healthcare, group by reviewed condition or service. Optionally include full-text sections for the two or three pages you most want understood correctly. Reconcile the two. Every URL in llms.txt must be Allowed in robots.txt. Run this check as the final step, and re-run it whenever either file changes.

This is the loop that the Access vs Attention Framework and the Read-Path Audit both close. Then maintain. Put both files in version control. Add review dates. Schedule a quarterly Read-Path Audit.

When your services, products, or applicable regulations change, update llms.txt so the read path stays current. What I have found is that the maintenance discipline matters more than the initial build. A perfect llms.txt written once and abandoned becomes a liability, quietly pointing models at outdated content.

A modest llms.txt reviewed every quarter compounds, because it stays accurate as adoption of the convention grows. The same is true for robots.txt: a stale block from a project three years ago can suppress content you now want visible. One system, two jobs, one maintenance rhythm.

That is the whole approach.

• Both files live at the domain root; place robots.txt first in your workflow.
• Write Access decisions in robots.txt, avoiding overly broad Disallow rules.
• Format llms.txt as markdown with an intro, grouped priority links, and honest descriptions.
• Every llms.txt URL must be Allowed in robots.txt; reconcile as a final step.
• Version-control both files and attach review dates.
• Run a quarterly Read-Path Audit and update after any regulatory or product change.

What I Wish I Had Told Clients Sooner

Early on, I answered the "robots.txt or llms.txt" question too literally. Clients asked which to use, and I explained each in turn without stopping to correct the premise. The premise was the problem. What I have found is that the moment I reframed it as access versus attention, the confusion cleared instantly. A managing partner does not lock the file room instead of indexing the case files. They do both, for different reasons. Once that clicked, clients stopped seeing these as competing options and started maintaining them as one system. The second thing I underestimated was maintenance. The initial setup is the easy part. The value comes from the quarterly review, the reconciliation, the changelog. In regulated work, that documented rhythm is often worth more than any single directive, because it demonstrates a deliberate, reviewable process to anyone who scrutinizes how your content reaches automated agents.

Your 30-Day Action Plan

1. Days 1-3 — Audit your current robots.txt. List every Disallow rule and confirm none accidentally block authoritative content. Identify overly broad directory blocks.
2. Days 4-7 — Run the Access pass of the Access vs Attention Framework. Build a table of URL patterns with access decisions, and make a deliberate, provider-by-provider decision on AI crawlers with recorded rationale.
3. Days 8-14 — Run the Attention pass. Identify your cornerstone, current, high-intent pages by practice area or reviewed service. Confirm each is Allowed in robots.txt.
4. Days 15-21 — Draft llms.txt at your root: organization intro with jurisdiction and scope, grouped priority links, honest descriptions, optional full text for two or three key pages.
5. Days 22-27 — Run the Read-Path Audit. Check freshness, description accuracy, coverage of top client questions, and reconciliation with robots.txt for every listed URL.
6. Days 28-30 — Put both files under version control, add review dates, create a shared changelog, and schedule the next quarterly Read-Path Audit.

Frequently asked questions

Does llms.txt replace robots.txt?

No. They do different jobs and neither replaces the other. robots.txt is an access-control file that tells crawlers which paths they may reach, and reputable crawlers have honored it for two decades. llms.txt is a proposed attention file that suggests to language models which pages to prioritize and how to interpret them. One governs reachability, the other proposes reading priority. You maintain both as a single system. If you remove robots.txt and rely on llms.txt, you lose enforceable access control and gain only a hint that no major provider is bound to follow. The reverse is also incomplete. Treat them as complementary, not as alternatives.

Will adding llms.txt guarantee AI assistants cite my content?

No, and any guide claiming otherwise is overstating it. llms.txt has no binding commitment from major AI providers at this time. It is a community proposal that some tools read and many ignore. Models read broadly across the web, not only from your listed pages. What llms.txt can do is offer a clean, curated map that may help agents that do read it, and it costs little to maintain. For developer documentation and large knowledge bases, that hint has genuine value. But if your goal is guaranteed citation, no file provides that. Focus instead on making your best content reachable, current, and clearly structured, which helps regardless of whether any given agent reads your llms.txt.

If I block an AI crawler in robots.txt, does my content disappear from AI answers?

Not necessarily, and this is widely misunderstood. Blocking a crawler in robots.txt is forward-looking. It can reduce future crawling by that agent, but it does not retract content the model already ingested. If your pages were crawled before you added the block, that knowledge may persist. Also, robots.txt governs crawling, not display, so content reachable through other links can still surface. For firms in regulated verticals, the practical takeaway is that blocking rarely produces the clean disappearance people expect. If accuracy is your concern, keeping the current version reachable and well-structured usually serves you better than a blanket block that leaves the model relying on whatever it already learned.

Do small websites need an llms.txt file at all?

Often not urgently. llms.txt tends to help large, complex sites with sprawling documentation far more than a tight ten-page practice site. On a small site, a language model can reasonably map your content without a curated reading list. That said, for regulated small sites, a brief llms.txt still has value: it lets you state your scope and jurisdiction plainly and point agents at your reviewed, current pages rather than an old blog post. The effort is modest. My guidance is to prioritize a clean robots.txt first, then add a lean llms.txt if you have specific pages you want interpreted correctly. Do not spend a week hand-writing full-text sections for a small site.

Where do robots.txt and llms.txt files go, and what format do they use?

Both files sit at the root of your domain, so they are reachable at yourdomain.com/robots.txt and yourdomain.com/llms.txt. robots.txt uses a specific directive syntax with User-agent, Disallow, and Allow lines, and it must follow that format to be read correctly. llms.txt uses markdown, typically opening with your organization name and a short description, followed by grouped links to priority pages, each with a brief description, and optional full-text sections. The key implementation rule is reconciliation: every URL you list in llms.txt must be Allowed in robots.txt. A page that is featured as priority attention but blocked from access sends a contradictory signal, and access always wins.

How often should I review these files?

At least quarterly, and immediately after any material change to your services, products, or applicable regulations. An unmaintained llms.txt is the most common real problem I see. It is rarely badly written; it is simply out of date, quietly pointing models at last year's version of the firm. In YMYL fields, a stale disclosure or superseded product term listed as priority content is a genuine risk. robots.txt needs review too, because a block added for a project years ago can suppress content you now want visible. Put both files under version control, attach review dates, keep a shared changelog, and schedule the Read-Path Audit as a recurring task. The maintenance rhythm matters more than the initial build.