The llms.txt I shipped was wrong

llms.txt is a small convention: a plain-text file at the root of your site that tells a language model what the site is and what is on it. It is a nice idea. Machines were going to summarise the site anyway; you may as well hand them a summary you wrote.

I wrote one for FeatureJet. I wrote it the way everybody writes one — by hand, in one sitting, from memory of the product.

It was wrong when it went live.

What "wrong" means here

Not stylistically wrong. Factually wrong. It described capabilities in terms that did not match what had actually shipped, because I was writing from my mental model of the product rather than from the product.

The gap was not large. That is the part I keep coming back to. It was the sort of drift that happens to any hand-written description of a moving thing, over the two or three weeks between writing it and shipping it. Nobody was careless. The file was simply a hand-maintained copy of a moving target, and those are always sliding toward wrong; the only variable is how fast.

And this file has a specific quality that makes the drift worse than usual: its audience is machines that will repeat it confidently, without the hedging a human reader would apply to marketing copy on a website.

The fix that does not work

The obvious fix is to be more careful. Read it again before shipping. Add it to a checklist.

That fails for the same reason the original failed. The file is written by a person, from memory, at a moment in time, and the product keeps moving after that moment. Care makes the drift slower. It does not make it stop.

The fix that does

Do not write the file. Generate it, from the same content the pages render, and then test that it agrees with them.

On this site, llms.txt is a route handler, not a file in the repo. It reads the same content objects the product pages and blog index read, and formats them:

export function buildLlmsTxt(
  posts: readonly Post[],
  products: readonly Product[],
  providers: Providers = {}
): string {
  // ...
  for (const metric of resolved.metrics) {
    // The only branch that may print a number is the one that HAS a number.
    const rendered =
      metric.state === "value"
        ? `${formatMetricValue(metric.value, metric.unit)} (as of ${formatDate(metric.asOf)})`
        : `not reported — ${metric.reason}`;
    lines.push(`- ${metric.label}: ${rendered}`);
  }
}

That alone removes most of the failure mode. If a product's status changes, the status in llms.txt changes, because it is the same string. There is no second copy to forget.

But "it is generated" is a property of today's code, not a property that stays true. Someone refactors, someone hardcodes a status into a template to fix a formatting bug, and the second copy is quietly back. So the invariant is a test.

The honesty test

The test does not check the file against a golden snapshot — a snapshot is just another hand-maintained copy, with the same disease. It re-derives what the file must contain from the content tree and asserts the relationship:

it("states each product's real status", () => {
  for (const product of products) {
    expect(txt).toContain(`- Status: ${PRODUCT_STATUS_LABEL[product.status]}`);
  }
});

it("omits no published post", () => {
  for (const post of posts) {
    expect(txt).toContain(absoluteUrl(`/blog/${post.slug}`));
  }
});

it("prints no number that has no resolved source", () => {
  for (const metric of unresolvedMetrics) {
    expect(txt).toContain(`- ${metric.label}: not reported`);
  }
});

Plus one that is less obvious and has earned its place: a scan for language that asserts traction. trusted by, thousands of, coming soon, join 400+. None of that can be generated by the code, so if it ever appears, it got there through a hand-written field in a content file — which is exactly the leak worth catching, because that is how the first one happened.

The general version

Any file that describes your product to somebody else, and is maintained by hand, is drifting away from your product right now.

The list is longer than it looks. llms.txt, the meta description, the structured data, the README's feature list, the pricing table you copied into a comparison page, the API docs you keep next to the API. Each one is a claim about the software written in a place the software cannot see.

You cannot generate all of them. But you can usually notice which ones are claims — and for a claim, the question is not "is this accurate?" It is "what would tell me if it stopped being accurate?" If the answer is a person remembering to check, it will be wrong eventually, and you will find out from somebody else.