This documentation is also published as Markdown for efficient machine reading: the whole site is indexed at /llms.txt, and every page has a clean Markdown copy under /_llms/. These are generated from the same source and cost far fewer tokens to read than this rendered HTML.

Skip to main content

May 4, 2026

Built for LLMs — llms.txt support

By Phil Scott

When an LLM crawler reads your documentation, it parses rendered HTML — nav chrome, sidebars, highlighting spans — to recover the prose underneath. The llms.txt convention skips that by handing crawlers clean markdown directly. Pennington now generates it.

An index and a markdown copy of every page

During the static build, Pennington produces two things:

  • llms.txt — an index of the whole site, organized by your navigation tree. Drop a custom llms.txt in your content root and it becomes the index header.
  • _llms/ — a directory holding a front-matter-stripped markdown copy of every page.

The markdown is real markdown, not a degraded HTML dump. The converter handles highlighted code blocks, tabbed groups, and GFM alerts, so a code sample survives the round trip intact. The llms.txt how-to covers the setup.

Agents find it on their own

Generating the files isn't enough if nothing points to them. DocSite and BlogSite emit a <link rel="alternate" type="text/markdown"> tag in every page's head — the standard way to advertise an alternate representation. And because tools like WebFetch strip <head> before an agent sees it, Pennington also drops a paired hidden hint at the top of the <body>, so an agent reading the page the hard way still learns there's a clean copy.

Content that's only for agents

Two opt-in conventions let you write content for llms.txt without producing an HTML page. The first is a *.llms.md file. The second is WithLlmsTxtEntry on a MapGet endpoint, for serving the markdown dynamically:

csharp 
app.MapGet("/_llms/architecture.md", () => Results.Text(overview, "text/markdown"))
   .WithLlmsTxtEntry("Architecture overview", "A high-level map for agents.");

Useful for an agent-oriented summary, or machine-readable detail you'd rather keep off the human-facing site.

llms-txtai-agents