Skip to content

Frontmatter to OKF: 61/63 Agent Score, One Source of Truth

• Updated
18 min read
Markdown Share this post: Share on X Share this post on X Share on LinkedIn Share this post on LinkedIn

Curating agent context by hand guarantees drift as the documentation grows; deriving it from source removes that failure mode entirely. Every post on this site now carries a machine-readable knowledge surface that agents can fetch and answer from, generated from one clusters frontmatter field and packaged as an Open Knowledge Format (OKF) v0.3 bundle served at /okf/. On a preregistered benchmark of 63 sealed questions per arm, an agent grounded on the derived knowledge graph scored 61/63; the same agent grounded on the OKF bundle (whole payload) scored 31/63 and lost to both other arms. A later extension of the same benchmark added a best-case per-question OKF slice, and it scored lower still: 22/63, significantly below the bundle (Holm p = 0.023).

This post documents the pipeline end to end: why frontmatter is the single source of truth, what OKF v0.3 adds, which conformance and determinism guarantees run in CI, and what the measurement says about graph-shaped context. It pairs with the agent-ready website stack, which positions /okf/ and the knowledge graph in the five-layer discovery model, and with the earlier finding that structured grounding did not fix reading-chain recall.

A link list tells an agent what exists; a graph tells it how things relate. When an agent must answer “which posts cover evaluation gating” or “which posts does this one cite,” a flat index forces the model to guess from titles. A graph exposes cluster membership, citation edges, and reading chains as data, so the answer is a lookup rather than a plausibility judgment. Retrieval research supports the shape: prepending knowledge-graph triples to a question improves zero-shot answer quality without training the model (Baek et al., 2023), and partitioning a corpus into graph communities lifts answer comprehensiveness on global questions that flat chunk retrieval fails (Edge et al., 2024). Google Cloud frames the same problem at enterprise scale: agents repeatedly solve an identical context-assembly problem, and a portable, markdown-first knowledge format lets producers publish that context once for any consumer (McVeety and Hormati, 2026).

What Does the Literature Say About Payload Shape?

The broader evidence is mixed, which is exactly why measurement matters. Retrieval-augmented generation grounds answers in external documents rather than parametric memory (Lewis et al., 2020), but models degrade when relevant evidence sits mid-payload instead of at the edges of the context window (Liu et al., 2023). Observational studies of llms.txt find little or no citation lift from the file itself, with value concentrated in developer-tooling retrieval (ALLMO, 2026). Shape, size, and the retrieval strategy around them interact; no format wins by decree. Graph-shaped context also degrades honestly: when a payload cannot answer a question, a well-formed graph makes the gap visible, and a disciplined agent abstains. The measurement below shows exactly this split: the arms differ less in what they can answer than in what they do when their grounding runs out of answers.

How Does One Frontmatter Field Drive the Whole Pipeline?

The pipeline has one edit point per post: the clusters frontmatter field. Listing a cluster id there is the only act of curation an author performs, and it simultaneously drives the on-site knowledge graph, the /llms.txt concept index, and the /okf/ bundle. There is no separately maintained catalog to drift out of date, because no separately maintained catalog exists.

Code Snippet 1: The frontmatter fields that feed the pipeline, title truncated for display. Cluster ids come from a closed set validated at build time; the highlighted lines are the entire curation surface.
title: "Frontmatter to OKF: 61/63 Agent Score..."
description: "Frontmatter drives a pure pipeline..."
pubDatetime: 2026-11-21T16:30:00Z
modDatetime: 2026-11-22T14:59:00Z
clusters:
  - agent-infrastructure
  - evaluation-reliability
draft: falsesrc/data/blog/frontmatter-okf-knowledge-graph.md

What Does Derive Actually Mean Here?

The derivation is a pure function, deriveGraph, in the site’s data layer: it takes post metadata and returns the graph of nodes, cluster membership, citation edges, concept seeds, and reading chains. The citation miner is a deterministic rule rather than a heuristic: an edge exists exactly when one published post’s body links to another’s /posts/{slug}/ URL; links to pages outside the corpus and self-links are excluded, and repeats deduplicate. Purity is not aesthetic. Because the function has no side effects and no build-clock input, two builds from identical sources produce identical output, which makes graph diffs reviewable in pull requests and makes the published knowledge-graph.json a reproducible artifact rather than a snapshot of whatever the CI machine saw that day.

Code Snippet 2: The validation and derivation halves of deriveGraph, condensed from the site data layer. Drafts are excluded, missing or empty clusters arrays fail, every cluster id is checked against the closed enum, and every violation fails the build naming the offending slug.
export function deriveGraph(
  posts: MinimalPost[],
): DerivedGraph {
  const included = posts.filter((p) => !p.draft);
  const validClusterIds = new Set<string>(CLUSTER_IDS);
  for (const post of included) {
    if (
      !Array.isArray(post.clusters)
      || post.clusters.length === 0
    ) {
      throw new Error(
        `Post "${post.slug}" has missing`
        + ` or empty clusters frontmatter`,
      );
    }
    for (const id of post.clusters) {
      if (!validClusterIds.has(id)) {
        throw new Error(
          `Post "${post.slug}" references`
          + ` unknown cluster "${id}"`,
        );
      }
    }
  }

  // citation edges mined from post bodies, plus curated
  // thematic edges, deduplicated by from|to|type
  const allEdges = [
    ...included.flatMap((p) =>
      extractCitationEdges(
        p.slug, p.title, p.body, slugs, titles,
      ),
    ),
    ...THEMATIC_EDGES,
  ];
  const edges = [...new Map(
    allEdges.map((e) => [`${e.from}|${e.to}|${e.type}`, e]),
  ).values()];

  // one node per post; timestamps come from
  // frontmatter, never from the build clock
  const inbound = new Map<string, number>();
  const outbound = new Map<string, number>();
  for (const e of edges) {
    outbound.set(e.from, (outbound.get(e.from) ?? 0) + 1);
    inbound.set(e.to, (inbound.get(e.to) ?? 0) + 1);
  }
  const nodes = included.map((post) => ({
    slug: post.slug,
    url: `https://clouatre.ca/posts/${post.slug}/`,
    pubDate: post.pubDatetime,
    concepts: CONCEPT_SEED_MAP[post.slug] ?? [],
    edgeStats: {
      inbound: inbound.get(post.slug) ?? 0,
      outbound: outbound.get(post.slug) ?? 0,
    },
  }));
  return { nodes, edges, clusters, readingChains };
}src/data/knowledge-graph.ts
Code Snippet 3: The emitted output for this post, condensed from the published public/knowledge-graph.json: one node with derived edge counts, and one citation edge mined from the corpus. Nothing here is edited at emit time: edgeStats, the mined edge, and its evidence are computed by the derivation, while concepts propagate curated seed data that the build validates against the corpus.
{
  "kind": "post",
  "slug": "frontmatter-okf-knowledge-graph",
  "title": "Frontmatter to OKF: 61/63 Agent Score...",
  "url": "https://clouatre.ca/posts/...",
  "pubDate": "2026-11-21T16:30:00.000Z",
  "tags": ["agentic-ai", "architecture", "..."],
  "concepts": ["knowledge-graph", "single-source-of-truth", "okf"],
  "category": "agent-infrastructure",
  "edgeStats": { "inbound": 1, "outbound": 1 }
}

{
  "from": "agent-ready-website",
  "to": "frontmatter-okf-knowledge-graph",
  "type": "citation",
  "weight": 0.7,
  "evidence": "Agent-Ready Websites: The 5-Layer..."
}public/knowledge-graph.json

What Breaks When the Field Is Wrong?

Validation fails loudly and precisely, and every failure is a build failure. An empty clusters array fails. A cluster id outside the closed enum fails. A thematic edge or concept seed that references a slug absent from the corpus fails. The failure mode “the graph silently disagrees with the corpus for a week” is designed out: the build names the slug and stops, so the error surface is a pull-request comment, not a production incident.

What Does the OKF v0.3 Bundle Add?

The Open Knowledge Format is a Google-backed specification that represents organizational knowledge as directories of markdown files with YAML frontmatter, so any editor, git host, or agent can read a bundle without translation (McVeety and Hormati, 2026). Version 0.2 added trust signals (provenance, freshness, lifecycle); v0.3, released November 18, 2026, tightened conformance rules and moved the spec to the standalone open-knowledge-format organization. License and provenance are first-class fields, which matters for a public site: a consumer can verify what it is reading and under what terms before trusting it. The graph and the bundle are two deterministic emitters behind the same pure derivation, and the bundle’s stages mirror the site’s build: derive once, emit knowledge-graph.json, then emit the OKF payload from that graph.

Code Snippet 4: Bundle metadata from the emitted /okf/index.md. All four fields are fixed constants in the emitter, so the root metadata is byte-deterministic; concept-file timestamps, by contrast, derive from post frontmatter, never the build clock.
---
okf_version: "0.3"
title: "clouatre.ca knowledge bundle"
description: "Concepts derived from the clouatre.ca..."
license: "CC-BY-4.0"
---public/okf/index.md

How Is Conformance Enforced?

The site emits its bundle at /okf/: one concept file per published post, body-only cluster indexes, and a root index.md with the bundle-level metadata. Every emitted file passes a strict conformance check against the v0.3 spec, and the build fails on violation, so conformance is enforced on every push rather than audited occasionally. The emitter is byte-deterministic by construction: output is sorted, and every generated timestamp derives from post frontmatter rather than the build clock, so the emitted files are as reproducible as the graph they are built from. Delivery reuses the existing Worker: .md payloads are served as text/markdown, so an agent fetching a concept file gets markdown, not an HTML-rendered page.

How Is Spec Provenance Pinned?

The spec is pinned by hash: it is vendored into the repository, and CI hashes the vendored copy against the pinned upstream commit; the pin was verified byte-identical to the canonical v0.3 release in the standalone organization before it shipped. A consumer auditing the site can reproduce the same hash check against the canonical repository, which is the trust property OKF’s provenance model asks producers to make verifiable. The gate itself is four lines in the bundle emitter.

Code Snippet 5: The vendored-spec drift gate, condensed from scripts/build-okf.ts. The hash pins references/okf/SPEC.md to upstream commit 527e81a; any drift fails the build.
const SPEC_SHA256 =
  "a1fcc51be23fce5c1e4e198f0a71af6046a5286503"
  + "959caa4009311f74fac3f1";
const hash = createHash("sha256")
  .update(await readFile(specPath))
  .digest("hex");
if (hash !== SPEC_SHA256) {
  throw new Error(
    `references/okf/SPEC.md sha256 mismatch:`
    + ` expected ${SPEC_SHA256} (spec commit 527e81a)`,
  );
}scripts/build-okf.ts

What Did the Grounding Comparison Measure?

A preregistered experiment compared three grounding payloads on identical work: frozen llms.txt, frozen knowledge-graph.json, and the frozen OKF v0.3 bundle. An extension then added a fourth arm, a per-question okf-index-relevant slice with gold-cluster routing, run under the extension seal rather than the original one; gold-cluster routing is best-case, oracle routing. “The slice” below always means this arm. The same model (gemma-4-26b-a4b-it, served on Cloudflare Workers AI), at temperature 0, with one deterministic attempt per item, answered 63 sealed questions per arm across four preregistered families, with an answerability map sealed before any run (Clouatre, 2026). A correct answer scores 1 and so does a correct abstention on a question the payload cannot answer. The secondary partial-credit metric scores set answers by Dice overlap with the sealed ground truth and edge-count items by the fraction of correct integers; abstentions earn zero. The frozen payloads differ sharply in size: llms.txt averaged 4,201 prompt tokens per run, the OKF bundle, serialized as a 35,616-byte concatenation of 32 markdown files, averaged 10,081, the knowledge graph averaged 18,767, and the slice averaged 4,350, just 43% of the bundle. The largest arm by tokens scored best; size alone predicts nothing here, so the argument below reads shape and delivery, not byte counts.

Table 1: Item score per grounding arm, n=63 per arm; a correct answer or a correct abstention scores 1. Extraction and abstention accuracy are reported separately because the composite mixes the two; truncations are runs whose context exceeded the 4096-token completion budget.
Arm Item score Rate Wilson 95% Extraction acc. Abstention acc. Truncations
knowledge-graph.json 61/63 0.968 [0.891, 0.991] 0.968 (63 items) n/a (0 items) 2
llms.txt 55/63 0.873 [0.769, 0.934] 1.000 (23 items) 0.800 (40 items) 2
OKF bundle (whole payload) 31/63 0.492 [0.373, 0.612] 0.375 (16 items) 0.532 (47 items) 23
OKF slice (okf-index-relevant) 22/63 0.349 [0.243, 0.472] 0.375 (16 items) 0.340 (47 items) 14
The knowledge graph scored 61/63; the bundle and its slice failed Item score (answer or abstention)95% CI (bootstrap) 0 0.25 0.5 0.75 1 Knowledge graph 0.968 llms.txt 0.873 OKF bundle 0.492 OKF slice 0.349 no gap McNemar exact, Holm p = 0.070
Figure 1: Item score per grounding arm, n=63 sealed questions each. The knowledge graph and llms.txt are not separated by the preregistered McNemar exact test with Holm correction (p = 0.070); the OKF bundle (whole payload) lost to both arms (Holm p < 1e-5), and the slice lost to llms.txt (Holm p = 1.05e-8) and to the bundle (Holm p = 0.023); a result of this emitter’s payload and delivery under a fixed budget, not a verdict on the format.

Two Readings Survive; One Does Not

First, at n=63 the design’s minimum detectable difference is roughly 22 to 24 percentage points, and the 61/63 versus 55/63 gap is not separated (McNemar exact, Holm p = 0.070). “Not separated at this sample size” is the finding, not “llms.txt is as good as the knowledge graph.”

Second, the OKF bundle’s 31/63 is a delivery result, not a verdict on the format, though delivery is now the weaker of the two explanations. The bundle went to the model whole, under a 4096-token completion budget, on one model. On the one family the seal marks answerable for the bundle, F4 post-to-cluster, it scored 6/16, and every miss was a complete, confident wrong answer with a normal finish. Its F3 score of 3/7 sits on the abstention side by design: the sealed answerability map marks F3 answerable only for the knowledge graph and llms.txt, and the bundle’s four F3 misses were confident attempts where a correct abstention scored 1.

Answering confidently wrong where a careful consumer should abstain is a documented failure mode of instruction-tuned models, and it is trainable (Zhang et al., 2024). That is the pattern a publisher should weight most heavily. Where the graph does edge ahead of llms.txt, the scope is precise: its entire six-item edge sits in F1 citation edges (18/20 versus 12/20 abstention-graded), with identical perfect item scores on F2 (correct abstentions for llms.txt, which the seal marks unanswerable there), F3, and F4. The best-case slice weakens the delivery reading; it lost to the bundle on composite (Holm p = 0.023) and tied it on extraction (6/16 versus 6/16), so the better-supported explanation is content: the graph’s nodes carry per-page edge counts and directional citation links the OKF concept files lack.

Where arms separate, the bundle and its slice trail knowledge-graph.json llms.txt OKF bundle OKF slice 0 0.25 0.50 0.75 1 1.0 1.0 1.0 1.0 0.429 0.375 0.286 0.375 F3 clusters (7 items) F4 posts (16 items) Every bundle and slice F4 miss was a confident wrong answer
Figure 2: Item score on the two families where arms separate (exploratory per the seal). Per the sealed answerability map, F3 forward cluster membership is extraction-graded for knowledge-graph.json and llms.txt and abstention-graded for the OKF bundle; F4 post-to-cluster is extraction-graded for all arms. knowledge-graph.json and llms.txt scored 7/7 and 16/16; the bundle scored 3/7 on F3, where its three credits are correct abstentions and its four misses are confident attempts where abstaining scored 1, and 6/16 on F4, where every miss was a complete, confident wrong answer rather than a truncation artifact.

Did Truncation Cause the Misses?

It did not; the record rules it out. The whole-bundle payload truncated in 23 of 63 runs, but all 23 truncations landed on families it could not answer anyway, where the empty output scored as a correct abstention, so truncation inflated the abstention side of the composite rather than causing the extraction misses. Every miss on an answerable family was that same failure. The truncation rate still evidences size interacting with the completion budget; it just did not touch the extraction failures. The graph arm tells the mirror-image story: it attempted every item, so its only two misses were its only two truncations, both on F1 citation-edge items, scored as extraction misses rather than absorbed as abstentions. The slice cut truncations from 23 to 14 at 43% of the bundle token budget and still fell on composite, 31 to 22; truncation was not the cause.

What Does the Composite Endpoint Hide?

The two arms’ item scores diverge by 38 points, yet the secondary partial-credit metric cannot tell them apart: llms.txt earns its credit almost entirely through correct abstentions, while the bundle earns what credit it gets through its few extraction attempts. Figure 3 shows the pairing. This is why the conditional accuracies in Table 1, not the headline rates, carry the argument; a composite that mixes two different ways of earning a point can make a confident-wrong arm look comparable to a careful abstainer.

On partial credit, the bundle ties llms.txt and its slice Partial credit per item (mean)95% CI (bootstrap) 0 0.25 0.5 0.75 1 Knowledge graph 0.981 llms.txt 0.407 OKF bundle 0.465 OKF slice 0.482 no gap permutation p = 0.744
Figure 3: Mean partial credit per arm, n=63 sealed questions each, on bootstrap 95% intervals (10,000 resamples, seed matching the sealed permutation test). The knowledge graph separates from the other three arms (paired permutation p < 1e-4); llms.txt and the OKF bundle do not separate (p = 0.44) despite the 38-point item-score gap, and neither do the two OKF arms (p = 0.744). Read against Figure 1, the brackets flip: on the primary endpoint the top two arms do not separate, while on this secondary endpoint the graph separates from the other three.

What Are the Caveats?

OKF is a v0.x specification. The format has moved twice in six months, and a bundle emitted today may need regeneration as the spec matures; the pinned, hash-checked vendored spec keeps that risk visible in CI rather than latent. The measurement covers one model, one site, and one deterministic attempt per item, so the rates carry item-sampling variance only, and run-to-run variance is unquantified by design. The composite endpoint mixes extraction and abstention credit, so conditional accuracies, not headline rates, are the right lens. The OKF result is scoped to whole-bundle delivery under a fixed completion budget; the extension measured the best case beyond it and it still lost (Holm p = 0.023), so retrieval or budget changes beyond this best case have no supported upside. Positional degradation in long payloads (Liu et al., 2023) is a mechanism slices avoid, but the best-case slice avoided it and still lost; payload position is not the binding mechanism. The pipeline itself is independent of those caveats: the graph and the bundle regenerate from frontmatter, so a consumer that prefers slices over whole bundles can assemble its own context from the same published files.

What Should a Publisher Ship?

  1. Make frontmatter the single source of truth. One validated field per post feeds every machine surface; curation drift cannot happen because there is nothing separate to drift.
  2. Derive, never curate, the graph. A pure function plus byte-deterministic emitters makes published knowledge surfaces reproducible and reviewable.
  3. Enforce conformance in CI, not in audits. A build that fails on spec violation and on vendored-spec drift keeps a v0.x format honest.
  4. Measure shape and delivery, not format. The OKF bundle (whole payload) lost as a delivery mode on one model, and the measured per-question slice lost too; cheaper was not better, with 14 truncations against 23 at 43% of the tokens, a worse composite, and no partial-credit separation (p = 0.744). If OKF must ground agents, the levers are content fixes (per-page edge counts, directional citation edges), not delivery.
  5. Report nulls and non-separations precisely. “Not separated at n=63” and “confident wrong answers inflate partial-credit scores against disciplined abstention” are the findings; neither collapses into a slogan.

References


Previous Post
When Accuracy Misleads: Cost Matrix Metrics and Thresholds

Related Posts