This experiment never published a result, because its first run nearly published a wrong one. That run reported a 41-point completion gap favoring structured exhibit grounding, but its metric measured schema-key presence, not correctness: an agent could pass while returning a nonsense sequence, as long as a specific key appeared. A pre-publication audit caught the flaw, and the experiment was re-run under a different metric. The re-run scores each run by sequence-match, meaning a run succeeds only when the returned sequence deep-equals the runner-resolved ground truth.
What did the re-run find?
Zero exact matches in both arms. The KG-grounded arm retrieved real slugs from knowledge-graph.json, but got the order wrong and substituted a third slug that is a real post on this site, just not a member of this chain. 95 of its 103 runs returned the identical wrong answer, ["mcp-tool-docs", "agent-ready-website", "orchestrating-ai-agents-subagent-architecture"], against the ground truth ["agent-ready-website", "mcp-tool-docs", "webmcp-beyond-checkout"]. The ungrounded arm degenerated, word-splitting the chain name into ["agent", "tool", "surface", "chain"] in 72 of 103 runs and echoing the chain id as the sole element in 31. Grounding changed failure quality, not success: a grounded agent produces plausible but mis-ordered sequences, while an ungrounded one produces noise.
Both arms sit at a rate of 0.0 with a Wilson 95% interval of [0.0, 0.036]. With zero successes in 103 runs, the interval says the true rate is at most about 3.6% per arm. The two-proportion score test is degenerate here, with statistic 0 and p = 1, because both rates are 0 and the test carries no information.
| Arm | Grounding | n | Exact matches | Rate | 95% CI |
|---|---|---|---|---|---|
| a | KG-grounded | 103 | 0 | 0.0 | [0.0, 0.036] |
| b | Ungrounded | 103 | 0 | 0.0 | [0.0, 0.036] |
How was it measured?
Each run asks the agent to list the posts of the agent-tool-surface-chain reading path in order, and the run counts only when the returned slug array deep-equals the ground truth the runner fetches at scoring time: agent-ready-website, then mcp-tool-docs, then webmcp-beyond-checkout. Arm a receives KG-grounded prompting, fetching knowledge-graph.json and reading the readingChains entry before answering. Arm b receives no machine-readable surface and must recall the path from the prompt alone. Both arms run the same model, gemma-4-26b-a4b-it, with 103 samples per arm in interleaved single attempts.
{
"protocol": {
"summary": "Two-arm comparison of reading-chain recall by an AI agent on this site, scored by sequence-match: a run succeeds only when the returned sequence deep-equals the runner-resolved ground-truth ordered slug array [agent-ready-website, mcp-tool-docs, webmcp-beyond-checkout]. Arm a is KG-grounded: the agent fetches https://clouatre.ca/knowledge-graph.json, reads the readingChains entry, and lists the sequence in order. Arm b is ungrounded: no machine-readable surface, asked to list the posts of the agent-tool-surface-chain reading path in order. Both arms respond with the same JSON shape {\"sequence\": [<slugs>]}.",
"metric_name": "sequence-match",
"metric_type": "binary",
"sequence_chain": "agent-tool-surface-chain",
"ground_truth_sequence": ["agent-ready-website", "mcp-tool-docs", "webmcp-beyond-checkout"],
"samples_per_arm": 103,
"model": "@cf/google/gemma-4-26b-a4b-it"
}
}agent-idea-2026-10-29-0-rerun.jsonThe chain itself is stable ground truth. Its sequence is unchanged in knowledge-graph.json throughout, so there is no ground-truth drift between the two runs. What changed between them is the metric, not the target.
What are the caveats?
The metric is strict. Exact-match scoring gives no partial credit, and a partial-credit metric would score the grounded arm above zero, since its dominant response carries two correct slugs. The result covers a single model on a single site, so it is a case study rather than a domain-general law. Two large observational studies point the other way: structured, statistics-rich content earns more AI citations (Aggarwal et al., 2024), and across 174,048 pages cited in AI Overviews, length barely predicted citation, with format matching query intent mattering more (Ahrefs, 2025). Those studies measure what gets cited; this run measures whether an agent completes a traversal, and the two can disagree. At n = 103 per arm the design detects only large effects, and here the effect estimate is exactly zero, with the Wilson intervals ruling out anything above roughly 3.6% per arm. The two-proportion score test adds nothing in this configuration; its statistic is 0 and its p-value is 1, a degenerate output when both arms fail completely.
What should a publisher take away?
Agents that fetch the structured exhibit still mis-order and embellish sequences, so exact-order recall is the open problem, not surface formatting. In this run, a knowledge graph got the model to the right slugs, and that is real, but ordering three known items defeated both arms. If agents cannot order a three-page reading path, structured surfaces alone will not make a site legible to them; the deeper question is how agents plan and verify sequences, which is the thread the agent-ready website thesis follows. Treat any experiment measuring agent comprehension by key presence rather than correctness with suspicion, because the two can disagree completely.
References
- Aggarwal, P. et al., “GEO: Generative Engine Optimization” (2024) — https://arxiv.org/abs/2311.09735
- Ahrefs, “Short vs. Long Content in AI Overviews: The Data Says Both Work” (2025) — https://ahrefs.com/blog/short-vs-long-content-in-ai-overviews/