Hugues Clouâtre - AI & Platform Engineering

SRE for AI Agents: Error Budgets, Trust Ladders, and 90 Trials

Hugues Clouâtre — Mon, 09 Mar 2026 17:36:00 GMT

AI tooling budgets hit record highs. We ran 90 file-prediction trials to measure what an AI agent gets wrong before it touches production. The model predicted 1.7 files beyond the actual change set on average, even on a well-structured codebase. Meanwhile, operational toil is climbing for the first time in years. SRE is not ceremony. It is the empirical gate between velocity and blast radius.

Contents

Why Is AI Widening the Dev/Ops Gap?
- The Perception Gap: Dashboards vs. Practitioner Reality
How Did We Measure AI’s Scope Creep?
- Why Medium-Tier PRs Underperformed
What Does SRE Mean in a Regulated Enterprise?
- Error Budgets as Compliance Evidence
How Does SRE Act as AI’s Production Conscience?
- Decision Provenance and Error Budget Separation
- Blast Radius Containment and the Trust Ladder
Why Does Platform Maturity Gate AI Readiness?
- The Learning Time Deficit
Where Should You Start?
- The Four Actions in Order
References

Why Is AI Widening the Dev/Ops Gap?

Toil is the repetitive, manual operational work that scales with system load rather than adding lasting value. Catchpoint’s annual SRE reports tracked toil rising from 25% to 34% between 2024 and 2026, the first sustained increase since the survey began in 2020. DORA’s 2025 report independently confirmed that AI tooling has not reduced operational toil. Over the same period, 92% of developers report that AI tools increase the blast radius of bad code needing to be debugged (DevOps.com, 2025). AI-generated code flows through pipelines designed for human-paced change, and the human-in-the-loop guardrails have not kept up. Defensive pipeline architectures can close part of this gap, but they address the pipeline, not the production governance layer.

Three root causes keep surfacing in post-mortems:

AI babysitting: someone has to review generated runbooks and roll back missed-context remediations
Configuration drift: accelerating faster than humans can audit it
Validation overhead: compounding because every AI output needs a trust-but-verify pass before production

More frequent deploys, multiplied by more autonomous agents, against review capacity that has not scaled to match. METR found experienced developers took 19% longer with AI tools despite perceiving a 20% speedup (Becker et al., 2025). DORA 2025 confirmed faster deployment frequency but flat organizational delivery.

The Perception Gap: Dashboards vs. Practitioner Reality

The perception gap makes this harder to fix. Directors reviewing dashboards see ticket counts drop and declare victory. Practitioners on the ground feel increased friction because toil shifted from “boring but predictable” to “novel and unpredictable.” Both views are correct, which is exactly why the problem persists.

Dimension	AI Accelerated	Human Judgment Required
Code generation	Higher code volume	Architectural review
Test creation	Unit test scaffolding	Integration test design
Deploy frequency	Higher deploy cadence	Change risk assessment
Incident detection	Faster alert correlation	Root cause judgment
Compliance	Automated scanning	Regulatory interpretation

Table 1: AI accelerates delivery tasks (left column), but human judgment gaps (right column) are where incidents originate.

Figure 1: AI investment and measured toil both climbing, 2021-2026.

How Did We Measure AI’s Scope Creep?

Inspired by the SWE-bench methodology (Jimenez et al., 2024), we designed a file-prediction study: given a GitHub issue description and the repository file tree at a PR’s base commit, can Claude Sonnet 4.6 predict which files a human engineer modified? We targeted tobymao/sqlglot, an MIT-licensed SQL transpiler with 9k+ stars and a predictable dialect-file structure. We curated 30 merged PRs, stratified by complexity (10 simple, 10 medium, 10 complex), and ran 3 predictions per PR for 90 total trials. A single Bedrock API call per prediction: no agent loop, no tools, no retrieval. For detailed methodology and raw data, see Supplementary Materials.

Each prediction was scored against the human’s actual change set using file-level precision, recall, F1, and Jaccard similarity (set overlap: intersection over union). Scope creep, what we term scope hallucination, counts how many extra files the model predicted beyond the human’s set. These are changes the model hallucinated that do not exist in the human’s actual changeset.

Tier	Precision / Recall	F1	Scope Creep
Simple (1-2 files)	0.645 / 0.850	0.708	1.3 files
Medium (3-5 files)	0.540 / 0.585	0.552	2.2 files
Complex (6-15 files)	0.769 / 0.673	0.712	1.6 files

Table 2: Results by complexity tier, 30 PRs x 3 runs = 90 total predictions. F1 is the harmonic mean of precision and recall. Full methodology, metrics, and per-tier results.

Why Medium-Tier PRs Underperformed

The non-monotonic curve is the headline finding: Jaccard similarity was 0.60 (simple), 0.41 (medium), and 0.58 (complex). Medium-tier PRs underperformed both simple and complex tiers. The model struggles most with moderate-scope changes, not the largest ones. Medium PRs occupy the worst of both tiers. Too many candidate files to guess by elimination (as with simple PRs), but not enough structural regularity for the model to infer the set from sqlglot’s dialect-file conventions (as with complex PRs). Complex PRs had the highest precision, likely because sqlglot’s rigid directory structure (dialect file plus corresponding test file) makes multi-file change sets predictable from the dialect name alone. 12 of 30 PRs failed (Jaccard < 0.5), spread across all tiers; no tier is immune. In a shadow-mode deployment, where the agent proposes changes but a human applies them, every over-predicted file is a false positive the reviewer filters out before the change reaches production.

What Does SRE Mean in a Regulated Enterprise?

SRE is not DevOps with a different name. The distinction is structural. In regulated financial services, production reliability carries regulatory weight: OSFI’s B-13 guideline mandates technology risk management with board-level accountability, and the EU’s DORA regulation sets equivalent requirements for operational resilience across European financial services (European Parliament and Council, 2022). The compliance surface extends beyond infrastructure to AI-driven supply chain risks that traditional dependency scanning does not catch.

Error Budgets as Compliance Evidence

SRE answers this with a reliability contract. Error budgets define how much unreliability a service can tolerate before feature work stops. SLOs (service level objectives) make reliability measurable rather than aspirational. Blameless postmortems treat incidents as system failures, not personnel failures. Google codified this framework in 2016 and enterprises have since adapted it, but in regulated environments the stakes include regulatory censure, not just customer churn.

Platform Engineering provides capability: the tools, the internal developer platform, the golden paths. Where model governance and operational resilience frameworks intersect with SRE practices, the regulatory surface extends from infrastructure to inference. SRE provides accountability: the error budgets, the incident response, the production governance. The question is whether that accountability holds when the agent making changes is not human.

How Does SRE Act as AI’s Production Conscience?

Deploying an AI agent is not a monitoring problem. It is a reliability problem. Monitoring tells you something broke. A reliability framework tells you how much breakage you can tolerate, who caused it, and whether to keep going.

Decision Provenance and Error Budget Separation

Decision provenance, the AI observability requirement that every agent action links to its inputs, reasoning, and authorization chain, goes beyond logging what an agent did. You need to trace why it made a choice, what context it consumed, and which prior decisions influenced the outcome. Without this, debugging an autonomous system is archaeology, not engineering. Under OSFI B-13 and DORA, an agent action without decision provenance is not just a debugging gap; it is a compliance liability.

Separate error budgets for AI-generated changes keep machine-authored deployments from hiding behind human baselines. If an AI agent burns through its error budget, its write permissions get revoked automatically, not the entire team’s. Our results showed 0.769 precision even on the best-performing tier, meaning roughly 1 in 4 predicted files was wrong. That error rate needs its own budget.

groups:
  - name: sli.deploys
    rules:
      - record: sli:deploy_success:ratio1h
        expr: |
          sum(rate(deploy_success_total{author_type="ai"}[1h]))
          / sum(rate(deploys_total{author_type="ai"}[1h]))
        labels:
          author_type: ai
          slo_target: "0.995" # Stricter than human baseline of 0.990
      - record: sli:deploy_success:ratio1h
        expr: |
          sum(rate(deploy_success_total{author_type="human"}[1h]))
          / sum(rate(deploys_total{author_type="human"}[1h]))
        labels:
          author_type: human
          slo_target: "0.990"
      - alert: AIChangeErrorBudgetBurnRate
        expr: |
          (1 - sli:deploy_success:ratio1h{author_type="ai"})
          / (1 - 0.995) > 14.4
        for: 5m
        labels:
          severity: critical
          team: sresli-and-burn-rate.yaml

Code Snippet 1: Prometheus recording rules and burn-rate alert for AI-authored deployments. Separate SLIs per author type; 14.4x burn-rate threshold from the Google SRE Workbook.

Production teams consistently trade agent capability for reliability, preferring narrower but predictable automation over broad but brittle autonomy (Pan et al., 2026). Separate error budgets formalize that trade-off.

Blast Radius Containment and the Trust Ladder

Blast radius containment means progressive rollout gates. No agent ships to 100% on day one. The trust ladder, a graduated set of AI guardrails where each rung grants broader blast radius only after the agent demonstrates reliability at the current level, makes this concrete: start with read-only agents, graduate to shadow mode for 30 days where the agent proposes but a human applies, then supervised write access, and finally autonomous operation once the agent sustains consistent accuracy against your SLOs.

Our experiment is a proxy for what shadow mode catches. At the medium tier, where Jaccard dropped to 0.409, shadow mode would have flagged more than half the predicted change set as incorrect. The specific SLO threshold is yours to define; what matters is that it is explicit, measured, and tied to your error budget rather than a gut feeling.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-readonly
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, replicasets]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-scoped-write
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments]
    verbs: [get, list, watch, update, patch]
    resourceNames: [canary-payments]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-production-write
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [apps]
    resources: [deployments, replicasets]
    verbs: [get, list, watch, create, update, patch, delete]sre/trust-ladder-rbac.yaml

Code Snippet 2: Kubernetes RBAC ClusterRoles for each trust ladder tier. Promotion from readonly to scoped-write to production-write is a ServiceAccount rebinding; demotion reverses it.

Figure 2: Trust ladder for agentic AI: read-only, shadow mode, supervised write, autonomous.

Accuracy alone cannot distinguish an agent that fails on a fixed subset of tasks from one that fails unpredictably at the same rate (Rabanser et al., 2026). Our 90 runs confirmed consistency (27 of 30 PRs showed zero variance across runs, 3 showed near-zero) but exposed robustness and safety gaps on complex refactoring tasks. Consistent failures are exactly what shadow mode is designed to catch: the model’s errors are systematic, not random, and a human reviewer can filter them. Early evidence supports this approach: STRATUS, a multi-agent SRE system operating under similar progressive constraints, achieved a 1.5x improvement over baselines in automated failure mitigation (Chen et al., 2025).

Why Does Platform Maturity Gate AI Readiness?

The 2025 DORA report is explicit: AI’s impact depends on the quality of the underlying organizational system. Bolt AI onto a fragile platform and you get faster fragility. An AI agent that auto-scales a misconfigured service does not fix the misconfiguration; it scales the blast radius. The AIOpsLab framework shows agent performance varies significantly with the quality of instrumented infrastructure underneath (Chen et al., 2025).

The maturity sequence matters. Build the IDP (internal developer platform) first, layer SRE practices including LLMOps telemetry for token consumption, latency, and decision traces on top, then introduce agentic AI. Skip a step and the agents inherit your tech debt at machine speed.

The Learning Time Deficit

Only 6% of SREs have dedicated, protected learning time (Catchpoint, 2026). You cannot build an SRE practice when the people staffing it have no time to learn the discipline. Concretely, 10% protected time means one half-day per week where an SRE studies agent failure modes, reviews postmortems from other teams, or shadow-tests a new observability tool without on-call interruptions. The organizations with the lowest toil trends treat learning hours like error budgets: protected, measured, and non-negotiable.

Maturity Level	Platform State	SRE State	AI Readiness
Foundation	Manual provisioning	Reactive ops, no SLOs	Not ready
Standardized	Self-service IDP	SLOs defined, error budgets	Read-only agents
Measured	Golden paths adopted	Toil tracked, burn-rate alerts	Shadow mode agents
Optimized	Platform-as-product	Blameless culture, SLO-driven	Supervised write agents
Autonomous	Full self-service	Proactive reliability	Agentic AI with guardrails

Table 3: Platform + SRE maturity levels and what each unlocks.

Figure 3: Maturity sequence, platform engineering and SRE prerequisites gate each AI readiness level.

Read-only agents need SLOs because without a defined “good,” the agent cannot distinguish signal from noise. Supervised write agents need blameless culture because humans must feel safe overriding the machine.

Where Should You Start?

Enforce the prerequisites before enabling agentic AI on any service:

package sre.ai.readiness

import rego.v1

default allow_agentic_ai := false

allow_agentic_ai if {
    input.slo_defined
    input.error_budget_policy
    input.shadow_period_days >= 30
    input.decision_provenance      
    input.rollback_automated
    input.toil_measured
}sre-readiness-check.rego

Code Snippet 3: OPA policy gate enforcing the minimum bar before any service receives agentic AI write access. Shadow period and decision provenance are the two gates most commonly skipped in practice.

The Four Actions in Order

Four actions, in order:

Audit your toil budget. Measure actual toil against perceived toil. If practitioners report higher friction while dashboards show fewer tickets, you have shifted toil rather than eliminated it. That distinction determines whether your AI investment compounds value or accelerates the same failure modes at higher velocity.
Define SRE boundaries. One team owns the IDP. Another owns the error budgets. Overlap is where accountability dies. Ambiguity here is the single most common reason SRE functions fail to scale in regulated enterprises.
Split error budgets by author type. Human-authored and AI-authored deployments have different failure profiles. Track them independently and revoke AI write access when the budget burns too fast.
Protect learning time. Budget 10% of engineering hours for skill development or accept compounding operational risk. At 6% of teams with protected learning time, the industry is not doing this, and the toil trend reflects it.

Start with the toil audit, the only prerequisite you can measure without instrumentation already in place. Measure first. Then automate.

References

Becker et al., “Evidence on the Impact of Generative AI on Software Development” (2025) — https://arxiv.org/abs/2507.09089
Catchpoint, “SRE Report 2026” (2026) — https://www.catchpoint.com/learn/sre-report-2026
Chen et al., “AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds” (2025) — https://arxiv.org/abs/2501.06706
Chen et al., “STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds” (2025) — https://arxiv.org/abs/2506.02009
Clouatre, H., “SRE Shadow Replay: File-Prediction Experiment Data” (2026) — https://github.com/clouatre-labs/sre-shadow-replay
DevOps.com, “Survey: AI Tools are Increasing Amount of Bad Code Needing to be Fixed” (2025) — https://devops.com/survey-ai-tools-are-increasing-amount-of-bad-code-needing-to-be-fixed-2/
DORA (Google Cloud’s DevOps Research and Assessment), “2025 State of AI-assisted Software Development” (2025) — https://dora.dev/research/2025/dora-report/
European Parliament and Council, “Digital Operational Resilience Act (DORA)” (2022) — https://www.digital-operational-resilience-act.com/
Google, “Site Reliability Engineering” (2016) — https://sre.google/sre-book/table-of-contents/
Jimenez, C. E. et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (ICLR 2024) — https://arxiv.org/abs/2310.06770
OSFI, “Technology and Cyber Risk Management Guideline B-13” (2022) — https://www.osfi-bsif.gc.ca/en/guidance/guidance-library/technology-cyber-risk-management
Pan et al., “Measuring Agents in Production” (2026) — https://arxiv.org/abs/2512.04123
Rabanser et al., “Towards a Science of AI Agent Reliability” (2026) — https://arxiv.org/abs/2602.16666

What a Null Result Taught Us About AI Agent Evaluation

Hugues Clouâtre — Thu, 26 Feb 2026 13:34:00 GMT

A Google Research paper demonstrates that repeating the entire user prompt verbatim can lift accuracy by up to 76 percentage points at zero output cost. No chain-of-thought overhead. No reasoning budget. Just send the same instruction twice.

We ran 20 parallel agents across two experiments: 10 per experiment, 5 control vs. 5 treatment, blind-scored against a pre-registered rubric.

We found nothing. The nothing is the finding.

Contents

What Did the Paper Claim?
- Why Our Agent Seemed Like a Good Candidate
- Why This Matters for Engineering Teams
How Did We Design the Test?
What Happened in the FastMCP Refactor Test?
Did a Stricter Methodology Change the Result?
What Infrastructure Confound Did We Miss?
Why Did Both Experiments Hit 100%?
- Where the Boundary Falls
What Did We Learn About AI Evaluation Design?
When Should You Use Prompt Repetition?
- What Transfers to Your Team
Did Prompt Repetition Change Anything Else?
References

What Did the Paper Claim?

A 2025 paper by Leviathan et al. at Google Research proposes a simple technique: repeat the entire user prompt once, verbatim, before sending to the model.

The mechanism is structural, not empirical. Decoder-only transformers use causal masking: each token attends only to tokens before it. In a single-pass prompt, early tokens never see later context. Repeating the prompt creates a second copy where every token attends to the full instruction during prefill. This reduces the positional attention decay documented as the “lost in the middle” phenomenon (Liu et al., 2023). This is a fundamental limitation of the decoder-only architecture, not a quirk of specific benchmarks. A 675B-parameter Mixture-of-Experts frontier model and a 3B-active-parameter small language model (SLM) (NVIDIA, 2025) share it equally.

Standard prompt (single pass):
  Token 1  sees: [Token 1]
  Token 5  sees: [Token 1, 2, 3, 4, 5]
  Token 50 sees: [Token 1, 2, ..., 50]
  --> Early tokens are blind to later context

Repeated prompt (two copies):
  Token 51 sees: [Token 1, 2, ..., 50, 51]
  Token 55 sees: [Token 1, 2, ..., 50, 51, 52, 53, 54, 55]
  --> Every token in the second copy attends to the full first copy
  --> Full context available during prefill, zero generation costcausal-masking.txt

Code Snippet 1: Causal masking creates an asymmetry where early tokens cannot attend to later context. Repeating the prompt gives the second copy full visibility over the first.

The reported gains:

Gemini 2.0 Flash-Lite on NameIndex: 21.33% to 97.33% accuracy
GSM8K and MMLU-Pro gains across Gemini 2.0 Flash, GPT-4o, Claude 3.7 Sonnet, DeepSeek V3, and others
Input tokens double; output tokens unchanged in fixed-format benchmarks (no latency increase, unlike chain-of-thought)

The paper positions this as a Pareto improvement over reasoning-heavy approaches: same output budget, better accuracy.

Why Our Agent Seemed Like a Good Candidate

Our Scout delegate, the research agent in our subagent architecture (full recipe), runs on claude-haiku-4-5 at temperature 0.5 with extended thinking off. Haiku 4.5 is structurally a non-reasoning model (extended thinking is opt-in, not default), making it precisely the class of LLM the paper’s title targets.

The paper tested Claude 3 Haiku alongside six other models; its strongest gains came from Gemini 2.0 Flash-Lite and GPT-4o-mini. We tested Claude 4.5 Haiku, a different model generation. Anthropic does not publish architectural details for either model. Whether the technique transfers across generations is an open question this experiment cannot answer, because our ceiling effects prevented any treatment from showing lift.

Why This Matters for Engineering Teams

Teams adopt AI techniques from papers without field-testing them first. BCG reports that 50% of companies are stagnating with AI (BCG, 2025), partly because they ship optimizations without measuring baselines. Shipping an unvalidated prompt change to production would cost more: doubled input tokens on every request, with no accuracy gain to show for it. As we covered in observability for AI agents, optimizing without measuring before and after is flying blind.

How Did We Design the Test?

Both experiments shared the same core structure: 10 parallel async Scout delegates, split 5 control vs. 5 treatment, scored blind against a pre-registered rubric. For detailed methodology and raw data, see Supplementary Materials.

# Shared config across all 10 delegates
model: claude-haiku-4-5
temperature: 0.5
extensions:
  - developer
  - context7
  - brave_search
output: "scout-run-{{ id }}.json"delegate-config.yaml

Code Snippet 2: Shared delegate configuration. All 10 runs use the same model, temperature, and extensions.

Control group: standard Scout instructions (~3,805 characters, instructions x1).

Treatment group: instructions repeated verbatim (~7,633 characters, instructions x2), mimicking the paper’s <QUERY><QUERY> pattern applied to the agent’s system prompt.

The orchestrator spawned all 10 delegates simultaneously via Goose’s background task system and handed off structured JSON. A separate blind-scoring delegate received only the output files (no group labels) and scored each against the rubric. Group assignments were sealed in a label map before scoring began.

Figure 1: Blind evaluation pipeline. Group labels are stripped before scoring to prevent bias.

What Happened in the FastMCP Refactor Test?

Target: FastMCP session ID refactor in math-mcp-learning-server. Open and unimplemented at the time of the experiment.

Rubric: 6 binary criteria, pre-registered before any runs were examined.

9 of 10 delegates produced valid output. control-1 ran 93 messages and wrote no output file. Session log analysis confirmed the file-write instruction appeared only at the end of the delegate prompt, and the model drifted past it. This is a delegate authoring flaw with a known fix: bookend critical instructions at the start and end.

Across the 9 valid runs, 5 of 6 criteria scored 100% in both groups. The only variance was C5 (must-not constraint violations): control 5.50/6, treatment 5.80/6, delta +0.30. The treatment scored marginally higher, but at n=4 vs n=5 with binary outcomes, Fisher’s exact test is degenerate (p = 1.0). Full per-criterion scores are in the raw data.

The task was too easy. The rubric could not discriminate. We needed a harder target.

Did a Stricter Methodology Change the Result?

Target: aptu#737, a tree-sitter AST (Abstract Syntax Tree)-based security scanner evaluation. Harder task, requiring synthesis from source code rather than retrieval from issue text. Unimplemented when tested.

Rubric: 7 binary criteria. C5, C6, and C7 required the delegate to read and reason about actual source code, not just summarize the issue. Pre-registered before any runs began.

Methodology improvements over Experiment 1:

Blinded file naming from the start (scout-run-01.json through scout-run-10.json with sealed label-map.json)
Mann-Whitney U test pre-specified (two-tailed, alpha = 0.05)
Wall-clock latency recorded per delegate

Results:

Group	Score	Wall-clock median
Control (x1)	7/7 all runs	6m 21s
Treatment (x2)	7/7 all runs	7m 29s
Delta	0	+17.8%

Table 1: Experiment 2 results. Zero variance in either group. Mann-Whitney U = 12.5, p = 1.0 (degenerate: complete ties, test cannot be evaluated).

Every Scout, in every run, in both groups, scored 7/7. Even C5, C6, and C7, the synthesis criteria we specifically designed to require source code reasoning, hit 100% across the board.

The 17.8% latency difference is in the expected direction (longer prompt, longer prefill), which is consistent with the paper’s Anthropic-specific latency caveat. At scale, that delta compounds: doubled tokens cost money, and the added prefill time costs throughput across every agent invocation. But n=5 cannot support any inference here, and the finding is further confounded by an infrastructure issue we discovered afterward.

The scores told us nothing. The session logs told us something we did not expect.

What Infrastructure Confound Did We Miss?

Post-hoc session log analysis revealed a confound present in both experiments.

Goose enforces a hard cap of 5 concurrent background delegates. When all 10 delegates were spawned simultaneously, runs 06-10 hit the cap and were queued into a second batch after runs 01-05 completed.

The resulting batch structure was unbalanced:

Batch	Runs	Control	Treatment	Condition
1 (runs 01-05)	5	C1, C2, C3	T1, T2	Resource contested
2 (runs 06-10)	5	C4, C5	T3, T4, T5	No contention

Table 2: The 5-delegate concurrency cap split 10 simultaneous spawns into two unbalanced batches. Exact run assignments are in the raw data.

Treatment delegates landed disproportionately in the less-contested second batch, making any latency comparison between groups uninterpretable. Accuracy was unaffected (ceiling effects dominated regardless), but the exposure is worth naming: pre-registration does not protect against runtime infrastructure behavior you did not know existed.

The confound matters for latency. But the bigger question is why accuracy showed zero variance in the first place.

Why Did Both Experiments Hit 100%?

Two experiments, two rubrics designed to be harder than the last, two 100% results.

This is itself a finding. A well-designed Scout delegate on a well-scoped engineering issue is already operating above the baseline accuracy threshold where prompt repetition shows lift. The paper’s largest gains came from synthetic positional tasks, NameIndex (Leviathan et al., 2025), where the answer is a name buried in a list. Real engineering issues, even unimplemented ones, give the agent structured context, code references, and acceptance criteria. The agent finds what it needs without help from the prefill geometry.

This is the core finding: prompt repetition solves an attention problem that well-scoped engineering tasks do not have. The technique’s value is real, but the paper’s benchmarks do not cover agentic engineering tasks. Our experiments tested that boundary. When the agent already has structured context pointing it to the right code, repeating the instruction adds input tokens without adding accuracy signal. Understanding where SLMs succeed and fail on their own is not academic: hybrid architectures like SMART (Kim et al., 2025) use SLMs as the primary reasoning engine, with LLMs intervening only at critical junctures. Every prompt-level optimization that improves standalone SLM accuracy reduces how often the expensive backstop fires.

For teams evaluating prompt techniques at scale, the implication is financial: doubling input tokens across every agent invocation is a measurable cost increase. If your agents already converge correctly on well-scoped tasks, that spend returns nothing. Embracing negative results as a research practice (Berger et al., 2024) prevents exactly this kind of waste: publication bias toward positive results means the null findings that would have saved you the experiment often go unpublished.

Where the Boundary Falls

The gap is between task types, not between models:

Positional retrieval tasks (NameIndex, needle-in-haystack): high positional attention decay, repetition helps
Structured engineering tasks (scoped issues with code context): low positional decay, Scout already converges correctly

Dimension	Paper (Leviathan et al.)	Experiment 1	Experiment 2
Task type	Standard + custom retrieval (MMLU-Pro, NameIndex, others)	Issue analysis (FastMCP refactor)	Source code synthesis (AST scanner)
Model	Gemini 2.0 Flash-Lite, Claude 3 Haiku, 5 others	Claude 4.5 Haiku	Claude 4.5 Haiku
Sample size	McNemar test on full benchmark datasets (7 benchmarks, 7 models)	n=4 vs n=5 (1 dropped)	n=5 vs n=5
Accuracy delta	47/70 pairs improved, 0 regressed; +76pp on NameIndex (Flash-Lite)	+0.30 (noise)	0.00 (ceiling)
Confounds	None reported	Delegate authoring failure	5-delegate batch cap

Table 3: Comparison of experimental conditions. The paper’s gains concentrate on positional retrieval tasks; our structured engineering tasks hit ceiling effects before any treatment could show lift.

Designing a rubric that discriminates between good and very good on the second category is harder than it looks. Both of ours failed. The criteria require synthesis and judgment under genuine ambiguity, not retrieval from a well-scoped document.

What Did We Learn About AI Evaluation Design?

Rubric Design Is Harder Than Experiment Design

We iterated twice and hit the ceiling both times. A 7-point rubric with “source code synthesis” criteria is not automatically harder. It depends on whether the task actually creates ambiguity the agent must resolve. Ours did not.

A practical calibration target: if your scoring delegate can answer any criterion by reading the issue alone (without running the code), the criterion will not discriminate.

Infrastructure Behavior Is a Confounder

The 5-delegate cap is undocumented. It is enforced as a hard rejection in source (GOOSE_MAX_BACKGROUND_TASKS defaults to 5), with no queuing or retry. Excess delegates are dropped, not deferred. It silently split our groups into unbalanced batches. This category of confound (runtime resource limits, queue behavior, model routing) is endemic to agent systems and invisible without structured logging.

Future experiments: spawn delegates in explicit batches of 5 with documented batch assignments. Record session IDs. Treat infrastructure state as a variable, not background noise.

Delegate Authoring Has a Turn-Length Problem

Long sessions drift from instructions that appear only once. The control-1 failure (93 messages, no output) demonstrated the fix: bookend critical actions at both the start and end of delegate prompts. This class of failure is predictable and preventable, but only if you treat delegate prompt structure as part of your experimental design.

The blind scoring infrastructure proved its value here. Each run produced a structured justification the scorer generated without knowing group assignment:

{
  "run_id": "run-01",
  "C1": 1, "C2": 1, "C3": 1, "C4": 1,
  "C5": 1, "C6": 1, "C7": 1, "total": 7,
  "justifications": {
    "C1": "Issues #735/#736 explicitly identified as regex limitation.",
    "C5": "Backward compatibility addressed via hybrid approach.",
    "C7": "Synthesis connects tree-sitter AST parsing to existing rules."
  }
}scorer-output.json

Code Snippet 3: Blind scorer output for a single run. Each criterion includes a justification generated without knowledge of group assignment.

When Should You Use Prompt Repetition?

The null result is not a failure of the paper. Prompt repetition won 47 out of 70 benchmark-model combinations with zero losses (Leviathan et al., 2025). The technique works. The question is where.

The paper’s gains concentrate on benchmarks with positional retrieval components: NameIndex, MiddleMatch, options-first multiple choice. Tasks where the answer depends on information placement in the context window. The paper also notes a neutral-to-slight effect with reasoning prompts (5 wins, 1 loss, 22 neutral with step-by-step). Reasoning appears to compensate for the same attention decay that repetition addresses.

The industry trend is not exclusively toward reasoning models. Capable SLMs are gaining ground. NVIDIA’s Nemotron 3 Nano (NVIDIA, 2025) activates 3 billion of its 30 billion parameters per token, delivering 3.3x the throughput of Qwen3-30B on a single H200, designed explicitly for multi-agent systems at scale. RLP (Hatamizadeh et al., 2025) embeds reinforcement learning into pretraining itself, lifting math and science accuracy by 19% on a 1.7B-parameter model without post-training reasoning. These models are non-reasoning by default. The causal masking limitation that prompt repetition addresses is structural to the decoder-only architecture all of them share. Their users are also the most cost-sensitive to doubled input tokens. Every token matters when you are optimizing for throughput at the edge.

Our null result came from the other side of that boundary: structured engineering tasks where the agent already has scoped context, code references, and acceptance criteria. The ceiling was in the task, not the technique.

Figure 2: When to use prompt repetition. Three decision points, two outcomes.

What Transfers to Your Team

Three things that transfer directly to any team evaluating AI agent behavior:

Baseline accuracy determines whether any prompt technique has room to work. Measure it before testing an optimization.
Infrastructure constraints are confounder candidates. Audit your delegate system’s limits before attributing latency or throughput differences to treatment variables.
Rubric discrimination is the bottleneck. Two rubrics, two ceiling effects. If your scoring criteria can be satisfied by reading the issue description alone, the rubric will not discriminate.

Did Prompt Repetition Change Anything Else?

One observation worth noting: treatment agents in both experiments used fewer output tokens and messages to reach the same scores. This is consistent with information-theoretic expectations; redundancy in the input reduces decoder uncertainty at each generation step, which should reduce exploratory turns in an agentic loop. The original paper’s benchmarks (MMLU, GSM8K) produce fixed-format answers where output length does not vary, making this effect invisible. Agentic workloads, where the model decides how many turns to take, may be where the efficiency signal surfaces.

The economics are also different than single-turn benchmarks suggest: in a multi-turn session, the doubled prompt adds single-digit overhead to accumulated input, not 100%. The growing conversation history dominates each API call. In our data, treatment agents used 13.1% fewer input tokens and 15.4% fewer output tokens despite the longer prompt. Each avoided turn eliminates an entire context window from the running total. The effect is confounded and too small to draw conclusions, but it is a pattern worth investigating with a discriminating rubric.

For how file-prediction accuracy maps to production governance, see SRE for AI Agents: Error Budgets, Trust Ladders, and 90 Trials.

References

BCG, “AI Adoption Puzzle: Why Usage Is Up But Impact Is Not” (2025) — https://www.bcg.com/publications/2025/ai-adoption-puzzle-why-usage-up-impact-not
Berger et al., “Position: Embracing Negative Results in Machine Learning” (2024) — https://arxiv.org/abs/2406.03980
Clouatre, H., “Orchestrating AI Agents: A Subagent Architecture for Code” (2025) — https://clouatre.ca/posts/orchestrating-ai-agents-subagent-architecture/
Clouatre, H., “Prompt Repetition Experiments: Supplementary Materials” (2026) — https://github.com/clouatre-labs/prompt-repetition-experiments
Hatamizadeh, A. et al., “RLP: Reinforcement as a Pretraining Objective” (2025) — https://arxiv.org/abs/2510.01265
Kim, Y. et al., “Guiding Reasoning in Small Language Models with LLM Assistance” (2025) — https://arxiv.org/abs/2504.09923
Leviathan, Y., Kalman, M., and Matias, Y., “Prompt Repetition Improves Non-Reasoning LLMs” (2025) — https://arxiv.org/abs/2512.14982
Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023) — https://arxiv.org/abs/2307.03172
NVIDIA, “Nemotron 3: Efficient and Open Intelligence” (2025) — https://arxiv.org/abs/2512.20856

Why Your AI Agent Failed in Production

Hugues Clouâtre — Tue, 03 Feb 2026 12:12:00 GMT

Your AI agent just approved a $50,000 invoice for office supplies. A legitimate vendor. The PO number matches. But the quantity is wrong by a factor of 10. By the time finance catches it, you’ve already paid, the goods already shipped, and you’re stuck negotiating a return.

The agent’s logs show “decision: approved” but nothing about why it ignored the quantity anomaly that a human would have caught instantly. Without proper instrumentation, root cause analysis stretches from minutes to days. This is what happens when observability is treated as “nice to have” instead of foundational infrastructure.

This post covers the production architecture, the vendor-neutral stack, and why you need to instrument before deployment, not after the first failure.

Contents

Why Should Observability Be Foundational Infrastructure?
- The Observer Effect Paradox
What Is Decision Provenance and Why Does Compliance Require It?
How Do Silent Integration Failures Kill AI Agents?
Why Do Token Costs Spiral Out of Control?
- Alerting on Token Budgets
What Does a Vendor-Neutral Observability Stack Look Like?
- How the Components Connect
- Why Trace-Metric Correlation Matters
What Is the ROI and How Do You Start?
Are You Ready for Production?
References

Why Should Observability Be Foundational Infrastructure?

When your agent fails in production, you need to answer three questions immediately: What decision did it make? What data did it use? How much did it cost? Without observability built in from day one, you’re flying blind. The difference between a 5-minute fix and a 5-hour war room is whether you instrumented decision provenance, integration health, and cost tracking before deployment.

The Observer Effect Paradox

Instrumentation changes what you measure. Synchronous logging to external systems adds latency to every LLM call. In multi-agent systems, this can trigger timeout-based retries where observability causes the failures it detects.

OpenTelemetry’s BatchSpanProcessor solves this by queuing spans in memory and exporting in batches, minimizing per-request overhead.

What Is Decision Provenance and Why Does Compliance Require It?

How do you prove your AI agent made the right decision six months ago when a regulator asks? Logging outputs without reasoning fails every major compliance framework.

Why Every Framework Requires Decision Trails

Every major compliance framework mandates reconstructible reasoning.

SOC 2 Type II: audit trails of system access and user activity. The gen_ai.conversation.id attribute ties every decision to a user and timestamp.
GDPR Article 30: records of processing activities. Structured logs with trace IDs link inputs to outputs.
HIPAA: audit controls for ePHI access. Span attributes capture what data the agent accessed.
PCI DSS 4.0.1 Requirement 10: tracking cardholder data access with automated log reviews. Prometheus metrics enable real-time anomaly detection.

Structured Logging with Correlation IDs

The fix links every decision to its inputs.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
import logging

tracer = trace.get_tracer(__name__)
logger = logging.getLogger(__name__)

def make_decision(invoice_data, retrieved_context):
    with tracer.start_as_current_span("make_decision") as span:
        span.set_attribute("invoice.id", invoice_data["id"])
        span.set_attribute("invoice.amount", invoice_data["amount"])
        span.set_attribute("context.sources", len(retrieved_context))
        
        decision = analyze(invoice_data, retrieved_context)
        confidence = calculate_confidence(decision)
        
        span.set_attribute("decision.result", decision["action"])
        span.set_attribute("decision.confidence", confidence)
        
        logger.info(
            "Decision made",
            extra={
                "trace_id": format(span.get_span_context().trace_id, "032x"),
                "invoice_id": invoice_data["id"],
                "decision": decision["action"],
                "confidence": confidence,
                "context_count": len(retrieved_context)
            }
        )
        
        return decisionagent/decision_logger.py

Code Snippet 1: OpenTelemetry structured logging captures decision provenance with trace IDs, span attributes, and correlation across the entire request lifecycle.

This gives you a complete audit trail: trace ID links the decision to all upstream data retrievals, span attributes capture the decision logic, and structured logs provide queryable records. When the regulator asks “why did you approve invoice #12345?”, you can show exactly what data the agent saw and how it weighted each factor.

Tracking Tool Calls with GenAI Semantic Conventions

Multi-agent systems make dozens of tool calls per decision. OpenTelemetry’s GenAI semantic conventions provide standard attributes for tracking these interactions:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def execute_tool_call(tool_name, arguments, conversation_id):
    with tracer.start_as_current_span("execute_tool") as span:
        # Standard GenAI attributes
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", tool_name)
        span.set_attribute("gen_ai.conversation.id", conversation_id)
        span.set_attribute("gen_ai.tool.call.arguments", str(arguments))
        
        result = call_tool(tool_name, arguments)
        
        span.set_attribute("gen_ai.tool.call.result", str(result))
        
        return resultagent/tool_tracking.py

Code Snippet 2: GenAI semantic conventions enable cross-platform analysis across LangChain, LlamaIndex, and custom agents.

Standard attributes like gen_ai.tool.name let you answer operational questions across your entire stack: “Which tools fail most often?” or “Which conversations require the most tool calls?” When you swap frameworks, your dashboards still work.

How Do Silent Integration Failures Kill AI Agents?

Your AI agent calls a legacy API that returns HTTP 200 with an empty result set. The agent interprets “no data” as “no problem” and proceeds. But the API actually failed silently because the database connection pool was exhausted. By the time you notice, you’ve processed 500 transactions with incomplete data.

AI agents don’t fail loudly. They fail gracefully, hiding problems until they cascade. You need distributed tracing that correlates agent decisions with integration health across every dependency.

from opentelemetry import trace, propagate
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def retrieve_from_legacy_api(query):
    with tracer.start_as_current_span("legacy_api_call") as span:
        span.set_attribute("api.endpoint", "/legacy/search")
        span.set_attribute("query", query)
        
        headers = {}
        propagate.inject(headers)  # Inject trace context 
        
        response = requests.get(
            "https://legacy.example.com/search",
            params={"q": query},
            headers=headers
        )
        
        span.set_attribute("http.status_code", response.status_code)
        span.set_attribute("response.size", len(response.content))
        
        if response.status_code == 200 and len(response.json()) == 0:
            span.set_status(Status(StatusCode.ERROR, "Empty result set"))
            span.add_event("Suspicious empty response from healthy endpoint")
        
        return response.json()agent/trace_integration.py

Code Snippet 3: Distributed tracing propagates correlation IDs and flags suspicious patterns like empty responses from healthy endpoints.

Correlation ID propagation (line 12) and explicit error marking for suspicious patterns (lines 23-25) are what matter. When you see a spike in “empty result set” errors correlated with database saturation metrics, you know the integration is degraded even though HTTP status codes look fine.

Why Do Token Costs Spiral Out of Control?

Your agent works in testing. Then production traffic hits and your LLM bill explodes. AI costs are surging 36% year-over-year, yet only half of organizations can confidently evaluate ROI (CloudZero, 2025). Without per-operation cost tracking, you can’t identify which workflows are burning money.

Consider a Claude 4.5 Sonnet deployment: input tokens cost $3/million, output tokens cost $15/million. A single complex query might use 50K input tokens and 4K output tokens, costing $0.21. At 10,000 queries per day, that’s $2,100 daily, or $63,000 monthly, just for one workflow. If your agent retries on failures or chains multiple calls, costs multiply fast.

from prometheus_client import Counter, Histogram

# Token counter with model and operation labels
tokens_total = Counter(
    'ai_tokens_total',
    'Total tokens consumed',
    ['model', 'operation', 'user_tier']
)

# Latency histogram with cost correlation
request_duration = Histogram(
    'ai_request_duration_seconds',
    'Request duration',
    ['operation'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, float('inf')]
)

def process_query(query, user_tier):
    with request_duration.labels(operation='query').time():
        embedding = embed(query)
        tokens_total.labels(
            model='text-embedding-3-small',
            operation='embed',
            user_tier=user_tier 
        ).inc(len(query.split()))
        
        results = vector_search(embedding)
        
        response = generate_response(results)
        tokens_total.labels(
            model='claude-4.5-sonnet',
            operation='generate',
            user_tier=user_tier 
        ).inc(response['usage']['total_tokens'])
        
        return responseagent/metrics.py

Code Snippet 4: Prometheus metrics track token usage with labels for model, operation, and user tier, enabling real-time cost monitoring and per-operation granularity to prevent budget overruns.

Alerting on Token Budgets

Labels let you slice cost by model, operation, and user tier. When free-tier token usage spikes on expensive models, you can throttle, switch to cheaper models, or convert users to paid tiers before costs spiral.

groups:
- name: ai-cost-alerts
  rules:
  - alert: TokenBudgetExceeded
    expr: sum(rate(ai_tokens_total[5m])) by (user_tier) > 1000
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Token budget exceeded for {{ $labels.user_tier }}"
      description: "{{ $labels.user_tier }} tier consuming {{ $value }} tokens/sec"ai-alerts.yml

Code Snippet 5: Prometheus alerting rule triggers when any user tier exceeds 1,000 tokens per second sustained over 5 minutes, enabling proactive cost control before budget overruns.

What Does a Vendor-Neutral Observability Stack Look Like?

Enterprise platforms like Datadog and Splunk offer polished, integrated experiences. For teams prioritizing cloud-native portability, OpenTelemetry handles instrumentation, Prometheus stores metrics, and Grafana visualizes everything. Zero licensing cost, no vendor lock-in, and production-proven.

Already invested in an enterprise platform? OpenTelemetry collectors export directly to these platforms, preserving full trace context and semantic attributes. You can adopt incrementally without disrupting existing dashboards, gaining richer observability now and portability for future migrations.

Figure 1: OpenTelemetry + Prometheus + Grafana stack

How the Components Connect

Your agent emits traces, metrics, and logs via OpenTelemetry SDKs. The OpenTelemetry Collector receives, processes, and routes telemetry to backends. Prometheus scrapes metrics and stores time-series data. Grafana queries Prometheus for metrics, Tempo for traces, and Loki for logs, correlating them in unified dashboards.

Why Trace-Metric Correlation Matters

When a user reports “the agent is slow”, start in Grafana, filter metrics by user ID, see elevated p95 latency, drill down to the request, and find the bottleneck in 30 seconds. Without correlation, you’re grepping logs for hours.

Figure 2: Trace ID propagation with per-operation latency breakdown

Every request gets a correlation ID that propagates through RAG retrieval, API calls, and decision logic. When you need to audit a decision, query by trace ID to reconstruct the entire flow: what data was retrieved, which APIs were called, response times, and token consumption.

What Is the ROI and How Do You Start?

Setup cost for the vendor-neutral stack is roughly 40-80 hours of engineering time ($8K-$16K at $200/hour), with payback in 1-3 months. Chronosphere’s Forrester TEI study shows 165% ROI with 6-month payback for observability investments (Forrester, 2022). Microsoft Sentinel’s TEI study shows 201% ROI for security observability (Forrester, 2020). This stack delivers comparable ROI with full portability.

Without	With
$5K-$20K/month untracked token spend	Per-request cost attribution
4-8 hours debugging per incident	30 minutes with trace correlation
$20K-$50K manual audit reconstruction	Query-ready decision logs

Table 1: AI Observability ROI

Start small: instrument one critical path (the highest-risk decision your agent makes) with decision provenance logging. Add integration health tracing for your most fragile API dependency. Implement cost tracking for your most expensive model. Expand based on what breaks. This is the same incremental approach I described in my AI agents ROI post: start with 5% of workflows, prove value, then scale.

Already using RAG for legacy systems? Add distributed tracing to correlate retrieval failures with agent decisions. Implementing AI-augmented CI/CD? Instrument the feedback loop to measure latency reduction. Building multi-agent orchestration? Add trace IDs to handoff files to debug cross-agent failures. Even without a full tracing backend, a shared ID lets you grep the entire workflow chain.

Decision provenance, integration health, and cost runaway are not edge cases. They cause production AI failures. Fix them before deployment, not after the invoice arrives.

Are You Ready for Production?

Before your next AI deployment, verify these four capabilities:

Decision provenance: High-risk workflows log inputs, reasoning, and outputs with trace IDs using OpenTelemetry and structured logging
Integration health: Distributed tracing covers legacy APIs and third-party services, with alerts on silent failures like empty responses from healthy endpoints
Cost attribution: Token usage tracked per model, operation, and user tier, with budget alerts via Prometheus metrics
Audit reconstruction: Any decision from the past 12 months can be fully reconstructed in under 30 minutes

If you can’t check all four, you’re not ready for production.

For the SRE framework that operationalizes decision provenance with error budgets and trust ladders, see SRE for AI Agents: Error Budgets, Trust Ladders, and 90 Trials.

References

AICPA, “2017 Trust Services Criteria (With Revised Points of Focus - 2022)” (2022) - https://www.aicpa-cima.com/resources/download/2017-trust-services-criteria-with-revised-points-of-focus-2022
CloudZero, “The State of AI Costs in 2025” (2025) - https://www.cloudzero.com/state-of-ai-costs/
European Union, “GDPR Article 30: Records of Processing Activities” (2018) - https://gdpr-info.eu/art-30-gdpr/
Forrester Consulting, “The Total Economic Impact of Chronosphere” (2022) - https://chronosphere.io/forrester-total-economic-impact-chronosphere/
Forrester Consulting, “The Total Economic Impact of Microsoft Sentinel” (2020) - https://tei.forrester.com/go/microsoft/microsoft_sentinel/
Kiteworks, “HIPAA Audit Logs: Complete Requirements for Healthcare Compliance in 2025” (2025) - https://www.kiteworks.com/hipaa-compliance/hipaa-audit-log-requirements/
OpenTelemetry, “Semantic Conventions for Generative AI Systems” — https://opentelemetry.io/docs/specs/semconv/gen-ai/
OpenTelemetry, “Tracing SDK Specification” — https://opentelemetry.io/docs/specs/otel/trace/sdk/
PCI Security Standards Council, “PCI DSS v4.0.1” (2024) - https://docs-prv.pcisecuritystandards.org/PCI%20DSS/Standard/PCI-DSS-v4_0_1.pdf

RAG for Legacy Systems: 7,432 Pages to 3s Answers

Hugues Clouâtre — Tue, 17 Feb 2026 12:00:00 GMT

Your legacy system documentation is 20 years old, 7,432 pages, and locked in PDFs. Manual search takes 15-30 minutes per query. We made it queryable in 170 seconds. Query response time: 3-5 seconds. ROI break-even: one day.

This isn’t a prototype. It’s Retrieval-Augmented Generation (RAG) on Amazon Bedrock, a system that retrieves relevant documentation and uses an LLM to generate answers without retraining models. Validated across four LLM families with 480 measurements. The implementation indexes 20,679 chunks and delivers sub-5-second responses with model-agnostic reranking. Overhead: 27.2ms ± 4.6ms regardless of which LLM you use.

Yes, 7,432 pages fit in any search index. But ranked results aren’t answers.

Here’s the production architecture, the multi-model validation data, and why you can switch providers without re-tuning.

Contents

Why RAG, Not Fine-Tuning?
How Does RAG Turn PDFs Into Answers?
Does Reranking Work Across Different Models?
What Are the Real Performance Numbers?
What’s the ROI Without Modernization?
When Does RAG Fail?
How Do You Migrate from Prototype to Production?
What Should You Do Next?
References

Why RAG, Not Fine-Tuning?

Fine-tuning trains a model on your docs. It bakes knowledge into weights (making provenance verification difficult), requires retraining for every update, and costs $1.32-6.24 per run on A100 GPUs (Thunder Compute, 2025). RAG costs $0 setup with local embeddings, $0.0011 per query on Bedrock, updates in 2 seconds, and keeps sources verifiable.

For legacy systems, choose RAG for operational factors, not economics. Documentation is scattered across wikis and PDFs. It evolves as reverse-engineering uncovers new system behaviors, while fine-tuning would require retraining each time. Query volume is low (dozens per week). The deciding factors: instant updates (2 seconds vs retraining), source citations for compliance, and simpler maintenance. We chose RAG for agility: 170s setup, 2s updates, under $20/year.

How Does RAG Turn PDFs Into Answers?

The Ingestion Pipeline

The pipeline has six stages:

Extract - Pull text from PDFs using PyMuPDF (44 pages/second)
Transform - Convert to Markdown with heading detection
Chunk - Split at headings (1,000-char limit, 200-char overlap)
Embed - Generate vectors with all-MiniLM-L6-v2 (local, free)
Index - Store in ChromaDB vector database
Retrieve - Hybrid search (BM25 + vector) with FlashRank reranking

import fitz  # pymupdf

doc = fitz.open(pdf_path)
for page_num in range(len(doc)):
    page = doc[page_num]
    text = page.get_text()
    
    for line in text.split("\n"):
        # Detect chapter headings (e.g., "Chapter 1. Title")
        if line.startswith("Chapter ") and ". " in line:
            cleaned_lines.append(f"\n## {line}\n")
        # Detect section headings
        elif len(line) < 80 and line[0].isupper():
            cleaned_lines.append(f"\n### {line}\n")
doc.close()src/ingest.py

Code Snippet 1: PyMuPDF extracts text and converts to Markdown with heading detection, processing 44 pages/second.

The pipeline converts PDFs to Markdown before chunking. This preserves document structure (chapters, sections, headings) and enables Markdown-aware chunking that respects semantic boundaries. Chunks split at heading boundaries (## , ### ) instead of mid-paragraph, keeping related content together. The Markdown files are cached, so subsequent runs skip PDF extraction and complete in 2 seconds instead of 170 seconds.

Hybrid Retrieval: BM25 + Vector Search

Why not just Elasticsearch or Ctrl+F? Pure keyword search fails when you search “memory error” but the 2005 docs say “data file cache exhaustion.” Pure vector search misses exact terms like “port 5432.” Hybrid retrieval solves ranking. The LLM solves synthesis: combining fragments from multiple documents into an actionable answer. Reciprocal Rank Fusion (RRF) consistently outperforms single-method search (Mandikal & Mooney, 2024).

# RRF combines BM25 + vector scores
doc_scores: dict[str, tuple[Document, float]] = {}

for rank, idx in enumerate(bm25_top_indices[:retrieve_k]):
    doc = chunks[idx]
    doc_id = doc.metadata.get("source", "") + str(hash(doc.page_content[:100]))
    rrf_score = 1 / (rank + 60)  # RRF with k=60
    if doc_id in doc_scores:
        doc_scores[doc_id] = (doc, doc_scores[doc_id][1] + rrf_score)
    else:
        doc_scores[doc_id] = (doc, rrf_score)src/rag.py

Code Snippet 2: RRF formula combines keyword and semantic search scores with k=60 constant.

Hybrid retrieval returns 16 candidate chunks. A cross-encoder model (FlashRank) scores each query-document pair and returns the top 8. This fixes the precision problem: high recall from hybrid search, high precision from reranking.

Example: Error Lookup in 3.4 Seconds

User query: “What is error 1006030 and how do I fix it?”

Generated answer:

Error 1006030: “Failed to bring a data file page into cache. Data file cache is too small.”

Cause: Essbase cannot store the data file page in the data file cache.

Solution: Increase the data file cache size. After fixing, check for database corruption (Error Message Reference, p. 126).

Timing: 3.4s total (retrieval: 80ms, reranking: 31ms, generation: 3.3s)

Retrieved from: Error Message Reference v11.1.1 (ranked 3rd of 8 after reranking)

The system retrieved error 1006030 from the Error Message Reference (ranked 3rd of 8 after reranking) and synthesized an actionable answer. Manual search would require opening the 1,200-page Error Message Reference PDF and using Ctrl+F.

Figure 1: RAG pipeline with hybrid retrieval and reranking (FlashRank adds 31ms overhead for 6-8% accuracy gain) (George, 2025)

Local Embeddings and Model-Agnostic Design

Why local embeddings? Cost, simplicity, and performance. Cloud embedding APIs charge $0.10-0.50 per million tokens. Local models are free, require no API keys, and embed 1,000 chunks in under 10 seconds on CPU. The all-MiniLM-L6-v2 model is 80 MB and runs without GPU acceleration.

The architecture is model-agnostic by design. We use Amazon Bedrock, but the same pipeline works with Azure OpenAI, Google Vertex AI, or local models.

Does Reranking Work Across Different Models?

We tested four LLM families across two providers (Amazon Bedrock, OpenRouter) to validate portability. Mean latency: 27.2ms ± 4.6ms across 480 measurements, with no statistically significant difference (ANOVA p=0.34). Cross-provider variance was only 4.1ms.

Model	Family	Latency	Provider
Claude Haiku 4.5	Anthropic	+31.3ms	Amazon Bedrock
Mistral Devstral-2512	Mistral	+32.5ms	OpenRouter
Llama 3.3 Instruct	Meta	+24.1ms	OpenRouter
Qwen 2.5 Coder	Alibaba	+25.1ms	OpenRouter

Table 1: Latency is consistent across models and providers (480 measurements, ANOVA p=0.34)

The latency is dominated by FlashRank’s cross-encoder, not the LLM. This means you implement once and switch providers without re-tuning.

from flashrank import Ranker, RerankRequest

def _rerank(self, query: str, docs: list[Document]) -> list[Document]:
    passages = [
        {"id": i, "text": doc.page_content, "meta": doc.metadata}
        for i, doc in enumerate(docs)
    ]
    
    rerank_request = RerankRequest(query=query, passages=passages)
    results = self.ranker.rerank(rerank_request)
    
    return [docs[result["id"]] for result in results[:RERANK_TOP_N]]src/rag.py

Code Snippet 3: FlashRank reranks 16 candidates in 31ms using cross-encoder scoring.

Reranking is infrastructure, not model-specific configuration. Build it into your retrieval pipeline and forget about it.

What Are the Real Performance Numbers?

With model-agnostic reranking validated, here are the production metrics.

We indexed 7,432 pages in 170 seconds. First-time setup includes PDF extraction (120s), chunking (20s), embedding (25s), and indexing (5s). Cached runs skip extraction and take 2.2 seconds. Query response time averages 3-5 seconds: retrieval (80ms), LLM generation (4s), overhead (200ms).

Cost per query is $0.01-0.05 on Amazon Bedrock. Input tokens (context from retrieved chunks) cost $0.25 per million. Output tokens (LLM answer) cost $1.25 per million. A typical query uses 2,000 input tokens and 500 output tokens, totaling $0.0011.

Metric	System A (Docs)	System B (Notes)
Documents	14 PDFs (7,432 pages)	94 markdown files
Chunks	20,679	Auto-chunked
First run	170s	40s
Cached run	2.2s	2s
Query time	3-5s	3-5s
Cost/query	$0.01-0.05	$0.01-0.05

Table 2: Performance metrics across two production RAG systems (System A handles technical docs, System B processes meeting notes)

Reranking adds 31ms to retrieval time. That’s a 65% increase in retrieval latency but only 0.3% of total query time. Users don’t notice 31ms in a 9-second end-to-end response. The 6-8% accuracy improvement compounds with hybrid retrieval’s gains over single-method search, making the overhead negligible compared to the final quality benefit. For detailed methodology and raw data, see Supplementary Materials.

What’s the ROI Without Modernization?

Manual search through 7,432 pages takes 15-30 minutes (median: 25 min). You open PDFs, use Ctrl+F, read context, cross-reference sections. RAG reduces this to 3-5 seconds.

Assume 10 queries per day during a 6-month migration project. Labor cost: $100/hour (mid-market technical consultant). Time saved: 25 minutes per query. Success rate: 87% (28 of 32 evaluation queries returned useful results; 4 required human review).

Daily savings: 10 queries × 25 min × ($100/hr ÷ 60) × 87% success rate = $362/day

Setup cost: 170 seconds of compute time plus $0 for local embeddings. Query cost: $0.01-0.05 on Amazon Bedrock. Break-even happens in one day.

When Does RAG Fail?

RAG fails on multi-step reasoning, ambiguous questions, and knowledge not in the docs. We’ve seen three failure modes in production.

Hallucination

The LLM invents answers not in the retrieved chunks. Mitigation: show source citations, add confidence scores, constrain responses to retrieved context only. We display the top 3 source documents with page numbers for every answer.

Context Overflow

Complex queries need more context than fits in the LLM’s window. Mitigation: break queries into sub-questions, use query expansion for domain terms, implement multi-hop retrieval for connected concepts.

Stale Data

Documentation changes but embeddings don’t update. Mitigation: hash-based cache invalidation for PDFs, timestamp-based for markdown files, automated re-indexing on file changes.

Corpus Limitations

Not all failures are system failures. The evaluation revealed three corpus-related issues:

Corpus gap: Knowledge doesn’t exist (e.g., specific error codes not documented). The system correctly responds “I don’t know.”
Scattered information: Knowledge exists but spread across sections, making synthesis incomplete.
Query formulation: Symptom-based queries (“out of memory errors”) outperform code-based queries (“error 1012001”) when exact codes aren’t indexed.

These are honest limitations, not RAG failures. The mitigation is corpus expansion, not system tuning.

What is the Overall Failure Rate?

12.5% of queries (4 of 32) need human review, primarily for multi-step reasoning or ambiguous questions. The alternative is searching 7,432 pages manually. RAG handles the straightforward cases autonomously, while experts focus on edge cases.

Query Category	Success Rate	Common Failure Mode	Mitigation
Error lookup	62%	Exact code not in corpus	Symptom-based queries; corpus expansion
Conceptual	100%	Rare; scattered information	Query expansion with domain terms
Procedural	100%	None observed (n=8)	Query expansion with command names
Multi-hop	88%	Knowledge scattered or missing	Corpus expansion; honest “not found”

Table 3: 32 scored queries across 4 categories (n=8 each), 98.1% ground truth accuracy, 0% false positive rate. Success rate includes partial matches. (methodology)

The key is transparency. Users see which documents were retrieved, can verify claims, and know when to escalate. Trust comes from citations, not blind faith in LLM outputs.

How Do You Migrate from Prototype to Production?

We started on OpenRouter’s free tier. Model: Devstral-2512. Cost: $0. Limits: rate-limited, no compliance guarantees. We validated quality with 20-30 test queries.

Migration to Amazon Bedrock took under 30 minutes. Code changes: swap dependencies (langchain-openai to langchain-aws), replace ChatOpenAI with ChatBedrock, update authentication to use AWS credentials instead of API keys. Benefits: no rate limits, SOC 2 compliance, governance controls, better answer quality from Claude Haiku 4.5.

The migration path: start small with one document set and one use case. Validate quality with test queries comparing RAG answers to ground truth from source documents. Measure adoption by tracking query volume and user feedback. Iterate by adding more docs, tuning chunking strategy, and improving retrieval.

Figure 2: Migration path from free tier validation to enterprise production (iterate on quality before investing in infrastructure)

Scale by building multiple RAG systems for different domains. We run two: one for technical documentation, one for meeting notes and tribal knowledge. Same architecture, different corpora. Total maintenance: under 1 hour per month.

What Should You Do Next?

Identify high-value document sets. Look for onboarding materials, compliance docs, or migration guides. Estimate ROI using queries per day, time saved per query, and hourly labor cost. If the math works, start with a free tier.

Use OpenRouter or local models for validation. Run 20-30 test queries. Compare RAG answers to ground truth from source documents. Measure accuracy, check for hallucinations, verify source citations. If quality is acceptable, invest in enterprise infrastructure.

Amazon Bedrock and Azure OpenAI offer compliance, governance, and better models. Cost is $0.01-0.05 per query. For 100 queries per day, that’s $1-5 daily or $30-150 monthly. Compare that to $9,000 in labor savings.

The decision framework: RAG wins when documentation changes frequently, source citations matter for compliance, or you need operational agility. Fine-tuning wins when knowledge is stable, you need specialized behavior beyond retrieval, or query volume is extreme (thousands per day) with strict latency requirements.

For legacy systems, RAG delivers ROI without modernization. No need to rewrite docs, migrate databases, or retrain staff. Layer RAG over existing PDFs and get 3-second answers to 20-year-old questions.

For broader integration patterns and ROI frameworks, see AI Agents in Legacy Systems: ROI Without Modernization.

References

Braintrust, “RAG Evaluation Metrics: How to Evaluate Your RAG Pipeline” (2025) — https://www.braintrust.dev/articles/rag-evaluation-metrics
Clouatre, H., “RAG Reranking Benchmarks: Supplementary Materials” (2026) — https://github.com/clouatre-labs/rag-reranking-benchmarks
de Luis Balaguer et al., “RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture” (2024) — https://arxiv.org/abs/2401.08406
Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023) — https://arxiv.org/abs/2305.14314
Gan et al., “Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey” (2025) — https://arxiv.org/abs/2504.14891
George, Sherine, “Enhancing Retrieval-Augmented Generation with Two-Stage Retrieval: FlashRank Reranking and Query Expansion” (2025) — https://arxiv.org/abs/2601.03258
LangChain Documentation, “Contextual Compression and Reranking” (2025) — https://python.langchain.com/docs/how_to/contextual_compression/
Mandikal & Mooney, “Sparse Meets Dense: A Hybrid Approach to Enhance Scientific Document Retrieval” (2024) — https://arxiv.org/abs/2401.04055
Oche et al., “A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions” (2025) — https://arxiv.org/abs/2507.18910
Thunder Compute, “AI GPU Rental Market Trends December 2025: Complete Industry Analysis” (2025) — https://www.thundercompute.com/blog/ai-gpu-rental-market-trends

AI Agents in Legacy Systems: ROI Without Modernization

Hugues Clouâtre — Thu, 05 Feb 2026 14:06:00 GMT

You run a company with SAP, mainframe, or AS400 systems that work but won’t win awards. The board wants AI. Your team wants modernization budgets. You’re stuck in the middle.

Every AI agent case study assumes clean APIs, cloud-native apps, and real-time data. Your world is batch jobs, COBOL, and integration layers built in 2003. The conventional answer is “modernize first, then AI.” That’s a 2-5 year, $5M-$50M bet before you prove a single dollar of AI value.

This post shows where AI agents make economic sense on top of legacy systems, how to measure ROI without enterprise-wide transformation, and which integrations work when your data lives in places LLMs have never heard of. You’ll walk away with a decision framework for identifying agent use cases that pay back in quarters, not years.

Contents

Why Legacy Systems Became the #1 AI Adoption Obstacle
The Reverse Modernization Strategy: Layer AI First, Upgrade Later
- Why This Works
- When Reverse Modernization Doesn’t Apply
How Do AI Agents Actually Integrate with Legacy Systems?
Why Observability Infrastructure Is Non-Negotiable
What ROI Can You Actually Expect?
Why Do 40% of AI Agent Projects Still Fail?
How to Start: A Practical Implementation Framework
Why Should Your Board Fund This Now?
References

Why Legacy Systems Became the #1 AI Adoption Obstacle

Legacy systems top the list of AI adoption obstacles, but the conventional fix is worse than the problem. Traditional modernization projects require multi-year timelines and eight-figure budgets before proving a single dollar of AI value. No wonder 40% of agentic AI projects will be canceled by 2027 (Gartner, 2025) due to escalating costs and unclear business value.

The real bottleneck isn’t legacy systems. It’s the false choice between “modernize everything” and “do nothing.” You need integration patterns that work with what you have.

The Reverse Modernization Strategy: Layer AI First, Upgrade Later

Layer AI agents over existing systems first. Capture ROI in months, then fund selective modernization. Prove value before investing in infrastructure.

Atera reduced sales response times by 60% (World Economic Forum, 2026) by integrating AI agents with their existing CRM and ticketing systems. They didn’t rebuild their infrastructure. They built a layer on top. Armis accelerated RFP response capacity by 73% (AutoRFP, 2026) without adding headcount. Both companies proved the business case before investing in modernization.

Why This Works

Reverse modernization works because legacy systems keep running (no disruption), ROI arrives in 3-6 months (30-80% productivity gains), and you start small: one workflow, one team, one agent.

Modernization is a binary bet. Agent layering is incremental: prove value, fund upgrades, repeat.

When Reverse Modernization Doesn’t Apply

Three scenarios require modernization first. End-of-life systems without vendor support expose you to compliance violations and security breaches (Cyber Snowden, 2026). Agent integration can’t fix missing security patches. Regulatory mandates that explicitly require infrastructure upgrades (e.g., PCI-DSS 4.0, GDPR data residency) make layering non-compliant. Systems scheduled for decommissioning within 12 months don’t justify integration investment. In these cases, accelerate modernization or sunset the system entirely.

For everything else, reverse modernization applies.

Figure 1: Reverse modernization flow (agents first, infrastructure later)

How Do AI Agents Actually Integrate with Legacy Systems?

But integration is where most projects fail. Agents need access to data and business logic buried in legacy systems. You have three options. Each has tradeoffs.

API Mediation Layer

Build a facade that abstracts legacy complexity. Agents interact with clean, modern interfaces while the mediation layer handles authentication, data translation (EBCDIC to UTF-8, fixed-width to JSON), and error handling. When the legacy system changes, you update the facade, not the agents. You also get a single point for logging, monitoring, and compliance audits.

from fastapi import FastAPI
from pydantic import BaseModel

class Customer(BaseModel):  # Modern JSON schema
    id: str
    name: str
    balance: float

@app.get("/customers/{customer_id}", response_model=Customer)
async def get_customer(customer_id: str) -> Customer:
    raw = legacy_client.call("CUSTINQ", customer_id.ljust(10))
    return Customer(
        id=raw[0:10].strip(),
        name=raw[10:40].encode("cp037").decode("utf-8"),  # EBCDIC to UTF-8  
        balance=int(raw[40:52]) / 100,  # Packed decimal
    )mediation/legacy_facade.py

Code Snippet 1: FastAPI facade translates COBOL fixed-width records to validated JSON.

Event-Driven Architecture

Legacy systems publish state changes through Dapr, which supports Kafka, Azure Event Hub, and others. Agents subscribe and react in near real-time. This pattern scales better than API mediation: the system pushes updates when they matter instead of agents polling constantly. Dapr’s abstraction avoids vendor lock-in.

The tradeoff: you need to instrument the legacy system to publish events, which isn’t trivial if the system is old and undocumented.

from dapr.ext.grpc import App
from cloudevents.sdk.event import v1
import json

app = App()

@app.subscribe(pubsub_name="legacy-events", topic="orders")
def handle_order(event: v1.Event) -> None:
    data = json.loads(event.Data())  # CloudEvents envelope  
    # Agent processes order without polling legacy system
    agent.process_order(data["order_id"], data["customer_id"])

app.run(6002)events/order_subscriber.py

Code Snippet 2: Dapr subscriber receives legacy system events via CloudEvents (swap Kafka/RabbitMQ/Azure without code changes).

Model Context Protocol (MCP)

Anthropic’s open standard for agent-to-data connections. You write one MCP server for your legacy system, and any agent can use it. No custom integration code for each agent. This matters when coordinating multiple agents, a problem I’ve written about in orchestrating multiple AI agents with subagent architecture.

from fastmcp import FastMCP  # FastMCP 3.0  

mcp = FastMCP("Legacy ERP")

@mcp.tool
def query_customer(customer_id: str) -> str:
    """Query customer from mainframe. Any MCP-compatible agent can call this."""
    result = mainframe_client.execute(f"SELECT * FROM CUSTMAST WHERE ID='{customer_id}'")
    return json.dumps(result)

if __name__ == "__main__":
    mcp.run(transport="http", port=8000)  # Remote agents connect via HTTPmcp/legacy_server.py

Code Snippet 3: FastMCP 3.0 server exposes legacy data to any MCP-compatible agent (one server, many agents).

Which Pattern Should You Choose?

Pattern	Best When	Timeline
API Mediation	Stable APIs, 1-2 agents, tight control needed	4-8 weeks
Event-Driven	1,000+ transactions/hour, sub-second response	8-12 weeks
MCP	3+ agents, standardization priority	6-12 weeks

Table 1: Integration pattern selection guide

Figure 2: Three integration patterns for legacy systems

When to Consider Platform-Level Solutions

Enterprise platforms like Palantir Foundry take a different approach: containerizing legacy code itself through Foundry Container Engine (FCE). This “lift and shift” strategy runs your COBOL or Fortran logic in a modern environment (Palantir, 2023) without rewriting it.

The tradeoff is vendor commitment. Palantir engagements typically start at $1M+/year and require dedicated integration teams. For organizations with complex, multi-system landscapes and enterprise budgets, this can make sense.

The integration patterns in this post work differently. You build lightweight facades around legacy systems using open tools (FastAPI, Dapr, MCP). Implementation takes weeks, not quarters. You prove value before committing to platforms.

These approaches complement each other. Demonstrate ROI with integration patterns, then justify platform investments for deeper modernization. Deloitte calls this “data-first ERP modernization” (Deloitte, 2025).

Why Observability Infrastructure Is Non-Negotiable

Whatever integration pattern you choose, log everything. Every integration call. Every agent decision. Every error. This isn’t optional.

Why Log Everything?

Compliance: Auditors ask “why did your agent approve this transaction?” You need logs showing the decision path.
Debugging: When an agent fails, trace which integration call failed, what data it received, and why it made the wrong decision.
Improvement: You can’t optimize what you don’t measure.

Track three categories of metrics.

Integration Health Metrics

Monitor API latency (p50, p95, p99), error rates by type, and timeout frequency—metrics that align with OpenTelemetry GenAI conventions.

Agent Performance Metrics

These require deliberate instrumentation design. Task completion rate needs a definition of “complete” per task type (e.g., “ticket resolved without escalation”). User satisfaction comes from thumbs up/down on responses, escalation rate, and support ticket correlation. Decision accuracy is the hardest to measure, see below.

How Do You Measure Decision Accuracy?

Ground truth is often available. The question is where to find it.

For RAG systems, use categorized query benchmarks with validation subsets. Test accuracy by query type (conceptual, procedural, error lookup, multi-hop) since each fails differently.

For approval workflows, compare agent decisions against eventual outcomes. Was the approved invoice paid? Was the flagged transaction actually fraudulent? The business process itself provides ground truth.

When ground truth is unavailable, sample decisions for human or AI-assisted review. The question is: how many?

Sample 300-400 decisions monthly. This achieves +/-5% margin of error at 95% confidence regardless of total volume (Cochran, 1977). The math: n = (1.96² x 0.5 x 0.5) / 0.05² = 385. For systems under 500 decisions/month, review all or accept wider uncertainty.

Business Impact Metrics

Calculate these monthly against pre-deployment baselines. These are not real-time dashboard metrics:

Metric	Formula
Response time reduction	Agent-handled avg vs. pre-deployment baseline
Throughput increase	Tickets/hour after vs. before deployment
Cost savings	(Hours saved x labor cost) - (API costs + infrastructure)

Table 2: Business impact calculation formulas

These metrics prove ROI and guide your next investments. For a complete observability implementation guide, see closing the AI observability gap.

What ROI Can You Actually Expect?

The numbers are compelling. Let me walk through real examples.

Bank of America’s Erica reduced IT service desk calls by 50% (Bank of America, 2025) across 213,000 employees. Insurance companies using agentic AI reduced claims processing time from 9.6 days to 3.2 days (Superhuman, 2026), a 67% reduction.

Atera’s 60% improvement in sales response times translates to faster deal closure. Armis’s 73% increase in RFP response capacity means the same team handles 73% more business. Both captured these gains without hiring.

BCG reports AI can reduce core insurance modernization costs by 30-50% (BCG, 2026). Agents pay for themselves, then fund the upgrades.

Company	Pattern	Metric	Result
Atera	API Mediation	Sales Response Time	60% reduction
Armis	API Mediation	RFP Response Capacity	73% increase
Bank of America	API Mediation	IT Service Desk Calls	50% reduction
Insurance Industry	Event-Driven	Claims Processing Time	67% reduction (9.6 -> 3.2 days)

Table 3: ROI examples across integration patterns (note: API Mediation dominates early wins due to faster implementation)

Why Do 40% of AI Agent Projects Still Fail?

Projects fail when teams skip fundamentals.

Unclear Business Value. Gartner predicts over 40% of agentic AI projects will be canceled by 2027 due to escalating costs, unclear value, and inadequate risk controls. Launching with vague goals like “improve productivity” makes it impossible to measure success. Define exact metrics before development: “Reduce invoice processing time from 8 days to 2 days while maintaining 99.5% accuracy” is measurable; “handle invoices better” is not.

Security Vulnerabilities. Prompt injection attacks are ranked #1 in OWASP 2025 Top 10 for LLMs (OWASP, 2025). Treat agents as privileged service accounts with these controls:

Tool allowlisting (no arbitrary network/file access)
Schema validation on tool inputs/outputs
Output sanitization (no untrusted content forwarded)
Secrets isolation (no secrets in prompts; short-lived tokens)
Rate limiting + anomaly detection
Approval gates for high-impact actions
Audit logs (immutable, centralized)

A compromised agent can make thousands of requests per minute. For deeper coverage, see AI supply chain security risks.

Governance Retrofitting. Adding compliance controls after deployment requires painful redesigns. Plan audit trails, role-based access, and compliance testing from the start.

How to Start: A Practical Implementation Framework

Step 1: Set Specific Targets. Pick one workflow with high volume, predictable, with success metrics, and low regulatory risk. Good first candidates: customer support routing, RFP response compilation, IT service desk triage, or invoice processing.

Step 2: Audit Data Quality. Check for duplicates, format inconsistencies, missing values, and access permissions. Fix the top three issues. Aim for 80% clean data, not perfection.

Step 3: Choose Your Integration Pattern. API mediation for stable APIs and 1-2 agents (4-8 weeks). Event-driven for 1,000+ transactions/hour (8-12 weeks). MCP for 3+ agents or standardization priority (6-12 weeks).

Step 4: Build Observability from Day One. Track integration health (latency, error rates), agent performance (completion rate, accuracy), and business impact (your target KPI). Set alerts for error rates above 5%, latency p95 spikes, and approval override surges.

Step 5: Start with Single-Agent Workflows. Run your first agent in shadow mode for 2-4 weeks (longer for complex workflows). Compare agent decisions against human decisions. Exit criteria: Switch to production when accuracy exceeds 95% (adjust based on cost-of-failure analysis). Production gate: Error rate below 5% and manual override rate trending down. Expansion gate: KPI sustained for 4 weeks with no Sev-1 incidents.

Step 6: Fund Modernization with Agent ROI. Track which legacy systems create the most integration friction. If agents generate $200K annual savings, allocate 30-50% to infrastructure upgrades. This creates a self-funding cycle.

Why Should Your Board Fund This Now?

ROI before major infrastructure investment: your board sees results next quarter, not in three years.

Competitive advantage: while competitors wait for budgets, you’re winning deals with faster response times, higher throughput, and lower support costs. Risk mitigation: agents layer over existing systems with no business disruption.

Explore how subagent architectures can orchestrate multiple AI agents without coordination complexity.

References

AutoRFP, “RFP AI Agents: Revolutionizing How Companies Win More Deals in Less Time” (2026) — https://autorfp.ai/blog/rfp-ai-agents-revolutionizing-how-companies-win-more-deals-in-less-time
Bank of America, “A Decade of AI Innovation: Erica Surpasses Milestones” (2025) — https://newsroom.bankofamerica.com/content/newsroom/press-releases/2025/08/a-decade-of-ai-innovation—bofa-s-virtual-assistant-erica-surpas.html
BCG, “Agentic AI Power Core Insurance AI Modernization” (2026) — https://www.bcg.com/publications/2026/agentic-ai-power-core-insurance-ai-modernization
Cochran, W.G., Sampling Techniques, 3rd ed. (1977) — https://en.wikipedia.org/wiki/Sample_size_determination
Cyber Snowden, “Difference Between End of Life and Legacy Cyber Security” (2026) — https://cybersnowden.com/difference-between-end-of-life-and-legacy-cyber-security/
Deloitte, “Deloitte & Palantir: Driving Value in Enterprise Operations” (2025) — https://www.deloitte.com/us/en/Industries/government-public/perspectives/deloitte-palantir-collaboration.html
Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027” (2025) — https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
OpenTelemetry, “Semantic Conventions for Generative AI” (2024) — https://opentelemetry.io/docs/specs/semconv/gen-ai/
OWASP, “LLM01:2025 Prompt Injection” (2025) — https://genai.owasp.org/llmrisk/llm01-prompt-injection/
Palantir, “Safely Modernize Legacy Systems with Palantir Foundry Container Engine (FCE)” (2023) — https://blog.palantir.com/safely-modernize-legacy-systems-with-palantir-foundry-container-engine-fce-d8900464da7c
Superhuman, “AI Agent Useful Case Studies” (2026) — https://blog.superhuman.com/ai-agent-useful-case-studies/
World Economic Forum, “AI Mid-Market Business Growth” (2026) — https://www.weforum.org/stories/2026/01/ai-mid-market-business-growth/

AI Supply Chain Attacks: New Vectors in Your Dependencies

Hugues Clouâtre — Wed, 11 Feb 2026 20:45:00 GMT

A CI pipeline trusts 400 packages. Last week, one of them laid off 75% of its engineering team. Two years ago, another nearly shipped a backdoor to every major Linux distribution. Attackers are now registering package names that only exist because an AI hallucinated them.

Three incidents. Three attack vectors. One common thread: AI is reshaping software supply chain risk faster than most security programs can adapt.

Most organizations scan for CVEs and maintain an SBOM. Few monitor for maintainer burnout, AI-driven revenue collapse, or packages that only exist because an LLM invented them. The following framework closes those gaps.

Figure 1: Three AI-driven attack vectors targeting a software dependency chain.

Contents

What Are the Three Attack Vectors?
Why Does This Matter at Enterprise Scale?
How Does the Two-Tier Ecosystem Create Different Risks?
- Tier 1: Foundation-Backed Projects
- Tier 2: Indie and VC-Backed Projects
How Should Organizations Assess AI Exposure Risk?
How Does Vibe Coding Multiply Supply Chain Risk?
How Can Organizations Defend Against AI Supply Chain Attacks?
- Extend the Toolchain
- Sponsor Strategically
How Will Relicensing Reshape the Stack?
- Before the Next Sprint
References

What Are the Three Attack Vectors?

AI does not just accelerate existing supply chain risks. It creates new ones.

Vector 1: Maintainer Collapse (AI-Accelerated)

On January 6, 2026, Adam Wathan attributed Tailwind’s layoffs directly to AI’s brutal impact on their business. Documentation traffic dropped 40%. Revenue collapsed 80%. Yet Tailwind CSS downloads keep climbing.

The mechanism is simple: developers ask Copilot for a Tailwind grid layout. The AI generates it. No documentation visit. No discovery of Tailwind UI. No conversion. The developer gets value. The maintainer gets nothing.

This is not a failing product. It is a failing business model, and not unique to Tailwind. Any project monetizing through documentation traffic faces the same exposure.

Meanwhile, curl maintainer Daniel Stenberg scrapped the project’s bug bounty program on January 21, 2026, citing “intact mental health.” Twenty AI-generated vulnerability reports flooded HackerOne in January alone. None identified actual vulnerabilities. Researchers paste code into LLMs, submit the hallucinated analysis, then loop follow-up questions through the same models. The result: maintainers spend hours triaging garbage instead of shipping code.

Different mechanism, same outcome. Two forces compete: AI lowers the cost of writing software (beneficial for maintainers), but it also diverts users away from the direct engagement that funds maintainers (unsustainable for business models). Koren et al. (2026) model this formally and show the demand-diversion channel dominates. Stack Overflow activity has declined roughly 25% since ChatGPT’s launch, following the same pattern as Tailwind.

In March 2024, Microsoft engineer Andres Freund discovered a backdoor in XZ Utils days before it would have shipped to most Linux distributions. CVE-2024-3094 scored a perfect 10.0. The backdoor enabled remote code execution through SSH on affected systems.

The attack took two years. A contributor using the name “Jia Tan” gained the sole maintainer’s trust through legitimate contributions, then inserted malicious code. The maintainer was burned out, working alone, grateful for help.

This pattern repeats. The event-stream incident in 2018 followed the same playbook: abandoned maintainer transfers control, attacker inserts cryptocurrency-stealing code. XZ Utils proved the technique works at infrastructure scale.

Two years later, the technique has evolved. Attackers now use LLMs to maintain technically helpful, perfectly patient personas over months, bypassing the “vibe check” that once caught human bad actors.

Vector 3: Slopsquatting (AI-Native)

This attack vector did not exist before LLMs. Slopsquatting exploits models that confidently recommend nonexistent packages. One in five AI suggestions points to a package that was never published (Spracklen et al., 2025).

The attack flow:

Researchers run popular LLMs and collect hallucinated package names
Attackers register those names on npm, PyPI, or RubyGems with malicious payloads
Developers install AI-suggested packages without validation
Malicious code executes

Unlike typosquatting, attackers do not need to guess which names developers might mistype. The model identifies exactly which fake packages to register. Names like “aws-helper-sdk” and “fastapi-middleware” appear in AI-generated code but never existed until attackers registered them.

The defense is straightforward: verify that packages existed before the commit date. This check catches AI-hallucinated packages that attackers registered after the model suggested them:

import requests
from datetime import datetime

def check_package_age(name: str, commit_date: str) -> bool:
    resp = requests.get(f"https://registry.npmjs.org/{name}")
    if resp.status_code == 404:
        return False  # Package does not exist
    pkg = resp.json()
    published = datetime.fromisoformat(pkg['time']['created'].replace('Z', '+00:00'))
    return published < datetime.fromisoformat(commit_date)scripts/validate_deps.py

Code Snippet 1: Detect slopsquatting by checking package age. Conceptual implementation only.

Why Does This Matter at Enterprise Scale?

Sonatype’s 2024 State of the Software Supply Chain report documented 512,847 malicious packages discovered between November 2023 and November 2024. A 156% year-over-year increase. The Verizon 2025 DBIR found that 30% of breaches now involve third-party components, double the previous year.

Log4Shell proved how fast these risks materialize. According to Wiz and EY, 93% of enterprise cloud environments were affected. Four years later, 13% of Log4j downloads from Maven Central still contain the vulnerable version, roughly 40 million vulnerable downloads per year.

These vectors do not operate in isolation. They cascade. The same “software-begets-software” feedback loop that drove OSS growth now amplifies contraction: fewer maintainers produce fewer packages, which reduces ecosystem quality, which further weakens incentives to share. At 70% vibe coding adoption, engagement-based monetization drops roughly 70%, but OSS entry can only sustain an 11% decline before projects start disappearing. That 59-percentage-point gap is the crisis window.

Monitoring the right signals matters more than counting CVEs.

How Does the Two-Tier Ecosystem Create Different Risks?

Not all dependencies face equal exposure. A two-tier structure is emerging.

Tier 1: Foundation-Backed Projects

CNCF, Apache, and Linux Foundation projects operate differently. Kubernetes maintainers are typically employed by member companies. Corporate membership dues fund development. Governance structures distribute responsibility.

These projects face sustainability challenges: burnout, security maintenance burdens, contributor fatigue. Foundation backing buffers against documentation-traffic collapse, but not full immunity. Corporate membership dues often correlate with ecosystem health; if the broader ecosystem contracts, so does corporate willingness to fund.

Tier 2: Indie and VC-Backed Projects

Tailwind, Bun (pre-acquisition), curl, and thousands of smaller projects depend on sponsorships, consulting revenue, or VC runway. Many have single maintainers. The bus factor is often one.

These projects underpin the modern web. They are also most exposed to all three attack vectors.

Most enterprise stacks span both tiers. Kubernetes (Tier 1) might orchestrate containers running applications built with Tailwind (Tier 2). The risk profiles differ, and monitoring should reflect that.

How Should Organizations Assess AI Exposure Risk?

Traditional dependency scanning catches CVEs. It does not catch maintainer burnout, revenue collapse, or AI-hallucinated packages. Additional signals are needed.

The 5-Signal AI Exposure Audit

Funding model: Corporate-backed or sponsorship-dependent? Check GitHub Sponsors, Open Collective, or company backing. Sponsorship-dependent projects face higher AI exposure.
Contributor count: Bus factor greater than three? XZ Utils had one active contributor. Look at commit history and active contributors over the past 12 months.
Governance: Foundation membership or solo maintainer? CNCF and Apache projects have succession plans. Solo projects often do not.
AI exposure score: Docs-driven monetization (high exposure) or infrastructure utility (lower exposure)? UI libraries and developer tools face higher risk than compression utilities or parsers.
Recent signals: Layoffs, acquisition talks, burnout posts? Monitor project blogs, maintainer social media, and GitHub discussions.

Decision Thresholds

The following framework maps signals to response levels:

Risk Level	Signals	Action
Monitor	1-2 signals, Tier 1 project	Quarterly review
Watch	2-3 signals, any tier	Monthly review, identify alternatives
Mitigate	3+ signals, Tier 2 project	Sponsor, fork, or migrate
Critical	Active incidents (layoffs, security events)	Immediate review, contingency plan

Table 1: Decision thresholds for dependency risk response.

Enterprise Application

Consider a fintech platform with 400 npm dependencies. Traditional scanning surfaces CVEs. The AI exposure audit surfaces different risks:

Dependency Type	Count	High AI Exposure	Action
UI frameworks	12	4	Review monetization models
Build tools	8	2	Track bus factor, burnout signals
Infrastructure	45	1	Lower priority
Utility libraries	335	23	Automate monitoring

Table 2: AI exposure audit across dependency categories.

The 23 high-exposure utility libraries are not all equal. Prioritize by criticality: is it in the authentication path? The payment flow? The deployment pipeline?

How Does Vibe Coding Multiply Supply Chain Risk?

Andrej Karpathy coined “vibe coding” in February 2025: developers who “fully give in to the vibes” and let AI generate entire applications. The practice is accelerating, and it compounds every vector above.

Slopsquatting is the visible risk. The less visible one: license laundering. AI training data includes GPL and AGPL (copyleft) code. When developers accept AI-generated output without review, they risk shipping restrictive-licensed code as proprietary. The provenance is untraceable. Tools like FOSSA and Snyk License Compliance can scan for license violations in CI, but they catch dependencies, not AI-generated source code. Manual review remains essential.

jobs:
  license-scan:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v6
      - name: Scan licenses with FOSSA
        uses: fossas/fossa-action@v1
        with:
          api-key: ${{ secrets.FOSSA_API_KEY }}
          run-tests: true.github/workflows/license-check.yml

Code Snippet 4: FOSSA scans dependencies for GPL/AGPL violations and fails the build on policy breach.

AI-assisted commits warrant validation: flagging new dependencies, verifying packages existed before the commit date (Code Snippet 1), and checking maintainer history. For teams using AI agents extensively, subagent architectures can dedicate a validation agent to check every dependency before acceptance.

How Can Organizations Defend Against AI Supply Chain Attacks?

Extend the Toolchain

Existing tools catch CVEs. Tools that catch sustainability risks fill the gap:

OpenSSF Scorecard (scorecard.dev) scores projects on maintainer activity, security practices, and bus factor
deps.dev provides dependency graphs with contributor data
Socket.dev detects supply chain attacks including slopsquatting patterns

Scorecard and Socket integrate directly into AI-augmented CI/CD pipelines via GitHub Actions, flagging risky dependencies before merge.

For a quick CLI check:

# Check a project's health score (0-10)
scorecard --repo=tailwindlabs/tailwindcss

Code Snippet 2: OpenSSF Scorecard CLI checks a repository’s security health score (0-10).

To enforce this in CI, integrate Scorecard into a GitHub Actions workflow. The workflow below fails the build if a dependency scores below 7 out of 10, preventing risky dependencies from entering production:

- name: Run OpenSSF Scorecard
  uses: ossf/scorecard-action@v2
  with:
    results_file: scorecard.json

- name: Block on low score
  run: |
    jq -e '.score >= 7' scorecard.json || exit 1.github/workflows/scorecard.yml

Code Snippet 3: GitHub Actions workflow to enforce minimum dependency health score of 7/10 in CI.

Tooling catches symptoms. Sponsorship addresses the cause. If a Tier 2 project in a critical path shows stress signals, $500/month establishes a relationship with the maintainer and early warning on sustainability issues.

But individual sponsorship is a stopgap, not a solution. The systemic fix requires platform-level change. One proposed model: AI coding platforms already track which packages they import, so they could redistribute subscription revenue to maintainers based on attributed usage, a “Spotify for open source.” The infrastructure exists; the coordination does not. Until it does, direct sponsorship remains the best lever CTOs have.

The companies that sponsored Log4j before Log4Shell had maintainer relationships when the crisis hit. The ones that did not scrambled with everyone else.

How Will Relicensing Reshape the Stack?

The OSS ecosystem is adapting. Bun’s acquisition by Anthropic shows one path: AI companies absorbing critical infrastructure. More acquisitions are likely.

Experimental protocols like x402 attempt to let AI agents pay for resource access. Still early, but architecturally sound.

The two-tier ecosystem is crystallizing. Foundation-backed projects will weather it better. Indie projects will face consolidation pressure, through acquisition, abandonment, new business models, or relicensing. HashiCorp moved Terraform to BSL in 2023. Redis followed months later. Sentry, MariaDB, Elastic. The pattern is clear. When sponsorships fail and AI erodes documentation revenue, restrictive licenses become the survival strategy. For enterprises, this means dependencies assumed to be permissively licensed may not stay that way.

Before the Next Sprint

Run OpenSSF Scorecard on the top 10 dependencies
Check bus factor on Tier 2 critical-path projects
Add slopsquatting validation to CI (Code Snippet 1)

AI adoption is accelerating. So are the attacks exploiting it. The question is not whether a supply chain has risk, but whether teams that see it first have the advantage.

References

CrowdStrike, “CVE-2024-3094 and XZ Upstream Supply Chain Attack” (2024) - https://www.crowdstrike.com/en-us/blog/cve-2024-3094-xz-upstream-supply-chain-attack/
Goodin, Dan, “Overrun with AI slop, cURL scraps bug bounties” (Ars Technica, 2026) - https://arstechnica.com/security/2026/01/overrun-with-ai-slop-curl-scraps-bug-bounties-to-ensure-intact-mental-health/
Infosecurity Magazine, “Vibe Coding: A Hidden Security Risk of the AI Era” (2025) - https://www.infosecurity-magazine.com/opinions/vibe-coding-security-risk-ai/
Karpathy, Andrej, “vibe coding” (X, 2025) - https://x.com/karpathy/status/1886192184808149383
Koren, Miklos et al., “Vibe Coding Kills Open Source” (arXiv, 2026) - https://arxiv.org/abs/2601.15494
Snyk, “Slopsquatting: New AI Hallucination Threats” (2025) - https://snyk.io/articles/slopsquatting-mitigation-strategies/
Sonatype, “10th Annual State of the Software Supply Chain” (2024) - https://www.sonatype.com/state-of-the-software-supply-chain/introduction
Spracklen, Joseph et al., “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs” (arXiv, 2025) - https://arxiv.org/abs/2406.10279
Sumner, Jarred, “Bun is joining Anthropic” (2025) - https://bun.sh/blog/bun-joins-anthropic
Verizon, “2025 Data Breach Investigations Report” - https://www.verizon.com/business/resources/reports/dbir/
Wathan, Adam, “GitHub comment on Tailwind layoffs” (2026) - https://github.com/tailwindlabs/tailwindcss.com/pull/2388#issuecomment-3717222957

Orchestrating AI Agents: A Subagent Architecture for Code

Hugues Clouâtre — Tue, 24 Feb 2026 12:46:00 GMT

Single-agent AI coding hits a ceiling. Context windows fill up. Role confusion creeps in. Output quality degrades. The solution: multiple specialized models with structured handoffs. One model plans, another builds, a third validates. Each starts fresh. Each excels at its role.

Basic code assistants show roughly 10% productivity gains. But companies pairing AI with end-to-end process transformation report 25-30% improvements (Bain, 2025). The difference isn’t the model. It’s the architecture, specifically how you engineer the context each agent receives.

Anthropic’s research on multi-agent systems confirms what we observe: architecture matters more than model choice. Their finding that “token usage explains 80% of the variance” reflects the impact of isolation: focused context rather than accumulated conversation history.

This post documents a production workflow using Goose, an open-source AI assistant. The architecture separates planning, building, and validation into distinct phases, each with a different model optimized for the task.

Contents

Why Do Single-Agent AI Coding Workflows Hit a Ceiling?
How Does Subagent Architecture Solve Context Problems?
How Does Model Selection Affect Cost and Quality?
How Do Subagents Communicate?
Where Should Human Judgment Stay in AI Workflows?
What Results Does This Produce?
- Design Targets
When Does This Work (and When Doesn’t It)?
Takeaways
References

Why Do Single-Agent AI Coding Workflows Hit a Ceiling?

A single AI model handling an entire coding task accumulates context with every interaction. By implementation time, the model carries baggage from analysis, research, and planning phases. This stems from three core problems.

Context Rot

Long conversations consume token budgets. The model forgets early instructions or weighs recent context too heavily. Chroma Research calls this context rot: performance degrades consistently as input tokens increase, even on simple tasks, and worsens for multi-step reasoning like coding (Chroma Research, 2025). On-demand retrieval adds another failure mode: agents miss context 56% of the time because they don’t recognize when to fetch it (Gao, 2026).

Role Confusion

A model asked to analyze, plan, implement, and validate lacks clear boundaries. It starts implementing during planning. It skips validation steps. Outputs blur together.

Accumulated Errors

Mistakes in early phases propagate. A misunderstanding in analysis leads to a flawed plan. A flawed plan leads to incorrect implementation. Fixing requires starting over.

How Does Subagent Architecture Solve Context Problems?

The fix: spawn specialized subagents for each phase. An orchestrator handles high-level coordination and human interaction. Subagents handle execution with fresh context.

Figure 1: Core subagent workflow. Orchestrator handles RESEARCH (with human gate) and PLAN. Builder and Validator run as separate subagents with fresh context. SETUP and COMMIT/PR phases omitted for clarity.

The orchestrator (Claude Opus 4.5) handles RESEARCH and PLAN phases. RESEARCH requires human judgment at a single gate to decide the approach. After plan completion, it spawns a BUILD subagent (Claude Haiku 4.5) that receives only the plan, not accumulated history. The builder writes code, runs tests, then hands off to a CHECK subagent (Claude Sonnet 4.5) for validation.

Each subagent starts with clean context. The builder knows what to build, not how we decided to build it. The validator knows what was built, not what alternatives we considered. This is context engineering in practice: designing what each agent sees rather than letting context accumulate. This isolation directly counteracts context rot and drives the performance gains research attributes to architecture over model selection (Anthropic Engineering, 2025).

How Does Model Selection Affect Cost and Quality?

Different phases need different capabilities. Planning requires reasoning. Building requires speed and instruction-following. Validation requires balanced judgment.

Model	Role	Temperature	Rationale
Opus	Orchestrator	0.5	High reasoning for research and planning
Haiku	Builder	0.2	Fast, cheap, precise instruction-following
Sonnet	Validator	0.1	Balanced judgment, conservative (catches issues)

Table 1: Model selection by phase. Temperature decreases as tasks become more deterministic.

Cost Optimization

Building involves the most token-heavy work: reading files, writing code, running tests. Routing this volume to cheaper models cuts costs significantly.

Model	Input	Output	Role in Workflow
Opus	$5/MTok	$25/MTok	Planning (~20% of tokens)
Sonnet	$3/MTok	$15/MTok	Validation (~20% of tokens)
Haiku	$1/MTok	$5/MTok	Building (~60% of tokens)

Table 2: Anthropic API pricing, December 2025. Building consumes the most tokens at the lowest cost.

Research on multi-agent LLM systems shows up to 94% cost reduction through model cascading (Gandhi et al., 2025). This architecture targets 50-60% savings by routing building work to Haiku while preserving Opus for planning.

Beyond cost, fresh context enables tasks that fail with single agents. A 12-file refactor that exhausts a single model’s context window succeeds when each subagent starts clean.

Why Minimalist Instructions Matter

Smaller models like Haiku excel with focused, explicit prompts. Complex multi-step instructions cause drift. The recipe went through multiple iterations to find the right balance: enough context to execute correctly, minimal enough to avoid confusion. Each phase prompt fits in under 500 tokens. The builder receives a structured JSON plan, not prose. Constraints beat verbosity.

How Does Project Context Reach Subagents?

Recipes define the workflow, but subagents also need project context: build commands, conventions, file structure. That’s where AGENTS.md comes in, a portable markdown file that provides the baseline knowledge every subagent inherits. Goose, Claude Code, Cursor, Codex, and 40+ other tools read it natively. In Vercel’s evals (Gao, 2026), an AGENTS.md file achieved a 100% pass rate on build, lint, and test tasks where skills-based approaches maxed out at 79%.

Think of it as CSS for agents: global rules cascade into every project, project-specific rules override where needed. The orchestrator and every subagent it spawns inherit both layers without explicit prompting.

## Commits
- Conventional commits, GPG signed and DCO sign-off
- Feature branches only, PRs for everything
- Never merge without explicit user request

## Security
- Treat all repositories as public
- No secrets, API keys, credentials, or PII~/.config/goose/AGENTS.md

## Stack
Rust 2024 + Tokio + Clap (derive) + Octocrab

## Project-Specific Patterns
- Apache-2.0 license with SPDX headers
- cargo-deny for dependency auditsaptu/AGENTS.md

Code Snippet 1: Global and project-level AGENTS.md files. The builder subagent from Table 3 inherits both layers: it knows to GPG-sign commits (global) and use cargo-deny (project) without either appearing in the handoff JSON.

How Do Subagents Communicate?

Subagents communicate through JSON files in $WORKTREE/.handoff/. Each session uses an isolated git worktree, so handoff files are scoped to that execution context. This creates an explicit contract between phases.

$WORKTREE/.handoff/
├── 02-plan.json      # Orchestrator → Builder
├── 03-build.json     # Builder → Validator  
└── 04-validation.json # Validator → Builder (on failure)

Code Snippet 2: Handoff directory structure showing the JSON files that pass context between phases.

The plan file contains everything the builder needs:

{
  "overview": "Remove 4 dead render_with_context methods",
  "files": [
    {"path": "src/output/triage.rs", "action": "modify"},
    {"path": "src/output/history.rs", "action": "modify"},
    {"path": "src/output/bulk.rs", "action": "modify"},
    {"path": "src/output/create.rs", "action": "modify"}
  ],
  "steps": [
    "Remove render_with_context impl blocks from each file",
    "Remove #[allow(dead_code)] annotations",
    "Remove unused imports",
    "Run cargo fmt && cargo clippy && cargo test"
  ],
  "risks": ["None - confirmed dead code"]
}.handoff/02-plan.json

Code Snippet 3: Plan handoff file with structured task definition for the builder subagent.

The validator reads both 02-plan.json and 03-build.json to verify implementation matches requirements. It writes structured feedback to 04-validation.json:

{
  "verdict": "FAIL",
  "checks": [
    {"name": "Remove #[allow(dead_code)] annotations", "status": "FAIL",
     "notes": "Annotations still present in history.rs:145, bulk.rs:31, create.rs:63"}
  ],
  "issues": ["Plan required removing annotations, but these are still present"],
  "next_steps": "Fix issue: Remove the three annotations, then re-validate"
}.handoff/04-validation.json

Code Snippet 4: Validation handoff file with actionable feedback for the builder to address.

The builder reads this feedback, fixes the specific issues, and triggers another CHECK cycle until validation passes.

Why files instead of memory? Three reasons:

Auditable. Every decision is recorded. Debug failures by reading the handoff chain.
Resumable. Interrupt and resume without losing state. Start a new session with the same handoff files and no work is lost.
Debuggable. Failed validations include exact locations and actionable next steps.

Where Should Human Judgment Stay in AI Workflows?

Not every phase needs human approval. The workflow distinguishes between decisions (require judgment) and execution (follow the plan).

Phases with gates (human approval required):

RESEARCH: “Which of these approaches should we take?”
CHECK (conditional): On FAIL or PASS WITH NOTES, human decides whether to fix issues or proceed

Phases without gates (auto-proceed):

SETUP: Initialize context and gather requirements
PLAN: Design solution based on approved research direction
BUILD: Execute the approved plan
CHECK (on PASS): Validation passed, proceed to commit
COMMIT/PR: Push validated changes

This separation preserves governance without creating bottlenecks. Humans make strategic decisions. AI executes. If validation fails, the system loops back to BUILD with specific feedback. No human intervention for mechanical fixes.

What Results Does This Produce?

This architecture powers development across multiple projects. Three examples from aptu:

PR	Scope	Files Changed
#272	Consolidate 4 clients → 1 generic	9 files
#256	Add Groq + Cerebras providers	9 files
#244	Extract shared AiProvider trait	9 files

Table 3: Representative PRs using subagent architecture. All passed CI, all merged without rework.

The validation phase caught issues the builder missed. In PR #272, the CHECK subagent identified a missing trait bound that would have failed compilation. The builder fixed it on the retry loop. No human intervention required.

Design Targets

Research on multi-agent frameworks for code generation shows they consistently outperform single-model systems (Raghavan & Mallick, 2025). The architecture is designed to achieve:

Metric	Single Agent	Subagent Architecture
Context at build phase	~50K tokens	~5K tokens (fresh)
Rework loops	2-3 typical	0-1 expected
Human interventions	Throughout	Only at gates

Table 4: Design targets based on context isolation and structured handoffs.

When Does This Work (and When Doesn’t It)?

Works well for:

Multi-file refactors where context isolation prevents confusion
Feature additions following established patterns
Complex changes requiring distinct planning and execution
Teams wanting audit trails (handoff files document decisions)

Less effective for:

Simple one-file fixes (overhead exceeds benefit)
Legacy systems without clear patterns (builder lacks context)
Exploratory work where plans change during implementation

For teams integrating AI agents with legacy systems, see AI agents in legacy environments for integration patterns that work when your data lives in mainframes and AS400 systems.

Takeaways

Separate reasoning from execution. Use capable models for planning, fast models for building.
Fresh context beats accumulated context. Subagents start clean. They follow instructions without historical baggage.
Structured handoffs create audit trails. JSON files document what was planned, built, and validated.
Gates at decisions, not execution. Human judgment for strategy. Automated loops for implementation.

The full recipe is available as a GitHub Gist. It builds on patterns from AI-Assisted Development: From Implementation to Judgment.

References

Anthropic Engineering, “How we built our multi-agent research system” (2025) — https://www.anthropic.com/engineering/multi-agent-research-system
Bain & Company, “From Pilots to Payoff: Generative AI in Software Development” (2025) — https://www.bain.com/insights/from-pilots-to-payoff-generative-ai-in-software-development-technology-report-2025/
Chroma Research, “Context Rot: How Increasing Input Tokens Impacts LLM Performance” (2025) — https://research.trychroma.com/context-rot
Gao, Jude, “AGENTS.md outperforms skills in our agent evals” (2026) — https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals
Gandhi et al., “BudgetMLAgent: A Cost-Effective LLM Multi-Agent System” (2025) — https://arxiv.org/abs/2411.07464
Raghavan & Mallick, “MOSAIC: Multi-agent Orchestration for Task-Intelligent Scientific Coding” (2025) — https://arxiv.org/abs/2510.08804

AI-Augmented CI/CD - Shift Left Security Without the Risk

Hugues Clouâtre — Fri, 06 Feb 2026 09:01:00 GMT

Code reviews are a bottleneck. Engineering teams lose measurable velocity waiting for feedback. This delay compounds when security vulnerabilities escalate. Fixing a defect in production costs 30-100× more than fixing it during design (Boehm & Basili, 2001). The economics are clear: early detection reduces downstream costs exponentially.

AI in CI/CD augments human review by analyzing code patterns and tool outputs before human reviewers see the changes. Analysis completes in 2-7 seconds compared to 4-22 hour human review cycles.

Contents

What Is the Real Cost of Manual Code Review?
How Does AI Integration Work Without Prompt Injection Risk?
Tier 2 and Tier 3: Speed vs. Security Trade-offs
Evolution From Uncontrolled to Managed AI Analysis
What Outcomes Does AI-Augmented CI/CD Deliver?
Implementation Guide
References

What Is the Real Cost of Manual Code Review?

The Review Bottleneck

Development velocity correlates with code review latency. Code review bottlenecks are well-documented across engineering teams. Feedback loops stretch from hours to days while developers context-switch or wait on reviewers. Research from Forsgren et al. (2024) shows context-switching during code review significantly reduces developer productivity and satisfaction.

GitHub’s 2024 Octoverse reports median time from PR open to first review is 4 hours in large organizations, 22 hours in enterprises. AI summaries reduce this to under 3 minutes.

Traditional CI/CD pipelines run automated linters and security scanners, generate reports, then stop. A human reads the output, interprets it, decides if it matters, and either approves or comments. This handoff creates velocity bottlenecks. Eight-hour review windows delay production deployments. Critical insights get buried in noise. Studies confirm developers fear review delays will slow delivery, even though they recognize reviews’ long-term quality benefits (Santos et al., 2024). The cost of this wait scales with engineer compensation.

The Security Cost Multiplier

Security defects amplify this cost multiplier. IBM and Software Engineering Institute research confirms production fixes can be orders of magnitude more expensive than early detection. The exact multiplier depends on when the defect surfaces. The expenses compound: rework costs, deployment delays, and potential security incidents all increase exponentially downstream.

Shift-left automation detects issues before a PR merges, before human review begins. AI analyzes linter output, security scan results, and code patterns in seconds. Analysis time: 2-7 seconds compared to 4-8 hour human reviews. Developers receive immediate feedback, iterate faster, and ship with higher confidence.

The Prompt Injection Risk

Raw AI analysis of code diffs introduces a critical vulnerability: prompt injection. If a CI/CD pipeline feeds user-submitted code directly to an AI model, an attacker can craft a PR with embedded instructions that manipulate the AI’s behavior. The AI might approve malicious code, disable security checks, or expose sensitive information. This is not theoretical. It represents a live attack surface in every AI-augmented system.

Defensive architecture mitigates this risk. The AI analyzes tool output (structured, deterministic results from linters, security scanners, and static analysis) rather than untrusted input directly. The pipeline sequence: linter runs first, generates JSON, AI summarizes the findings, human approves. This reduces the attack surface to near zero.

Threat models vary by repository type. A private repository with a trusted five-person team tolerates different risk than open-source projects accepting external contributors. Three security tiers match different threat models while maintaining analysis speed.

How Does AI Integration Work Without Prompt Injection Risk?

Tier 1 eliminates prompt injection risk. The linter runs first, produces JSON output, and the AI analyzes only that structured data. The AI never sees the raw code, never processes user input, and never runs in the context of potentially malicious diffs.

name: AI Analysis - Maximum Security
on: [pull_request]

permissions:
  contents: read

jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v6
      
      - name: Lint Code
        run: pipx run ruff check --output-format=json . > lint.json || exit 0
      
      - name: Setup Goose
        uses: clouatre-labs/setup-goose-action@v1
      
      - name: AI Analysis
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
        run: |
          echo "Summarize these linting issues:" > prompt.txt
          cat lint.json >> prompt.txt
          # Only structured tool output appended. Never raw source code.
          goose run --instructions prompt.txt --no-session --quiet > analysis.md
      
      - name: Upload Analysis
        uses: actions/upload-artifact@v5
        with:
          name: ai-analysis
          path: analysis.mdtier1-maximum-security.yml

Code Snippet 1: In Tier 1, AI analyzes only JSON output from the linter, never raw code. Full example

The AI sees only JSON. No code, no comments, no user input. Attack surface: zero. This pattern applies to public repositories, open-source projects, and any system where external contributors submit PRs.

Figure 1: Tier 1 defensive pattern. AI analyzes tool output, never sees raw code. Immune to prompt injection.

Tier 2 and Tier 3: Speed vs. Security Trade-offs

Tier 2 provides additional context (file paths, change stats, commit metadata) without exposing raw code. This represents a middle ground: more insight than Tier 1, lower risk than Tier 3.

name: AI Analysis - Balanced Security
on: [pull_request]

permissions:
  contents: read

jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v6
      
      - name: Get Changed Files
        id: files
        run: |
          git diff --name-only origin/main...HEAD > files.txt
          wc -l files.txt >> summary.txt
      
      - name: Setup Goose
        uses: clouatre-labs/setup-goose-action@v1
      
      - name: AI Analysis
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
        run: |
          echo "Review these file changes:" > prompt.txt
          cat files.txt summary.txt >> prompt.txt
          # File names and stats. Not the actual code content.
          goose run --instructions prompt.txt --no-session --quiet > analysis.md
      
      - name: Upload Analysis
        uses: actions/upload-artifact@v5
        with:
          name: ai-analysis
          path: analysis.mdtier2-balanced-security.yml

Code Snippet 2: In Tier 2, AI sees file scope and metadata, but not code diffs. Full example

The AI sees file-level patterns but not line-by-line changes. Injection risk is low but non-zero: an attacker could craft filenames or commit messages to manipulate analysis. This tier applies to private repositories with trusted contributors.

Tier 3 applies to small, trusted teams where analysis speed outweighs defense-in-depth requirements. The AI sees full code diffs. Injection risk exists but is controlled through human approval gates.

name: AI Analysis - Advanced Patterns
on: [pull_request]

permissions:
  contents: read

jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v6
      
      - name: Get Full Diff
        run: git diff origin/main...HEAD > changes.diff
      
      - name: Setup Goose
        uses: clouatre-labs/setup-goose-action@v1
      
      - name: AI Analysis
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
        run: |
          echo "Deeply analyze these code changes:" > prompt.txt
          cat changes.diff >> prompt.txt
          # Complete code diffs for maximum context and detail
          goose run --instructions prompt.txt --no-session --quiet > analysis.md
      
      - name: Upload Analysis
        uses: actions/upload-artifact@v5
        with:
          name: ai-analysis
          path: analysis.mdtier3-advanced-patterns.yml

Code Snippet 3: In Tier 3, AI sees full diffs for subtle patterns. Full example

Each tier trades visibility for security. Tier 1 eliminates injection risk by sacrificing some context. Tier 2 accepts low risk for moderate context. Tier 3 prioritizes insight over security and is typically used sparingly.

Tier selection depends on three factors:

Repository access model (external contributors vs internal team)
Required AI context (tool output vs full diffs)
Risk tolerance (injection risk vs deeper analysis)

Figure 2: Three security tiers. Selection depends on threat model and team trust level.

Tier	Input	Injection Risk	Approval Gate	Typical Feedback Time	Recommended For
1	Tool output (JSON)	None	Human reviews artifact	2-5 min	Public repos, OSS, any external contributors
2	File stats + metadata	Low	Human pre-approval	1-3 min	Private repos, internal teams
3	Full code diff	Controlled	Optional	<60 sec	Tiny trusted teams only

Table 1: Tier comparison: speed, risk, and recommended use.

The decision framework is simple: start at Tier 1. Measure deployment velocity, security posture, and developer satisfaction. Only move to Tier 2 or 3 if team consensus is that the additional AI context outweighs the injection risk. Most teams never need to leave Tier 1.

Evolution From Uncontrolled to Managed AI Analysis

The naive approach feeds AI the code diff directly and allows it to comment on the PR. This is fast, appears intelligent, and creates an injection surface. The improved approach layers security tiers on top, providing a decision framework that matches the threat model.

Figure 3: Evolution from uncontrolled AI analysis to risk-managed tiers.

The shift is architectural, not just operational. The evolution moves from “AI sees everything and decides” to “AI sees what’s safe and humans decide what matters.” This distinction enables both security and speed improvements.

What Outcomes Does AI-Augmented CI/CD Deliver?

First-review latency drops from 4–22 hours (Octoverse 2024) to under 5 minutes, a 50–250× reduction. Developers iterate faster because they receive feedback immediately. CI/CD pipelines do not stall waiting for human review availability.

Quality improves because AI catches patterns humans miss during late-night reviews or context-switching. Linting issues get flagged automatically. Security tool outputs get analyzed for severity and context. Fewer critical issues reach production because they are caught earlier in the workflow.

For broader observability patterns in AI agent workflows, including legacy system integration, see AI agents in legacy systems.

Developer satisfaction increases when velocity and quality both improve. Engineers are not blocked by the review process. They receive comprehensive feedback without waiting. They trust the pipeline because it combines deterministic tools with AI insight and human judgment.

Business outcomes are measurable. Deployment frequency increases, mean time to resolution decreases, and security incidents reduce. Engineering teams ship faster without sacrificing safety.

Implementation Guide

Start with Tier 1. It provides maximum security with zero prompt injection risk. The example workflow demonstrates the complete pattern. For AWS-native environments, setup-kiro-action offers SIGV4 authentication without API keys in secrets.

Baseline measurement establishes the starting point: current review latency, deployment frequency, and security incident rate. A two-week measurement period provides sufficient data for comparison. After AI integration, the same metrics reveal impact.

Tier selection depends on threat model. External contributors and public repositories warrant Tier 1. Internal teams with trusted code may benefit from Tier 2 or Tier 3 context. The key is matching exposure level to trust level.

The human gate remains essential. AI generates artifacts for review, not merge approvals. Engineers validate recommendations before acting. This preserves accountability while accelerating feedback cycles.

For observability patterns in AI agent workflows, see AI Observability Gaps.

References

Boehm & Basili, “Software Defect Reduction Top 10 List” (2001) — https://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.78.pdf
Forsgren et al., “DevEx in Action: A study of its tangible impacts” (2024) — https://dl.acm.org/doi/10.1145/3639443
GitHub Octoverse 2024 — https://github.blog/news-insights/octoverse/octoverse-2024/
OWASP LLM Top 10 (2025 edition) – Prompt Injection #1 — https://owasp.org/www-project-top-10-for-large-language-model-applications/
Santos et al., “Modern code review in practice: A developer-centric study” (2024) — https://www.sciencedirect.com/science/article/pii/S0164121224003327

AI-Assisted Development: From Implementation to Judgment

Hugues Clouâtre — Thu, 05 Feb 2026 20:47:00 GMT

Software developers spend roughly equal time on meetings (12%) and coding (11%), with the remaining time distributed across debugging, architecture, reviews, and operational tasks. This fragmentation correlates with decreased productivity and satisfaction when it creates a gap between actual and ideal time allocation (Kumar et al., 2025).

Traditional workflows treat implementation as the bottleneck, forcing a costly trade-off: explore multiple solutions (expensive) or ship the first working approach (fast but suboptimal). Large-scale projects face 50+ key decisions where “it’s simply infeasible to thoroughly research every single decision” before implementation (Backlund, 2024). The real constraint isn’t typing speed. It’s decision quality under fragmented time.

AI-assisted development breaks this trade-off. Implementation becomes a review task. Expert judgment focuses on strategic decisions.

Contents

How Does AI-Assisted Development Shift Time Allocation?
- Real Example: CI Modernization
- Real Example: Matrix Operations Feature
What Business Value Does AI-Assisted Development Deliver?
How Do Recipes Codify Engineering Judgment?
When Does This Approach Work?
The Transformation
References

How Does AI-Assisted Development Shift Time Allocation?

AI assistants shift the focus from implementation to judgment. You spend minimal time reviewing code and most time applying strategic thinking.

Controlled studies show 55% faster task completion with AI assistance. The gain isn’t typing speed. It’s preserving cognitive capacity for critical thinking.

Critical thinking scales. Implementation doesn’t. You can evaluate 3 architectural approaches in the time it takes to implement one.

Example: Goose (open-source AI assistant) handles codebase analysis, implementation, testing, and documentation. You provide business context, architectural judgment, and approve decisions at critical gates.

Figure 1: Traditional approach focuses on implementation overhead, AI-assisted approach maximizes strategic thinking

Real Example: CI Modernization

Context: math-mcp-learning-server had no CI workflow. Legacy mypy was slow and unused.

The judgment call: Build CI from scratch or adapt patterns from a similar project?

AI identifies reusable patterns: Ruff (linter/formatter) + uv (package manager) + pytest-cov. I review the risk assessment, verify the tooling choices match project needs, and confirm zero regressions.

Results:

~20 minutes vs 3-4 hours from scratch
CI runtime: 5 seconds (Ruff is 10-100x faster than legacy tooling)
67 tests passing, 83% coverage

Source: PR #52 - Add modern CI workflow

Real Example: Matrix Operations Feature

Context: math-mcp-learning-server needed 5 matrix operation tools with NumPy integration.

The judgment call: Implement incrementally (one tool per PR) or batch with shared validation patterns?

AI identifies common infrastructure: dimension validation, ToolError handling, DoS prevention via size limits. I review API design, error handling conventions, and security limits.

Results:

2 minutes from PR creation to merge
5 tools, 21 tests, 395 lines added
Patterns reusable for future tools

Source: PR #109 - Implement 5 matrix operation tools

What Business Value Does AI-Assisted Development Deliver?

Decision Quality

The fundamental shift is from “implement first, evaluate later” to “evaluate first, implement once”. With AI handling implementation, you can explore 2-3 alternatives per decision instead of committing to the first working approach. Research confirms “the best solution must first become a candidate before being selected… More candidates increase the likelihood the ideal solution is among them.”

Time to validated options drops from 1.5-6 hours to 15 minutes to 1 hour. Architectural reversals decrease because upfront analysis improves.

Senior Engineer Leverage

Metric	Traditional	AI-Assisted	Business Impact
Time allocation	Implementation-heavy	Judgment-focused	Maximize expert leverage
Scope per engineer	1-2 specialties	Full stack	Eliminate specialist bottlenecks
Exploration cost	High (must implement)	Low (preview and abandon)	Ship best solution, not first

Table 1: Comparison of senior engineer time allocation and scope between traditional and AI-assisted approaches

This transformation aligns with research on AI-assisted development: GitHub studies found 60-75% of developers report increased job fulfillment and 87% preserve mental effort on repetitive tasks when using AI coding assistants.

Measured Time Savings

Task	AI-Assisted	Traditional	Savings
CI modernization	~20 minutes	3-4 hours	~90%
Matrix operations (5 tools)	2 minutes	1-2 hours estimated	~95%
DNS migration	2 hours	4-6 hours	~60%

Table 2: Measured time savings across infrastructure and DevOps tasks

Industry research validates these gains: 26% overall productivity increase across 4,867 developers, with 30-50% time savings on repetitive tasks in enterprise settings.

At 10 infrastructure tasks per month, this recovers ~60 hours per year per engineer. That is 1.5 weeks of productive time returned to strategic work.

Strategic Impact

Outcome	Shift	Business Value
Engineer capability	1-2 specialties → Full-stack	Eliminate specialist bottlenecks
Production risk	Manual review → AI + gates	Governance without slowdown
Knowledge retention	Tribal → Codified recipes	Team continuity
Onboarding time	Weeks → Hours	Faster scaling

Table 3: Strategic outcomes from traditional to AI-assisted workflows

How Do Recipes Codify Engineering Judgment?

Goose uses “recipes”: YAML workflow definitions that codify your judgment and process. The key innovation is mandatory STOP points where AI proposes and you approve before proceeding.

5-phase workflow:

ANALYZE - Understand codebase and problem
RESEARCH - Explore 2-3 solution approaches with trade-offs
PLAN - Detailed implementation plan
IMPLEMENT - Code, tests, documentation
PREPARE - Create PR, verify branch, push

Figure 2: Recipe workflow enforces governance through 5 mandatory approval gates - AI proposes, human judges

This matters for four reasons. First, repeatable process replaces ad-hoc prompting. Second, audit trails capture every decision in PR history. Third, human judgment gates ensure governance without blind automation. Fourth, codified expertise becomes an onboarding tool.

Example GATE pattern:

## Phase 1: RESEARCH

Understand scope and constraints:
- Read issue/PR description, linked discussions
- Identify affected files with `rg` and `analyze`
- Note CI requirements, test patterns, coding standards

### GATE: Research Summary  

**STOP - Present to user:**
- Problem statement (1-2 sentences)
- Affected files and scope
- Constraints discovered (CI, tests, dependencies)
- 2-3 possible approaches with trade-offs

**ASK:** "Which approach do you prefer?"~/.config/goose/recipes/goose-coder.yaml

Code Snippet 1: GATE pattern from production recipe. AI presents constrained options, human selects direction before any code is written.

Branch hygiene is enforced by global githooks: pre-push blocks protected branches, commit-msg requires conventional commits with DCO. The recipe ensures work starts on feature branches. Full recipe: goose-coder.yaml on GitHub Gist

# Conventional commit format
CONVENTIONAL_REGEX='^(feat|fix|docs|...)(\([a-z0-9_-]+\))?(!)?: .{1,100}$'

if ! echo "$COMMIT_MSG" | grep -qE "$CONVENTIONAL_REGEX"; then
    echo "BLOCKED: Commit message must follow conventional format"
    exit 1
fi

# DCO required
if ! grep -q "^Signed-off-by:" "$COMMIT_MSG_FILE"; then
    echo "BLOCKED: Missing DCO (Signed-off-by)"
    exit 1
fi~/.githooks/commit-msg

Code Snippet 2: Global commit-msg hook enforces conventional commits and DCO. Githooks provide hard blocks; recipes provide guidance.

When Does This Approach Work?

Task Type	AI-Assisted Fit	Evidence
CI/DevOps automation	High	20 min vs 3-4 hrs (PR #52)
Feature implementation	High	2 min for 5 tools (PR #109)
Boilerplate generation	High	Common pattern in both PRs
Greenfield architecture	Medium	More judgment gates needed

Table 4: Task type fit for AI-assisted development based on production experience

Low-fit tasks include security-sensitive code (AI may miss edge cases), regex/parsing logic (subtle bugs compound), and legacy systems without documentation (AI lacks context).

Critical success factor: You must have expertise to evaluate proposals. AI amplifies judgment, it does not replace it.

The Transformation

Traditional software economics fragments expert time across operational overhead. When senior engineers spend only 11% of their time coding, the bottleneck isn’t implementation speed. It’s context-switching between debugging, reviews, meetings, and operational tasks.

AI-assisted development consolidates this fragmentation. AI handles the operational work (debugging patterns, boilerplate, documentation), freeing human attention for architectural decisions and strategic judgment. The scarce resource shifts from “time to implement” to “ability to decide.”

When implementation becomes cheap (minutes instead of hours), exploration becomes affordable. You can evaluate three approaches, prototype two, and ship the best one, all in less time than the traditional single-path approach.

For technical leaders, this amplifies your most expensive resource: expert judgment. When your bottleneck is making the right decision, not finding time to code, AI becomes a strategic multiplier.

References

ACM, “Measuring GitHub Copilot’s Impact on Productivity” (2024) — https://cacm.acm.org/research/measuring-github-copilots-impact-on-productivity/
Backlund, Emil, “A Cost-Based Decision Framework for Software Engineers” (2024) — https://www.emilbacklund.com/p/a-cost-based-decision-framework-for
GitHub, “Research: Quantifying GitHub Copilot’s impact on developer productivity and happiness” (2024) — https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/
GitHub Resources, “Measuring the Impact of GitHub Copilot” (2024) — https://resources.github.com/learn/pathways/copilot/essentials/measuring-the-impact-of-github-copilot/
IEEE Computer Society, “Software Engineering Economics and Declining Budgets” (2024) — https://www.computer.org/resources/software-engineering-economics
Kumar et al., “Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven Era” (2025) — https://arxiv.org/abs/2502.15287
Pandey et al., “Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects” (2024) — https://arxiv.org/abs/2406.17910

Migrating to Cloudflare Pages: One Prompt, Zero Manual Work

Hugues Clouâtre — Thu, 29 Jan 2026 02:30:05 GMT

We migrated website infrastructure from Amazon Route53 + GitHub Pages to Cloudflare in 2 hours, during business hours. This included hosting, DNS, and CI/CD. Zero downtime. Zero manual commands.

The entire migration: One prompt. Then review and approve AI-proposed changes.

Why this matters for executives: DNS migrations traditionally require specialized DevOps knowledge, extended maintenance windows, and carry significant risk. A single misconfigured record can break email, take down services, or disrupt business operations for hours. This approach eliminates that risk through programmatic validation and automation.

The only manual step: Creating a Cloudflare API token.

Contents

The Starting Point
The Goal
Why This Matters for Your Business
The Starting Prompt
What Got Automated
The Only Manual Step
- BEFORE: Fragmented Infrastructure
- AFTER: Unified Platform
Results
What Business Impact Does AI-Assisted Migration Deliver?
Key Lessons
When Does This Approach Apply?
What Is the ROI of AI-Assisted Infrastructure Migration?
References

The Starting Point

Infrastructure:

Code repository: GitHub (unchanged after migration)
Hosting: GitHub Pages
DNS: Amazon Route53 (20+ DNS records)
Domain: Squarespace (registrar)
CI/CD: GitHub Actions → GitHub Pages

Critical services:

Email (5 MX records for Google Workspace)
Google Workspace services (Calendar, Contacts, Sites)
SSL validation records
Legacy service records (needed cleanup)

The Goal

Migrate everything to Cloudflare:

Faster DNS globally
Faster deployments
Simplified infrastructure (one platform)
Preview deployments for PRs
Zero downtime (email cannot break)

Why This Matters for Your Business

The Traditional Challenge

DNS migrations can be done with zero downtime, but they require extensive planning and careful execution. One misconfigured MX record means email down for hours, and human error causes 66-80% of outages (Uptime Institute, 2025). Imagine missing customer orders, support tickets, or sales inquiries during your peak season.

The Difference with AI Assistance

Same zero-downtime outcome, but with programmatic validation instead of manual checklists. Business hours execution becomes feasible because pre-validation eliminates guesswork. Teams without specialized DevOps expertise can execute complex migrations confidently.

Strategic Value

Infrastructure changes shift from high-stress, weekend events to business-hours execution with automated validation. Experienced engineers still evaluate proposals, but with dramatically reduced risk and time investment. Preview deployments enable stakeholder review before release.

The transformation: From possible-but-stressful to routine-and-confident.

The Starting Prompt

What we told Goose (open-source AI assistant powered by Claude Sonnet 4.5):

I want to migrate from GitHub Pages to Cloudflare Pages. The domain clouatre.ca is registered at Squarespace. I need zero downtime - email and Google Workspace cannot break. Check if DNSSEC is enabled and handle it appropriately. Use a risk-adverse approach.

This prompt started a 5-phase recipe workflow with mandatory approval gates: I reviewed and approved each phase (Analyze → Research → Plan → Implement → Prepare). Not autonomous execution, AI-assisted with human governance at every decision point.

We didn’t need to specify where DNS was hosted (discovered Route53 automatically), how many DNS records existed (found 20+), which records were critical vs obsolete, how to configure Cloudflare Pages, or how to set up GitHub Actions for Cloudflare. The AI handled discovery, analysis, and execution.

Claude Sonnet 4.5 (reasoning): Analyzes context, decides what to do, discovers infrastructure, validates approach
Goose (execution layer): Provides tool access (shell, git, AWS CLI, gh), manages conversation state, enforces approval gates
Human (governance): Reviews proposals at gates, approves/rejects, maintains control

Critical: We reviewed every decision. The AI proposed, we approved. The combination of automation + human judgment enabled confidence.

Figure 1: AI-assisted migration workflow with two human approval gates ensuring governance and confidence

What Got Automated

The migration workflow orchestrated five critical phases.

Discovery and Cleanup

Claude analyzed 20+ Route53 records and separated signal from noise: 15 critical records (email, Google Workspace, SSL validation) and 5 obsolete entries (old servers, expired validations). DNSSEC verification came back negative, confirming no migration blocker.

Pre-Migration Validation

Records were exported from Route53 and imported to Cloudflare via APIs, then tested against Cloudflare nameservers before switching. This included verifying email servers (MX priorities), SPF, DKIM, DMARC (exact TXT values), CNAMEs (Google Workspace), and comparing TTL values between source and target. The validation report confirmed 100% match.

# Verify records match before switching nameservers
dig @nameserver1.cloudflare.com clouatre.ca MX +short
# Output: 1 aspmx.l.google.com. (matches Route53)
diff <(aws route53 list-resource-record-sets) <(curl cloudflare-api)
# Output: (empty = 100% match, zero risk)scripts/validate-cloudflare-dns.sh

Code Snippet 1: Pre-validation against Cloudflare nameservers before switching (zero output from diff = zero risk)

CI/CD Reconfiguration

GitHub Actions were updated to deploy to Cloudflare Pages via wrangler (Cloudflare’s CLI), with base URL fixes (GitHub’s /repo/ path to root /) and a preview deployment workflow with 7-day auto-cleanup. Result: 38-second deploys, down from 5-8 minutes.

# Cloudflare Pages deployment (38-second deploys)
- name: Deploy to Cloudflare Pages
  uses: cloudflare/wrangler-action@v3
  with:
    apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
    accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
    command: pages deploy dist --project-name=clouatre-ca.github/workflows/deploy.yml

Code Snippet 2: GitHub Actions deployment to Cloudflare Pages (replaced GitHub Pages action for 88% faster deploys)

Governance Trail

The assistant created PRs with migration context, rationale, and rollback procedures. Every change was reviewable before production, creating an audit trail for compliance.

Preview Infrastructure

Every branch gets a preview URL automatically. Stakeholders can review before merge, and the system handles auto-cleanup with zero maintenance.

The Only Manual Step

Creating a Cloudflare API token (2 minutes):

Cloudflare dashboard → API Tokens
Create token with Pages permissions
Store in GitHub secrets

Everything else: automated.

BEFORE: Fragmented Infrastructure

AFTER: Unified Platform

Figure 2: Infrastructure transformation - from fragmented AWS/GitHub setup to unified Cloudflare platform

Results

Traditional manual DNS migrations typically require 4-6 hours of focused work and weekend execution windows to minimize business risk: planning, exporting records, importing, testing, monitoring propagation. The stakes are high. Downtime costs Global 2000 companies $400B annually (Splunk/Oxford Economics, 2024).

Metric	Before	After	Business Impact
DNS Resolution	20-30ms	10-15ms	50% faster global access
Deploy Time	5-8 min	38 sec	88% reduction - 10x faster iteration
Platform Cost	Route53: $12/year	Cloudflare: Free	Cost-neutral migration
Preview Deployments	None	Per PR	Catch issues before production
Migration Window	Weekend (risk mitigation)	2 hours, business hours	Eliminates deployment stress

Table 1: Before and after metrics - Complete migration (DNS + Hosting + CI/CD) completed in 2 hours, zero downtime, zero manual commands

What Business Impact Does AI-Assisted Migration Deliver?

What this approach enables:

Reduce specialized knowledge dependency - DevOps tasks no longer require memorizing cloud provider CLIs, DNS record formats, or deployment configurations
Lower operational risk - Programmatic validation means migrations happen with confidence, not guesswork
Faster iteration - Preview deployments enable stakeholder review before production release
Cost efficiency - Reduced deployment time by 88% (5-8min → 38sec), freeing developer time for feature work

Who benefits:

Small businesses without dedicated DevOps teams
Technical leaders managing infrastructure migrations
Teams wanting to reduce deployment anxiety and increase velocity

Key Lessons

1. The AI Stack Handles Implementation Details

You still need to understand what you’re migrating, but you don’t need to remember exact API syntax, AWS CLI flags for Route53 operations, Cloudflare API endpoints, DNS record format specifics, or YAML workflow syntax.

Claude discovered our infrastructure (Route53) and analyzed the records. Goose orchestrated the execution with tool access. We provided the goals and constraints, reviewed the approach, and approved changes.

Value: Reduces specialized knowledge requirement, eliminates manual typos, compresses migration timeline from 4-6 hours to 2 hours. McKinsey research shows developers complete tasks up to 2x faster with AI assistance (2023).

2. Pre-Validation Eliminates Risk

All DNS records were tested against Cloudflare’s nameservers before switching. This included email servers, Google Workspace records, and SSL validation. The process queried Cloudflare nameservers for each record type, verified all 5 MX records, verified TXT records (SPF, DKIM, DMARC), verified CNAME records (Google Workspace services), and generated a validation report.

Business outcome: We knew email, Google Workspace, and website would work before changing nameservers. Zero guessing.

3. Automate Record Migration

20+ DNS records, each with specific formats, priorities, TTLs. Manual copying guarantees typos. APIs provided accuracy: export from Route53 (AWS CLI), import to Cloudflare (API), and programmatic comparison to verify all matched.

Result: Zero typos. Zero manual record editing.

4. Preview Deployments Change Everything

Preview deployments reduce deployment anxiety, catch issues early, enable stakeholder review, and enable faster iteration. For technical leaders, preview deployments shift risk from production to staging, enabling confident releases.

5. The Paradigm Shift: From Careful Planning to Confident Execution

Traditional DNS migrations rely on weekend deployment windows (lower risk, higher stress), manual command execution (careful, but one typo equals disaster), sequential testing after switching (discover errors in production), specialized knowledge (dig syntax, DNS formats, cloud CLIs), and extensive planning with checklists (mitigates risk but time-intensive).

AI-assisted migrations enable business hours execution (confidence through pre-validation), programmatic execution (eliminates manual typos), pre-validated testing (know it works before switching), offloaded domain expertise (AI handles implementation syntax), and less planning overhead (validation happens automatically).

The transformation: From “plan exhaustively and execute carefully” to “validate programmatically and execute confidently.”

We knew every record worked before switching. No deployment anxiety, no weekend stress, no contingency planning. Just confidence through programmatic validation.

When Does This Approach Apply?

This approach works best for infrastructure migrations (DNS, hosting, CI/CD platforms) where teams lack specialized DevOps resources but need zero-downtime execution and audit trails. Requirements include AI assistant with CLI/API access (Goose or similar), API access to source and target platforms, clear migration constraints, and human review processes.

The trade-offs: reviewing AI decisions takes time, complex migrations may need human judgment on priorities, and initial setup requires configuring API tokens. Not suitable for instant-execution scenarios, environments prohibiting API access, or situations where teams lack domain knowledge to evaluate AI proposals.

What Is the ROI of AI-Assisted Infrastructure Migration?

Time savings compound quickly. Deployment speed improved 88% (5-8min to 38sec), saving ~7 minutes per deploy. At 5 deploys per day, that’s 35 minutes daily or 213 hours yearly of developer time recovered. Migration execution took 2 hours versus typical 2-3 day weekend projects.

Risk avoidance delivers additional value. Zero-downtime migrations eliminate revenue loss windows. Pre-validation prevents email outages (typical cost: hours of missed customer communications). Preview deployments catch production issues before customer impact.

Platform economics favor Cloudflare. The free tier (500 builds/month, unlimited bandwidth) serves most businesses. High-traffic sites may need paid plans ($20-$200/month), but deployment speed gains alone justify the cost through developer productivity.

The real ROI: Developer time back for feature work, not infrastructure babysitting.

References

BlueCat Networks, “What causes a DNS outage? Humans, mostly” (2024) — https://bluecatnetworks.com/blog/what-causes-a-dns-outage-humans-mostly/
Cloudflare, “Change your nameservers (Full setup)” — https://developers.cloudflare.com/dns/zone-setups/full-setup/setup/
DNSPerf, “DNS Performance Benchmarks” — https://www.dnsperf.com/
DORA, “Accelerate State of DevOps Report” (2024) — https://dora.dev/research/2024/dora-report/
McKinsey, “Unleashing Developer Productivity with Generative AI” (2023) — https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/unleashing-developer-productivity-with-generative-ai
Splunk and Oxford Economics, “Downtime Costs Global 2000 Companies $400B Annually” (2024) — https://www.splunk.com/en_us/newsroom/press-releases/2024/conf24-splunk-report-shows-downtime-costs-global-2000-companies-400-billion-annually.html
Uptime Institute, “Annual Outage Analysis Report 2025” (2025) — https://uptimeinstitute.com/about-ui/press-releases/uptime-announces-annual-outage-analysis-report-2025

Hugues Clouâtre - AI & Platform Engineering

SRE for AI Agents: Error Budgets, Trust Ladders, and 90 Trials

Table of contents

Why Is AI Widening the Dev/Ops Gap?

The Perception Gap: Dashboards vs. Practitioner Reality

How Did We Measure AI’s Scope Creep?

Why Medium-Tier PRs Underperformed

What Does SRE Mean in a Regulated Enterprise?

Error Budgets as Compliance Evidence

How Does SRE Act as AI’s Production Conscience?

Decision Provenance and Error Budget Separation

Blast Radius Containment and the Trust Ladder

Why Does Platform Maturity Gate AI Readiness?

The Learning Time Deficit

Where Should You Start?

The Four Actions in Order

References

What a Null Result Taught Us About AI Agent Evaluation

Table of contents

What Did the Paper Claim?

Why Our Agent Seemed Like a Good Candidate

Why This Matters for Engineering Teams

How Did We Design the Test?

What Happened in the FastMCP Refactor Test?

Did a Stricter Methodology Change the Result?

What Infrastructure Confound Did We Miss?

Why Did Both Experiments Hit 100%?

Where the Boundary Falls

What Did We Learn About AI Evaluation Design?

Rubric Design Is Harder Than Experiment Design

Infrastructure Behavior Is a Confounder

Delegate Authoring Has a Turn-Length Problem

When Should You Use Prompt Repetition?

What Transfers to Your Team

Did Prompt Repetition Change Anything Else?

References

Why Your AI Agent Failed in Production

Table of contents

Why Should Observability Be Foundational Infrastructure?

The Observer Effect Paradox

What Is Decision Provenance and Why Does Compliance Require It?

Why Every Framework Requires Decision Trails

Structured Logging with Correlation IDs

Tracking Tool Calls with GenAI Semantic Conventions

How Do Silent Integration Failures Kill AI Agents?

Why Do Token Costs Spiral Out of Control?

Alerting on Token Budgets

What Does a Vendor-Neutral Observability Stack Look Like?

How the Components Connect

Why Trace-Metric Correlation Matters

What Is the ROI and How Do You Start?

Are You Ready for Production?

References

RAG for Legacy Systems: 7,432 Pages to 3s Answers

Table of contents

Why RAG, Not Fine-Tuning?

How Does RAG Turn PDFs Into Answers?

The Ingestion Pipeline

Hybrid Retrieval: BM25 + Vector Search

Example: Error Lookup in 3.4 Seconds

Local Embeddings and Model-Agnostic Design

Does Reranking Work Across Different Models?

What Are the Real Performance Numbers?

What’s the ROI Without Modernization?

When Does RAG Fail?

Hallucination

Context Overflow

Stale Data

Corpus Limitations

What is the Overall Failure Rate?

How Do You Migrate from Prototype to Production?

What Should You Do Next?

References

AI Agents in Legacy Systems: ROI Without Modernization

Table of contents

Why Legacy Systems Became the #1 AI Adoption Obstacle

The Reverse Modernization Strategy: Layer AI First, Upgrade Later

Why This Works

When Reverse Modernization Doesn’t Apply

How Do AI Agents Actually Integrate with Legacy Systems?