The Self-Improving Agent — Build AI That Measures Its Own Performance and Gets Better

The most sophisticated workflow in the trilogy — an agent that evaluates its own output weekly, identifies what went wrong, tests improvements and auto-deploys better prompts based on evidence.

Most AI agent systems are static. The prompts that were good on launch day are still the same prompts a year later — even as the input distribution has changed, new edge cases have emerged, and better approaches have been discovered. The Self-Improving Agent fixes this by treating prompt optimisation as a continuous, automated process.

The Four-Phase Loop

Phase 1 — Measure: Every interaction is logged with input, output, latency, cost and any available user feedback signals (thumbs up/down, regeneration request, conversation abandonment). A weekly scheduled job samples 100 recent interactions and evaluates each using an LLM-as-judge prompt: rate this response on faithfulness, relevance and completeness (1-10 each). Log the scores to a database.

Phase 2 — Diagnose: A failure analysis agent clusters low-scoring interactions by root cause. Common categories: prompt ambiguity (the prompt does not specify the edge case clearly), context gap (the agent lacked information it needed), format mismatch (the output format was not what the downstream step expected), hallucination on uncertain topics. Each cluster gets a specific failure description and a hypothesis about what prompt change would fix it.

Phase 3 — Test:

For each high-priority failure cluster, generate a specific prompt improvement and implement A/B testing. Route 10% of production traffic to the improved prompt version for one week while the remaining 90% continues on the current prompt. Log quality scores for both versions independently. The test runs for a full week to capture representative traffic patterns including edge cases.

Phase 4 — Deploy:

At the end of the test week, a statistical evaluation compares quality scores between control and treatment groups. If the improvement is statistically significant (p below 0.05) and practically significant (at least 5% improvement in average quality score), the new prompt is automatically deployed to 100% of traffic. The old prompt is archived with its performance data for reference.

Over time, this system produces a compounding quality improvement. Prompts that were 75% quality at launch consistently reach 90%+ quality within six months — without any manual intervention in the optimisation process.