How SIA improves itself

a deep dive into the self-improvement loop

← Back to demo
The one-line version

An AI agent that rewrites its own code generation after generation — with no human edits and without changing the underlying model — measured on a real, ungameable task: turning English questions into SQL.

75.8% → 93.3%
▲ self-improvement, model held fixed

1 · The task

Given a plain-English question (“How many singers do we have?”) and a database, produce a SQL query that returns the right answer. This is text-to-SQL, benchmarked on Spider — 200+ real databases and thousands of questions from Yale.

We grade with execution accuracy: actually run the predicted SQL and the gold SQL against the database and check the result sets match. It’s exact, instant, and hard to fake — you can’t bluff it with plausible-looking SQL. It either returns the right rows or it doesn’t.

2 · The self-improvement loop

SIA (a real framework, hexo-ai/sia) runs three AI agents in a loop:

1
Meta-agentwrites the first version of the “target agent” (the program that answers questions).
2
Target agentanswers all 120 questions → produces SQL predictions.
3
Verifierruns every query read-only and scores execution accuracy.
4
Feedback-agentreads which questions failed and why, writes improvement.md, and rewrites the target agent for the next generation.

Repeat. The crucial constraint: the answering model never changes (Claude Haiku, every generation). The only thing that changes is the harness — the agent’s own scaffolding, prompt, and code. So any accuracy gain is provably the agent rewriting itself, not a bigger model.

3 · Why the score can’t be gamed

The gold answers live in data/private/ and are never given to the agent — it only sees the questions, the schemas, the databases, and a separate labeled training pool. So gains are real generalization, not memorizing an answer key. The verifier runs every query read-only with a timeout and rejects anything that isn’t a single SELECT/WITH. (Judges call this the “Goodhart” test — a verifier you can’t cheat.)

4 · What the agent discovered — on its own

By reading its own failures, the feedback-agent invented concrete, non-obvious fixes:

  • Noticed Spider always writes SELECT count(*), col (aggregate first) and added that rule.
  • Invented a helper tool to show exact string casing (the model kept writing Jetblue when the data said JetBlue).
  • Learned Spider’s quirk that “larger than any” means > MIN, not > MAX.
  • Added same-database few-shot examples from the training pool, and an execute-and-repair loop.

It even caught and fixed a regression it caused itself: gen 2 over-applied a rule, then gen 3 detected that from the new failures and corrected it. That self-correction — reasoning about the consequences of its own past edits — is the heart of the demo.

5 · The result

Across generations, execution accuracy climbed 75.8% → 79.2% → 88.3% → 93.3% — every point a self-discovered harness change, with the model held fixed the whole time.

See it live in the demo →