case study
RECALL-Agent
A coding agent that learns reusable lessons from its own runs
Most coding agents start from zero on every task. RECALL-Agent keeps short, typed lessons from earlier runs and checks every answer by actually executing it, so I could measure whether memory helps instead of taking the model's word for it.
- Type
- Research project
- Stack
- Python, LangGraph, ChromaDB, SQLite, FastAPI, Streamlit
- Models
- OpenAI, Anthropic, and a deterministic offline provider
The problem
Two things bothered me about "self-improving" agent demos. First, the same model often solves the task and grades it, so any reported gain can't be checked. Second, memory is usually a dump of old conversations, which fills the context without saying what to do differently next time.
I wanted an agent where success comes from running code, and where memory stores compact lessons with a measurable usefulness score.
How it works
A Decision Layer supervises a Planner, Worker and Critic. The Worker's solution is run against the task's check() function in an isolated, time-limited Python subprocess, and that pass or fail result overrides an over-optimistic critic.
After notable runs (a failure, low confidence, a repeated mistake or something new) a Reflection step writes exactly one lesson: its scope, a trigger condition, a recommended action, an anti-pattern to avoid and a confidence score. Lessons are stored in SQLite and ChromaDB, retrieved by scope, semantic relevance and learned utility, merged when they are near-duplicates, and pruned when they stop helping.
Key decisions
- Execution decides success. A model grading itself can't be falsified; running check() can.
- Lessons, not transcripts. Each lesson is a typed record with a trigger and an anti-pattern, so retrieval returns advice rather than old chat.
- Utility is learned. Each lesson's score is updated with a relevance-weighted moving average of the reward from runs that used it, and low-utility lessons are pruned.
- A real ablation. With memory off there is no retrieval, no lesson writing and no utility update, so the comparison is clean.
How I evaluated it
The benchmark has nine coding task families, each with an executable check. I ran it with memory on and off. In the offline setup the deterministic provider can only pass a task by using a stored lesson, which isolates the memory loop. I also ran a smaller live experiment with an OpenAI model.
| Condition | Overall | Round 1 | Round 2 | Round 3 | Round 4 | Avg. retries |
|---|---|---|---|---|---|---|
| Memory off | 0% | 0% | 0% | 0% | 0% | 2.0 |
| Memory on | 75% | 0% | 100% | 100% | 100% | 0.5 |
| Condition | Overall success | Spread across seeds |
|---|---|---|
| Memory off | 48% | ± 4 points |
| Memory on | 54% | ± 9 points |
What the results do and don't show
The offline result shows that the memory loop works end to end: every task family failed on first sight, produced a lesson, and was solved in every later round, with retries cut from 2.0 to 0.5. It is not a claim about real models, because the offline baseline is 0% by construction.
The live run is the honest external check. Success went from 48% to 54%, which is a small gain, and with two seeds I wouldn't call it significant. Showing a real effect on a real model needs more seeds and harder, unseen task families.
Limitations and next steps
- Only tasks that ship a check() get a hard signal; open-ended tasks fall back to the critic's score.
- Offline runs use a hashing embedding, so semantic retrieval is conservative and relies on tag-scoping.
- Lessons are family-specific. Measuring transfer across task families is the next experiment.
- Planned: a held-out generalisation suite, counterfactual credit assignment (which lesson caused a win) and a Qdrant backend for scale.
- The verifier strips credentials and enforces a timeout, but it is not a security boundary. Untrusted code should run in a locked-down container.