Your Agent Has the Memory of a Goldfish With a Fresh Context Window
You told your local agent on Monday that you moved from Neovim to Helix. On Tuesday it suggested a Neovim plugin. Every session starts blank, so the agent has no way to know what you said yesterday. The fix is long-term memory, and the internet’s answer is a vector database, an embedding model, a chunking strategy, and a weekend you won’t get back.
Skip all of that at first. For one person’s agent with hundreds to low thousands of memories, a vector DB is the wrong first tool. Start with plain Markdown files. Move to SQLite FTS5 when the index no longer fits your context budget. Add embeddings only when keyword recall measurably fails, and it fails on paraphrase, not on size. Buying a vector database for 300 notes is hiring a forklift to move a couch. It works, but the neighbors will have questions.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/llm/agent-memory-without-vector-db
The code below runs on the Python standard library. It talks to any OpenAI-compatible endpoint, so llama.cpp’s server and Ollama both work. Tested here with Python 3.14.7 and SQLite 3.53.4.
What Actually Belongs in Memory
Most memory systems die from hoarding, not from forgetting. Store things the agent cannot rediscover from the current conversation or the code in front of it.
- Facts about you: your editor, your NAS hostname convention, the fact that you run Debian on the servers and Arch on the laptop.
- Preferences: “short answers, no bullet lists,” “always show the rollback command.”
- Decisions with the reason attached: “Chose Postgres for the billing app because it needs row-level locking; SQLite was rejected.”
That last category is the valuable one. A decision without its “why” turns into a rule nobody can evaluate. Six months later the agent follows the rule after the reason has expired, and nobody can tell.
Do not store chat transcripts, things you can grep out of a repo, or anything you would be annoyed to see surface in a prompt. Memory is a prompt injection point you wrote yourself. Treat it like one.
The Contract: Three Tools and One Key
Both backends expose the same three operations, so the agent never knows which one is underneath.
remember(key, description, body)saves or updates one fact.recall(query)returns the best matches.forget(key)deletes one.
The key is the part that matters. Reusing a key overwrites the old memory. That is your update path, and it is the whole defense against duplicates. Without it the agent happily writes “user uses Neovim” and “user uses Helix” as two memories and then confidently cites the wrong one.
In the OpenAI-compatible request, each tool is a JSON schema in the tools array. This is the shape from agent.py:
{"type": "function", "function": { "name": "recall", "description": "Search saved memories by keywords.", "parameters": {"type": "object", "properties": { "query": {"type": "string"}}, "required": ["query"]}}},The model answers with tool_calls, each holding an id and a function with a name and a JSON string of arguments. You run the tool and send the result back as a message with role: "tool" and the matching tool_call_id.
Two server notes, because this is where local setups stall at 2 AM. llama.cpp’s server only does tool calls when you start it with --jinja, and the model’s chat template has to support tools. Ollama needs a model that lists tool support in its library. Current families such as Gemma 4 and Qwen 3.6 are listed as tool-capable in community guides, but check the badge on the exact tag you pull. A model without tool support does not error out. It writes you a polite paragraph about what it would have saved.
Option 1: One Markdown File Per Memory
This is the approach Claude Code uses for its own memory files: one small file per fact, plus an index file that always loads into context. The store is a directory.
agent_memory/ MEMORY.md # the index, always in the system prompt editor.md db-choice.md nas-backup-schedule.mdEach memory file is a one-line description, a blank line, and the body. The index has one line per memory. memory_files.py rebuilds it from the files, so it cannot drift:
def _rebuild_index(self) -> None: # One line per memory: "- [key](key.md): description". Rebuilt from the files. lines = [] for p in sorted(self.root.glob("*.md")): if p.name == "MEMORY.md": continue desc = p.read_text().split("\n", 1)[0].removeprefix("description: ") lines.append(f"- [{p.stem}]({p.name}): {desc}") self.index_path.write_text("\n".join(lines) + ("\n" if lines else ""))The read path is the clever part: you paste MEMORY.md into the system prompt on every request. The model sees every description, picks the ones that matter, and calls recall to pull the full body. There is no search engine in the loop for the common case. The model is the search engine, and it reads descriptions the way you scan a table of contents.
Things this gets you for free: you can open the files in any editor, git diff shows exactly what the agent learned today, and fixing a wrong memory is rm. When the agent says something odd at 2 AM, you can read its entire brain with cat.
Overwriting by key is one line of logic:
existed = (self.root / f"{key}.md").exists()(self.root / f"{key}.md").write_text(f"description: {description}\n\n{body}\n")self._rebuild_index()return f"{'updated' if existed else 'saved'} {key}"The recall tool in this backend is deliberately dumb: it counts how often the query words appear in each file and returns the top hits. It is a fallback for when the model wants a body it did not pick from the index.
The Context Budget Math
The index is the cost. Every line rides along on every request, so count it. English text runs roughly 4 characters per token, so treat these as estimates and measure with your model’s tokenizer before you commit to a number.
A good index line is a key plus a description of about 100 characters, call it 25 to 30 tokens. Then:
| Memories | Index tokens (estimate) | Fits in an 8K context? |
|---|---|---|
| 50 | ~1,400 | Yes |
| 200 | ~5,500 | Barely, and nothing else fits |
| 500 | ~14,000 | No |
Local models make this worse than hosted ones. You are probably running 8K to 32K of context to keep the KV cache inside your VRAM, and the index competes with the conversation, the tool schemas, and the answer. A reasonable rule: keep the index under 10 percent of the window. On a 16K window, that is about 60 memories. Past that, the always-loaded index stops being a feature.
Option 2: SQLite FTS5 and BM25
When the index no longer fits, stop loading it. Put memories in SQLite, give the agent a recall tool, and let the database do the finding. FTS5 ships inside Python’s sqlite3 module in current builds, so there is nothing to install.
self.db.execute( "CREATE VIRTUAL TABLE IF NOT EXISTS mem USING fts5(" "key UNINDEXED, description, body, tokenize='porter unicode61')")Two choices in that line.
UNINDEXEDonkeystores the column without indexing it. A search for “editor” will not match a memory merely because its key iseditor.tokenize='porter unicode61'stacks the Porter stemmer on top of the defaultunicode61tokenizer. A search for “backups” then finds “backup”. The other built-in tokenizers areasciiandtrigram. Trigram does substring matching and makes the index larger. For English facts and preferences, porter over unicode61 is the right default.
Upsert by key
FTS5 virtual tables cannot carry UNIQUE or PRIMARY KEY constraints, so there is no ON CONFLICT. Delete, then insert:
def remember(self, key: str, description: str, body: str) -> str: # FTS5 tables have no UNIQUE constraint, so upsert is delete + insert. cur = self.db.execute("DELETE FROM mem WHERE key = ?", (key,)) self.db.execute( "INSERT INTO mem(key, description, body) VALUES (?, ?, ?)", (key, description, body), ) self.db.commit() return "updated " + key if cur.rowcount else "saved " + keyThe delete scans the unindexed key column, which is fine at a few thousand rows.
The query escaping gotcha
FTS5 has its own query language. Hand it raw user text and it will throw. A hyphen reads as a column filter or a NOT, a stray double quote opens an unterminated string, and AND, OR, NOT, and NEAR are operators. A model that searches for “peer-to-peer” crashes your agent with sqlite3.OperationalError, and the tool result the model sees is a stack trace.
The SQLite docs say barewords may contain only letters, digits, underscore, and non-ASCII characters. Anything else must be quoted, and a double quote inside a string is escaped by doubling it. The simplest safe approach is to throw away punctuation and quote every word:
def fts_query(text: str) -> str: words = re.findall(r"\w+", text) return " OR ".join(f'"{w}"' for w in words)Because \w+ keeps only word characters, no quote can survive into the output, so there is nothing to escape. Quoting also defuses AND and NOT typed as plain words. peer-to-peer becomes "peer" OR "to" OR "peer".
Why OR and not the implicit AND? A model writes queries like “which editor does the user like.” With AND, every word must appear in one memory, and you get nothing back. With OR, any match qualifies and BM25 sorts out which memory matches best. test_memory.py feeds ten hostile strings through recall (say "hello, NEAR(x, col:val, a lone () and asserts none of them raise. It also asserts that the raw, unescaped peer-to-peer does raise, so you can see the trap fire.
Ranking: lower is better
FTS5’s bm25() returns negative numbers. The SQLite docs explain that the value is multiplied by -1 so that better matches get numerically lower scores, which makes a plain ORDER BY do the right thing. Newcomers sort descending and get the worst matches first.
rows = self.db.execute( "SELECT key, description, body, bm25(mem, 0.0, 3.0, 1.0) AS score " "FROM mem WHERE mem MATCH ? ORDER BY score LIMIT ?", (q, limit),).fetchall()The extra arguments to bm25() are per-column weights in table order: key (unindexed, so 0.0), description at 3.0, body at 1.0. A hit in the one-line summary counts three times as much as a hit buried in the body. That single knob usually fixes “why did it return the long rambling memory first.”
Stale Memory Is the Real Failure
Retrieval gets all the attention, but the bug you will actually hit is a confident wrong answer from a memory that was true in March. Three habits keep it in check.
- Update by key, never append. The agent should call
recallbeforeremember, and reuse the key if one exists. Say so in the system prompt. The loop inagent.pydoes. - Store dates and reasons. “Using Neovim (as of Oct 2026)” lets the model, and you, judge staleness. A decision with a “why” tells you when it expires.
- Give the agent
forget. When you say “that’s wrong now,” the right outcome is a deleted file or row, not a polite acknowledgment that evaporates at the end of the session.
Also review the memory store like you review a config. With files that is git log -p. With SQLite, a nightly sqlite3 memory.db .dump > memory.sql into a git repo gives you the same diff.
Where Keyword Search Falls Over
BM25 matches words, not meaning. Store “Drives a Subaru” and search “automobile”, and you get nothing. No shared token, no hit. test_memory.py asserts exactly this miss. Porter stemming handles “backup” versus “backups,” but it does not know that “car” and “automobile” are related, or that “my home VPN” and “WireGuard” are about the same thing.
In practice this hurts less than the vector-DB vendors suggest, for one reason: the writer and the reader are the same model. The agent that wrote “WireGuard” tends to search for words it would use itself. You can push this further by telling remember to include likely synonyms in the description, which costs a few tokens at write time and saves a lot of misses at read time.
You still hit the wall sometimes, so measure instead of guessing. Write 20 questions you expect the agent to answer from memory, phrased differently from how the memory was saved. Run them through recall. If 18 of 20 land in the top three, you are done. If paraphrase misses are common, that is a real reason to add embeddings.
When to Finally Add Vectors
Add vector search when all three of these are true:
- Your paraphrase test shows keyword recall failing on questions you care about.
- Synonym-stuffing descriptions did not fix it.
- You have enough memories that scanning them by hand stopped being an option.
Even then, you probably do not need a new server. sqlite-vec is a SQLite extension that stores vectors in the same database file as your FTS5 table, so you can run both queries and merge the results. If your stack already runs Postgres, our pgvector with local embeddings post covers that path, and RAG beyond vector search covers why hybrid retrieval beats either method alone. Both routes need an embedding model running next to your chat model, which means more VRAM, more latency on every write, and a re-embed job when you swap models.
Plan for that cost before you pay it. A dedicated vector DB service is the last stop, not the first.
The Loop That Ties It Together
agent.py is about 80 lines. The shape is: build the system prompt, call /chat/completions with the tools, run any tool_calls, feed the results back, stop when the model answers in plain text or after six rounds.
# The files backend keeps its index in the prompt. FTS keeps it behind recall.if a.backend == "files": system += "\n\nMemory index:\n" + (mem.index() or "(empty)")Run it against either backend with a flag:
export OPENAI_BASE_URL=http://localhost:8080/v1export MODEL=your-model-namepython3 agent.py --backend files "I switched my editor to Helix"python3 agent.py --backend fts "Which editor do I use?"The six-round cap exists because small local models sometimes loop on a tool call. A hard stop beats a runaway process holding your GPU.
I ran the loop against a stub server that returns one scripted tool call, which checks the plumbing for both backends. I did not run it against a live model for this post, so the model-specific part is yours to test: does your pick format tool arguments cleanly? Smaller quantized models are the usual culprits for malformed arguments JSON.
Which One to Pick
Start with files. Move to FTS5 when the index outgrows its budget, and keep the same three tools so the agent never notices. Add sqlite-vec or pgvector only after a paraphrase test fails. Your 2 AM self will thank you when the whole memory is something you can read with cat.
Common Questions
Do I need a vector database for LLM agent memory?
No. For one user with hundreds to low thousands of memories, Markdown files or SQLite FTS5 recall well enough. A vector database pays off only when keyword search measurably fails on paraphrased questions. Test that with 20 real queries before you add an embedding model and a second service.
How many memories fit in a local model’s context window?
About 60 index lines fit in a 16K window if you hold the index to 10 percent of context. At roughly 25 to 30 tokens per line, that is 1,500 to 1,800 tokens. Past that, stop loading the index and search with a recall tool instead.
Why does my FTS5 query throw a syntax error on a hyphen?
FTS5 parses queries with its own syntax, and a hyphen is not a legal bareword character. Strip punctuation and wrap every word in double quotes before running MATCH. The fts_query helper does this with re.findall(r"\w+", text), so quotes and operators never reach the parser.
Does SQLite FTS5 rank results with the highest score first?
No, the best match has the lowest score. FTS5’s bm25() returns negative numbers, and the SQLite docs say better matches are numerically smaller. Use ORDER BY bm25(...) ascending. Sorting descending returns your worst matches first.
Can llama.cpp or Ollama call tools for a memory agent?
Yes, with conditions. llama.cpp’s server needs the --jinja flag and a model whose chat template supports tools. Ollama needs a model that lists tool support in its library. Without those, the model answers in text and never calls remember or recall.
