Vector RAG, graph RAG, a database, and what agent memory still needs

People ask why Willder is not "just" vector RAG, or graph RAG, or a Postgres table. The short answer is that it is all three. Willder runs vector search in Postgres with pgvector, keeps a knowledge graph in Neo4j, and stores everything else in Postgres.

So this is not a post about why those tools are wrong. Each is good at something real. It is about three questions none of them answers on its own, and that I had to answer once more than one agent shared the memory.

I.What vector RAG is good at

You split text into chunks, embed each one, and at question time you return the chunks whose embeddings sit closest to the question's, usually by cosine similarity between the question's vector $q$ and a chunk's vector $d$ :

\cos(q, d) = \frac{q \cdot d}{\lVert q \rVert \, \lVert d \rVert}

It finds text that means roughly the same thing even when the words differ, it handles long documents well, and it is cheap and well understood.

Willder's vector store is a MemoryChunk table in Postgres with pgvector [1]: 384-dimension embeddings from BGE-small-en-v1.5 [2], behind an IVFFlat cosine index. Long-form content goes there.

What a vector index does not have is any idea who is asking. Nothing in that formula names the caller: it returns the nearest chunks. It does not know which of them the caller may read, whether a chunk is the current version of what it says, or who wrote it.

II.What graph RAG is good at

A knowledge graph stores entities and the relationships between them, so a question like "which services depend on the session store, and who decided that?" can follow edges instead of hoping the right chunks look similar.

Willder's graph is Graphiti on Neo4j. Zep describes Graphiti as its open source framework for temporal knowledge graphs [3] [4]. Willder's design notes record that it extracts clean subject, verb, object facts from short content and does worse on long documents [5]. That is the reason for having both stores. Retrieval is hybrid: every read asks all the stores its account tier allows, the vector store (its embedding search and a text match) and the graph, in parallel, and fuses the results into one answer [6]. It does not pick stores by the kind of question; the same stores are asked for every query [7].

A graph has the same gap as a vector index. Its edges say how facts connect, not who may see them.

III.What a database is good at

Transactions, constraints, indexes, and a permissions system you already trust. If you are going to put access control anywhere, a database is the obvious place, and that is where Willder puts it.

The catch is that the database gives you the parts, not the rules. You still write the scope model, the token model, the check every read goes through, the audit trail, the version history and the refusal when a write is based on stale data. That layer is most of Willder.

IV.What agent memory still needs

IV.1Permissions before the search, not after

The tempting design is to search everything, then drop the results the caller may not see. It has two problems. Your top ten becomes your top three after filtering, so good in-scope answers lose their place to results that were never going to be shown. And every piece of the pipeline between the search and the filter has handled data it should never have touched.

In Willder, the access decision comes first. Scopes form a tree stored as Postgres ltree paths [8], and a grant covers a target when the granted path is an ancestor of it, or the target itself:

SQL

granted_path @> target_path

Every memory read goes through check() before any store is queried, and the search then runs only against the partitions that check allowed [9]. In the graph, each scope is its own partition, keyed by its path. Figure 1 traces one read. The same rule holds for smaller things. The "since your last session" count an agent sees at the start of a session never counts a change outside its scope, so a number cannot leak what the agent may not read [10].

Figure 1. One memory read. The access decision comes before any search, and the search runs only where it allowed.

IV.2The current version, not just a similar one

Similarity search has no idea that a fact was replaced. If an agent wrote "we store sessions in Redis" in March and another wrote "we moved sessions to Postgres" in June, both chunks are close to "where do we store sessions?", and the old one may rank higher.

Willder keeps every version of a fact. When one is replaced, the old version stops being served, but it stays readable by id [11]. If a search ranks a replaced version, the read serves the version that replaced it in its place [12]. Each answer also says who wrote it, when, and which version it is:

Text

• we store sessions in Postgres, not Redis, so they survive a restart. (as of <when it was stored>) — from: <agent name> (agent) · version <version id>

That last part matters more than it looks. An agent that knows which version it read can say so when it writes back, and the server can refuse the write if someone changed the fact in between. That is the subject of another post.

IV.3A deletion that reaches every copy

In a RAG setup, the same fact lives in several places: the source document, its chunks, the embeddings, maybe a graph node, maybe a summary that quoted it. Deleting the source rarely deletes the rest.

In Willder, deleting a fact in the app removes it from search, from the knowledge graph, and from every agent's task-start brief, and each of those has its own test. On 29 October 2026 a production measure deleted a fact and watched all seven read paths an agent's token can use, each read by a different agent. All seven agent read paths stopped presenting it within the 30-second limit, the slowest after 2.3 seconds (Figure 2).

Horizontal bar chart of how many milliseconds after a delete each of the seven agent read paths stopped presenting the deleted fact: memory_query 0, memory_entity 557, memory_version 900, memory_changes 1,049, change notice 1,274, GET /api/changes 1,870, task-start brief 2,311. — Figure 2. Time from a delete until each of the seven agent read paths stopped presenting the deleted fact, on production on 29 October 2026, one agent per path; the signed-in web routes were not measured. The limit was 30 seconds; the slowest path took 2.3.

V.Where this is still weak

Wrapping access control around two stores is more work than one, and I have not finished it.

The check runs in the application, not in the database. No row-level security is enabled, so anyone holding the database URL reads every table.
The memory service API key trusts whatever partition id the caller sends. With the key, you can read or write any organization's graph.
A lint rule fails CI when code queries a memory table in Postgres outside the access and memory services. It does not cover the graph, and a few routes and jobs still write the graph without the guarded wrapper.
Two stores mean a deduplication layer, two ingest paths per write and two ways to fail.

The README lists the first three, each with the file that makes it true [14], and the architecture page's design decisions list the last. I would rather you read those lists than find them.

VI.The short version

Use vector search to find similar text. Use a graph to follow relationships. Use a database for anything that has to be correct. When several agents share one memory, you also need a layer that decides what each of them may see before the search runs, serves the current version with its writer and date, and makes a deletion reach every store. Willder is that layer, on top of the three.