This is a writeup of a short talk I gave on Accord Book, the AI memory and change-control project I've been building for software dev agencies. You can find more about the project on the Vector Intelligence site and blog, and watch the original talk on YouTube.
The talk is really about how good architectural choices and strategies help when you're building something that requires a lot of experimentation, and you don't know in advance which bets will pay off.
Source of truth first
The first choice was to treat all the data we ingest as a source of truth. We extract source records from our ingestion channels and back them up before doing anything else with them. Only afterward do we derive indexes from that data: embeddings for vector search, and structured memory items — categories, entities, and relations.
The bet I made was that graph traversal over those entities and relations should help vector search. The reasoning was that it mirrors how people actually think: we're constantly attaching new information to entities in our heads, and we hold all kinds of connections between them.
Where the benchmark misled us
Part of the problem was the benchmark dataset we picked. It looked good at the time — it contains long chains of conversation, similar to what we were ingesting from Slack and other channels. We got a decent benchmark score (around 90%), and we spent time tuning retrieval latency. But from the start, the graph channel's contribution to retrieval quality was zero.
The actual issue was the structure of that dataset (LongMemEval). It's optimized for long-context recall, but it doesn't have the kind of relational structure a graph retriever could actually exploit.
What actually moved the needle
What did improve our vector search was source-aware reranking. I used the event-sourcing pattern for this: we always keep the original memory record and layer updates — including rank — on top of it. A fact or decision can be updated, and every trace of that decision process stays live in the database.
Because retrieval was designed with two lanes from the start, we were able to simply disable the graph lane once it was clear it wasn't contributing — no rearchitecting needed, just a flag flip.
The takeaway
In the age of AI, development cycles keep getting faster, but that's exactly why it's worth holding onto good practices and the architectural lessons that got us here — and to actually understand the systems we build, rather than just shipping on vibes.
In this concrete case, having a source of truth meant we could always rebuild from scratch if something went wrong. And once we had real data from a pilot, we could re-test the graph-traversal hypothesis and directly measure its contribution — rather than just assuming it was helping.