At that point, I thought I had found the solution.
The previous problem was clear enough: feeding the model more history didn’t make it understand me better. It made it slower, less consistent, and prone to ignoring details that were sitting right there in the prompt. More context wasn’t the fix. So the next idea followed naturally.
“Maybe I don’t need more context. Maybe I just need better retrieval.”
So I turned to RAG.
Why RAG looked like the obvious move
Retrieval-Augmented Generation is built on a simple premise: don’t hand the model everything, hand it what’s relevant. Store the conversation somewhere outside the prompt, retrieve only the pieces that matter for the current turn, and keep the actual context window small and focused.
On paper, this maps almost exactly onto the problem I’d just run into. The issue wasn’t a lack of information — I had plenty. The issue was that all of it was being pushed into the model at once, undifferentiated. RAG offered a way to separate storage from delivery: keep everything, surface only what’s needed, when it’s needed.
It felt elegant. Almost obvious, in hindsight, which should have been my first hint to look harder before trusting it.
Where it actually helped
To be fair to the approach, it did work — for a while, and along real dimensions:
- Responses got faster, since the prompt was no longer carrying the full conversation history.
- Prompts stayed cleaner, easier to reason about, easier to debug.
- Some answers became more relevant, because retrieval was pulling in details that mattered for the specific question being asked.
If the story stopped there, RAG would have been the full answer. It wasn’t.
Where it broke down
Pushing the system further surfaced a different kind of problem — quieter, harder to catch than “the model is slow” or “the model ignored something.”
Not everything retrieved was actually useful. Some important details got missed entirely, even though they were technically stored somewhere in the system. And in the other direction, the system sometimes brought back fragments that made complete sense on their own — a past preference, a decision made three turns ago — but broke the flow of reasoning when dropped back into the current conversation.
That second failure mode is the one worth sitting with. A retrieved fragment can be locally correct and globally wrong. It answers the literal similarity match — this text resembles that text — without answering the question that actually matters: does this belong in the reasoning the model is doing right now?
That gap is structural, not incidental. Retrieval systems are built to find related content. They are not built to track what a conversation is currently doing, has already resolved, or still has open. Similarity is not the same thing as relevance to an evolving thread of reasoning, and the more the system got pushed, the more that distinction showed up in the output.
The realization
RAG is powerful. But it doesn’t understand conversations.
It retrieves fragments. And conversations aren’t fragments — they’re evolving state. A goal that’s been set. Decisions already made. Open issues not yet resolved. Preferences expressed along the way. None of that is captured by pulling back the top-k most similar chunks to a query, no matter how good the embedding model is.
That distinction changed how I started thinking about memory entirely. The question stopped being “how do I retrieve the right pieces of the past” and became “what does this system need to actually track, on purpose, as the conversation moves forward.” Retrieval still has a place in that answer — but as one component serving a structured state, not as the memory system itself.
That’s where the next part of this starts.