Once I started thinking in terms of state, everything became clearer. The issue was never just “context size” or “retrieval quality,” even though both of those were the explanations I’d reached for earlier in this series. The real issue, underneath both, was the lack of structure.
So I stopped treating the model like it should remember everything, and started designing the system around it instead — around the fact that the model is a reasoning engine, not a memory store, and building accordingly.
The four layers
Here’s what actually worked, broken into distinct responsibilities:
Short-term memory — keep the last few messages, for local coherence. This handles the immediate back-and-forth: pronoun references, follow-up questions, the natural flow of a single exchange. It doesn’t need to be smart. It just needs to be recent.
Structured state — track goals, decisions, open issues, preferences. This is the object described in the previous post: a small, current, explicit representation of where the conversation actually stands, updated as things change rather than accumulated indefinitely.
Retrieval (RAG) — bring back only a few relevant past elements. Notice the scope here is deliberately narrow. Retrieval isn’t the memory system anymore; it’s a supplement, called in for specific past details that structured state doesn’t need to carry permanently but that might still matter occasionally.
The LLM — used for reasoning, not memory. This is the reframing that ties the other three together. The model’s job is to think with whatever context it’s handed. It is not the model’s job to remember what it was handed three turns ago, or five, or fifty. That job belongs to the layers above it.
Why the separation matters
At each step, the context is assembled intentionally. Not dumped, not guessed at. Every piece that reaches the model earned its place there for a specific reason tied to one of those four layers, rather than being included because it might be relevant, or because excluding it felt risky.
The difference that separation makes is significant, and it shows up in dimensions that actually matter for a working system:
- More consistent answers, because the model is no longer expected to sort signal from noise inside an oversized prompt.
- Lower latency, because the prompt is only ever as large as it needs to be for that turn.
- Better reasoning over time, because the structured state carries forward what matters without degrading as the conversation extends.
That’s when it really clicked for me: the model doesn’t manage the conversation. Your architecture does. Everything the model appears to “remember” or “understand” about an ongoing discussion is really a reflection of how well the system around it assembled the context it received — not a property of the model itself.