A bigger context window is not memory. It's a bigger whiteboard that still gets wiped every session — and refilling it costs tokens every single time. Long-term memory for an LLM means a store outside the model, written during conversations and retrieved into context only when relevant. Here's how that actually works.
The three memory patterns
- Fact memory: short durable statements the agent saves ("the staging DB is read-only"). Cheap to store, precise to retrieve.
- Conversational memory: summaries of past sessions, useful for assistants that pick up threads.
- Document memory: whole files and pages, chunked and embedded, retrieved by meaning — this is where memory meets RAG.
Most real agents need fact memory plus document memory — and they work best sharing one retrieval system, so a recalled decision can cite the document behind it. The trade-offs between the two are worth understanding.
What good retrieval looks like
- Semantic search catches paraphrases — the question never matches the memory word-for-word.
- Keyword search catches exact identifiers — error codes, names, flags.
- Hybrid retrieval blends both, and near-duplicate detection keeps the store clean over months of use.
Wiring it up
With Kit for AI, memory is two operations: remember stores a fact, recall retrieves by meaning. Agents that speak MCP get them as native tools; everything else calls REST or the SDK. Documents ingest into the same knowledge bases, so one query searches everything the model should know.
Give your model a memory that outlives the session — Start free.