Much has been said, published and iterated on the topic of memory systems over the last few years. Most of that work, and most of the benchmarks, is about finding a specific piece of information in a large amount of data. I think that part is mostly solved. The bigger problem in daily work is that the agent often doesn't even know there is something to search for.
Where I'm coming from
I have been playing with memory systems on the side for years, and for about a year professionally, too. I first got interested in this through Notion and the whole second brain idea, where you capture everything, link it and end up with a system that helps you organize. I soon realised that the manual labour to do this well is just too high, at least for my taste.
LLMs remove most of that labour, since reading, sorting, summarising and linking are all cheap now. But they bring their own problem of relevance realization, of deciding what matters, and that is exactly the kind of judgment where humans currently still have an advantage. So we have to give the system a seed of that judgment it can build on, and ideally extract a lot of it implicitly from what gets ingested.
The platonic ideal I keep coming back to is a system you give a rough overview of your goals. It takes in arbitrary streams of information and assembles them into whatever shape is most navigable. The reader of that shape may be me or another agent. Their needs differ in some ways, but they overlap a lot, for instance in not wanting to be overloaded with information that isn't relevant to the task.
The question guiding my design thinking is whether there is a fundamental solution here that works across people and organisations, or whether this is inherently specific.
Search is mostly solved
Two simple approaches already get you surprisingly far. One is Karpathy-style Markdown wikis, where an agent maintains a folder of plain notes and reads them back in. The other is to embed everything and let agents search, since retrieval over large amounts of data works well now.
Fundamentally, memory is a problem of compression and search, and of the two, search is in much better shape.
There are many benchmarks that measure whether a system can find a specific piece of information in a large amount of data. I tested my own system on one of them, LongMemEval, and it does about as well as plain retrieval, which is what I would expect from a system that only contains a basic best-practice version of search and isn't optimized for this.
Benchmarks like this don't separate the approaches much anymore, because they assume the question is already known.
What I would want to measure instead is whether an agent knows what to look for, rather than how well the search works for a known needle, and to my knowledge current benchmarks don't capture this.
Why the agent needs a map
If I ask what to gift my girlfriend, that's easy, because the agent knows what it is missing and can write a query for it. But if I ask what I should read this weekend, there is no good query. The agent could search for my interests and would get back a few entries that happen to match, but it can't tell what it isn't seeing, which of these are current and which are central. The answer depends less on any single fact than on the shape of everything that is there, and that is exactly what a search result doesn't show and a map does.
The more open-ended a request becomes, the more this matters. A request like "tell me something interesting" depends almost entirely on that shape.
Importantly, i also think this will matter more over time. AI work seems to be moving towards higher levels of abstraction, and the higher the level, the more an answer depends on context that was never mentioned in the request.
How do we design the map?
This is what my recent project, pd-memory, tries to answer. It is my attempt at a general solution, though what it looks like is itself shaped by how context works in current AI systems.
The core constraint for an agent is token bloat, since everything in the standing context is read again on every turn and competes for the model's attention.
There are two failure modes to avoid: by trying to achieve completeness, one may be tempted to increase the token budget and bloat the context, and on the other end there is the search-only case, where the agent doesn't know what it can search for.
So the map has to be compressed strongly, and once you do that it starts to look like a long list of one-liners. What I landed on is a dictionary.
The central piece is this dictionary-style overview, which is assembled on demand and in variable length. It can cover the whole system, a specific area or subtopic, or whatever a specific conversation is about, as a starting point for the agent. From there the agent can notice where it lacks detail, open the relevant entry, follow links to related entries and come back with what it needs. Retrieval then becomes part of the agent's thinking instead of a separate step.
Here is a heavily cut slice of the whole-system overview my own agents get at the start of a session.
## daimon (artifact) — Daimon – graphical interface and backend for coding conversations, sessions, context handoff, presented artifacts, and probability-benchmark work. (125)
- 9dbcbe [2+] Daimon should create the conversation thread and durable submission together on the first Send, so Pi starts from recorded user intent rather than an empty pre-created session. (3)
...
## pd-memory (project) — pd-memory – experimental project for progressively disclosed, navigable, editable, and uncertainty-aware durable memory for agents. (45)
- 0d1c84 [2+] pd-memory should retain only knowledge likely to remain useful several weeks later, prioritizing lasting importance over mere truth. (5)
- 403282 [2+] pd-memory must attribute a decision to Mark only when Mark explicitly confirms it; an assistant proposal is never Mark's decision without that confirmation. (1)
...
### local-conversation-archive-ingestion (project) — Local conversation archive ingestion – planned local processing of Mark's Codex, Claude, and related conversation archives for durable memory. (7)
...
## pi (artifact) — Pi – authoritative coding-session runtime providing session history, native tools, extensions, and runtime semantics. (25)
...
## backup-and-recovery (topic) — Backup and recovery – durable backup coverage, restoration practices, and retention decisions for Mark's Macs and project data. (6)
- 7f14eb [2+] Restic provides deduplicated, versioned snapshots, so deleting a source file does not remove it from earlier snapshots until retention removes the last snapshot containing it. (1)
...
## mark (actor) — Mark – Person whose durable views, decisions, and working preferences are recorded here. (7)
...
## forecasting-and-calibration (topic) — Forecasting and calibration – prediction under uncertainty, prospective evaluation, and the reliability of reported confidence. (5)
- 52722a [2+] Mini-Jev may initially use logits as a confidence signal, but they must be calibrated and validated rather than treated as trustworthy probabilities by default. (1)
...
## battlesim (project) — BattleSim – project for experimenting with and evaluating combat simulation models and implementations. (8)
### battlesim-renderer (artifact) — BattleSim renderer – code-first renderer work for inspecting replayed battles with rich terrain, free-camera exploration, and individually represented soldiers. (3)
...How to read this
Headings are pages, which are durable subjects and can be nested. The lines below them are observations. The id is used to open an entry, the number in brackets shows how prominent it is, a + means there is more detail behind it, and the count at the end is the number of sources that back it.
Even this cut slice shows that I am building a memory system and an agent interface, and working on forecasting and a combat simulation on the side. For the question of what to read this weekend, that says more than any search for my interests would.
Getting this format right is mostly a question of cognitive ergonomics for the model. Raw UUIDs waste tokens, and if opening the next layer takes several awkward tool calls, the agent will guess from the summary instead.
This slice, like my memory in general, is currently strongly biased towards engineering, because coding sessions are what I have ingested so far. The mechanism itself is meant to be general. It should work the same way for a company knowledge base or a personal memory built from chat logs, and for almost any topic, because the dynamics of what an agent needs to know and when it needs it are the same.
Where it does lose out is when the big picture itself is the core information, because extracted observations tend to lose it. An earlier system I built worked the other way around, by summarising sources and progressively compressing older summaries into higher-level ones. It kept the big picture well and works well for clearly scoped projects. But as soon as a claim is relevant to several projects, it gets tricky, because the claim sits somewhere inside a mass of summary text that has to be combed through. Smaller units of information are much handier here, especially for assembling an overview on demand. That is what pushed me towards individual entries.
Getting the big picture back into them is a problem I haven't fully solved yet. The direction I want to go is to make the basic entry wide enough to also hold observations that cut across many sources, so that statements about the big picture live in memory next to the individual facts and can be opened and corrected the same way.
Is text the right level?
Not really. Ultimately much stronger compression will be possible that isn't based on text at all. Cursor's post on training Composer to summarise itself mentions research on compaction in latent space, where the model keeps context as vectors rather than text. As far as I understand, this requires much more direct access to the model, and for now it is also slower.
Their own result is interesting from another angle. They trained the model to do the compression itself, and its summaries came out at around a fifth of the size of a carefully prompted baseline while losing about half as much. So relevance realization can be learned when there is a reward signal. In a coding task that signal arrives quickly, because you soon see whether the summary kept the right things, while with long-term memory it may take weeks to find out, if you find out at all.
For now I see text as an intermediate stage. Its big advantage is that it is readable, which means humans and agents consume the same representation, and that makes it easy to think about and iterate on. It's also quite interesting to read what your agents think they know about you, I can recommend it.
The code is here.