Spectre Mnemonic: Memory for Agents That Actually Forgets Correctly

This is the third article in the Spectre series.

First came Spectre Kinetic, where the model expresses intent and Elixir decides what reality accepts. Then came Spectre Lens, where agents get browser perception without being forced to work directly with raw DOM.

Now I want to write about memory, because this is one of the parts of agent systems that looks simple at the beginning and then becomes annoying very fast.

Most of the time people say “agent memory” and what they actually mean is: save chunks, create embeddings, search by similarity, and put the result back into the prompt. This is usefull, of course. I use this kind of approach too. But after some time you start to notice the problem: the system isnt really remembering. It is mostly finding similar text.

Vector Search Is Useful, But It Is Not Memory

I do not want to make this sound like vector search is bad. It is not. Vector search is very useful, especially when you need to find related notes, old messages, docs, code fragments, or anything that is too messy for normal keyword search.

But memory needs more than similarity.

If an agent remembers an old payment policy that was replaced two weeks ago, that is not helpful memory. That is a bug. And it is a dangerous bug, becasue the answer can sound correct while being completely outdated.

This is something I kept seeing while building agent systems. The model was not only missing context. That would be easy to understand. The harder problem was when it found the wrong context and used it with confidence.

Real memory has some kind of lifecycle. Some things are temporary. Some things become important. Some things go stale. Some facts are later contradicted. Some data should be forgotten, especially if it is personal or no longer useful.

That is why I started working on Spectre Mnemonic. It is not meant to replace your application database. It is also not a magic brain for agents. It is a memory layer that lives beside your application and helps agents work with context in a more controlled way.

The goal is simple: let the system remember useful things, recall them with boundaries, promote important knowledge, mark stale or contradicted facts, and forget things when they should not stay around forever.

Yes, The Name Is A Johnny Mnemonic Joke

The name is obviously a Johnny Mnemonic joke. The movie is about a man carrying important data in his head, which is somehow not very far from how we treat context windows today.

Agents are often forced to carry too much in their “head”. Chat history, summaries, user preferences, tool results, project notes, random extracted HTML, partial plans, old decisions. Then we hope the model picks the right thing from all of that.

Sometimes it works. Many times it does not.

Memory is not only about storing data. It is about choosing what still matters.

Spectre Mnemonic tries to make that process more explicit. It records moments. It recalls context. It can promote useful knowledge. It can mark facts as stale or contradicted. It supports scoped recall, so memory from one project, tenant, user, or session does not leak into another one.

This part is important for real applications. In a demo, global memory feels fine. In production, global memory becomes a problem very quickly.

The Basic Flow: Remember, Recall, Search, Consolidate

The public API is intentionally simple. I dont want the agent to care too much about where things are stored or how they are promoted. The agent should mostly say what happened, and the runtime should decide how to keep it.

{:ok, memory} =
  SpectreMnemonic.remember("Alice email is alice@example.com",
    stream: :chat,
    kind: :personal_fact,
    persist?: true
  )

{:ok, packet} = SpectreMnemonic.recall("Alice email")
{:ok, results} = SpectreMnemonic.search("Alice email")
{:ok, durable} = SpectreMnemonic.consolidate()

The idea is that remember is used when something happens. It can be a chat message, a task note, a parsed document, a code note, a tool result, or any other event that may become useful later.

recall is for getting active context back. search can look across active and durable memory. consolidate is where the system can decide that some moments are important enough to become more durable knowledge.

{:ok, packet} =
  SpectreMnemonic.remember("TODO: implement durable graph search",
    title: "Planner note",
    stream: :planning,
    task_id: "alpha",
    scope: {:project, "alpha"},
    metadata: %{source: :agent},
    persist?: true
  )

packet.root
packet.chunks
packet.summaries
packet.categories
packet.associations

The returned packet is not just stored text. It can include chunks, summaries, categories, associations, and graph information. This gives the runtime something structured to work with later.

That was important for me. I did not want memory to be only a table of strings and embeddings. I wanted something that the application can inspect and reason about.

Recall Needs Scope, Budget, And Time

One of the easiest mistakes with agent memory is making it too global.

At the beginning, global memory feels convenient. The agent remembers everything, so it can answer more things. But this also means it can mix contexts that should never be mixed. One customer’s details can affect another customer’s answer. One project can pollute another project. Old test data can come back in a real workflow.

So Spectre Mnemonic supports scoped memory.

{:ok, packet} =
  SpectreMnemonic.recall("how is alpha going?",
    scope: {:project, "alpha"},
    max_tokens: 2_000,
    budget: :mid
  )

packet.moments
packet.observations
packet.mental_models
packet.active_status
packet.associations
packet.knowledge

Recall returns a packet, not just a list of similar chunks. The packet can contain moments, observations, mental models, active status, associations, and compact knowledge.

This makes the result easier to use in an agent prompt. Instead of dumping random search results into the context window, the agent receives something closer to a small briefing.

Time is also part of the problem. A fact can be stored today but describe something that happened last month. Another fact can be valid only from a certain date. A policy can be true now but not true anymore next week.

This sounds like too much detail until you build something that uses old business rules because they matched the query better than the new ones.

SpectreMnemonic.remember("Payment retry policy is stable",
  scope: {:tenant, "acme"},
  mission: :code_agent,
  occurred_at: ~U[2026-05-01 12:00:00Z],
  valid_from: ~U[2026-05-01 00:00:00Z],
  persist?: true
)

SpectreMnemonic.recall("payment retry",
  scope: {:tenant, "acme"},
  valid_at: ~U[2026-05-30 00:00:00Z]
)

This is not the most exciting part of the library, but it is the kind of thing that makes memory usable outside of a toy example.

Contradictions Are Normal

Real systems change all the time.

Alice changes email. A task changes status. A project deadline moves. A payment policy is updated. A decision from last week becomes wrong today. This is normal.

The memory layer should not treat all remembered facts as equally true forever.

Spectre Mnemonic has governance states like candidate, short term, promoted, pinned, stale, contradicted, and forgotten. I think this model is closer to what applications actually need. Some memories are fresh. Some are important. Some are old but still useful. Some should stay in history but not be used as current truth.

{:ok, %{moment: old}} =
  SpectreMnemonic.signal("Alice email is old@example.com",
    persist?: true
  )

{:ok, %{moment: new}} =
  SpectreMnemonic.signal("Alice email is new@example.com",
    persist?: true
  )

SpectreMnemonic.Governance.state_for(old.id)
#=> :contradicted

SpectreMnemonic.Governance.state_for(new.id)
#=> :promoted

In this example, the old fact does not have to disappear completely. It can still exist as history. But it should not be treated as the current answer anymore.

Thats one of the main reasons I wanted governance inside the memory layer. If everything is only appended forever, the agent eventually has to guess what is still true. And the model is not the right place to make that decision alone.

Forgetting is also part of this. Spectre Mnemonic can forget active memories and write tombstones. This is not only for privacy, even if privacy is the obvious reason. It is also because systems become messy when nothing ever leaves.

Observations, Mental Models, And Reflection Packets

Raw memories are useful, but they are not always the best thing to give back to an agent.

Sometimes the system needs a higher-level observation. Sometimes it needs a stable answer to a repeated question. Sometimes it needs raw evidence too, but not as the first thing.

This is where observations and mental models come in.

{:ok, observations} =
  SpectreMnemonic.consolidate_observations(
    scope: {:project, "alpha"}
  )

{:ok, matches} =
  SpectreMnemonic.search_observations("payment retry",
    scope: {:project, "alpha"}
  )

{:ok, model} =
  SpectreMnemonic.put_mental_model(%{
    title: "Payment Retry Policy",
    query: "payment retry",
    answer: "Use bounded retries with idempotency keys.",
    scope: {:project, "alpha"},
    source_ids: ["mom_123"]
  })

Mental models are useful when the same question keeps coming back. Instead of rebuilding the answer from raw fragments every time, the system can keep a stable answer with source ids.

That last part matters. I do not want the memory layer to produce mysterious conclusions with no evidence. If a mental model exists, it should be possible to know where it came from.

{:ok, packet} =
  SpectreMnemonic.reflect("What is the payment retry policy?",
    scope: {:project, "alpha"},
    max_tokens: 4_096
  )

packet.mental_models
packet.observations
packet.raw_memories
packet.citations
packet.response

reflect is meant to return a more complete packet. Mental models first, observations after that, raw memories when needed, and citations so the system can show evidence.

The goal is not to give the model everything. The goal is to give it enough of the right context to answer without inventing missing pieces.

The Spectre Pattern

At this point the Spectre pattern is becoming more clear to me.

Kinetic makes intent inspectable. Lens makes pages readable for agents. Mnemonic makes memory more controlled.

None of these libraries try to become the whole agent. I think that is the right direction. I do not want one huge abstraction that hides everything behind the word “autonomy”. I prefer smaller pieces that each own one part of the problem.

Spectre Mnemonic focuses on memory. It knows about heat, durability, scope, facts, contradictions, observations, mental models, secrets, action recipes, and compact knowledge. It can work locally and deterministically without embeddings, and use embeddings when they are actually useful.

Hot working memory can live in ETS. Durable records can live in append-only storage. Recall packets can stay small enough to fit into the context window without dragging the whole history with them.

The model should not remember everything by itself. It should ask for context. The runtime should decide what matters. The memory layer should keep evidence, lifecycle, and boundaries.

That is the split I keep coming back to: the model can reason and write, but the system around it should control what gets remembered, what gets recalled, and what should be ignored.

An agent is not useful because it remembers everything. It becomes useful when it remembers the right things, in the right scope, at the right time. Eventually.If it also forgets my old bugs, even better.