Experimenting with imperfect LLM memory

Research
1

I’ve been playing with a weird idea for LLM memory:

Humans are singular because they have different memories. Our memories are not factual représentations of the past, they distord, they are selected, forgotten, beautified, uglified, ruminated etc…

That’s what makes a human unique, more or less interesting than another one…

I was wondering what would happen if LLM had such kind of memory… A path-dependent memory, reconstructive which can select, forget, distord, ruminate… A memory which ‘hallucinate’ on purpose…

I designed a quick Rust project to experiment this

Two agents start identical. They go through the same interactions. At some point, one of them experiences something different. Then I remove the original context and keep talking to them.

I’m basically asking: does the difference survive?

SelMem doesn’t keep a perfect copy as the agent’s memory. Memories are compressed, some are forgotten, and recalled memories can be reconstructed and modified over time.

The original inputs are kept separately so I can compare what was actually said with what the agent eventually remembers. They’re not used for recall.

I’m not sure yet whether this leads anywhere useful. It could just produce interesting-looking noise. That’s partly what I want to find out.

The repo is here: GitHub - jbsalles/Selmem: Selective memory for LLMs · GitHub

If anyone working on LLM agents / memory wants to try to break the idea, I’d be very interested.

2

Long story short of what the repo does:

Instead of treating memory as immutable information storage, SelMem maintains mutable lived traces:

input → semantic segmentation → selective encoding → trace → recall/reconstruction → reconsolidation → forgetting/merging

Each input is stored in two forms:

sealed archive — immutable original input, used only for auditing/evaluation;

lived memory — compressed and mutable representation used by the agent.

Lived traces contain semantic content plus attributes such as salience, affect, self-relevance, fidelity, access history and permanence. Memories can decay, become latent, merge, or be reconstructed differently when recalled.

Recall uses retrieval to select memories, then a grounding/judging layer compares the reconstruction against the trace’s semantic core. It distinguishes cases such as preservation, compression, elaboration, reframing, contradiction and unrelated recall. Repeated grounding failures can trigger corrective reconsolidation