This is a really interesting direction. Most current “memory” implementations for LLMs are essentially append-only logs or vector stores, but modeling things like salience, decay, and consolidation is much closer to how useful memory actually works.
One of the biggest challenges seems to be avoiding memory bloat while still preserving important long-term context. I’m curious how you balance forgetting vs retention — especially how salience is calculated and updated over time.
Thanks! The bloat problem is exactly what pushed me toward forgetting-as-a-feature rather than just accumulating everything.
Salience is scored on four dimensions when a memory is first ingested: novelty (how surprising/new), relevance (how useful for future interactions), emotional weight (personal significance), and predictive value (does this change expectations). A Sonnet instance does the scoring, so it adapts to context.
For retention vs forgetting, each memory's strength is calculated dynamically:
Decay rate is 0.015/day, so an unaccessed memory with average salience of 0.3 fades to near-zero in about three weeks. But accessing a memory boosts it (+0.12 per access, capped at 0.5), and consolidated memories get a flat +0.2 bonus. So memories that keep being relevant naturally survive.
The consolidation cycle (modeled on slow-wave sleep) is the other half. It merges redundant memories into stronger single entries, extracts generalized patterns from recurring themes, and prunes anything below threshold. This is what prevents bloat at scale while preserving the signal.
Honestly the balance isn't perfectly tuned yet. The decay rate and consolidation frequency are constants right now. Ideally, they'd adapt based on how the user actually interacts with their memories. That's on the roadmap.
This is a really interesting direction. Most current “memory” implementations for LLMs are essentially append-only logs or vector stores, but modeling things like salience, decay, and consolidation is much closer to how useful memory actually works.
One of the biggest challenges seems to be avoiding memory bloat while still preserving important long-term context. I’m curious how you balance forgetting vs retention — especially how salience is calculated and updated over time.
Thanks! The bloat problem is exactly what pushed me toward forgetting-as-a-feature rather than just accumulating everything.
Salience is scored on four dimensions when a memory is first ingested: novelty (how surprising/new), relevance (how useful for future interactions), emotional weight (personal significance), and predictive value (does this change expectations). A Sonnet instance does the scoring, so it adapts to context.
For retention vs forgetting, each memory's strength is calculated dynamically:
strength = avg_salience + retrieval_boost + consolidation_bonus - (decay_rate × age_in_days)
Decay rate is 0.015/day, so an unaccessed memory with average salience of 0.3 fades to near-zero in about three weeks. But accessing a memory boosts it (+0.12 per access, capped at 0.5), and consolidated memories get a flat +0.2 bonus. So memories that keep being relevant naturally survive.
The consolidation cycle (modeled on slow-wave sleep) is the other half. It merges redundant memories into stronger single entries, extracts generalized patterns from recurring themes, and prunes anything below threshold. This is what prevents bloat at scale while preserving the signal.
Honestly the balance isn't perfectly tuned yet. The decay rate and consolidation frequency are constants right now. Ideally, they'd adapt based on how the user actually interacts with their memories. That's on the roadmap.