Skip to content

Commit ebee09a

Browse files
A memory's age is how much of the vault is newer, not how many days have passed
A fact does not get less true in six months. What dates it is the vault having moved on since, and how far it has moved is a question about how much was written rather than how long it took. So the fact recency prior no longer reads the clock: a memory's age is the share of the vault's own live memories in the namespaces being searched that were written after it. Two memories added in a fortnight leaves a fortnight-old memory alone; five hundred added in the same fortnight halves the odds behind its semantic evidence. The starting memories and installed packs are outside that population and carry no recency at all, because thousands of them arrive at one instant and counting them would leave the prior inert for anyone who just installed. The shift is applied in odds to the semantic term only, so a strong match is nudged and a faint one is scaled, and a literal identifier does not age: the one memory holding the sha that was typed is the answer whatever year it was written. Membership is decided before any of this, on the unaged score, so recency chooses the order of the results and can never remove one. Without that rule a vault would answer "nothing found" about the single old memory it holds on a subject. The store keeps the population as one sorted array, rebuilt when a write moves a record in or out of it, so a search pays a binary search per candidate. Ranking and the post-ranking filters now read a narrow row instead of the whole record, and only the memories actually returned are decrypted, which leaves search a little faster than before. W_RECENCY and RECENCY_HALF_LIFE_DAYS are gone; the opinion recency prior, which is a different mechanism on a real clock, is untouched. The new recency_half_life_share setting overrides the 0.5 default, and 0 turns the prior off.
1 parent 2686197 commit ebee09a

10 files changed

Lines changed: 899 additions & 101 deletions

File tree

‎README.md‎

Lines changed: 34 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -112,8 +112,8 @@ the old record instead of inserting, and the caller resends with
112112
Restating a live opinion refreshes its date instead of storing a copy.
113113
Superseded records are removed from search but kept in the audit chain and
114114
readable by id, with a pointer to their replacement. `supersedes` also works
115-
on facts, for corrections. Opinion ranking weights recency much more than
116-
fact ranking, so the newest opinion wins. `compartment opinions audit` finds
115+
on facts, for corrections. Opinion ranking carries a recency bonus that fact
116+
ranking does not, so the newest opinion wins. `compartment opinions audit` finds
117117
overlapping live opinions in older vaults and keeps the newest, or reports
118118
them for manual merging.
119119

@@ -320,15 +320,19 @@ does well, sized to break ties:
320320
+ w_rrf · k · [ 1/(k + rank_vec) + 1/(k + rank_lex) ] w_rrf = 0.10, k = 20
321321
```
322322

323-
### Importance and recency multiply the score
323+
### Importance and recency reorder the score
324324

325325
```
326-
final(d) = score(d) · ( 1 + w_imp · (2·importance(d) - 1)
327-
+ w_rec · 2^( -age_days(d) / half_life ) )
326+
evidence(d) = aged( vec term, q(d) ) + lex term + rank residue
327+
final(d) = evidence(d) · ( 1 + w_imp · (2·importance(d) - 1) + w_op )
328328
329-
facts: w_imp = 0.15 w_rec = 0.10 half_life = 180 days, from `created`
330-
opinions: w_imp = 0.15 w_rec = 0.30 half_life = 30 days, from the last
331-
re-affirmation (`affirmed`)
329+
aged(s, q) = log( 1 + 2^( -q / half_life_share ) · (e^s - 1) )
330+
331+
q(d) = share of the vault's own memories written after this one
332+
half_life_share = 0.5, so the median memory is worth half the odds
333+
w_imp = 0.15, for facts and opinions alike
334+
w_op = 0.30 · 2^( -age_days / 30 ) for an opinion, from the last
335+
re-affirmation (`affirmed`); 0 for a fact
332336
```
333337

334338
**Multiplicative, so a prior can only reorder memories that already
@@ -342,9 +346,27 @@ centring they would all get the same boost and importance would do nothing.
342346
Centred, an unweighted memory is neutral and only a deliberate weight moves
343347
it.
344348

345-
A fact's recency bonus halves every 180 days from when it was stored. An
346-
opinion's halves every 30 days from when it was last re-affirmed, at three
347-
times the weight, so the newest opinion on a subject wins.
349+
**A memory's age is counted in memories, not in days.** A fact does not get
350+
less true in six months; what makes an old memory the wrong answer is that
351+
the vault has moved on, and how far it has moved is a question about how
352+
much was written. So a memory is as old as the share of the vault written
353+
after it. Two memories added in a fortnight leaves a fortnight-old memory
354+
untouched; five hundred added in the same fortnight halves the odds behind
355+
its semantic evidence. The population counted is the vault's own live
356+
memories in the namespaces being searched: not the reference facts and not
357+
an installed pack, which arrive in their thousands at one instant and have
358+
no age in this sense.
359+
360+
The shift is applied to the semantic channel only, and in odds, so a strong
361+
match is nudged and a faint one is scaled away. A literal identifier is left
362+
alone: a commit SHA that appears in exactly one memory names that memory
363+
whether it was written yesterday or last year. And the relevance floors read
364+
the unaged score, so recency chooses the ORDER of the results and never
365+
which ones come back.
366+
367+
An opinion carries a second prior on top of that, and that one is on the
368+
clock: it halves every 30 days from when it was last re-affirmed, heavily
369+
enough that the newest opinion on a subject wins.
348370

349371
### Retrieval order
350372

@@ -681,6 +703,7 @@ copy; a later install backs up a changed copy rather than overwriting it.
681703
| `auto_lock_minutes` | `30` | idle time before it locks. `0` never locks |
682704
| `search_starter_facts` | `true` | whether the seeded facts join search results |
683705
| `include_packs_in_search` | `true` | the same, for installed packs |
706+
| `recency_half_life_share` | `0.5` | how much of the vault has to be newer than a memory before its semantic evidence is worth half the odds. `0` turns the recency prior off |
684707
| `expire_memories` | `true` | remove expired memories automatically |
685708
| `duplicate_threshold` | `0.97` | cosine similarity at which a store is a duplicate |
686709
| `max_memory_chars` | `200` | the one-claim length limit for authored memories. `0` disables the length and layout checks |

‎docs/MEMORY.md‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -135,14 +135,19 @@ accumulate is contradictory stances.
135135
soft OR over independent evidence, in log space:
136136
`score = -0.75·log(1 - p_vec) - 0.25·log(1 - p_lex)` plus a small
137137
rank-agreement residue, so either channel alone can establish relevance
138-
and neither can veto the other. Importance and recency then MULTIPLY:
139-
`final = score · (1 + 0.15·(2·importance - 1) + recency)`, where recency
140-
is `0.10·2^(-age/180d)` for a fact and `0.30·2^(-age/30d)` for an opinion,
141-
measured from its last re-affirmation - so the current stance outranks a
142-
stale one, and a prior can only reorder memories that already matched. In
143-
practice: ask "what theme does the user like?" and the dark-mode
144-
preference wins on relevance; ask "did the user say to email the client?"
145-
and the consent decision wins.
138+
and neither can veto the other. A memory's age is then counted in MEMORIES
139+
rather than in days: the semantic half of its evidence is shifted, in
140+
odds, by `2^(-q/0.5)`, where `q` is the share of the vault's own memories
141+
written after it. So a fortnight-old memory is barely touched in a quiet
142+
vault and halved in one that wrote five hundred memories in the same
143+
fortnight. Importance MULTIPLIES what is left,
144+
`final = evidence · (1 + 0.15·(2·importance - 1))`, and an opinion adds
145+
`0.30·2^(-age/30d)` from its last re-affirmation, so the current stance
146+
outranks a stale one. A prior can only reorder memories that already
147+
matched, and the relevance floors read the unaged score, so recency cannot
148+
remove one either. In practice: ask "what theme does the user like?" and
149+
the dark-mode preference wins on relevance; ask "did the user say to email
150+
the client?" and the consent decision wins.
146151

147152
**The agent-directed path.** The host model (Hermes, Claude, anything)
148153
also holds `compartment_store` / `compartment_forget` tools, so the intelligence

‎src/compartment/dash.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -226,7 +226,7 @@ def snapshot_search(v: Vault, query: str, caller: str = "dash",
226226
v._require_open()
227227
allowed = set(v._readable_namespaces(caller))
228228
qvec = v.embedder.embed_query(query)
229-
boosted, vec_score, scores = v._rank_candidates(query, qvec,
229+
boosted, vec_score, static = v._rank_candidates(query, qvec,
230230
CANDIDATE_POOL)
231231

232232
results = []

‎src/compartment/ranking.py‎

Lines changed: 160 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -57,16 +57,78 @@
5757
5858
PRIORS
5959
------
60-
Importance and recency MULTIPLY the result rather than adding to it, so they
61-
can only reorder memories that already matched. An additive prior lets a very
62-
important memory surface for a question it has nothing to do with, which is how
63-
a memory system starts feeling haunted.
60+
Importance MULTIPLIES the result rather than adding to it, so it can only
61+
reorder memories that already matched. An additive prior lets a very important
62+
memory surface for a question it has nothing to do with, which is how a memory
63+
system starts feeling haunted.
6464
6565
Importance is centred on the 0.5 default, so an unweighted memory is exactly
6666
neutral. Without centring, a vault's thousands of starting facts all sit at the
6767
same 0.5 and collect the same silent boost as everything else, which is another
6868
way of saying importance did nothing at all.
6969
70+
Opinions carry a second, deliberate prior on top of that: a stance decays on a
71+
short wall-clock half-life from the day it was last re-affirmed, so among live
72+
opinions on one subject the newest wins. That one is about a claim that
73+
REVISES rather than accumulates, and it is separate from the recency below.
74+
75+
The net doctrine: importance and recency reorder matches. Neither can invent
76+
one, and recency cannot delete one either.
77+
78+
RECENCY, COUNTED IN MEMORIES RATHER THAN DAYS
79+
---------------------------------------------
80+
A fact does not get less true in six months. What makes an old memory the wrong
81+
answer is that the vault has moved on since, and how far it has moved is a
82+
question about how much was written, not about how long it took. A vault that
83+
gained two memories in a fortnight has not moved on; a vault that gained five
84+
hundred has. So a memory's age here is the SHARE of the vault that is newer
85+
than it,
86+
87+
q(m) = |{ r in P : t(r) > t(m) }| / |P|
88+
89+
with t = affirmed if set, else created, so a re-affirmed memory is a recent
90+
memory. The newest record has q = 0, the median one q = 0.5, the oldest one
91+
q just under 1.
92+
93+
P, the population, is the LIVE ORGANIC records of the namespaces being
94+
searched. Not the starting memories: every install seeds thousands of them at
95+
one instant, and counting them would park a user's first few hundred own
96+
memories at q below 0.07, leaving the prior inert for exactly the people the
97+
product ships to. A pack arrived all at once and has no age in this sense, so
98+
a pack record carries no recency prior at all (q = None). Namespace-scoped
99+
because in a shared vault one agent writing five thousand records in a week
100+
must not push another agent's whole history to q = 1.
101+
102+
The prior enters in ODDS, not as a multiplier on the score. An evidence term s
103+
is -log P(not relevant), so its odds are e^s - 1, a prior multiplies odds, and
104+
the shifted term is log(1 + rho . (e^s - 1)). Doing it in odds is what makes
105+
the shift behave: strong evidence is NUDGED (it loses at most log 4 nats, even
106+
at q = 1) while weak evidence is CUT in proportion, which is the right way
107+
round. A multiplier on the score would take the same fraction off both.
108+
109+
It shifts the SEMANTIC term only. Recency is a prior on which memory a
110+
question MEANS. A literal identifier says which memory a question NAMES, and
111+
naming does not age: the record holding the one commit sha the user typed is
112+
the answer whether it was written yesterday or last year. Leaving the literal
113+
channel alone also keeps the forced bound above intact under the new prior,
114+
since the semantic term can now only go down.
115+
116+
cosine semantic term q=0 q=0.1 q=0.5 q=1.0
117+
0.50 0.404 1.00 0.89 0.55 0.29
118+
0.65 0.824 1.00 0.91 0.60 0.34
119+
0.80 1.590 1.00 0.93 0.68 0.43
120+
121+
MEMBERSHIP IS DECIDED WITHOUT RECENCY. The floors that choose how many results
122+
come back read the UNAGED evidence; recency then orders what was admitted. The
123+
absolute floor asks whether the evidence is real, which is not a question age
124+
can answer, and without this rule a vault's only relevant memory would drop
125+
below the floor for being old and the vault would answer "nothing found" about
126+
something it knows.
127+
128+
Cost: one SQL scan per mutation to rebuild the population (about 10 ms at
129+
7,000 records), then a masked binary search per candidate, well under a
130+
millisecond per query.
131+
70132
MEASURED, on 44 queries against a real 6,705-memory vault, comparing this
71133
against the previous scorer end to end:
72134
@@ -97,15 +159,25 @@
97159

98160
# --- priors -------------------------------------------------------------------
99161
W_IMPORTANCE = 0.15
100-
W_RECENCY = 0.10
101-
RECENCY_HALF_LIFE_DAYS = 180.0
102-
# Opinions age differently from facts. A fact learned in March is as true in
103-
# September; a stance held in March may have been revised twice since, and
104-
# what a reader wants is the CURRENT one. So an opinion's recency runs from
105-
# the day it was last re-affirmed (falling back to created), decays on a much
106-
# shorter half-life, and carries three times the weight - enough that among
107-
# live opinions on one subject the newest wins decisively, while staying
108-
# multiplicative: it reorders matches and can never manufacture one. The
162+
# How much of the vault has to be newer than a memory before its semantic
163+
# evidence is worth half the odds. 0.5 says the median memory, the one with
164+
# half the vault written after it, is the half-way point: the newest memory
165+
# keeps odds 1, the oldest keeps 1/4. A smaller share ages memories faster and
166+
# starts hiding a vault's older half behind whatever was written last month; a
167+
# larger one flattens the prior until a busy vault stops preferring what it
168+
# just learned. 0.5 is also the only value with a plain reading, which is what
169+
# a user editing the setting has to reason with.
170+
RECENCY_HALF_LIFE_SHARE = 0.5
171+
# Opinions age differently from facts, and on a different clock. A fact
172+
# learned in March is as true in September, and what dates it is the vault
173+
# having moved on, which is the count above. A stance held in March may have
174+
# been revised twice since, and what a reader wants is the CURRENT one, which
175+
# is a question about the calendar however quiet the vault has been. So an
176+
# opinion carries this SECOND prior on top: it runs from the day the stance
177+
# was last re-affirmed (falling back to created), decays on a short wall-clock
178+
# half-life, and is weighted heavily enough that among live opinions on one
179+
# subject the newest wins decisively, while staying multiplicative: it
180+
# reorders matches and can never manufacture one. The
109181
# stronger mechanism for replaced opinions is superseding, which removes the
110182
# old record from retrieval entirely; this prior settles the remainder, the
111183
# same-subject opinions nobody has explicitly reconciled yet.
@@ -213,45 +285,100 @@ def information_coverage(term_info: dict[str, float], text: str) -> float:
213285
return min(_CERTAINTY_CAP, max(0.0, got / total))
214286

215287

288+
def _vec_term(p_vec: float) -> float:
289+
"""The semantic channel's contribution, on its own so recency can shift
290+
it without touching what the literal channel established."""
291+
return -W_VEC * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_vec)))
292+
293+
294+
def _lex_term(p_lex: float) -> float:
295+
return -W_LEX * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_lex)))
296+
297+
298+
def _rrf_residue(vec_rank: int | None, lex_rank: int | None) -> float:
299+
if not W_RRF:
300+
return 0.0
301+
residue = 0.0
302+
if vec_rank is not None:
303+
residue += 1.0 / (RRF_RESIDUE_K + vec_rank + 1)
304+
if lex_rank is not None:
305+
residue += 1.0 / (RRF_RESIDUE_K + lex_rank + 1)
306+
return W_RRF * residue * RRF_RESIDUE_K
307+
308+
216309
def evidence(p_vec: float, p_lex: float,
217310
vec_rank: int | None = None, lex_rank: int | None = None) -> float:
218311
"""Combine two independent channels as a soft OR, in log space."""
219-
score = (-W_VEC * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_vec)))
220-
- W_LEX * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_lex))))
221-
if W_RRF:
222-
residue = 0.0
223-
if vec_rank is not None:
224-
residue += 1.0 / (RRF_RESIDUE_K + vec_rank + 1)
225-
if lex_rank is not None:
226-
residue += 1.0 / (RRF_RESIDUE_K + lex_rank + 1)
227-
score += W_RRF * residue * RRF_RESIDUE_K
228-
return score
312+
return (_vec_term(p_vec) + _lex_term(p_lex)
313+
+ _rrf_residue(vec_rank, lex_rank))
314+
315+
316+
def recency_odds(newer_share: float | None,
317+
half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float:
318+
"""Prior odds for a memory with this share of the vault newer than it.
319+
320+
1.0 for the newest memory and for anything outside the population, which
321+
is how a pack record and a brand new one both come through unaged. A
322+
half_life_share of 0 turns the whole prior off."""
323+
if newer_share is None or half_life_share <= 0.0:
324+
return 1.0
325+
q = min(1.0, max(0.0, float(newer_share)))
326+
return 2.0 ** (-q / float(half_life_share))
327+
328+
329+
def aged_term(term: float, newer_share: float | None,
330+
half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float:
331+
"""One evidence term, shifted by the recency prior IN ODDS.
332+
333+
A term is -log P(not relevant), so its odds are e^term - 1; the prior
334+
multiplies those odds and this reads the result back as a term. Monotone
335+
in `term`, so equally aged memories keep their order, and never above
336+
`term`, so ageing can only cost a memory rank."""
337+
rho = recency_odds(newer_share, half_life_share)
338+
if rho >= 1.0 or term <= 0.0:
339+
return term
340+
return math.log1p(rho * math.expm1(term))
341+
342+
343+
def aged_evidence(p_vec: float, p_lex: float, newer_share: float | None = None,
344+
vec_rank: int | None = None, lex_rank: int | None = None,
345+
half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float:
346+
"""Evidence with the semantic channel aged and the literal one left alone.
347+
348+
What a question MEANS is a guess that the vault having moved on can
349+
weaken. What a question NAMES it cannot: an identifier that occurs in one
350+
memory names that memory whatever year it was written."""
351+
return (aged_term(_vec_term(p_vec), newer_share, half_life_share)
352+
+ _lex_term(p_lex) + _rrf_residue(vec_rank, lex_rank))
229353

230354

231355
def prior(importance: float, created: float, now: float | None = None,
232356
kind: str = "fact", affirmed: float | None = None) -> float:
233357
"""Multiplicative modulation: reranks a match, never manufactures one.
234358
235-
`kind` and `affirmed` are optional so every existing caller keeps its
236-
behavior bit for bit: with the defaults this is the fact prior it always
237-
was. An opinion's recency runs from its last re-affirmation on the
238-
shorter, heavier opinion constants above."""
239-
now = time.time() if now is None else now
359+
For a fact this is importance alone, so a fact stored with the default
360+
0.5 carries no prior at all: its age is counted in memories rather than
361+
days and reaches the score through aged_evidence instead. `created`,
362+
`now`, `kind` and `affirmed` stay in the signature for the opinion
363+
branch, whose recency runs from the last re-affirmation on the shorter,
364+
heavier opinion constants above."""
240365
centred = 2.0 * float(importance) - 1.0 # 0.5 default -> exactly 0
241366
if kind == "opinion":
367+
now = time.time() if now is None else now
242368
ref = float(affirmed or created)
243369
age_days = max(0.0, (now - ref) / SECONDS_PER_DAY)
244370
recency = math.exp(
245371
-math.log(2.0) * age_days / OPINION_RECENCY_HALF_LIFE_DAYS)
246372
return W_IMPORTANCE * centred + W_RECENCY_OPINION * recency
247-
age_days = max(0.0, (now - float(created)) / SECONDS_PER_DAY)
248-
recency = math.exp(-math.log(2.0) * age_days / RECENCY_HALF_LIFE_DAYS)
249-
return W_IMPORTANCE * centred + W_RECENCY * recency
373+
return W_IMPORTANCE * centred
250374

251375

252376
def final_score(p_vec: float, p_lex: float, importance: float, created: float,
253377
vec_rank: int | None = None, lex_rank: int | None = None,
254378
now: float | None = None, kind: str = "fact",
255-
affirmed: float | None = None) -> float:
256-
return evidence(p_vec, p_lex, vec_rank, lex_rank) * (
379+
affirmed: float | None = None,
380+
newer_share: float | None = None,
381+
half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float:
382+
return aged_evidence(p_vec, p_lex, newer_share, vec_rank, lex_rank,
383+
half_life_share) * (
257384
1.0 + prior(importance, created, now, kind=kind, affirmed=affirmed))

0 commit comments

Comments
 (0)