|
57 | 57 |
|
58 | 58 | PRIORS |
59 | 59 | ------ |
60 | | -Importance and recency MULTIPLY the result rather than adding to it, so they |
61 | | -can only reorder memories that already matched. An additive prior lets a very |
62 | | -important memory surface for a question it has nothing to do with, which is how |
63 | | -a memory system starts feeling haunted. |
| 60 | +Importance MULTIPLIES the result rather than adding to it, so it can only |
| 61 | +reorder memories that already matched. An additive prior lets a very important |
| 62 | +memory surface for a question it has nothing to do with, which is how a memory |
| 63 | +system starts feeling haunted. |
64 | 64 |
|
65 | 65 | Importance is centred on the 0.5 default, so an unweighted memory is exactly |
66 | 66 | neutral. Without centring, a vault's thousands of starting facts all sit at the |
67 | 67 | same 0.5 and collect the same silent boost as everything else, which is another |
68 | 68 | way of saying importance did nothing at all. |
69 | 69 |
|
| 70 | +Opinions carry a second, deliberate prior on top of that: a stance decays on a |
| 71 | +short wall-clock half-life from the day it was last re-affirmed, so among live |
| 72 | +opinions on one subject the newest wins. That one is about a claim that |
| 73 | +REVISES rather than accumulates, and it is separate from the recency below. |
| 74 | +
|
| 75 | +The net doctrine: importance and recency reorder matches. Neither can invent |
| 76 | +one, and recency cannot delete one either. |
| 77 | +
|
| 78 | +RECENCY, COUNTED IN MEMORIES RATHER THAN DAYS |
| 79 | +--------------------------------------------- |
| 80 | +A fact does not get less true in six months. What makes an old memory the wrong |
| 81 | +answer is that the vault has moved on since, and how far it has moved is a |
| 82 | +question about how much was written, not about how long it took. A vault that |
| 83 | +gained two memories in a fortnight has not moved on; a vault that gained five |
| 84 | +hundred has. So a memory's age here is the SHARE of the vault that is newer |
| 85 | +than it, |
| 86 | +
|
| 87 | + q(m) = |{ r in P : t(r) > t(m) }| / |P| |
| 88 | +
|
| 89 | +with t = affirmed if set, else created, so a re-affirmed memory is a recent |
| 90 | +memory. The newest record has q = 0, the median one q = 0.5, the oldest one |
| 91 | +q just under 1. |
| 92 | +
|
| 93 | +P, the population, is the LIVE ORGANIC records of the namespaces being |
| 94 | +searched. Not the starting memories: every install seeds thousands of them at |
| 95 | +one instant, and counting them would park a user's first few hundred own |
| 96 | +memories at q below 0.07, leaving the prior inert for exactly the people the |
| 97 | +product ships to. A pack arrived all at once and has no age in this sense, so |
| 98 | +a pack record carries no recency prior at all (q = None). Namespace-scoped |
| 99 | +because in a shared vault one agent writing five thousand records in a week |
| 100 | +must not push another agent's whole history to q = 1. |
| 101 | +
|
| 102 | +The prior enters in ODDS, not as a multiplier on the score. An evidence term s |
| 103 | +is -log P(not relevant), so its odds are e^s - 1, a prior multiplies odds, and |
| 104 | +the shifted term is log(1 + rho . (e^s - 1)). Doing it in odds is what makes |
| 105 | +the shift behave: strong evidence is NUDGED (it loses at most log 4 nats, even |
| 106 | +at q = 1) while weak evidence is CUT in proportion, which is the right way |
| 107 | +round. A multiplier on the score would take the same fraction off both. |
| 108 | +
|
| 109 | +It shifts the SEMANTIC term only. Recency is a prior on which memory a |
| 110 | +question MEANS. A literal identifier says which memory a question NAMES, and |
| 111 | +naming does not age: the record holding the one commit sha the user typed is |
| 112 | +the answer whether it was written yesterday or last year. Leaving the literal |
| 113 | +channel alone also keeps the forced bound above intact under the new prior, |
| 114 | +since the semantic term can now only go down. |
| 115 | +
|
| 116 | + cosine semantic term q=0 q=0.1 q=0.5 q=1.0 |
| 117 | + 0.50 0.404 1.00 0.89 0.55 0.29 |
| 118 | + 0.65 0.824 1.00 0.91 0.60 0.34 |
| 119 | + 0.80 1.590 1.00 0.93 0.68 0.43 |
| 120 | +
|
| 121 | +MEMBERSHIP IS DECIDED WITHOUT RECENCY. The floors that choose how many results |
| 122 | +come back read the UNAGED evidence; recency then orders what was admitted. The |
| 123 | +absolute floor asks whether the evidence is real, which is not a question age |
| 124 | +can answer, and without this rule a vault's only relevant memory would drop |
| 125 | +below the floor for being old and the vault would answer "nothing found" about |
| 126 | +something it knows. |
| 127 | +
|
| 128 | +Cost: one SQL scan per mutation to rebuild the population (about 10 ms at |
| 129 | +7,000 records), then a masked binary search per candidate, well under a |
| 130 | +millisecond per query. |
| 131 | +
|
70 | 132 | MEASURED, on 44 queries against a real 6,705-memory vault, comparing this |
71 | 133 | against the previous scorer end to end: |
72 | 134 |
|
|
97 | 159 |
|
98 | 160 | # --- priors ------------------------------------------------------------------- |
99 | 161 | W_IMPORTANCE = 0.15 |
100 | | -W_RECENCY = 0.10 |
101 | | -RECENCY_HALF_LIFE_DAYS = 180.0 |
102 | | -# Opinions age differently from facts. A fact learned in March is as true in |
103 | | -# September; a stance held in March may have been revised twice since, and |
104 | | -# what a reader wants is the CURRENT one. So an opinion's recency runs from |
105 | | -# the day it was last re-affirmed (falling back to created), decays on a much |
106 | | -# shorter half-life, and carries three times the weight - enough that among |
107 | | -# live opinions on one subject the newest wins decisively, while staying |
108 | | -# multiplicative: it reorders matches and can never manufacture one. The |
| 162 | +# How much of the vault has to be newer than a memory before its semantic |
| 163 | +# evidence is worth half the odds. 0.5 says the median memory, the one with |
| 164 | +# half the vault written after it, is the half-way point: the newest memory |
| 165 | +# keeps odds 1, the oldest keeps 1/4. A smaller share ages memories faster and |
| 166 | +# starts hiding a vault's older half behind whatever was written last month; a |
| 167 | +# larger one flattens the prior until a busy vault stops preferring what it |
| 168 | +# just learned. 0.5 is also the only value with a plain reading, which is what |
| 169 | +# a user editing the setting has to reason with. |
| 170 | +RECENCY_HALF_LIFE_SHARE = 0.5 |
| 171 | +# Opinions age differently from facts, and on a different clock. A fact |
| 172 | +# learned in March is as true in September, and what dates it is the vault |
| 173 | +# having moved on, which is the count above. A stance held in March may have |
| 174 | +# been revised twice since, and what a reader wants is the CURRENT one, which |
| 175 | +# is a question about the calendar however quiet the vault has been. So an |
| 176 | +# opinion carries this SECOND prior on top: it runs from the day the stance |
| 177 | +# was last re-affirmed (falling back to created), decays on a short wall-clock |
| 178 | +# half-life, and is weighted heavily enough that among live opinions on one |
| 179 | +# subject the newest wins decisively, while staying multiplicative: it |
| 180 | +# reorders matches and can never manufacture one. The |
109 | 181 | # stronger mechanism for replaced opinions is superseding, which removes the |
110 | 182 | # old record from retrieval entirely; this prior settles the remainder, the |
111 | 183 | # same-subject opinions nobody has explicitly reconciled yet. |
@@ -213,45 +285,100 @@ def information_coverage(term_info: dict[str, float], text: str) -> float: |
213 | 285 | return min(_CERTAINTY_CAP, max(0.0, got / total)) |
214 | 286 |
|
215 | 287 |
|
| 288 | +def _vec_term(p_vec: float) -> float: |
| 289 | + """The semantic channel's contribution, on its own so recency can shift |
| 290 | + it without touching what the literal channel established.""" |
| 291 | + return -W_VEC * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_vec))) |
| 292 | + |
| 293 | + |
| 294 | +def _lex_term(p_lex: float) -> float: |
| 295 | + return -W_LEX * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_lex))) |
| 296 | + |
| 297 | + |
| 298 | +def _rrf_residue(vec_rank: int | None, lex_rank: int | None) -> float: |
| 299 | + if not W_RRF: |
| 300 | + return 0.0 |
| 301 | + residue = 0.0 |
| 302 | + if vec_rank is not None: |
| 303 | + residue += 1.0 / (RRF_RESIDUE_K + vec_rank + 1) |
| 304 | + if lex_rank is not None: |
| 305 | + residue += 1.0 / (RRF_RESIDUE_K + lex_rank + 1) |
| 306 | + return W_RRF * residue * RRF_RESIDUE_K |
| 307 | + |
| 308 | + |
216 | 309 | def evidence(p_vec: float, p_lex: float, |
217 | 310 | vec_rank: int | None = None, lex_rank: int | None = None) -> float: |
218 | 311 | """Combine two independent channels as a soft OR, in log space.""" |
219 | | - score = (-W_VEC * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_vec))) |
220 | | - - W_LEX * math.log(1.0 - min(_CERTAINTY_CAP, max(0.0, p_lex)))) |
221 | | - if W_RRF: |
222 | | - residue = 0.0 |
223 | | - if vec_rank is not None: |
224 | | - residue += 1.0 / (RRF_RESIDUE_K + vec_rank + 1) |
225 | | - if lex_rank is not None: |
226 | | - residue += 1.0 / (RRF_RESIDUE_K + lex_rank + 1) |
227 | | - score += W_RRF * residue * RRF_RESIDUE_K |
228 | | - return score |
| 312 | + return (_vec_term(p_vec) + _lex_term(p_lex) |
| 313 | + + _rrf_residue(vec_rank, lex_rank)) |
| 314 | + |
| 315 | + |
| 316 | +def recency_odds(newer_share: float | None, |
| 317 | + half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float: |
| 318 | + """Prior odds for a memory with this share of the vault newer than it. |
| 319 | +
|
| 320 | + 1.0 for the newest memory and for anything outside the population, which |
| 321 | + is how a pack record and a brand new one both come through unaged. A |
| 322 | + half_life_share of 0 turns the whole prior off.""" |
| 323 | + if newer_share is None or half_life_share <= 0.0: |
| 324 | + return 1.0 |
| 325 | + q = min(1.0, max(0.0, float(newer_share))) |
| 326 | + return 2.0 ** (-q / float(half_life_share)) |
| 327 | + |
| 328 | + |
| 329 | +def aged_term(term: float, newer_share: float | None, |
| 330 | + half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float: |
| 331 | + """One evidence term, shifted by the recency prior IN ODDS. |
| 332 | +
|
| 333 | + A term is -log P(not relevant), so its odds are e^term - 1; the prior |
| 334 | + multiplies those odds and this reads the result back as a term. Monotone |
| 335 | + in `term`, so equally aged memories keep their order, and never above |
| 336 | + `term`, so ageing can only cost a memory rank.""" |
| 337 | + rho = recency_odds(newer_share, half_life_share) |
| 338 | + if rho >= 1.0 or term <= 0.0: |
| 339 | + return term |
| 340 | + return math.log1p(rho * math.expm1(term)) |
| 341 | + |
| 342 | + |
| 343 | +def aged_evidence(p_vec: float, p_lex: float, newer_share: float | None = None, |
| 344 | + vec_rank: int | None = None, lex_rank: int | None = None, |
| 345 | + half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float: |
| 346 | + """Evidence with the semantic channel aged and the literal one left alone. |
| 347 | +
|
| 348 | + What a question MEANS is a guess that the vault having moved on can |
| 349 | + weaken. What a question NAMES it cannot: an identifier that occurs in one |
| 350 | + memory names that memory whatever year it was written.""" |
| 351 | + return (aged_term(_vec_term(p_vec), newer_share, half_life_share) |
| 352 | + + _lex_term(p_lex) + _rrf_residue(vec_rank, lex_rank)) |
229 | 353 |
|
230 | 354 |
|
231 | 355 | def prior(importance: float, created: float, now: float | None = None, |
232 | 356 | kind: str = "fact", affirmed: float | None = None) -> float: |
233 | 357 | """Multiplicative modulation: reranks a match, never manufactures one. |
234 | 358 |
|
235 | | - `kind` and `affirmed` are optional so every existing caller keeps its |
236 | | - behavior bit for bit: with the defaults this is the fact prior it always |
237 | | - was. An opinion's recency runs from its last re-affirmation on the |
238 | | - shorter, heavier opinion constants above.""" |
239 | | - now = time.time() if now is None else now |
| 359 | + For a fact this is importance alone, so a fact stored with the default |
| 360 | + 0.5 carries no prior at all: its age is counted in memories rather than |
| 361 | + days and reaches the score through aged_evidence instead. `created`, |
| 362 | + `now`, `kind` and `affirmed` stay in the signature for the opinion |
| 363 | + branch, whose recency runs from the last re-affirmation on the shorter, |
| 364 | + heavier opinion constants above.""" |
240 | 365 | centred = 2.0 * float(importance) - 1.0 # 0.5 default -> exactly 0 |
241 | 366 | if kind == "opinion": |
| 367 | + now = time.time() if now is None else now |
242 | 368 | ref = float(affirmed or created) |
243 | 369 | age_days = max(0.0, (now - ref) / SECONDS_PER_DAY) |
244 | 370 | recency = math.exp( |
245 | 371 | -math.log(2.0) * age_days / OPINION_RECENCY_HALF_LIFE_DAYS) |
246 | 372 | return W_IMPORTANCE * centred + W_RECENCY_OPINION * recency |
247 | | - age_days = max(0.0, (now - float(created)) / SECONDS_PER_DAY) |
248 | | - recency = math.exp(-math.log(2.0) * age_days / RECENCY_HALF_LIFE_DAYS) |
249 | | - return W_IMPORTANCE * centred + W_RECENCY * recency |
| 373 | + return W_IMPORTANCE * centred |
250 | 374 |
|
251 | 375 |
|
252 | 376 | def final_score(p_vec: float, p_lex: float, importance: float, created: float, |
253 | 377 | vec_rank: int | None = None, lex_rank: int | None = None, |
254 | 378 | now: float | None = None, kind: str = "fact", |
255 | | - affirmed: float | None = None) -> float: |
256 | | - return evidence(p_vec, p_lex, vec_rank, lex_rank) * ( |
| 379 | + affirmed: float | None = None, |
| 380 | + newer_share: float | None = None, |
| 381 | + half_life_share: float = RECENCY_HALF_LIFE_SHARE) -> float: |
| 382 | + return aged_evidence(p_vec, p_lex, newer_share, vec_rank, lex_rank, |
| 383 | + half_life_share) * ( |
257 | 384 | 1.0 + prior(importance, created, now, kind=kind, affirmed=affirmed)) |
0 commit comments