Smarter KV techniques (e.g. DKV)

[TomLucidor]

Saw this thing called "Differential KV" that seems to be better than TurboQuant and similar, but there might be more developments in the next few years that I think maybe Rapid-MLX could consider checking out. But this would require benchmarking pipes to see if each advancement is "worth it" Omc12/Differential-KV#2 (reply in thread)

[raullenchai]

@TomLucidor Thanks for the pointer, and sorry this went unanswered. I've added #5645 as a research-backlog item for Differential KV. Before considering integration, we need to check the implementation/license and MLX/Metal applicability, then define correctness/quality, memory, TTFT and decode comparisons against the current cache paths. Interactions with prefix reuse, sliding-window/hybrid caches and speculative decoding also matter. No advantage or compatibility is established yet; this is tracked for assessment, not a promise to ship it.

[raullenchai]

Research assessment is complete in #5645: pinned-source assessment and evaluation criteria.

Disposition: a separately approved, isolated evaluation may be worthwhile; production/default integration is not approved. No Rapid-MLX quality, memory, latency, or compatibility advantage has been established. The inspected approach changes attention behavior through approximate reconstruction; it is not a lossless cache serialization change. Existing clone/rollback code does not establish compatibility with Rapid-MLX prefix reuse, cancellation, continuous batching, hybrid/sliding attention, or speculative rejection.

Before experiments, quality tolerances must be fixed against the same checkpoint, quantization and tokenized inputs. Any evaluation must establish actual compression activation, held-out quality, active/peak memory, TTFT versus first-visible latency, and client versus native decode timing before lifecycle/serving qualification. Unsupported combinations must remain disabled. The source revision was pinned; the deposited paper PDF revision was not independently verified.

No experiment, model download, product change, benchmark submission, or production authorization is part of this closeout. Future evaluation or integration needs its own scoped decision.