cachedb_perf: high-performance local memory cache built on modern kernel features

[Lt-Flash]

Summary

This PR introduces cachedb_perf — a new, from-scratch cachedb backend for large, high-churn local caches, selected by URL scheme (perf://). I'm trying to build a much faster local cache module by combining a cache-conscious, lock-free-read design (CLHT / MemC3 lineage) with what recent Linux kernels make possible: overcommit hugetlb pools, shmem THP, MADV_COLLAPSE, MADV_POPULATE_WRITE and swap pinning.

It implements the same cachedb_funcs vtable as every other backend, so any module taking a cachedb_url works unchanged, and core script usage (cache_store("perf", ...)) only changes the backend name. v1 is a single-node in-memory cache with optional db_* persistence — whole collections can be saved to and loaded from an SQL backend, so state survives a restart (see the DB persistence section below). What it deliberately does not do is cachedb_local-style per-operation replication (cluster_id write-through); cross-node sharing is instead a shared-DB refresh model (the perf_sync cluster-sync below), not a streamed op log.

The module is functionally complete (data ops, expiry, runtime growth, statistics, huge-page arena, a full introspection MI, observability events, db_* persistence, and a perf_sync cluster refresh), validated by built-in selftests, a script-level end-to-end suite, and a multi-process correctness soak.

Motivation

Found while benchmarking topology_hiding's cacheDB state backend (#4114): setting cache_collections "th=16" cut the load balancer's CPU from 45% to 29% at 4000 CPS. The root cause is structural: cachedb_local's hash table is sized once from cache_collections and never resized. The default is 512 buckets, most deployments never set the parameter, and at 50 000 entries that is a load factor of ~98 — roughly 50 string compares and 50 dependent cache misses per lookup. The module also exports zero statistics, so the cliff is invisible in production.

Rather than progressively rewriting a module every deployment depends on, this is a clean backend: operators opt in per collection by changing a URL.

Configuration

Every parameter is optional — with none set you get a single default collection at 16384 buckets, plain shm, growth and the expiry sweep on. In practice you declare the collection(s) you use and point consumers at them.

modparam	type	default	what it does
cache_collections	string	default (14 → 16384 buckets)	Declares collections as name or name=size, ;-separated. size is the log2 of the initial bucket count, clamped to [4, 24]; the table grows past it at runtime, so it is a starting point, not a ceiling. The cachedb_local /r replication marker is rejected — this cache is single-node.
cachedb_url	string	perf:// (the default collection)	Connection URL(s) that scripts and other modules resolve. The collection is the URL's db part (perf:///th) or, equivalently, its host part (perf://th); perf:// alone selects default. Prefix a group (perf:grp:///th) to address a specific URL from the script. Naming an undeclared collection is a startup error. Repeatable.
expiry_sweep_period	int (seconds)	1	How often expired records are reclaimed. Expiry itself is instant — an expired entry reads as absent the moment it lapses; the sweep only frees the memory, hint-guided so idle collections cost next to nothing. 0 disables reclamation (expired entries hold their memory until overwritten).
growth_load_factor	int	2	Target entries-per-bucket the maintenance timer grows the table toward — the knob that keeps the ~84 ns bucket shape as the cache scales. 0 disables growth (fixed-size table, i.e. cachedb_local behaviour).
growth_budget	int	4096	Cap on bucket splits per maintenance tick, so a single growth pass can never stall the timer.
arena_hugepage_mb	int (MB)	0 (off)	Size of the 2 MB huge-page arena reservation (mlock-pinned, created pre-fork, shared by all workers). Chases the best tier by trying the hugetlb → THP → MADV_COLLAPSE → 4K ladder; 0 uses plain demand-faulted shm. Wants LimitMEMLOCK=infinity; warns and continues unpinned otherwise.
arena_selftest	int	0	Run the arena selftest at startup and fail startup on any mismatch. Permanent, cheap diagnostic.
htable_selftest	int	0	Run the hash-table + growth selftest at startup and fail startup on any mismatch.
db_url	string	(unset)	A db_* (SQL) backend to persist collections to (the matching db_* module must be loaded). Unset = no persistence.
db_table	string	cachedb_perf	Table holding the persisted rows.
db_mode	int	0 (off)	Automatic persistence for persist_collections: 1 = load at startup, 2 = load at startup + save on graceful shutdown. 0 = MI-only.
persist_collections	string	(none)	CSV of collections that db_mode auto-loads/saves. perf_save/perf_load still work on any collection on demand.
sync_cluster_id	int	0 (off)	Cluster to signal on perf_sync (needs clusterer loaded + a db_url). Soft: with either missing, perf_sync degrades to a DB save.

The motivating setup — topology_hiding state backend:

loadmodule "cachedb_perf.so"
modparam("cachedb_perf", "cache_collections", "th=16")   # 2^16=65536 buckets to start; grows with call volume
modparam("cachedb_perf", "expiry_sweep_period", 1)
modparam("cachedb_perf", "arena_hugepage_mb", 512)       # optional: 512 MB of 2M huge pages

loadmodule "topology_hiding.so"
modparam("topology_hiding", "th_state_url", "perf:///th")
# What the cache serves is whatever has no dialog. If a dialog exists by
# the time topology_hiding() runs, the dialog carries the state instead -
# and that includes a dialog the script created itself with
# create_dialog(), not just force_dialog=1. So on a proxy that creates
# dialogs for its calls, the cache serves the non-INVITE dialogs
# (SUBSCRIBE/NOTIFY/OPTIONS/INFO/PUBLISH) and the dialog serves the calls,
# which is the right split: a dialog's lifetime is the call's own, it is
# torn down on BYE rather than waiting out a TTL, it is replicated by the
# dialog module, and Call-ID hiding needs it. On a load balancer that
# wants no dialogs at all - no accounting, no BYE handling, nothing to
# replicate - leave force_dialog=0 and do not call create_dialog(), and
# INVITE state flows through the cache too (bounded by Session-Expires,
# else th_state_ttl); at high call rates that avoids paying for a dialog
# per call. See the topology_hiding docs for the full trade-off.

Generic script cache + glob operations:

modparam("cachedb_perf", "cache_collections", "sessions")
modparam("cachedb_perf", "cachedb_url", "perf:///sessions")   # groupless default -> "sessions"

route {
    cache_store("perf", "session-$ci", "$var(state)", 3600);   # -> "sessions"
    cache_fetch("perf", "session-$ci", $var(state));
    # the collection arg is optional: omitted, the glob ops use the groupless
    # cachedb_url's collection (here "sessions") - the same place cache_store
    # above writes. Pass it explicitly to target another collection.
    perf_del("session-$ci-*", "sessions");                  # glob delete -> count
    perf_mget("session-*", $avp(k), $avp(v), "sessions");   # matches -> index-paired AVPs
}

MI commands

The full management interface — the operator visibility cachedb_local never had. Every command is lock-free (seqlock reads), so a key scan or dump never stalls SIP traffic; every name carries the perf_ prefix to match the script functions and stay clear of the core's bare get/set. Each is invoked module-namespaced as cachedb_perf:<command> (OpenSIPS 4.0+). Arguments in <> are required, [] optional; an omitted collection means the groupless cachedb_url's collection (where cache_store("perf", …) writes).

command	arguments	returns / effect
perf_stats	[collection]	per-collection stats: entries, buckets, load factor, overflow occupancy, hits/misses/stores/removes/expired/destroyed, hit rate (with a note that reflects the measured value), seqlock retries (and per-1k-reads), plus arena bytes/chunks and the achieved memory tier. No arg = every collection
perf_stats_reset	[collection]	re-baseline the cumulative counters so the next reading covers a fresh interval instead of a lifetime average — useful after a restart, when the miss burst from dialogs older than the cache drags the hit rate down long after it has recovered. The counters are never rewound (each process owns its counter cache line); only a baseline is recorded and the difference reported. Live gauges — entries, buckets, overflow, load factor, arena — are unaffected
perf_keys	<glob> [collection] [limit]	names and TTL of keys matching the shell glob, bounded (default limit 1000; the reply flags truncation). The KEYS equivalent
perf_scan	<cursor> [glob] [count]	cursor-based incremental iteration (Redis SCAN) over the default collection: start at cursor 0, repeat with the returned cursor until it comes back 0. count bounds the buckets visited per call. The answer for a large cache, where perf_keys would truncate
perf_dump	<glob> [collection] [limit]	like perf_keys but includes values — opt-in, never the default
perf_get	<key> [collection]	one key: its value, remaining TTL (-1 = never) and size
perf_set	<key> <value> [ttl] [collection]	write one key; ttl in seconds (0 or omitted = never expires)
perf_ttl	<glob> <ttl> [collection]	re-arm the TTL of every key matching the glob without rewriting the value (the versionless bump — one atomic expires store, readers undisturbed); ttl in seconds (0 = never). Returns the count updated. A literal key matches exactly one
perf_del	<glob> [collection]	delete every key matching the glob; returns the count. The MI face of the perf_del() script function
perf_save	[collection]	snapshot a collection to the db_url backend (all declared collections if none named)
perf_load	[collection]	restore a collection from the db_url backend
perf_sync	[collection]	save to the DB, then signal the cluster to reload it (save-then-broadcast); also a script function

MI parameters are named, so any sensible subset resolves — e.g. perf_keys <glob> limit=N without a collection, or perf_set <key> <value> collection=C without a ttl.

Note the argument order: the glob-taking commands (perf_keys, perf_dump, perf_del, perf_ttl) take the glob first and the collection second, so perf_dump mycoll looks for keys named mycoll rather than dumping that collection — use perf_dump "*" mycoll. Only perf_get/perf_set lead with a key. Quote globs, or the shell expands them before opensips-cli sees them.

opensips-cli -x mi cachedb_perf:perf_stats
opensips-cli -x mi cachedb_perf:perf_stats_reset            # fresh interval for the rates
opensips-cli -x mi cachedb_perf:perf_stats_reset th         # just one collection

opensips-cli -x mi cachedb_perf:perf_keys "*"               # every key in the default collection
opensips-cli -x mi cachedb_perf:perf_keys "session-*" th 50 # glob, collection, limit
opensips-cli -x mi cachedb_perf:perf_scan 0                 # then: …:perf_scan <returned-cursor> … until 0
opensips-cli -x mi cachedb_perf:perf_dump "profile-*"       # names AND values
opensips-cli -x mi cachedb_perf:perf_dump "*" th 20         # a sample of one collection

opensips-cli -x mi cachedb_perf:perf_get session-abc123
opensips-cli -x mi cachedb_perf:perf_set greeting hello 300
opensips-cli -x mi cachedb_perf:perf_ttl "session-*" 1800   # re-arm matching keys to 30 min
opensips-cli -x mi cachedb_perf:perf_del "session-abc*"

perf_stats counters are cumulative since startup (or the last perf_stats_reset), so a hit rate read straight after a restart is dominated by sequential requests for dialogs older than the cache and recovers only as those age out. Either reset once the cache has warmed, or — better for monitoring — poll twice and difference the counters.

Events

Four EVI events let a script or monitor react to the cache. Each is gated by evi_probe_event(), so with no subscriber it costs one shared read, and none sit on the lock-free get/set path.

event	when	parameters
E_CACHEDB_PERF_EXPIRED	a record was reaped (opt-in per collection via event_expired_collections)	collection, key
E_CACHEDB_PERF_NOMEM	a write was dropped because the arena is full	collection, key, size
E_CACHEDB_PERF_GROWN	the table grew itself	collection, prev_buckets, buckets, splits, entries
E_CACHEDB_PERF_MEM_DEGRADED	huge pages requested but the arena landed below hugetlb (once at boot)	requested_mb, tier, backing, overcommit_pages
E_CACHEDB_PERF_SYNCED	this node reloaded a collection because a peer issued perf_sync	collection, source_node

event_route[E_CACHEDB_PERF_NOMEM] {
    xlog("L_ERR", "cachedb_perf full: dropped $param(key) ($param(size) B) in $param(collection)\n");
}

DB persistence

With db_url set, a whole collection can be persisted to any db_* (SQL) backend. The DB is a shared, durable store; the cache is an in-memory view over it. A save is a full snapshot — the collection's rows are deleted and every live entry re-inserted; a load restores them. TTLs are stored as absolute wall-clock time so they survive a restart (the cache's own expiry is monotonic ticks, which reset on reboot); already-expired rows are skipped on both save and load, and native counters round-trip as their decimal value. This is single-node durability; cross-node sharing over the same DB is perf_sync (below), still not per-operation replication.

loadmodule "db_mysql.so"
modparam("cachedb_perf", "cache_collections", "sessions")
modparam("cachedb_perf", "db_url", "mysql://opensips:pw@localhost/opensips")
modparam("cachedb_perf", "db_mode", 2)              # load at startup, save on graceful shutdown
modparam("cachedb_perf", "persist_collections", "sessions")

# on demand, from opensips-cli:
opensips-cli -x mi cachedb_perf:perf_save sessions   # -> {"collections":1,"saved":N}
opensips-cli -x mi cachedb_perf:perf_load sessions   # -> {"collections":1,"loaded":N}

The table (default cachedb_perf) has four columns: collection (string), pkey (string), pvalue (BLOB, binary-safe), expires (int, absolute unix time, 0 = never). The startup load runs before the workers fork, so every worker starts with a warm cache.

⚠️ A save/load is a full, blocking snapshot — one SQL statement per entry, synchronous in the issuing process. It is a maintenance / bootstrap operation — startup warm-up, shutdown flush, an occasional snapshot or a perf_sync refresh — never on a per-request path or a tight timer. If you need durable per-key writes on every operation, this is the wrong tool.

The snapshot is one transaction — measured

A save is one statement per row, so on a backend that commits each one separately it is not merely slow, it does not finish. Before this was addressed, 30 000 entries to db_sqlite ran at ~140 rows/s, blew the 60 s SHUTDOWN_TIMEOUT, and the core aborted the process with 7 918 of 30 000 rows written — and since a save deletes the collection first, what remained was a partially written table where a complete snapshot had been, with nothing in the log or the reply to say so.

The whole snapshot now runs inside one transaction, which fixes both halves:

backend, 30 000 entries	before	after
db_sqlite	aborted at the 60 s watchdog, 7 918 rows	0.35 s — 86 000 rows/s
db_redis	5.2 s	5.05 s — 5 900 rows/s

The ~600× on SQLite is the per-row fsync disappearing; it was never slow at inserting. db_redis has no raw_query, so it takes no transaction — it has no per-row commit to amortise, but it does pay a network round trip per row, which is why it is now the slower of the two.

More important than the speed: the delete and the inserts are atomic. Uncommitted work is rolled back when the connection closes, so an interrupted or failed save now leaves the previous snapshot in place rather than destroying it. The error paths return without committing deliberately.

No single spelling starts a transaction everywhere, so both are tried — SQLite and PostgreSQL take BEGIN TRANSACTION, MySQL takes START TRANSACTION. The bare BEGIN all three accept is unusable: db_sqlite's raw_query only reaches its exec path for statements at least as long as select, and mis-parses anything shorter as a SELECT.

A save also logs its duration and rate, and warns past 10 s that a shutdown save has a 60 s budget — noting when the backend took no transaction, since that is the case where the budget is at risk.

With db_redis

loadmodule "db_redis.so"
modparam("cachedb_perf", "db_url", "redis://127.0.0.1:6379/0")
modparam("cachedb_perf", "db_mode", 2)
modparam("cachedb_perf", "persist_collections", "default")

db_redis needs a schema declared for the table before first use — it fails at load without one, and none ships for cachedb_perf:

redis-cli -n 0 HSET schema:cachedb_perf \
  __cols "collection pkey pvalue expires" __pk pkey \
  collection string pkey string pvalue string expires int

Its primary key is a single column while a row here is identified by (collection, pkey), so persist one collection: with two, identical key names would collide and the later save would overwrite the earlier. The /0 database component is required by the core URL parser even though Redis-cluster mode ignores it.

Cluster sync

perf_sync [collection] (MI and script function) builds on the same DB: it saves the collection, then signals the cluster over clusterer to reload it — one message per sync, not per operation, so the hot path is untouched. A reload overwrites a peer's copy from the DB, so it's for single-writer / read-replica topologies (one authority updates the DB, the others refresh); a node reloads and raises E_CACHEDB_PERF_SYNCED. With no clusterer / sync_cluster_id 0 it degrades to a DB save. Same blocking cost as perf_save, so: an occasional refresh, not a live primitive.

loadmodule "clusterer.so"          # before cachedb_perf (a soft dep also enforces init order)
modparam("clusterer", "my_node_id", 1)          # 2, 3 on the other nodes
loadmodule "cachedb_perf.so"
modparam("cachedb_perf", "cache_collections", "profiles")
modparam("cachedb_perf", "db_url", "mysql://opensips:pw@dbhost/opensips")
modparam("cachedb_perf", "sync_cluster_id", 1)

# on the authority, after it updated "profiles":
#   opensips-cli -x mi cachedb_perf:perf_sync profiles   # save + tell peers to reload
#   perf_sync("profiles");                                # same, from script

event_route[E_CACHEDB_PERF_SYNCED] {   # fires on each replica after its reload
    xlog("L_INFO", "reloaded $param(collection) from node $param(source_node)\n");
}

The capability registers with the clusterer, so it shows in its clusterer_list_cap MI — verified on a single-node cluster (with cachedb_perf loaded before clusterer, confirming the soft dependency reorders init):

$ opensips-cli -x mi clusterer_list_cap
{
  "Clusters": [
    { "cluster_id": 1,
      "Capabilities": [
        { "name": "cachedb-perf-sync", "state": "Ok", "enabled": "yes" }
      ] } ]
}

The allocation-free read — get_buf (new optional cachedb endpoint)

Profiling the read path (production F_MALLOC allocator) showed the biggest removable cost is not the lookup — it is the vtable contract. get() must return a value the caller owns and pkg_free()s, so every hit pays a pkg_malloc + pkg_free + second memcpy that exist only to satisfy ownership: a fixed ~70 ns/op (15% of a 470 ns lookup over 50 000 entries; 30% of a 236 ns cache-resident one). memcpy itself never even appears in the profile.

So this PR adds a small, optional core endpoint — get_buf(), advertised by CACHEDB_CAP_GET_BUF — that reads into a buffer the caller already owns: no allocation, one copy instead of two. Any backend may implement it; every caller keeps get() as the fallback, and nothing about the existing endpoints changes (in particular get_counter() keeps its documented {-2,-1,0} return set).

50k entries, 200-byte values, 100% reads	run 1	run 2	run 3	median
get() ns/op	395.3	414.2	397.7	397.7
get_buf() ns/op	290.1	271.0	372.7	290.1 (−27%)

Runs alternate get/get_buf in one binary to cancel warm-up bias; run-to-run variance on this box is real (one pair shows only −6%), so the honest claim is 20–27%.

The contract is written for the failure cases, since those are what callers get wrong: the buffer must be private to the calling process; it may be written speculatively and abandoned, so its contents are undefined on any non-hit; *vlen/*needed are zeroed before anything else; and a value that does not fit reports its size via *needed while *vlen stays 0 — {buf, *vlen} is always a valid str. Inside cachedb_perf both entry points share one implementation of the seqlock read (optimistic loop, lock fallback, re-route retry, overflow leg, expiry), so they cannot disagree about which record they return.

The first consumer is topology_hiding's th_state_url path (#4114): the conversion is written and tested (both the in-place path and the too-small→allocated fallback), and will be pushed to that PR once this one merges, since it needs this core endpoint to compile.

Enabling the kernel memory backing

The huge-page arena (arena_hugepage_mb) climbs a detect-by-trying ladder at mod_init: it attempts each tier in turn and keeps the best one the running kernel actually grants — you do not pick a tier, you enable what you can and the module reports what it got. The tiers, fastest to slowest (the cost is the isolated 2 MB pointer-chase from §5 of the study):

tier	kernel feature the module uses	one-time admin action	cost (2M chase)	swap-pinning
1 (fastest)	overcommit hugetlb pool + MAP_HUGETLB	one sysctl	177 → 125 ns (1.42×)	inherent — hugetlb is unswappable, no mlock needed
2	shmem THP + MADV_HUGEPAGE	one sysfs write	177 → 158 ns	via mlock (see below)
3	MADV_COLLAPSE after fill	none (kernel ≥ 6.1)	177 → 156 ns	via mlock (see below)
4 (baseline)	plain demand-faulted 4 KB	—	177 ns	via mlock (still reserved+pinned)

Tier 1 — overcommit hugetlb (the one to prefer: on-demand, nothing held while the cache is small, and no memlock grant needed). Allow enough on-demand 2 MB pages for the arena (arena_hugepage_mb / 2, plus a small margin):

sysctl -w vm.nr_overcommit_hugepages=320          # e.g. a 512 MB arena = 256 pages + margin
echo 'vm.nr_overcommit_hugepages = 320' > /etc/sysctl.d/60-opensips-hugepages.conf

Tier 2 — shmem THP (used if tier 1 is unavailable). Put shmem THP in advise so it honours the module's MADV_HUGEPAGE:

echo advise  > /sys/kernel/mm/transparent_hugepage/shmem_enabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

Tier 3 — MADV_COLLAPSE needs no sysctl (kernel ≥ 6.1); on some 6.12 builds it also wants tier 2's shmem_enabled=advise. Tier 4 is the default and needs nothing.

Swap-pinning (mlock) — tiers 2–4 only. When tier 1 is unavailable the arena is a regular shared mapping, which the module mlock-pins pre-fork so it can't be swapped out from under the lock-free readers. systemd's default LimitMEMLOCK=65536 (64 KB) makes that mlock fail on any real arena — the module then warns and runs unpinned (swappable); the huge pages still form, they are just not pinned. Tier 1 (MAP_HUGETLB) is exempt and needs none of this. To pin tiers 2–4, grant it once:

mkdir -p /etc/systemd/system/opensips.service.d
printf '[Service]\nLimitMEMLOCK=infinity\n' > /etc/systemd/system/opensips.service.d/memlock.conf
systemctl daemon-reload

Turn it on and confirm what landed:

modparam("cachedb_perf", "arena_hugepage_mb", 512)   # 0 (default) = plain shm, tier 4

opensips-cli -x mi cachedb_perf:perf_stats     # -> memory_tier (1 hugetlb .. 4 plain 4K) + memory_backing

mod_init also logs the achieved tier and, when it falls short of tier 1, the exact sysctl to reach it and the measured cost of running without it.

The study

Everything below was measured, not assumed — the benchmark rig ships in-tree (modules/cachedb_perf/bench/, make run, no OpenSIPS build needed) and every figure is reproducible. Hosts: Xeon E5-2699 v4, kernels 5.4 / 6.8 / 6.12; the NUMA numbers come from a vNUMA-pinned two-socket guest on the same silicon. The rig models structures and cache behaviour (single process, threads); it ranks designs rather than predicting server throughput.

1. The index structure

design	@512 buckets (shipped default)	@65536 buckets
chained + strncmp (cachedb_local today)	2484 ns	111 ns
chained + hash cached in node	1837 ns	86 ns
sorted array per bucket + binary search	134 ns	100 ns
64B cache-line bucket + 1-byte tags (this module)	84 ns
flat open addressing (rejected: stop-the-world resize)	78 ns

Load factor alone is a 20× spread. The chosen design is within 8% of the fastest structure measured, and the fastest one (flat open addressing) is impossible to resize across processes in shm. Also checked: core_hash() is not at fault (chi²/df 0.65–1.18 vs FNV-1a on thids, dialog ids, AoRs and call-ids — statistically indistinguishable), so the module keeps it.

2. Concurrency — an honest negative result

The hypothesis was that cachedb_local's write-lock-on-every-read destroys scaling. It does not: with a well-sized table workers rarely collide on a bucket lock, and it scales 8.4× on 8 threads. The 4× gap is a per-operation constant factor (no atomic RMW on reads, one cache line per bucket, tag filtering) — not a scaling win. Against the shipped 512-bucket default the gap is ~90×.

3. The read protocol — measured before being believed

Readers take no locks: a per-bucket seqlock with bounded retries and a sleeping-lock fallback. What makes this legal in OpenSIPS specifically: shm is mapped once before fork and never unmapped, so a stale pointer read is garbage-but-not-a-fault, and the version re-check discards it — the value is always copied out inside the optimistic section, with every length clamped and every pointer extent-checked before use.

The one credible alternative (QSBR / pointer-publication, no version check at all) was implemented in the rig and rejected on the numbers: identical at 100% reads (on x86/TSO the version loads hit the already-loaded bucket line — the seqlock is free) and ahead only under single-hot-bucket write contention that SIP traffic doesn't exhibit (seqlock retries measured at 1.2 per 1000 reads on a uniform 95/5 mix). The useful piece survived without any grace-period machinery: a byte-identical set() that only refreshes the TTL — the dominant write in the motivating workload — takes the bucket lock but skips the version bumps and the memcpy entirely. One atomic expires store; concurrent readers of the bucket are undisturbed.

4. What was rejected: write staging and queueing

queued writes, 8-thread budget	applied Mops/s	vs direct	ring full
8 direct writers	116.3	1.00×	—
7 producers + 1 consumer	24.1	0.21×	99%
4 producers + 4 consumers	54.6	0.47×	97%

A shared staging buffer loses throughput as threads are added — one atomic append offset is a hotter point of coordination than thousands of bucket locks. Queued writes do less than half the work of writing directly, and break read-your-writes semantics. The rule this established shapes the whole module: per-process regions win for allocation (the arena uses them — zero atomics on the alloc fast path), but never for staging live entries.

5. Modern-kernel memory backing

OpenSIPS shm today is demand-faulted 4K pages — no MAP_POPULATE, no hugepages, no madvise, no mlock anywhere in mem/. A multi-hundred-MB cache pays for that in TLB misses. Four routes to 2M pages, ranked as a runtime fallback ladder:

route	admin action	6.8	6.12	chase latency
vm.nr_overcommit_hugepages + MAP_HUGETLB	one sysctl	works, no reservation held	works	177→125 ns (1.42×)
THP-shmem: shmem_enabled=advise + MADV_HUGEPAGE	one sysfs write	works	works	177→158 ns
MADV_COLLAPSE after fill	none	works even with shmem_enabled=never	EINVAL — needs advise	177→156 ns
plain 4K	—	—	—	baseline

Key findings:

• Overcommit hugetlb removes the classic reservation objection: pages are taken from free memory at fault time and returned on exit — nothing is held hostage while the cache is small. Pre-faulting is also 4–5× cheaper at 2M granularity.
• Every tier is detected at runtime by trying it, never by kernel version — the 6.8/6.12 MADV_COLLAPSE divergence proves version checks lie, and they lie in both directions: a later 6.12 (6.12.96, Debian 13) collapses fine with shmem_enabled=never again — same major version, opposite behaviour, and the probe silently got the better tier. The module already does this in mod_init: it reports the achieved tier and, when hugetlb is unavailable, logs the exact sysctl and the measured cost of running without it.
• Two kernel subtleties learned the hard way (both documented in the code comments): shmem THP requires the VA and the shmem file offset to be congruent mod 2M (a range VA-aligned inside an unaligned mapping is silently ineligible — THPeligible: 0, collapse EINVAL; the fix is reserve PROT_NONE, then MAP_FIXED the shmem at a 2M boundary), and a shmem MADV_COLLAPSE creates the huge folio without PMD-mapping the caller — verify via the ShmemHugePages meminfo delta, not smaps.
• 1 GB pages: ruled out (runtime allocation unobtainable after any uptime, and 2M pages already give a ≤1 GB arena full STLB residency on this class of hardware).
• Swap pinning via mlock in mod_init works across fork (locks are not inherited, but the pages are shared — one pre-fork lock pins the arena for every worker). Found a production blocker on our own SBCs while checking: systemd's default LimitMEMLOCK=65536 means mlock of any real arena fails — the unit needs a one-line drop-in.

6. Expiry

strategy	per sweep (50k entries, ~13 due)	locks/sweep
full sweep, lock every bucket	1.31 ms	65 536
per-bucket min_expires, unlocked skip	0.044 ms	13
timer wheel, O(expired)	0.0005 ms	—

Even the full sweep is 0.13% of a core — expiry is a memory reclamation problem (entries squatting up to cache_clean_period), not a CPU problem. The min_expires hint gets 30× for zero hot-path cost; the wheel's further 84× buys nothing and costs 74 ns/insert plus 16 B/entry.

7. NUMA — measured on a pinned two-socket testbed

binding	dependent pointer chase
local (same socket)	146.5 ns
remote (cross-socket)	194.6 ns (+33%)

Measured in-guest on a Proxmox VM with vNUMA bound per host socket (numaN: ...,hostnodes=N,policy=bind, dual E5-2699 v4 host — plain numa: 1 without hostnodes fabricates topology over one memory domain and measures nothing). Two consequences, both already reflected in the design rather than motivating changes:

• Cross-socket reads of a shared cache cannot be sharded away — a worker reading an entry written on the other socket pays the remote latency however memory is partitioned; only replication avoids it. So the table is deliberately not NUMA-sharded (and §2 shows there is no lock contention for sharding to relieve either).
• The write side is node-local by construction: the arena's per-process chunk ownership means each worker faults — and therefore first-touch places — its own records on its own node.

Where NUMA does matter for the roadmap: page walks against remote memory amplify TLB-miss cost, so the huge-page backing is expected to be worth more on two sockets than the 1.42× measured on one — to be quantified in the end-to-end benchmark. One refinement from the same testbed: with pdpe1gb exposed, 1 GB pages are allocatable at runtime on a fresh boot (2 granted right after boot) — the earlier "unobtainable" holds only once uptime fragments memory. The ruling against them stands on arithmetic: 2 M pages already give a ≤1 GB arena full TLB coverage on this hardware.

Measured: cachedb_perf vs cachedb_local, in-process

The bench rig above ranks designs in a single process. This measures the real modules — real OpenSIPS 4.1-dev, real shared memory, N real worker processes — driving the th_store access pattern (16-byte thid keys, 200-byte values, 95% get / 5% set). Both backends did byte-for-byte identical work (same 22.8 M hit count). Release build (-O3), Q_MALLOC, pinned to one 8-core socket.

Same conditions — both collections sized to 65536 buckets (th=16):

condition	cachedb_perf	cachedb_local	perf faster
no load (near-empty), 8 workers	29.7 Mops/s (271 ns/op)	9.9 Mops/s (812 ns)	3.0×
50 000 resident, 8 workers	18.0 Mops/s (448 ns/op)	7.9 Mops/s (1013 ns)	2.3×
50 000 resident, 1 worker	1.8 Mops/s (558 ns)	1.2 Mops/s (853 ns)	1.5×

Two effects drive the gap. Scaling: cachedb_perf's lock-free reads scale 10.0× from 1→8 workers vs cachedb_local's 6.6× (it takes a bucket lock on every read). The default: most deployments never set cache_collections, so cachedb_local runs at its 512-bucket default — at 50k entries that is a load factor of ~98, 3529 ns/op, and cachedb_perf is 7.8× faster than cachedb_local as typically shipped.

at 50 000 resident, 8 workers	ns per operation
cachedb_perf (65536 buckets)	448 ns
cachedb_local, tuned (65536 buckets)	1013 ns
cachedb_local, default (512 buckets)	3529 ns

Honest notes: numbers include a real pkg_malloc+free of the 200-byte value on every get (the th_store copy-out), so this is per-operation cost, not a bare lookup. This deliberately isolates the cache from the SIP layer. In a full LB the per-call cost is SIP parsing, header manipulation and transaction state plus the th_store put/get — there is no per-call encryption (th_store values are stored in the clear; the only crypto is one cheap MD5 to derive the key), so the cache is a direct share of that cost. An end-to-end run under 50 000 held calls confirms the direction below.

Is it only fast at reads? The mix swept, with cachedb_redis for scale

The numbers above use the th_store access pattern (95% get / 5% set), which invites a fair question: is this just a read cache that gives the win back on writes? It is not. Sweeping the read/write mix with cdbbench driving the cachedb_funcs vtable directly — no consumer module in the path — gives:

ns/op (median of 3), 50 000 entries, 200-byte values	1 worker	4 workers	8 workers
100% writes — cachedb_perf	562	639	632
100% writes — cachedb_local	3123	4239	6226
100% writes — cachedb_redis (loopback)	161 138	257 458	366 320
50/50 — perf / local	401 / 2638	549 / 3527	546 / 5014
95% reads — perf / local	328 / 2583	404 / 2866	385 / 2775

cachedb_perf is 5.6–9.9× faster than cachedb_local on pure writes, and the margin widens with concurrency — cachedb_perf stays flat (562 → 639 → 632 ns as workers go 1 → 4 → 8) while cachedb_local degrades (3123 → 4239 → 6226). Writers in both take a bucket lock; the difference is what happens inside it. cachedb_local's add/set path parses the stored value, reformats it and reallocs the entry under the lock, so the critical section grows with contention; cachedb_perf writes fixed-width fields in place under a seqlock bracket.

cachedb_redis over loopback is 250–580× slower here. That is not a criticism of Redis — it is the cost of a synchronous round trip per operation, and it is exactly why a local cache exists. Use Redis when state genuinely must be shared between nodes; use perf:// when it must not leave the box.

End-to-end: TH under 50 000 held calls

Same LB, topology_hiding with th_state_url pointing at each backend (65536 buckets), ramped while holding ~50 000 concurrent calls. "Sustained" means <5% failures and peak concurrency ≤75k (actually still holding 50k, not backlogging):

offered CPS	th + cachedb_perf	th + cachedb_local
4000	3875 achieved, 3.0% fail, 54k held	3941 achieved, 1.4% fail, 57k held
6000	5775 achieved, 3.7% fail, 68k held ✓	5490 achieved, 8.5% fail, 94k (backlogging) ✗
8000	6670, 16.6% fail (overloaded)	5874, 26.6% fail (overloaded)

cachedb_perf sustains the 6000-CPS rung where cachedb_local breaks — a sustained-ceiling lift from ~3941 to ~5775 CPS (~1.5×) at 50k live th_store states. The end-to-end gain is smaller than the isolated-cache 2.3–7.8× because SIP processing is the larger share of per-call cost, but it lands exactly where the cache matters: the high-concurrency point where cachedb_local's lock-on-every-read serializes the workers.

At 100 000 concurrent calls: cachedb_perf-TH vs dialog-TH vs no-TH

Pushing to 100 000 held calls, comparing three topology-hiding strategies on the same LB (huge pages / THP enabled): cachedb_perf-backed th_store, the in-memory dialog module (force_dialog), and plain record-routing with no topology hiding at all.

offered CPS (≈100k held)	no-TH (rr)	cachedb_perf-TH	dialog-TH
2000	55% CPU, 0% fail	67% CPU, 2.9%	85% CPU, 0.1%
3000	73% CPU, 0.1%	—	100% CPU, 1.0%
4000	96% CPU, 0.1%	93% CPU, 1.3%	100% CPU, 8.2% (breaks)

At 100k concurrency cachedb_perf-TH is nearly as cheap as doing no topology hiding at all — it tracks the no-TH curve and holds 4000 CPS at 93% CPU. dialog-TH is the loser here: it saturates CPU by 3000 CPS, breaks at 4000, and carries ~2.5× the resident memory (a full per-dialog state machine + timers vs one compact th_store entry). This is a crossover from lower concurrency, where dialog leads — cachedb_perf's flat per-entry cost wins as the live-state count climbs.

Caveats: the single load generator is unstable at 100k (some cachedb_perf mid-rungs showed generator-side failures at low LB CPU — discarded); and the huge pages here are whole-shm THP that benefits all three equally — the module's own huge-page arena is a separate mechanism, measured on its own in the next section.

Re-verified end to end, with the backend's own counters as proof

The runs above were re-done with one addition: every rung asserts, over MI, that the backend under test actually did the work — cachedb_perf:perf_stats must show stores in the th collection and zero dialogs; the dialog arm must show dialog:processed_dialogs and zero cache stores; the no-TH arm neither. A rung that fails its assertion is reported invalid and discarded rather than silently contributing a number. All twelve rungs passed.

50 000 held calls	no TH	cachedb_perf-TH	dialog-TH
CPU @2000 CPS	41%	43%	41%
CPU @3000 CPS	47%	52%	60% — 3.1% calls lost
CPU @4000 CPS	54%	58%	69% — 10.3% calls lost
RSS @4000 CPS	6.6 GB	6.9 GB	12.1 GB
peak concurrency @4000	49 989	50 033	56 504 (backlogging)

At 4000 CPS cachedb_perf-backed topology hiding costs 4 CPU points and 4% more memory than doing no topology hiding at all, while holding concurrency at exactly 50k. Dialog-backed hiding costs 15 CPU points, 1.8× the memory, and is dropping 10.3% of calls — its rising "concurrency" is a backlog, not held calls. This is the same result as the 100k run above, now with the cache proven to have been exercised rather than assumed.

Huge-page arena (CP-20)

The arena can now back its chunks with 2 MB huge pages instead of 4 K (modparam arena_hugepage_mb; the reservation is 2M-aligned, mlock-pinned, created pre-fork and shared by all workers). Measured on the real module (−O3, 8 workers, medians of repeated runs):

condition	4K pages	huge pages	gain
near-empty (clustered working set)	31.7 Mops/s	33.8 Mops/s	+7%
50 000 resident (~13 MB working set)	21.2 Mops/s	24.0 Mops/s	+13%

The gain is larger at 50k, where the working set spreads across enough memory to thrash the 4K TLB — exactly the case huge pages relieve. It's below the 1.19–1.43× the pointer-chase showed in isolation because each operation also pays for the hash, the tag scan and a copy of the value. Detection is by trying each tier (hugetlb → THP → collapse → 4K), never by kernel version; mlock wants LimitMEMLOCK=infinity and warns-and-continues otherwise.

Design in brief

struct pcache_bucket {          /* exactly one cache line - asserted */
    volatile unsigned version;  /* seqlock: even = stable, odd = writer inside */
    gen_lock_t        lock;     /* writers (+ reader fallback) */
    unsigned char     tags[6];  /* 1 byte of hash per slot - rejects ~255/256
                                   of non-matching slots without a deref */
    unsigned short    used:4,   /* slots in use */
                      owner:12; /* holder id, for dead-writer recovery */
    pcache_rec       *slot[6];
};

• Slab arena: entries live in fixed-size cells inside class-bound chunks that are never returned to shm — the invariant the lock-free read path stands on. Byte 0 of every cell is the class id, stamped at chunk-carve time and immutable, which is how a reader clamps a possibly-stale length without aligned chunks. Allocation state is per-process (bump chunk + private free stack per class; no atomics, no shared cache lines on the fast path).
• Growth: segmented directory + linear hashing — buckets never move, splits happen one bucket at a time (driven from the single maintenance timer), and the routing word is re-checked on a miss. The table grows at runtime instead of being sized once — the load-factor cliff that motivates this whole module cannot form.
• No allocator call ever happens under a bucket lock (cachedb_local nests the shm allocator inside bucket locks in five places); records are pre-built before lock_get, frees happen strictly after release.
• Native counters: add/sub store an int64 and accumulate fixed-width under the bucket lock; every user-facing read formats them as decimal. No parse/format/realloc in the critical section.
• Overflow: full buckets spill to a small chained side table gated by a counter readers check only after a stable miss; a key lives in its bucket or in overflow, never both.

Script interface

Single-key operations go through the core cache functions unchanged. The module's own multi-key operations are perf_-prefixed with Redis verbs — deliberately not the cachedb_local parity names (cache_remove_chunk / fetch_chunk), so migrating those two calls requires a script change; everything else is drop-in:

perf_del("session-*");                          # glob delete -> count
perf_mget("user-*", $avp(k), $avp(v));          # matches -> index-paired AVPs
perf_mget_json("*", $var(j));                   # -> {"hits":"6","user-alice":"a1",...}

All three ride one lock-free walker (Redis SCAN-class guarantee) with binary-safe JSON escaping; iter_keys uses the same walker. Two startup selftest modparams (arena_selftest, htable_selftest) ship as permanent diagnostics and fail startup on any mismatch.

The same walker backs the introspection MI (full command table in the MI commands section above) — the operator visibility cachedb_local never had, and lock-free so a key scan never stalls SIP traffic. perf_scan is the answer for a large cache where perf_keys would truncate: its cursor is an ascending bucket index, so it stays valid across a concurrent resize and returns every entry present throughout at least once — without Redis's reverse-binary cursor masking, because the table only grows (buckets never move).

Status

• Module shell, URL/collection parsing (size clamped to [4,24] — 1 << size on an unbounded unsigned is UB), memory-tier probe with actionable sysctl guidance

• Slab arena (size classes, per-process allocation, donation/refill pools)

• Table core: 64B buckets, SWAR tag scan, seqlock reads with full copy-out validation, versionless TTL bump, overflow

• cachedb vtable: get/set/remove/add/sub/get_counter + native counters; iter_keys

• perf_del / perf_mget / perf_mget_json

• Selftests + script-level end-to-end suite

• Expiry sweep — hint-routed (per-bucket min-expires hints in sweep-friendly parallel arrays, 16 per cache line; the hot TTL-bump path never writes them), timer-driven via expiry_sweep_period (default 1 s), reclamation through the global pool strictly after lock release

• Statistics — per-process sharded counters (one 64-byte line per process, summed only at read time; a shared update_stat counter would recreate the 0.72× collapse measured above), exported as ten cachedb_perf: core stats and a per-collection perf_stats MI (load factor, overflow, seqlock retries/1k, backing tier, expired/destroyed, and a hit rate whose accompanying note follows the measured value rather than asserting a verdict). perf_stats_reset re-baselines the cumulative counters for a fresh measurement interval without a restart, leaving live gauges alone

• Linear-hash growth + maintenance timer — the table now resizes itself (the thing cachedb_local fundamentally cannot do): one-bucket-at-a-time splits driven from the single-process maintenance timer, no rehash, overflow left findable; growth_load_factor keeps the bucket shape as entries scale. Verified: 1000 entries → 484 splits → 500 buckets, all keys intact

• Introspection MI — perf_keys / perf_scan / perf_dump / perf_get / perf_set / perf_ttl / perf_del as MI commands, all lock-free (a key scan never stalls writers, unlike cachedb_local's). perf_scan is cursor-based (Redis SCAN): an ascending bucket cursor, stable across a concurrent resize, every entry returned at least once. Verified over a datagram MI

• Observability events (EVI) — E_CACHEDB_PERF_EXPIRED (per reaped key, opt-in per collection), E_CACHEDB_PERF_NOMEM (a write dropped because the arena is full), E_CACHEDB_PERF_GROWN (a table resized, with the before/after span), E_CACHEDB_PERF_MEM_DEGRADED (huge pages requested but the arena landed below hugetlb). Each evi_probe_event()-gated (free with no subscriber) and off the hot path; verified end-to-end over event_routes

• Huge-page arena backing — 2M-aligned mlock-pinned reservation via the detect-by-trying ladder (arena_hugepage_mb), lock-free bump from it, shm_malloc fallback; measured +7–13% (see above)

• Multi-process correctness soak — forked worker processes hammer one live backend (get/set/remove/add) while the maintenance timer splits buckets underneath them, checking four invariants: no torn read, no lost update, no lost key across splits, no crash/UAF. Found and fixed a real fork-safety bug (see below). Post-fix: 8 processes, 24M ops, 3093 concurrent splits, 0 crashes, torn_reads=0, counter sum == adds, all immortals intact; clean under the Q_MALLOC_DBG redzone allocator and under all three core allocators (F_MALLOC / Q_MALLOC / HP_MALLOC, driving both pkg and the arena's shm chunk backing)

• Portability — built and exercised outside the usual glibc/x86 dev box, in containers: Alpine 3.24 (musl 1.2, gcc 15.2) and RHEL 9 / UBI9 (glibc 2.34, gcc 11.5). Zero warnings on either, with the arena/table selftests and the multi-process soak passing on both. The module needed no conditional compilation for musl; the one portability defect the exercise turned up was in the core rather than here (lib/url.c undefining _GNU_SOURCE before the headers that need it, which hides clock_gettime/ctime_r on musl), submitted separately as core: fix build on musl libc (Alpine) - stray #undef _GNU_SOURCE in lib/url.c #4119

• End-to-end th_state_url benchmark against cachedb_local (50k held calls) and against dialog-based topology hiding (100k held calls) — both sections above

• DB persistence — whole-collection save/load to any db_* backend (perf_save/perf_load MI, plus db_mode startup-load / shutdown-save), TTLs kept as absolute wall-clock time so they survive a restart. The snapshot runs in a single transaction where the backend supports one, which makes it both fast and atomic — 30 000 entries save in 0.35 s on db_sqlite (against ~60 s and an aborted process before), and an interrupted save now rolls back rather than replacing a good snapshot with a partial one. Verified end to end with db_sqlite (save → shutdown-save → startup-load, values intact, TTL decremented across the cycle) and with db_redis, which takes no transaction and measures 5.05 s for the same rows. Rows whose wall-clock expiry has passed are dropped at load rather than merely skipped — otherwise, with db_mode=1 or after any shutdown that was not graceful, dead rows accumulate in the table indefinitely. Single-node durability, not replication

• Cluster sync (perf_sync) — MI command + script function that saves this node's collection to the DB, then signals peers over the clusterer API to reload it (one message per sync, not per operation); each peer reloads from the DB and raises E_CACHEDB_PERF_SYNCED. Soft dependency — degrades to a DB save with no broadcast if clusterer/sync_cluster_id is absent. A pull-from-DB refresh model for single-writer/read-replica topologies — deliberately not /r-style per-operation replication. Verified on a single-node cluster: the capability registers and lists in the clusterer's clusterer_list_cap MI (cachedb-perf-sync, state Ok), a soft DEP_SILENT clusterer dependency reorders init so it works regardless of load order, and perf_sync degrades cleanly with no clusterer — no crashes. Each collection also reports last_sync_out / last_sync_in / last_sync_source in perf_stats (shown only when sync_cluster_id is set): the clusterer's own clusterer_list_cap state of Ok for this capability only means registered and enabled — the module registers with startup_sync=0 and takes no part in the clusterer's startup data-sync — so the honest convergence signal lives in the module's own stats. The multi-node broadcast→reload fan-out follows the ratelimit clusterer pattern and is verified on a two-node cluster: with both nodes pointed at one shared db_sqlite file and linked over proto_bin, cachedb-perf-sync registers on both, perf_sync on node 1 returns {collections:1, saved:3, broadcast:1}, and node 2 logs cluster sync: reloading <sync> from DB (issued by node 1), reloads every key with its TTL intact and raises E_CACHEDB_PERF_SYNCED carrying source_node=1. Verified in both directions (node 2 → node 1 likewise, source_node=2)

• Read/write mix swept — 5.6–9.9× over cachedb_local on pure writes (flat vs degrading as workers rise), and an assertion-verified end-to-end three-way where perf-TH lands within 4% of no-TH on CPU and memory

• Allocation-free read — the optional get_buf() cachedb endpoint (CACHEDB_CAP_GET_BUF) and its cachedb_perf implementation; one shared read path for both entry points; measured 397.7 → 290.1 ns/op at the median (table above)

Correctness: what the multi-process soak caught

A lock-free read path plus a table that resizes itself under live traffic is exactly the kind of code where a single-process selftest passes and production still corrupts memory. So the soak (bench/cdbstress.c) runs the real thing: 8 worker processes on one shared backend, a get/set/remove/add mix, with the maintenance timer splitting buckets the whole time. Every value is written all-bytes-equal so a torn read is visible; counters are hammered with add(+1) so a lost update shows as a shortfall; a set of keys is inserted once and never removed so a split that drops one shows as a miss.

It failed inside a second — a segfault on an impossible size class (88) read out of a cell's class byte. Root cause: after fork() every child holds a copy-on-write copy of the parent's private allocator hoard (same bump pointer, same free-list cell addresses), and pcache_arena_child_init had each child donate that hoard to the global pool. The identical physical cells were enqueued once per child, popped by several processes at once, and written through concurrently — one process's value byte landed on another's class id. The fix: a child drops its inherited copy and carves its own chunk on first use, never donating cells it doesn't own. After it, the full soak is clean — 24M ops, 3093 concurrent splits, no torn reads, counter sum equals total adds, every immortal key intact — and equally clean under the Q_MALLOC_DBG redzone allocator and under each of OpenSIPS' three core allocators (F_MALLOC, Q_MALLOC, HP_MALLOC), which back both the per-process state and the arena's shm chunk allocation.

And what production caught that the soak did not

The soak runs a single collection with generous TTLs, so it never combined
overflow chaining with expiry reclamation — and that combination was the one
that mattered. On a live SBC the module crashed repeatedly inside the arena's
free path, on an impossible size class, with topology-hiding keys going
missing.

struct povf, the overflow node, had its next pointer at offset 0 — but
byte 0 of every arena cell is the size class, read by both free paths. Linking
a node wrote the pointer's low byte over the class id (0x40/0x80/0xC0 →
"class" 64/128/192), indexing past the 21-entry class table and corrupting the
pool, so the crash surfaced far from the cause. It needs overflow and the
expiry sweep together to trigger, which is why a table with room to spare never
showed it.

The fix reserves byte 0 in struct povf and hardens both free paths to log and
leak on an invalid class rather than corrupt. Reproduced deliberately
afterwards — a churn set with a 2 s TTL, a 1 s sweep and growth enabled catches
it in under a minute — and the soak was extended to cover the same shape.

And what an adversarial design review caught before it shipped

Designing get_buf began with a red-team pass over the read/write protocol, which found three latent defects in the existing code — each now a separate commit:

1. A seqlock write-ordering hole on weak memory models. Every version bump used a RELEASE RMW; release is one-way and permits the payload stores that follow to be observed first, so on aarch64/ppc64le a reader could sample an even version, copy a half-written record, re-check the same even version and accept the tear. All bumps are now ACQ_REL (the same reason the kernel writes seq++; smp_wmb()); x86-64 was never affected, which is exactly why no soak had caught it.
2. rflags survived an in-place overwrite. Storing an 8-byte string over a key that had been a native counter left PCACHE_F_INT set, and the read path then formatted the ASCII as an int64: cache_store("…","12345678") read back as 4050765991979987505. The flag is now cleared inside the version bracket.
3. The copy clamp could wrap. PCACHE_REC_HDR + klen + vlen > bound on unsigned values wraps for a torn vlen, skipping the clamp on the one path that needs it; it is now subtractive and cannot wrap.

The standalone rig behind every figure above lives in modules/cachedb_perf/bench/ (make run, no OpenSIPS build needed) — full measurement history, every rejected alternative and why — so the numbers here are reproducible rather than asserted.

[Lt-Flash]

CP-15 preview: cross-node state sharing for cachedb_perf, measured on a 3-node containerized cluster

perf:// as submitted in this PR is deliberately node-local: every read and write touches only local shared memory, which is where its performance comes from. CP-15 is the follow-up layer (developed, working, not yet part of this PR) that answers the one question a node-local cache cannot: what happens when a request lands on a node that doesn't hold the state?

The answer is pull-on-miss read repair. On a local miss in an opted-in collection, the node asks the cluster, stores the answer locally with its remaining TTL, and serves it — so state migrates towards wherever it is actually requested, one key at a time, and a second request for the same key is an ordinary local hit. Nothing is pushed eagerly and nothing participates unless named in replicate_collections; a fully-warm cluster behaves exactly like today's node-local module.

Two transports are selectable via pull_transport: bin (clusterer generic messages over the proto_bin TCP mesh) and clctr — one encrypted multicast packet per miss over the clusterer_controller UDP plane (XChaCha20-Poly1305, Noise_NNpsk0 join), with unicast replies. A short negative cache (pull_negative_ms, default 300 ms) stops broadcast storms for keys that exist nowhere; a miss waits at most pull_timeout_ms (default 50 ms), and all peers answering "not here" completes the wait early. The blocking form (pull_on_miss=1) is off by default; consumers can instead drive the pull through an exported API with an eventfd, which is how topology_hiding suspends the transaction instead of holding a worker.

To see how it behaves under sustained load — not just in a unit rig — I ran a 3-node cluster of Alpine containers (containerd/nerdctl on one 16-core host, dedicated bridge network, IGMP snooping off), built from the development branch at -O3 with F_MALLOC. Workload: 30,000 keys, 12 synchronous loaders driving ~90,000 uniform-random reads/s cluster-wide through the SIP script path (cache_fetch on OPTIONS), every request timed, the control plane captured with tcpdump on the bridge.

1. Warm cluster: the feature costs nothing when it isn't needed

All 30,000 keys on every node, 45 s of load:

Measurement	Result
Throughput	90,457 req/s (4.38 M requests, all 200)
Hit rate	100.0%
Latency	p50 105 µs, p99 317 µs (dominated by the SIP round trip)
Cluster pulls	0
Multicast packets	0 (≈2 pkt/s of controller beacons only)

This is the steady state a production cluster with call-id affinity lives in, and it is indistinguishable from the node-local module.

2. Each node holds one third — how fast does everyone hold everything?

Fresh seed: node 1 gets keys 0–9,999, node 2 gets 10,000–19,999, node 3 gets 20,000–29,999. Same uniform load against all three nodes, so two thirds of initial reads are misses:

Half of each node's 20,000-key gap closes in about one second; 99% by 7 s; every node holds all 30,000 keys at 11.5 s — while the cluster keeps serving 87,000 req/s throughout. The exponential shape is inherent to read repair: hot keys arrive almost immediately, and the tail is just rarely-requested keys waiting to be asked for. (For a planned mass hand-over — node loss, re-hash — the perf_sync bulk path remains the right tool; it measured 23–45× faster than an equivalent pull storm.)

3. What a pull costs the request that triggers it

A pull is a 200 µs – 1.7 ms event (multicast ask + first positive reply), visible on ~3% of convergence-phase requests. Cluster-wide p99 rose from 317 µs to 576 µs during the heaviest miss phase and was back to baseline within seconds. Nothing approached the 100 ms budget; across 10 M+ requests in all scenarios there were zero timeouts and zero failures. Caveat for calibration: this is a single-host bridge network, so the wire is sub-millisecond — on a real LAN the pull cost is roughly one RTT plus these numbers, which is exactly the case where the eventfd/async form is worth using instead of the blocking one.

4. Wire accounting, to the packet

tcpdump on the bridge for a full thirds-seeded convergence:

Packets	Count	Meaning
multicast :4499	60,066	60,000 misses × exactly 1 broadcast, plus beacons
unicast :4499	120,048	exactly 2 replies per pull — every peer answers, positive or negative
proto_bin :5599	0	the clctr transport really carries everything

So the cost model for an N-node cluster is simply 1 multicast + (N−1) unicast replies per miss, independent of cluster size on the request side, and ~2 pkt/s of beacons when idle.

Where this lands

CP-15 lives on a development branch as module-scoped commits on top of this PR's branch (cachedb_perf: probe / pull / negative cache / async face / membership / stats; the consumer side on the topology_hiding branch; the messaging API on the clusterer_controller branch). It will be proposed once this PR merges, since it builds on the collection and stats layers introduced here. The defaults follow from the measurements above: everything opt-in, pull_on_miss off, blocking tolerable on a quiet LAN, suspension available where it matters, and bulk sync for planned hand-overs.

[Lt-Flash]

Follow-up: the same bench with pull_transport=bin — why the multicast transport is the default worth having

The pull layer deliberately supports two transports, so the choice ought to rest on a measurement rather than a preference. Same three containers, same 30,000 keys, same ~90,000 req/s workload as the previous comment; the only change between the two runs is the pull_transport modparam — clctr (one encrypted multicast ask, unicast replies) versus bin (clusterer generic messages over the proto_bin TCP mesh, one ask per peer).

	clctr — encrypted multicast	bin — clusterer TCP mesh
Warm steady state	90.5 k req/s, zero cluster traffic	89.8 k req/s, zero cluster traffic — identical
Slowest node fully converged	13.2 s	17.6 s
Convergence-phase p99	585 µs	1,319 µs
Throughput during the miss storm	~50 k req/s	~20 k req/s
Wire, same 60,000 misses	180,114 UDP packets (3 per miss)	367,540 TCP segments (≈6.8 per miss)

Same exponential shape — read repair fetches hot keys first regardless of transport — but BIN takes 4.4 s longer to finish, because each pull's slower round trip compounds through every blocked worker.

The medians are indistinguishable in both runs, which is the useful control: the transports genuinely differ only on the miss path. There, BIN's p99 starts near 2 ms against clctr's 1 ms and stays elevated roughly twice as long.

The operationally interesting one. With blocking pulls a slower transport bites twice — the request waits longer and the worker is unavailable longer — so BIN's throughput during the heaviest miss phase drops to ~20 k req/s against clctr's ~50 k. (The asynchronous form softens this for both transports; it does not change their ordering.)

The structural difference, counted rather than argued: BIN cannot broadcast, so every miss asks each peer separately over TCP and pays ACKs — twice the packets for identical work in a 3-node cluster. And the ratio is size-dependent in BIN's disfavour: the multicast ask stays one packet at any cluster size, while BIN's ask side grows as N−1.

Two honesty notes. First, this is a single-host bridge network — BIN's TCP overhead is at its kindest here; on a real network with loss and latency the gap widens. Second, BIN remains the right fallback where multicast is unavailable (some cloud fabrics), which is exactly why it exists as an option rather than being removed.

The full harness — image build, cluster bring-up, loaders, collector, capture wrapper, distilled per-run data and both chart generators — is committed on the development branch under modules/cachedb_perf/bench/containers/, so the whole comparison reproduces in a few minutes on any host with containerd.

[Lt-Flash]

Rebased onto current master (clean, no conflicts - this module wasn't touched by the topology_hiding refactor happening elsewhere) and added one more small fix, found from a real production reading.

Bug: hit_rate_note only looks at the hit/miss ratio, so a collection that's had zero stores - nothing has ever been written to it, e.g. it loaded 0 rows from persistence at startup - gets the exact same note as a collection that's genuinely churning through entries faster than they're used: "cached state is being lost or is expiring before it is used." That's actively misleading for the zero-stores case, since there's no state to have been lost or expired if nothing was ever stored in the first place.

Caught live on a production node: a debug collection (rtpdebug) with entries: 0, stores: 0, misses: 9 showed the "being lost or expiring" note, and the startup log confirmed pcache_db_load: collection <rtpdebug>: loaded 0 entries - it had been empty since the process started, not losing anything. The fix checks stores == 0 before falling into that branch and reports a different, more accurate note pointing at "nothing is reaching this collection" instead of "tune your eviction/TTL."

[Lt-Flash]

New commit: cross-node pull — ask the cluster before calling a miss

This is the follow-up trailed in the July benchmarks above, now landed as one
commit on top of the module (+3,799/−72, 22 files). It closes the gap between
the two sharing mechanisms the module already had: perf_sync moves whole
collections through a shared DB (right for bootstrap/failover, wrong for one
key), while this moves one key, on demand, at miss time.

The shape of it. Nodes behind a load balancer each run their own
cachedb_perf, so a key written on one node is a miss on every other. Now a miss
(on an opted-in collection) allocates a slot in a small shm pool, broadcasts
(collection, key, id) to the cluster, and any peer holding the key answers
with the value and its remaining TTL; the reply is stored locally with that
TTL. Opt-in is per collection via replicate_collections — a key is only worth
asking the cluster about if it means the same thing on every node, and only the
operator knows that.

Blocking is explicit and bounded. pull_on_miss=1 makes a cache_fetch
miss wait (poll + eventfd) up to pull_timeout_ms (default 50) for the answer —
that is a worker parked per miss, fine on a maintenance path, priced accordingly
on a SIP path, and the module says so at startup. With it off, the pull still
happens and benefits the next lookup. A slot reaper (utimer, twice per timeout
window) releases every slot whose deadline passed — without it, a request that
fewer peers than expected answered would leak its slot and hang the blocked
worker forever, which is exactly the failure we hit in an early deployment; the
regression test for it ships in bench/pullsoak/. A miss the whole cluster
confirmed is remembered in a negative cache (pull_negative_ms), so a key that
exists nowhere does not re-interrogate the cluster per lookup.

Transports, and what this PR does not depend on. The default transport
bin rides the clusterer module's BIN links — one unicast per peer, no new
dependency. There is a second transport, clctr, that rides the
clusterer_controller module (PR #4074): one encrypted datagram reaches every
peer. That module is optional at build time and at run time — the coupling
is gated behind CLUSTERER_CTRL_SUPPORT, which only a tree that carries the
controller ever defines, so this PR builds and runs standalone on master
(verified both ways: a full build of exactly this branch with no controller in
the tree produces a .so with zero clctr symbols, and a live start with
pull_transport=clctr + sync_cluster_id set but neither module available
comes up serving node-local, logging the three WARNs of the ladder below). At run time the degradation
ladder is uniform and non-fatal: clctr requested but unavailable → WARN, fall
back to bin; clusterer unavailable too → WARN, pull and sync disabled, the
cache runs purely node-local. Missing cluster infrastructure never stops the
module from starting; only a genuinely invalid config (an unknown transport
name) does.

Sizing is measured, not guessed. A pull slot embeds its key/value bounds:
pull_max_key (default 128) and pull_max_value (default 512) size them, so
the 64-slot pool costs 50 KB of shm at the defaults instead of the ~550 KB a
worst-case constant would take. Defaults come from measuring live collections
(values of 1–20 bytes under keys ≤33; sql_cacher rows ~70 bytes); dns_cache-
style users with multi-KB records raise pull_max_value and accept fewer slots
per KB. A value over the cap — or over what one datagram can carry — is
answered as held-but-unsendable rather than absent: the requester must not
conclude a key is missing from a node that demonstrably holds it.

Failover hook. sync_shtag="name/cluster_id" arms a sharing-tag callback:
the node becoming active reloads its collections from the DB snapshot, so a
standby that just took over does not serve stale state. Soft like everything
else — no DB or no clusterer means it logs and disarms.

Observability. Module stats pull_requested/answered/served/timeouts/ negative_hits, plus per-collection pulled_in/served_out (module-wide
counters cannot say which collection is converging). perf_stats gains a
cluster.topology object (this node's id, resolved IP and per-peer
reachability), and perf_cluster_probe actively asks every peer to answer for a
collection — "who is configured" and "who is actually reachable" fail
differently, so they are separate questions.

Running in production on our gateway fleet; the bench/pullsoak/ rig drives
the e2e burst, the slot-leak regression, the oversize leg and the max-key
boundary against a real two-node cluster.

[Lt-Flash]

Added: degraded-operation documentation, with captured outputs

Follow-up commit documenting exactly what each MI command, event and cachedb
surface does in the two degraded modes — every output below captured from a
live instance of this branch, not paraphrased. (Also in the admin guide as
§1.6 "Degraded operation", and the Dependencies section now names the two
optional modules instead of saying "None".)

Mode 1 — clusterer loaded, clusterer_controller absent. Only the clctr
transport is lost; pull and sync run over the clusterer's BIN links. One
startup WARN (two possible wordings — the build-time and run-time reasons read
differently):

WARNING:cachedb_perf:mod_init: pull_transport 'clctr' but this build carries
    no clusterer_controller support - falling back to 'bin'
WARNING:cachedb_perf:mod_init: pull_transport 'clctr' but clusterer_controller
    is not loaded - falling back to 'bin'

Mode 2 — neither module available. Pull and sync off, cache runs purely
node-local, three WARNs:

WARNING:cachedb_perf:mod_init: clusterer module not available - the cluster
    features are disabled; load clusterer before cachedb_perf
WARNING:cachedb_perf:mod_init: pull_transport 'clctr' but this build carries
    no clusterer_controller support - falling back to 'bin'
WARNING:cachedb_perf:mod_init: replicate_collections is set but the cluster
    is not available (needs sync_cluster_id + clusterer) - cross-node pull disabled

Unaffected in both modes: the whole local surface — cache_store/fetch/ add/sub/remove("perf",…), perf_del/perf_mget/perf_mget_json, the local
MI set (perf_get/set/probe/keys/scan/dump/ttl/del/stats/stats_reset),
perf_save/perf_load/db_mode, and the four local events
(E_CACHEDB_PERF_EXPIRED/NOMEM/GROWN/MEM_DEGRADED).

What changes, surface by surface:

surface	mode 1 (clusterer only)	mode 2 (neither)
perf_pull nokey	{"source": "no-answer"} — ran over bin, no peer held it	500 "cross-node pull not active (replicate_collections)"
perf_cluster_probe	probes over bin (lone node: 500 "…no peers, or no free pull slot")	400 "cross-node pull is not active for this collection…"
perf_sync	{"collections": 2, "saved": 1, "broadcast": 2}	{"collections": 2, "saved": 1, "broadcast": 0, "note": "cluster sync inactive (no clusterer / cluster_id 0) - saved to the DB only"} — the save still happens, it never fails for cluster reasons
perf_stats	full cluster object: ids, membership, pull counters, "pull_slots": 64, topology array	cluster object absent; per-collection stats identical, pulled_from_cluster/served_to_cluster stay 0
cache_fetch + pull_on_miss=1	a miss on an opted-in collection blocks ≤ pull_timeout_ms asking the cluster	plain immediate miss — no blocking, no negative cache
E_CACHEDB_PERF_SYNCED	still fires when a peer's sync arrives	can never fire (nothing can arrive); subscribing costs nothing

The one thing that still fails startup is a genuinely invalid config — an
unknown pull_transport name. Unavailable infrastructure never does.

[Lt-Flash]

New commit: make a pull that never left the node visible

A cross-node pull datagram that the transport refused to send was logged at LM_DBG. That made it unreachable on any normal deployment — log_level 3 is INFO and L_DBG is 4 — so in practice nobody ever saw it.

That is the wrong level for this event. A send that fails produces no packet, so the far end never learns there was a question and the requester simply times out. This line is the only direct evidence of why, and it is the actual cause behind a class of "cross-node pull is slow / does not converge" reports.

It cannot become an unconditional LM_WARN either: a partitioned or overloaded peer fails every send, and an unbounded warning is its own incident. So it now warns on the first failure and then at most once every 30 s, folding in the count it stands for.

Details worth noting:

• The rate-limit state is in shared memory, not a static. A pull reply is sent by whichever worker happened to receive the request, so a per-process limiter would let every SIP worker warn once per interval each. Two workers can still race the interval check and both warn — deliberate, and cheaper than taking a lock on a failure path.
• Applied to all four send sites, not just the reply path: both reply paths (clusterer_controller unicast and BIN) and both request paths. A request that never goes out fails in exactly the same way.
• New pulls_send_failed statistic and MI field, so the exact total stays available while the log is suppressing. It is deliberately distinct from pulls_timeout: a timeout means a peer was asked and stayed silent, this means no peer was ever asked. Conflating them would hide a broken transport as peer slowness.

[Lt-Flash]

New commit: store a pull answer that arrives after its caller gave up

A pull that times out can still be answered — the reply was merely late. Previously that answer was worthless: the waiter freed the request slot on its way out, a reply arriving after that failed the slot lookup and was dropped, and since the wire reply carries neither collection nor key, once the slot was gone the answer was uninterpretable. Net effect: the cache permanently re-asked for exactly the keys that are slowest to fetch — the worst possible convergence profile — and no counter recorded any of it.

Now a waiter that leaves empty-handed orphans the slot instead of freeing it. The slot keeps the collection, key and deadline, so a late reply still matches, and the reply path completes the store the waiter would have done. Final outcomes (a stored value, an oversize holder, a settled absence) still free the slot.

Design notes:

• The store happens after the pull lock is released, on copies taken under it — storing under it would serialise every node-wide pull behind one table write and nest the pull lock outside the bucket locks.
• The present-key check is pcache_ht_probe() == 0: allocation-free, and exactly 0 for a live key. A late answer never overwrites a live entry — this is read repair; a local write in the intervening window is by definition fresher.
• No in-flight TTL correction: the peer computes ttl_left immediately before sending, so charging the requester's elapsed time would subtract the peer's own delay from a figure that never included it.
• Optional pull_linger_ms (default 0 = off) bounds how late is too late, for deployments that delete keys; write-and-expire deployments need no bound because the value carries its own expiry.
• Orphans are bounded three ways: the existing reaper frees them at deadline + abandon; a stored orphan frees itself; and when the pool runs dry, allocation steals the longest-dead orphan — never a live pull. pulls_in_flight excludes orphans so the leak alarm doesn't fire on the ordinary outcome of a timeout.
• Six new statistics: pulls_orphaned, pulls_late_stored, pulls_orphan_evicted, pulls_late_superseded, pulls_late_expired, pulls_orphan_expired.

Verified fail-then-pass on two hosts

Two nodes on this branch over plain clusterer/BIN (no controller — the degraded-op path), pull_timeout_ms=50, and 100 ms of tc netem on the peer's BIN replies so every answer is guaranteed late (netem match confirmed by tc counters, not assumed):

	before	after
fetch #1	miss; reply vanished (pulls_received=0, nothing anywhere)	miss (same 50 ms give-up); pulls_orphaned=1 → pulls_late_stored=1
fetch #2	miss again, pulls_requested=2 — re-asks forever	hit, pulls_requested stays 1 — converged

Waiter latency is unchanged in both cases; the difference is purely that the answer now lands.