ReplicatedMergeTree: probable MergeTreeSettingsImpl use-after-free/lifetime corruption causes cross-thread SIGSEGVs in getStatus() and wakeupEarlierIfNeeded() on 25.8

[coreground]

Summary

We are seeing repeated production SIGSEGVs in ReplicatedMergeTree on ClickHouse 25.8.

Affected builds:

25.8.16.10002.altinitystable
25.8.28.10001.altinitystable

The same failure occurs on both versions and on multiple hosts.

The strongest evidence points to a lifetime/ownership problem involving:

MergeTreeSettings
    -> MergeTreeSettingsImpl

Multiple threads can successfully retain the table's mutex-protected std::shared_ptr<const MergeTreeSettings>, but later dereference the same unmapped MergeTreeSettingsImpl allocation.

We suspect use-after-free or corruption of the settings object graph, but have not yet proven the exact mechanism.

---

Environment

Rocky Linux 8.10
kernel 4.18.0-553.156.1.el8_10.x86_64

Docker CE
Ubuntu 22.04.5 container userspace
glibc 2.35

No investigated crash was associated with an OOM kill, host reboot, or Docker-initiated kill.

---

Strongest single-process example

On 25.8.28.10001, one ClickHouse process received two SIGSEGVs from separate threads before a single eventual exit with code 139.

First fault:

2026-09-10 02:46:38.011346 UTC

Address: 0x7e9e60479640
Access: read
Address not mapped to object

DB::StorageReplicatedMergeTree::getStatus(...)
DB::ServerAsynchronousMetrics::updateImpl(...)
DB::AsynchronousMetrics::update(...)

Second fault 1.41 seconds later:

2026-09-10 02:46:39.421861 UTC

Address: 0x7e9e60479850
Access: read
Address not mapped to object

DB::IMergeTreeCleanupThread::wakeupEarlierIfNeeded()
DB::StorageReplicatedMergeTree::scheduleDataProcessingJob(...)
DB::BackgroundSchedulePool::threadFunction()

The process later exited once with:

exitCode=139

Multiple system.crash_log rows therefore do not necessarily represent multiple process deaths.

---

Both faults resolve to the same MergeTreeSettingsImpl base

In 25.8.28, the binary contains these setting offsets:

replicated_can_become_leader
    = 0x640

cleanup_thread_preferred_points_per_iteration
    = 0x850

Subtracting those offsets from the two fault addresses:

0x7e9e60479640 - 0x640
    = 0x7e9e60479000

0x7e9e60479850 - 0x850
    = 0x7e9e60479000

So two independent threads accessed different fields through the exact same invalid MergeTreeSettingsImpl base:

0x7e9e60479000

---

getStatus() fault

Relevant 25.8.28 disassembly:

mov    0x210(%rsp),%rax

mov    ...,%rcx
# MergeTreeSetting::replicated_can_become_leader = 0x640

mov    (%rax),%rax
# MergeTreeSettings::impl

movzbl (%rax,%rcx,1),%eax
# fault: impl + 0x640

The observed fault address was:

0x7e9e60479640

---

wakeupEarlierIfNeeded() fault

Relevant 25.8.28 disassembly:

mov    ...,%r15
# MergeTreeSetting::cleanup_thread_preferred_points_per_iteration = 0x850

mov    (%r12),%rax
# MergeTreeSettings::impl

cmpq   $0x0,(%rax,%r15,1)
# fault: impl + 0x850

The observed fault address was:

0x7e9e60479850

---

Both readers retain the shared_ptr under the same mutex

Both code paths copy the settings shared pointer under the table settings mutex and increment the shared_ptr control-block refcount before accessing impl.

Observed layout:

storage + 0x4c0    settings mutex
storage + 0x4e8    MergeTreeSettings*
storage + 0x4f0    shared_ptr control block

For example:

call   mutex::lock()

mov    0x4e8(%storage), settings_ptr
mov    0x4f0(%storage), control_block

lock incq 0x8(control_block)

This makes a simple torn shared_ptr read during settings replacement unlikely.

---

changeSettings() publication also appears synchronized

MergeTreeData::changeSettings() replaces the same fields while holding that mutex:

call   mutex::lock()

mov    new_settings,0x4e8(%storage)
mov    0x4f0(%storage),old_control_block
mov    new_control_block,0x4f0(%storage)

# release old shared ownership afterward

We do not see an obvious reader-visible torn (object pointer, control block) state.

---

Allocation and destruction of MergeTreeSettingsImpl

StorageReplicatedMergeTree::getDefaultSettings() creates:

8-byte MergeTreeSettings outer object
0xe60-byte MergeTreeSettingsImpl

Relevant sequence:

mov $0x8,%edi
call operator new

...

mov $0xe60,%edi
call operator new

...

mov %new_impl,(%outer)

MergeTreeSettings::~MergeTreeSettings() later destroys outer->impl.

The deleting destructor for MergeTreeSettingsImpl ends in:

call MergeTreeSettingsTraits::Data::~Data()

mov $0xe60,%esi
mov %impl,%rdi
jmp operator delete(void*, unsigned long)

So normal destruction explicitly frees the same 0xe60 allocation that these threads later attempt to read.

This is consistent with use-after-free/lifetime corruption, although we do not yet have exact-address destructor logging proving that the specific faulting allocation had previously been freed.

---

Same failure exists in 25.8.16

The 25.8.16.10002 binary has the exact same offsets:

replicated_can_become_leader = 0x640
cleanup_thread_preferred_points_per_iteration = 0x850

Its getStatus() also performs:

MergeTreeSettings::impl + 0x640

and its ReplicatedMergeTreeCleanupThread::wakeupEarlierIfNeeded() performs:

MergeTreeSettings::impl + 0x850

Both acquire the same settings shared pointer under the same mutex and increment the shared_ptr control-block refcount first.

We have repeatedly observed old-build faults such as:

...640
StorageReplicatedMergeTree::getStatus()

...850
ReplicatedMergeTreeCleanupThread::wakeupEarlierIfNeeded()

as well as additional faults such as:

...9e0

on the same base page during the same dying process.

This therefore appears to be a cross-version failure mode, not something introduced in 25.8.28.

---

Frequency

On 25.8.16, we found:

614 system.crash_log records
12 hosts

with top frame:

DB::ReplicatedMergeTreeCleanupThread::wakeupEarlierIfNeeded()

These are crash-log signal records, not 614 confirmed process deaths.

The same semantic crash also occurs on 25.8.28.

---

Reproduction status

We do not have a compact deterministic SQL reproducer.

The workload includes:

• continuous high-volume inserts;
• ReplicatedMergeTree;
• materialized-view aggregation chains;
• frequent part creation/merge/retirement;
• replication activity;
• background cleanup;
• asynchronous metrics.

The crashes are stochastic and become frequent under sustained background activity.

The workload has run historically on older ClickHouse versions without this failure pattern.

---

Working hypothesis

The evidence is consistent with one of:

• premature destruction/use-after-free of MergeTreeSettingsImpl;
• corruption of MergeTreeSettings::impl;
• premature destruction of the outer MergeTreeSettings;
• corrupted shared_ptr/control-block state;
• another memory-corruption issue affecting this object graph.

The normal settings publication path appears synchronized, so we currently suspect a deeper lifetime/refcount or memory-corruption issue rather than a simple reader/writer race on storage_settings.

---

Questions

1. Are there known lifetime/refcount issues involving MultiVersion<MergeTreeSettings> / MergeTreeData::storage_settings in 25.8?
2. Is there any legal path that can destroy or move MergeTreeSettings::impl while a shared_ptr<const MergeTreeSettings> remains alive?
3. Are there fixes after 25.8.28 touching this ownership path?
4. Would maintainers recommend a diagnostic build that pins retired settings versions for the lifetime of the table/process?
5. Would logging MergeTreeSettings and MergeTreeSettingsImpl addresses in their destructors be useful for proving exact-address use-after-free?

We can provide full fatal logs, exact build IDs, more same-process crash sequences, system.crash_log data, Docker lifecycle timestamps, and additional disassembly.