Skip to content

Skip backtrace capture in trap handlers to eliminate lock contention - #11

Merged
ndr-ds merged 1 commit into
linera-wasmerfrom
ndr-ds/skip-trap-backtrace
Mar 17, 2026
Merged

ndr-ds merged 1 commit into
linera-wasmerfrom
ndr-ds/skip-trap-backtrace

Conversation

@ndr-ds

@ndr-ds ndr-ds commented Mar 17, 2026

Copy link
Copy Markdown

Description

Eliminate all Backtrace::new_unresolved() calls in trap handling paths.

Backtrace::new_unresolved() calls _Unwind_Backtrace, which invokes
_Unwind_Find_FDE for each stack frame. This takes a process-wide mutex (either
object_mutex in libgcc or dl_iterate_phdr's internal lock in glibc). Under high
concurrency — hundreds of async tasks executing WASM contracts simultaneously — this
causes severe lock contention. In production we observed 67% of CPU time spent in
native_queued_spin_lock_slowpath (the kernel futex spinlock beneath the contended
mutex), pinning workers at 100% CPU utilization.

Changes

  • lib/vm/src/trap/traphandlers.rs: Always use Backtrace::from(vec![]) in the
    signal-based trap handler instead of conditionally calling Backtrace::new_unresolved()
    (previously only skipped for stack overflow traps).
  • lib/vm/src/trap/trap.rs: Trap::lib() and Trap::oom() constructors use empty
    backtraces.
  • lib/compiler/src/engine/trap/stack.rs: Trap::User variant returns an empty
    WASM trace instead of capturing a new backtrace during Trap → RuntimeError
    conversion.

Trade-off

The WASM-level stack trace (FrameInfo frames shown in RuntimeError Display) is lost
— error messages will show RuntimeError: unreachable without the at function_name (module[N]:0xOFFSET) lines. The trap code and the original panic/error message are
fully preserved.

@ndr-ds
ndr-ds changed the base branch from main to linera-wasmer March 17, 2026 04:01
@ndr-ds
ndr-ds requested review from Twey and ma2bd March 17, 2026 11:25
@ndr-ds
ndr-ds merged commit 62b2c5d into linera-wasmer Mar 17, 2026
8 of 58 checks passed
@ndr-ds
ndr-ds deleted the ndr-ds/skip-trap-backtrace branch March 17, 2026 12:57
@Twey

Twey commented Mar 17, 2026

Copy link
Copy Markdown

I'm confused — does that mean we have some code path that is trapping in a loop? How could we be spending 67% of CPU time in any error-handling path?

(Nit: we execute Wasm in threads, not tasks — Wasm execution is currently fully synchronous, so blocks the thread)

ma2bd pushed a commit to linera-io/linera-protocol that referenced this pull request Mar 17, 2026
## Motivation

Under high concurrency, wasmer's `Backtrace::new_unresolved()` in trap
handlers takes a
process-wide mutex, causing severe lock contention. In production we
observed 67% of CPU
time spent in `native_queued_spin_lock_slowpath`.

## Proposal

Bump `linera-wasmer` and `linera-wasmer-compiler-singlepass` from
`4.4.0-linera.7` to
`4.4.0-linera.8`, which eliminates all `Backtrace::new_unresolved()`
calls in trap
handling paths.

See linera-io/wasmer#11 for the upstream
changes.

## Test Plan

CI
@ndr-ds

ndr-ds commented Mar 17, 2026

Copy link
Copy Markdown
Author

Yes, I was confused by that too, but the CPU profile seemed to point to the CPU usage coming from the WASM panics. That turned out to be wrong 😅 Even though it could, it wasn't in this particular situation.
The actual problem was something else, which is also already fixed. The WASM backtraces were not providing much useful information anyways, so it's still a positive thing we removed it I guess 🤷🏻‍♂️

github-merge-queue Bot pushed a commit to linera-io/linera-protocol that referenced this pull request Apr 1, 2026
…re (#5721) (#5844)

## Motivation

Under high concurrency, wasmer's Backtrace::new_unresolved() in trap
handlers takes a process-wide mutex, causing severe lock contention. In
production we observed 67% of CPU time spent in
native_queued_spin_lock_slowpath.

## Proposal

Bump linera-wasmer and linera-wasmer-compiler-singlepass from
4.4.0-linera.7 to 4.4.0-linera.8, which eliminates all
Backtrace::new_unresolved() calls in trap handling paths.

See linera-io/wasmer#11 for the upstream
changes.

Frontport of #5721.

## Test Plan

CI
github-merge-queue Bot pushed a commit to linera-io/linera-protocol that referenced this pull request Apr 1, 2026
…re (#5721) (#5844)

## Motivation

Under high concurrency, wasmer's Backtrace::new_unresolved() in trap
handlers takes a process-wide mutex, causing severe lock contention. In
production we observed 67% of CPU time spent in
native_queued_spin_lock_slowpath.

## Proposal

Bump linera-wasmer and linera-wasmer-compiler-singlepass from
4.4.0-linera.7 to 4.4.0-linera.8, which eliminates all
Backtrace::new_unresolved() calls in trap handling paths.

See linera-io/wasmer#11 for the upstream
changes.

Frontport of #5721.

## Test Plan

CI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants