Pointer arithmetic bug in `first_matrix__bitonic_topk_kernel` causes shared memory out-of-bounds write

[hjldegit]

🐛 Describe the bug

Bug

There is a pointer arithmetic bug in src/libtorchaudio/cuctc/src/ctc_prefix_decoder_kernel_v2.cu (line 298) that causes out-of-bounds shared memory writes.
The bug is latent on most GPU architectures but produces incorrect results or crashes on NVIDIA H100 due to stricter shared memory bounds checking.

Location

float* block_topk_key =
    reinterpret_cast<float*>(smem_buf_bytes + smem_result_byte_offset);
int* block_topk_value =
    reinterpret_cast<int*>(block_topk_key + sizeof(float) * beam);  // ← BUG

impact

• A10 / A100: The out-of-bounds write silently lands on physically present but unallocated shared memory regions. Tests pass, but results may be subtly incorrect.
• H100: Stricter shared memory bounds checking causes data corruption, leading to incorrect CTC decoding results or kernel failures.

Root Cause

block_topk_key is a float*. Pointer arithmetic operates in units of the pointed-to type, not bytes. Adding sizeof(float) * beam to a float* advances by sizeof(float) × sizeof(float) × beam = 16 × beam bytes, instead of the intended sizeof(float) × beam = 4 × beam bytes.

Solution

int* block_topk_value =
    reinterpret_cast<int*>(block_topk_key + beam);

Versions

main

[lanarkite99]

Thanks for the clear report and root-cause analysis.

I prepared a focused fix for this by changing the shared-memory result layout so block_topk_value starts after beam float entries rather than after sizeof(float) * beam float entries. I also added a CUDA decoder regression test using beam_size=128 with a vocabulary larger than the beam size, so the test exercises the full top-k result buffers.

I’ll open a PR shortly referencing this issue.

[hiSandog]

This looks like a good regression-test candidate because the failure is narrowed to “Pointer arithmetic bug in first_matrix__bitonic_topk_kernel causes shared memory out-of-bounds write”. If a minimal fixture can exercise first_matrix__bitonic_topk_kernel and assert the expected behavior around src/libtorchaudio/cuctc/src/ctc_prefix_decoder_kernel_v2.cu, it would make the eventual fix much easier to keep from regressing.