Unbuntu - p2p Patch applied - WSl2 no luck either.
I hope this info helps it was gathered by Ai .

cat > ~/buqi_bug_report.txt << 'ENDOFREPORT'
Bug Report: Buqi dual-GPU stalls at first collective

Component: buqi-minimax-h3-multigpu — Ulysses SP multi-GPU

Summary

Worker rank 1 spawns correctly on the second GPU and loads the model,
but the first NCCL collective deadlocks. Rank 0 spins at 100% SM with
0% memory traffic while rank 1 idles.

Environment

ComfyUI : 0.34.3
PyTorch : 2.14.0+cu130
NCCL : 2.30.7 (loaded from nvidia-nccl-cu13)
CUDA : 13.0 (PyTorch), 13.3 (system toolkit)
Python : 3.13.13
GPUs : 2x RTX 5090 (32 GB each), distinct UUIDs,
different PCIe root complexes
Driver : 610.57.04
P2P : Bidirectional enabled, 111 GB/s via patched driver
IOMMU : iommu=pt (passthrough)
Launch : --cuda-device 0,1 --highvram

Symptom

Worker spawns successfully. Both GPUs load ~20 GB each. Worker
reaches "rank 1: ready". Then:

GPU 0: 100% SM utilization, 0% memory controller activity,
13801 MHz memory clock, ~110 W
GPU 1: 18-25% SM, 11-21% memory, 7001 MHz memory clock, ~118 W
Sampler: progress bar stays at 0%|...| 0/8 indefinitely

This persists for at least 10 minutes with no error, no timeout,
no progress.

NCCL Init Trace (Rank 0)

ncclCommInitRankConfig rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 1000

• Init START
  NCCL_P2P_LEVEL set by environment to PXB
  Rank 0: 1 Net devices
  comm rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
  Channel 00/04 : 0 1
  Channel 01/04 : 0 1
  Channel 02/04 : 0 1
  Channel 03/04 : 0 1
  Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 ...
  P2P Chunksize set to 131072
  Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
  [Proxy Service] Device 0
  [INFO] [minimax_sp] process group up, waiting for workers to load weights...
  [INFO] [minimax_sp] all 2 ranks ready
  [INFO] [minimax_sp] sequence parallel enabled, world_size=2 devices=['0', '1']

Then no further output.

NCCL Init Trace (Rank 1)

ncclCommInitRankConfig rank 1 nranks 2 cudaDev 0 nvmlDev 1 busId e1000

• Init START
  NCCL_P2P_LEVEL set by environment to PXB
  Rank 1: 1 Net devices
  comm rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
  Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 ...
  P2P Chunksize set to 131072
  Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
  [Proxy Service] Device 0
  rank 1: loaded on cuda:0, 19.5 GiB allocated
  rank 1: ready

Note: Rank 1 shows Trees but never prints the "Channel 00/04"
lines that rank 0 shows. The channel setup appears to hang after
topology computation.

Environment Variables in Use

NCCL_P2P_LEVEL=PXB
NCCL_P2P_DISABLE=0
NCCL_IB_DISABLE=1
PYTHONPATH=<comfy_root>
MINIMAX_SP_LOGDIR=<log_dir>
MINIMAX_SP_DEVICES=0,1

Things Already Tried

NCCL_P2P_LEVEL=SYS -> Same stall
NCCL_P2P_LEVEL=PHB -> Same stall
NCCL_MIN_NCHANNELS=1 MAX_NCHANNELS=1 -> Same stall
MINIMAX_SP_AG_CHUNKS=1 -> Same stall
MINIMAX_SP_ATTN_CHUNKS=1 -> Same stall
Removing --cuda-device flag -> Buqi reports "only 1 GPU visible"
Clean reinstall of comfy-kitchen -> No change
Clearing all pycache and .pyc -> No change
Testing with FL2VA and Ref2VA models -> Same stall in both

Verification Confirming Rank 1 Uses Correct GPU

Running the same probe that Buqi's sp_group.py uses to spawn workers,
from inside the same environment:

import os, subprocess, sys
env = os.environ.copy()
env['CUDA_VISIBLE_DEVICES'] = '1'
env['PYTHONPATH'] = '<comfy_root>:' + env.get('PYTHONPATH', '')
probe = '''
import torch
print("Worker sees:", torch.cuda.device_count(), "GPU(s), UUID:",
torch.cuda.get_device_properties(0).uuid)
'''
r = subprocess.run([sys.executable, '-c', probe], env=env,
capture_output=True, text=True)
print(r.stdout)

Output: Worker sees: 1 GPU(s), UUID: <gpu1_uuid>

This confirms:

• CUDA_VISIBLE_DEVICES=1 is honored inside the spawned environment
• The worker sees GPU 1 as its only device
• The worker's cuda:0 correctly maps to physical GPU 1

Hypothesis

Rank 0 completes NCCL channel setup and enters the first collective.
Rank 1 never completes channel setup - it's blocked earlier, in a
phase rank 0 has already passed. Since NCCL channel setup is P2P-only,
the deadlock is likely in the P2P handshake between the two ranks.

The fact that NCCL_P2P_LEVEL=SYS (which routes through system memory
instead of direct P2P) doesn't resolve it suggests it's not a raw
PCIe transport issue - the ranks are stuck before even reaching the
transport layer.

Request

1. Is there a known issue with the gloo subgroup initialization on
   rank 1 blocking before rank 0's NCCL barrier?
2. Would adding a dist.barrier() on rank 0 BEFORE the initial
   dist.barrier() on the gloo group help?
3. Is there a diagnostic env var that logs per-rank NCCL progress at
   finer granularity than NCCL_DEBUG=INFO?
   ENDOFREPORT
   echo "Created: $(wc -l < ~/buqi_bug_report.txt) lines at ~/buqi_bug_report.txt"