fix(kernel): recover TraceLens reports after lost subagent completion

[akii96]

Summary

Recover TraceLens report generation when the parent workflow misses completion notifications from its final subagents.

Incident

During a real MI355X/vLLM optimization, the GEMM subagent completed and wrote a valid findings file:

[claude-sdk] Ran gemm_analysis.py (standalone scope)
[claude-sdk] Wrote category_findings/gemm_findings.md with Status SUCCESS
Findings file: .../category_findings/gemm_findings.md

However, approximately five minutes later, the parent still considered GEMM unfinished:

[claude-sdk] Now waiting for the final subagent (gemm).
ResultMessage: I'll wait for the subagent completion notifications before continuing to Step 7.4/7.5.

The SDK stream then ended without running the aggregation steps:

TraceLens SDK runner did not write .../analysis.md
TraceLens analysis.md was not produced

Because analysis.md is the authoritative TraceLens output, Hyperloom correctly refused to use the intermediate files. The optimization continued without a fresh kernel analysis, reducing the quality of downstream GEAK work.

Fix

When this final-subagent barrier occurs, the runner now:

• Reads the latest list of subagents the parent says it is waiting for.
• Confirms every named subagent has exactly one complete findings file.
• Supports both category and system findings formats used by TraceLens.
• Runs one aggregation-only recovery pass without launching new subagents.
• Skips recovery after SDK errors, timeouts, incomplete output, or ambiguous files.
• Preserves the original and recovery activity in the agent transcript.

analysis.md remains the only TraceLens result consumed by Hyperloom.

Testing

• Full TraceLens test module: 234 passed
• Covers the captured incident, multiple pending subagents, missing or duplicate findings, status variations, fenced templates, stale wait messages, SDK errors, and stream cleanup.

[tsrikris]

@akii96 This seems to add a lot of bloat to the codebase. Can you pls let us know how often you saw this error? Can you let us know how to reproduce this? We'll try to test things out to see if anything can be done on the agent itself

[akii96]

@tsrikris Happened both the times I tested it might to do with the fact that I do not run InfX configs 🤔

Combined with my other PR #1029 which corrects the vllm upstream source

export MODEL_PATH=MiniMaxAI/MiniMax-M3-MXFP8
export RUN_TAG="minimax-m3-mxfp8-c16-16k-$(date +%Y%m%d_%H%M%S)"
export RUN_DIR="$USER_DATA_PATH/optimizer_runs"
export RUN_LOG="$RUN_DIR/run_${RUN_TAG}.log"
export PID_FILE="$RUN_DIR/run_${RUN_TAG}.pid"
export LAUNCH_INFO="$RUN_DIR/launch_${RUN_TAG}.json"

mkdir -p "$RUN_DIR"

setsid nohup inference_optimizer -v optimize \
  --model "$MODEL_PATH" \
  --framework vllm \
  --tp 4 \
  --ep 1 \
  --conc 16 \
  --isl 16384 \
  --osl 512 \
  --profile-osl 512 \
  --precision mxfp8 \
  --max-model-len 82256 \
  --max-hours 9 \
  --tick-interval-sec 30 \
  --enable-roofline \
  --enable-conc-sweep \
  --conc-sweep-concs 16,8,4 \
  --conc-sweep-timeout-sec 600 \
  --conc-sweep-total-budget-sec 1200 \
  --phase-budget-prelude-pct 0.05 \
  --phase-budget-framework-pct 0.08 \
  --phase-budget-explore-pct 0.07 \
  --phase-budget-kernel-pct 0.68 \
  --phase-budget-sweep-pct 0.10 \
  --phase-budget-close-pct 0.02 \
  --skip-variants "add-shuffle-*,add-max-num-batched-tokens-32768,add-no-async-*,add-quick-reduce-*,add-fp8-kv-cache-*,add-indexer-kv-*" \
  --target-summary "Optimize MiniMax-M3 MXFP8 text serving at TP4, concurrency 16, ISL 16384, OSL 512 on gfx950; use upstream vllm-project/vllm PR intelligence and prioritize MXFP8 linear, sparse-GQA, QuickReduce, and fused-MoE" \
  --server-args "--distributed-executor-backend mp --trust-remote-code --block-size 128 --no-enable-prefix-caching --language-model-only --gpu-memory-utilization 0.95 --max-num-batched-tokens 32768 --max-num-seqs 64 --enable-chunked-prefill --long-prefill-token-threshold 32768 --attention-backend TRITON_ATTN --moe-backend aiter --linear-backend emulation --tool-call-parser minimax_m3 --enable-auto-tool-choice --reasoning-parser minimax_m3 --no-enable-log-requests" \
  --launch-info-file "$LAUNCH_INFO" \
  >"$RUN_LOG" 2>&1 </dev/null &

echo $! | tee "$PID_FILE"
echo "Log: $RUN_LOG"
echo "Launch info: $LAUNCH_INFO"

This is how I setup my container whereby I take a vllm nightly image and setup hyperloom on it with all its dependecies
hyperloom_prep.md