Skip to content

[WIP] rescore: LDA path + env-tunable GBM config - #71

Closed
jspaezp wants to merge 1 commit into
mainfrom
lda-rescore
Closed

[WIP] rescore: LDA path + env-tunable GBM config#71
jspaezp wants to merge 1 commit into
mainfrom
lda-rescore

Conversation

@jspaezp

@jspaezp jspaezp commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

What

Exploration of faster Phase 5 rescoring. The GBM rescorer dominates the
pipeline (~180-220s of a ~300s run on the Skyline library). Two additive,
opt-in paths; the default GBM rescore is unchanged.

Not for merge yet - captures the exploration and leaves the tuning knobs in
place.

Results (Skyline library, 250225 Hela off, 2.3M candidates)

GBM baseline on this run = 46539 targets @ 1% FDR, ~182s training.

config                         targets@1%   %GBM    train    speedup
default (1000 it, depth-6)       46539      100%     182s      1x
GBM 200 it / 31 leaves / lr.10   45566      97.9%     66s      2.75x
GBM 100 it / 15 leaves / lr.15   44160      94.9%     33s      5.5x
LDA (linear ceiling)             37984      81.6%     ~10s     ~18x

Path 1: env-tunable GBM config (the likely mergeable win)

The forust default (1000 iterations, depth-6, unlimited leaves) is heavily
over-provisioned. GBMConfig::from_env adds runtime overrides so the
accuracy/speed knob can be swept without recompiling:

TIMSSEEK_GBM_ITERS  _MAX_DEPTH  _MAX_LEAVES  _LR  _EARLY_STOP

200 it / 31 leaves keeps 98% of IDs at 2.75x. Far better trade than LDA.

Path 2: Sage-style shrinkage LDA (TIMSSEEK_RESCORE_MODEL=lda)

  • Two-pass Fisher LDA, flat row-major matrix, rayon-parallel and
    deterministic (fixed-chunk reductions). Ridge shrinkage keeps the
    collinear 108-dim feature set solvable without pruning.
  • Rank-based inverse-normal transform (van der Waerden) for the per-feature
    normality assumption; missingness indicators + abs() of signed error
    features as linear-friendly feature engineering.
  • ~40x faster (Phase 5 ~180s -> ~10s), but caps at ~82% of GBM IDs on the
    hard Skyline library. This is a genuine linear-model ceiling, confirmed
    offline against sklearn LDA / QDA / logistic (all cluster at 79-82%);
    HistGradientBoosting matches forust at 99.6%, so trees are required for
    the last ~18%. LDA is useful for QC/triage, not a drop-in replacement.

Methodology

TIMSSEEK_LDA_DUMP=/prefix writes the raw feature matrix + labels, which
drove an offline numpy/sklearn harness (feature selection, transforms,
model zoo, tree-config sweep). Kept local under bench_out/ (gitignored).
Feature selection did not help - LDA wants all 108 features; INT is worth
~+5600 IDs; the gap is model capacity, not FE.

Open question

Which operating point should become the default: keep 1000/depth-6, or
move to ~200/31 (2.75x, -2% IDs)? Leaving that decision for the merge.

Explore faster alternatives to the Phase 5 GBM rescorer, which dominates
the pipeline (~180-220s of a ~300s run on the Skyline library).

Two additive, opt-in paths (GBM `rescore` unchanged at defaults):

1. Sage-style shrinkage LDA (`TIMSSEEK_RESCORE_MODEL=lda`, ml::lda):
   - Two-pass Fisher LDA on a flat row-major matrix, rayon-parallel and
     deterministic (fixed-chunk reductions). Ridge shrinkage keeps the
     collinear 108-dim set solvable without pruning.
   - Rank-based inverse-normal transform (van der Waerden) to satisfy the
     per-feature normality assumption; missingness indicators + abs of
     signed error features as linear-friendly FE.
   - ~40x faster than GBM (Phase 5 ~180s -> ~10s at 2.3M candidates), but
     caps at ~82% of GBM IDs on the hard Skyline library (linear model
     ceiling, confirmed against sklearn LDA/QDA/logistic). Good for
     QC/triage, not a drop-in replacement.

2. Env-tunable GBM config (`GBMConfig::from_env`): TIMSSEEK_GBM_ITERS,
   _MAX_DEPTH, _MAX_LEAVES, _LR, _EARLY_STOP. The default forust config
   (1000 it, depth-6) is heavily over-provisioned. Measured on the real
   pipeline:
     - 200 it / 31 leaves / lr .1  -> 97.9% of IDs at 2.75x
     - 100 it / 15 leaves / lr .15 -> 94.9% of IDs at 5.5x
   This is the better speed/accuracy trade and the likely mergeable win.

Also adds an optional raw feature-matrix dump (TIMSSEEK_LDA_DUMP) used to
drive an offline model/feature-selection harness (kept local under
bench_out/, gitignored).

Not for merge yet — captures the exploration and the tuning knobs.
@github-actions

Copy link
Copy Markdown

Apex-finder bench

cargo run -p apex_sim --release --example bench -- 1000 2

Sensitivity + timing across canonical scenarios
=== summary (n=1000, tol=±2 cycles) ===
  scenario                      pass2%  pass1%   medErr    us/run
  clean                          100.0   100.0        0     31.98
  moderate_noise                  91.6    91.6        0     31.54
  high_noise+interference         57.8    57.8        1     31.71
  heavy_interference               5.8     5.8       85     35.67
  mismatched_library              56.3    56.3        1     31.37
  absent_top_fragment             16.5    16.5       69     31.45
  absent_precursor                57.8    57.8        1     31.53

=== broad apex-finding (n=500, tol=±2 cycles) ===
  scenario                      pass2%  pass1%   medErr    us/run
  broad_clean                    100.0   100.0        0    189.56
  broad_moderate_noise            50.6    50.6        1    186.13
  broad_high_noise+interf         27.6    27.6      308    185.27
  broad_hard_3x_density            0.8     0.8      487    185.42
  broad_mismatched_library        41.0    41.0      161    184.56

=== narrow recovery (n=1000, tol=±2 cycles) ===
  scenario                      pass2%  pass1%   medErr    us/run
  narrow_clean                   100.0   100.0        0     21.52
  narrow_moderate_noise           81.3    81.3        0     21.27
  narrow_high_noise+interf        65.8    65.8        1     21.11
  narrow_hard_3x_density          10.9    10.9       34     21.10
  narrow_mismatched_library       62.8    62.8        1     21.09

=== narrow score discrimination (AUC, n_pairs=1000) ===
  scenario                         AUC   med+signal    med-noise
  narrow_clean                   1.000     4.329e14      7.199e7
  narrow_moderate_noise          0.870      4.054e9      3.054e8
  narrow_high_noise+interf       0.830      2.761e9      3.602e8
  narrow_hard_3x_density         0.684      3.644e9      1.244e9
  narrow_mismatched_library      0.842      2.172e9      3.575e8

commit 809299d

@jspaezp jspaezp closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant