[WIP] rescore: LDA path + env-tunable GBM config - #71
Closed
jspaezp wants to merge 1 commit into
Closed
Conversation
Explore faster alternatives to the Phase 5 GBM rescorer, which dominates
the pipeline (~180-220s of a ~300s run on the Skyline library).
Two additive, opt-in paths (GBM `rescore` unchanged at defaults):
1. Sage-style shrinkage LDA (`TIMSSEEK_RESCORE_MODEL=lda`, ml::lda):
- Two-pass Fisher LDA on a flat row-major matrix, rayon-parallel and
deterministic (fixed-chunk reductions). Ridge shrinkage keeps the
collinear 108-dim set solvable without pruning.
- Rank-based inverse-normal transform (van der Waerden) to satisfy the
per-feature normality assumption; missingness indicators + abs of
signed error features as linear-friendly FE.
- ~40x faster than GBM (Phase 5 ~180s -> ~10s at 2.3M candidates), but
caps at ~82% of GBM IDs on the hard Skyline library (linear model
ceiling, confirmed against sklearn LDA/QDA/logistic). Good for
QC/triage, not a drop-in replacement.
2. Env-tunable GBM config (`GBMConfig::from_env`): TIMSSEEK_GBM_ITERS,
_MAX_DEPTH, _MAX_LEAVES, _LR, _EARLY_STOP. The default forust config
(1000 it, depth-6) is heavily over-provisioned. Measured on the real
pipeline:
- 200 it / 31 leaves / lr .1 -> 97.9% of IDs at 2.75x
- 100 it / 15 leaves / lr .15 -> 94.9% of IDs at 5.5x
This is the better speed/accuracy trade and the likely mergeable win.
Also adds an optional raw feature-matrix dump (TIMSSEEK_LDA_DUMP) used to
drive an offline model/feature-selection harness (kept local under
bench_out/, gitignored).
Not for merge yet — captures the exploration and the tuning knobs.
Apex-finder bench
Sensitivity + timing across canonical scenarioscommit 809299d |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Exploration of faster Phase 5 rescoring. The GBM rescorer dominates the
pipeline (~180-220s of a ~300s run on the Skyline library). Two additive,
opt-in paths; the default GBM
rescoreis unchanged.Not for merge yet - captures the exploration and leaves the tuning knobs in
place.
Results (Skyline library, 250225 Hela off, 2.3M candidates)
GBM baseline on this run = 46539 targets @ 1% FDR, ~182s training.
Path 1: env-tunable GBM config (the likely mergeable win)
The forust default (1000 iterations, depth-6, unlimited leaves) is heavily
over-provisioned.
GBMConfig::from_envadds runtime overrides so theaccuracy/speed knob can be swept without recompiling:
200 it / 31 leaves keeps 98% of IDs at 2.75x. Far better trade than LDA.
Path 2: Sage-style shrinkage LDA (TIMSSEEK_RESCORE_MODEL=lda)
deterministic (fixed-chunk reductions). Ridge shrinkage keeps the
collinear 108-dim feature set solvable without pruning.
normality assumption; missingness indicators + abs() of signed error
features as linear-friendly feature engineering.
hard Skyline library. This is a genuine linear-model ceiling, confirmed
offline against sklearn LDA / QDA / logistic (all cluster at 79-82%);
HistGradientBoosting matches forust at 99.6%, so trees are required for
the last ~18%. LDA is useful for QC/triage, not a drop-in replacement.
Methodology
TIMSSEEK_LDA_DUMP=/prefixwrites the raw feature matrix + labels, whichdrove an offline numpy/sklearn harness (feature selection, transforms,
model zoo, tree-config sweep). Kept local under bench_out/ (gitignored).
Feature selection did not help - LDA wants all 108 features; INT is worth
~+5600 IDs; the gap is model capacity, not FE.
Open question
Which operating point should become the default: keep 1000/depth-6, or
move to ~200/31 (2.75x, -2% IDs)? Leaving that decision for the merge.