Speed up cold indexing by 5–11x with adaptive context

[cktang88]

Impact

Cold indexing becomes 5.3x faster on React and 11.3x faster on a large private repository on an Apple M4 Pro with 48 GB RAM, while peak memory falls by 67–73%.

This revision replaces the earlier fixed 64-token policy with adaptive, structure-aware context budgets. That recovers substantially more retrieval quality, but the benchmark results below also show that the speedup is not quality-neutral on every workload.

What changes for users

• Fresh indexes are encoded and written in one pass instead of repeatedly rebuilding index-wide IVF data at each checkpoint. Resumed partial indexes retain checkpointing.
• Encoding uses a bounded worker pool with at most one worker per ONNX session.
• The default code model selects a 64, 128, 256, or 512-token budget from the parsed code unit's type, span, control flow, and complexity. Short/simple units remain cheap; larger functions retain more context.
• JavaScript files marked @flow are parsed with the existing TSX grammar so typed functions remain coherent units.
• Function-local declarations are not separately embedded for JavaScript-family languages when their enclosing function already contains them.
• COLGREP_INDEX_DOCUMENT_LENGTH=0 restores the model-native context. A positive value forces a fixed limit. Custom models continue to use their native limit by default.
• Existing indexes rebuild once because the embedding policy and index format changed.

Cold-index benchmarks

Release binaries used the same base commit, cached model, CPU INT8 inference, CoreML + Accelerate support, 14 ONNX sessions, and isolated cold index directories.

React

Pinned at react/react@28cd4bb, indexing the same 6,959 files:

Metric	main	This PR	Change
Wall time	196.67 s	37.15 s	5.29x faster
CPU time	1,380.17 s	362.50 s	3.81x less
Peak RSS	6.53 GB	1.79 GB	73% lower
Parsed/indexed units	90,602	52,903	Coherent parsing removes duplicate/fragmented units

Private large repository

Both runs indexed the same 10,406 files:

Metric	main	This PR	Change
Wall time	621.07 s	54.75 s	11.34x faster
CPU time	4,575.02 s	509.88 s	8.97x less
Peak RSS	6.49 GB	2.18 GB	66% lower
Parsed/indexed units	119,490	72,255	Coherent parsing removes duplicate/fragmented units

Only aggregate measurements are reported; no proprietary source, paths, fixtures, query text, or repository-specific behavior is included.

Retrieval evaluation

The evaluation uses 100 recent non-merge commit subjects as queries and treats files changed by the corresponding commit as relevant. Searches use --files-only -k 10. This is a reproducible proxy for code-search usefulness, not a human relevance judgment.

React

Search mode	Branch	Recall@1	Recall@5	Recall@10	MRR
Default/hybrid	main	0.41	0.72	0.80	0.536
Default/hybrid	This PR	0.50	0.74	0.83	0.602
Semantic only	main	0.42	0.67	0.75	0.536
Semantic only	This PR	0.47	0.63	0.72	0.542

Default/hybrid search improves at every cutoff. Semantic-only search improves Recall@1 and MRR but loses 4 points at Recall@5 and 3 points at Recall@10.

Private large repository

Search mode	Branch	Recall@1	Recall@5	Recall@10	MRR
Default/hybrid	main	0.52	0.77	0.84	0.637
Default/hybrid	This PR	0.49	0.72	0.79	0.593
Semantic only	main	0.49	0.73	0.79	0.593
Semantic only	This PR	0.36	0.59	0.67	0.462

The private workload shows a modest default/hybrid regression and a material semantic-only regression. The native-context environment override remains available, but reviewers should treat this PR as an explicit performance/quality tradeoff rather than a claim of universal quality parity.

Why cold builds were slow

Three costs compounded:

• Each checkpoint appended to a growing index and rebuilt index-wide IVF data, making a fresh build effectively quadratic.
• Each encoding chunk spawned one OS thread per batch even though only a bounded number of ONNX sessions could run.
• The default model accepted up to 2,048 document tokens, so a small long-document tail dominated transformer inference.

GPU/SIMD changes were not the primary lever on this workload: eliminating repeated index construction, thread oversubscription, fragmented units, and unnecessary transformer context produced the measured gains while retaining the existing cross-platform inference path.

Manual QA

• Compare isolated cold indexes on pinned React.
• Compare isolated cold indexes on the private large repository.
• Run the 100-query retrieval proxy in default/hybrid and semantic-only modes on both repositories.
• Confirm native and fixed-length environment overrides remain available.
• Confirm no proprietary source, paths, fixtures, or queries are present in the branch.
• Decide whether the private-repository semantic-only tradeoff is acceptable before merging.