Use direct I/O for merge-time reads of raw vectors

[john-mlika]

Open the merge-side reader with a direct-I/O context when direct I/O is enabled, created lazily on the first getMergeInstance() call and reading through a merge-sized (256 KiB) buffer rather than the 8 KiB rescore buffer. This stops merges from re-populating the page cache with the raw vectors that on_disk_rescore deliberately keeps out of memory. Mechanism, history around #127406, and benchmarks are in the issue.

Closes #155021

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[leemthompo]

The inline applies_to tag is floating between sentences, which makes its scope ambiguous. Moving the new sentence to its own paragraph at the end and prefixing it with the tag makes the scope clear.

For the new content, defer to #155919 (comment)

Suggested change

	: (Optional, boolean) Only applicable to quantized HNSW and `bbq_disk` index types. When `true`, vector rescoring will read the raw vector data directly from disk, and will not copy it in memory. This can improve performance when vector data is larger than the amount of available RAM. For `bbq_hnsw` indices, segment merges also read the raw vector data directly from disk where the platform supports it, so that merging is less likely to evict more frequently accessed data from the filesystem cache. {applies_to}`stack: ga 9.6` This setting only applies to newly-indexed vectors; after changing this setting, the vectors must be reindexed or force-merged to apply the new setting to the whole index. Defaults to `false`.
	: (Optional, boolean) Only applicable to quantized HNSW and `bbq_disk` index types. When `true`, vector rescoring will read the raw vector data directly from disk, and will not copy it in memory. This can improve performance when vector data is larger than the amount of available RAM. This setting only applies to newly-indexed vectors; after changing this setting, the vectors must be reindexed or force-merged to apply the new setting to the whole index. Defaults to `false`.
	{applies_to}`stack: ga 9.6` For `bbq_hnsw` indices, segment merges also read the raw vector data directly from disk where the platform supports it, so that merging is less likely to evict more frequently accessed data from the filesystem cache.

@thecoop: should the parent on_disk_rescore tag also be updated from

{applies_to}`stack: preview 9.3`

to

{applies_to}`stack: preview 9.3, ga X.x`

?

Or is on_disk_rescore still a preview thing?

[thecoop]

I'm not sure we can say directIO is ga yet, there's still various aspects we need to check - in particular whether we use direct IO for merges generally. We're due to come back to this soon, so we can re-evaluate it then

[jimczi]

Thanks for the deep dive here, the page cache analysis is really useful. I like the idea of
using direct I/O for merges.

I'd like to separate two things that are mixed together in this PR, a bug and a feature.

The bug is that MergeReaderWrapper returns its merge reader without calling
getMergeInstance() on it, and it doesn't implement finishMerge() either, so both calls
stop at the wrapper. Same problem I hit in IVFVectorsReader, so I've extended #153423 to
cover the wrapper and ES818BinaryQuantizedVectorsReader too.

I think we should land that one first since it changes the baseline you're comparing
against. I ran a local benchmark and just restoring the RANDOM to SEQUENTIAL advice was
worth about 2.2x on merge read throughput. On that box (capped around 450 MiB/s by the
device) mmap with the right advice was then as fast as direct I/O. So the -37.6% merge time
in the issue might be mostly the missing advice and not direct I/O. Would be good to
re-measure on your setup once #153423 is in. The page cache residency numbers aren't
affected by any of this, they stand on their own.

Then the feature part. Merges bypassing the page cache when you configure it sounds great
to me. Two things I'd like to figure out:

1. Should we do the same for writes? HybridDirectory only uses its DirectIODirectory in
   openInput, so merge writes still go through the buffered output. If the point is to stop
   merges from disturbing the page cache, writing the new segment through it seems like it
   would undo part of that. Did you look at whether the write side matters here?

2. Can we make merge and rescore independent? Today on_disk_rescore turns on direct I/O
   for both, but they want different things. Rescoring does small random reads where direct
   I/O can be slower, merges read everything sequentially. I think "use direct I/O for
   merges, keep mmap with random access for rescoring" is the combination most people
   actually want. The flag we persist is only a read hint and doesn't change the layout, so
   a separate knob for merges shouldn't need a format change.

[john-mlika]

sounds fair, once #153423 lands, i'll rebase and re-measure on top of it. and agreed the residency numbers are unaffected either way.

on the 2.2x: i think my numbers and yours are both right but they're measuring different bottlenecks. mine ran memory-constrained — during a merge the sources, the new-segment writeback and an unrelated pre-warmed file want ~38 GB between them, against ~25 GB of usable page cache.

the device wasn't the limit there — that NVMe does 3.4-3.9 GB/s under O_DIRECT. so the baseline wasn't bandwidth-bound, it was reclaim-bound: the merge faults its sources in repeatedly while the kernel reclaims underneath it. i would say direct I/O isn't faster than RAM, it's faster than faulting through a thrashing cache.

your box sounds device-bound at ~450 MiB/s, about 8x slower than mine. at that point mmap-with-the-right-advice and direct I/O should both saturate the device and tie, which is exactly what you saw. SEQUENTIAL advice fixes readahead, but it doesn't stop the merge from filling the cache, so i'd expect it to close much less of the gap once memory is the constraint rather than bandwidth. that's my prediction at least.

writes: yes, and it matters. even with direct-IO reads the merged output is still buffered-written — ~82% of the new .vec stays resident and the file cache still saturates at ~24.8 GB. so the read side alone only gets you part of the way. i have the write-side change working but it's stacked on this one and needs its own numbers, so i'd rather not fold it in here.

decoupling merge from rescore: agree, and that's most of the way done already. after thecoop's round 1 the PR builds two direct-IO directories and picks between them on context() — search gets 8 KiB with async prefetch, merges get 256 KiB with none. so the behaviours are already separate internally; what's missing is a knob that lets you turn merges on without turning rescore on. happy to add that here since it's a read hint either way and doesn't touch the format — or keep this one small and do it as a follow-up, whichever you prefer.

also relevant: the lucene half of the finishMerge cascade just landed — apache/lucene#16481 into branch_10x, moving finishMerge into a finally block (backport of #14977) so the advice actually gets reset after a merge. your wrapper fix is the other half of the same problem.