Fix UTF-8 partial-match ranking#101
Open
Uzer007-admin wants to merge 2 commits into
Open
Conversation
Calculate partial-match penalties from unmatched Unicode characters and cache character counts for indexed words.
MadSchemas
reviewed
Jul 21, 2026
| if (words.empty()) { | ||
| words.reserve(words_um.size()); | ||
| } | ||
| holder.ReserveWords(words.size() + words_um.size()); |
Contributor
There was a problem hiding this comment.
words.size() + words_um.size() does not looks correct for the cases, when words already contain something. Actually, current logic may overreserve here up to 2 times
| for (auto& w : words_) { | ||
| res += sizeof(w) + w.vids.heap_size(); | ||
| } | ||
| res += wordsCharsLen_.capacity() * sizeof(wordsCharsLen_[0]); |
Contributor
There was a problem hiding this comment.
wordsCharsLen_[0] is unsafe from ASAN/Debug standpoint. Size of value_type would be better
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes partial-match ranking for UTF-8 strings in the fast full-text index.
Problem
The previous ranking formula calculated the partial-match penalty using UTF-8 byte lengths:
For a valid substring match, this is equivalent to:
As a result:
Solution
The penalty is now calculated from the number of unmatched Unicode characters:
The Unicode character count of each unique indexed word is calculated during indexing and cached as a 16-bit value. This avoids repeated UTF-8 decoding in the query hot path.
The cache is kept synchronized during index clearing and recommit operations, and its capacity is included in memory statistics.
Behavior changes
This intentionally changes the ranking of:
There are no public API, configuration, or storage format changes.
The additional memory cost is approximately two bytes per unique indexed word, based on vector capacity.
s PR.