fix(engine): skip tag-shaped candidate entity names at intake - #3277
Open
JoshFunnell wants to merge 4 commits into
Open
fix(engine): skip tag-shaped candidate entity names at intake#3277JoshFunnell wants to merge 4 commits into
JoshFunnell wants to merge 4 commits into
Conversation
Production banks were measured (2026-08-08) with entity canonical_name values containing embedded newlines -- extraction artifacts -- which shears any line-oriented consumer (psql -A output, logs, exports). Add _normalize_entity_name() and apply it at candidate-entity intake in _prepare_entities_for_resolution(), collapsing all internal whitespace runs (including \n, \r, \t) to a single space and stripping ends. Case handling is unchanged (the registry matches on LOWER(canonical_name) separately). Tested by tests/test_entity_name_hygiene.py.
Measured (2026-08-08): extraction minted entities named domain:lens, domain:host, domain:memory -- category labels, not entities. They behave like broad tags and measurably poisoned entity-based scoping (61.5% vs 94.0% precision on one subject). Add a conservative tag-shape regex (_is_tag_shaped_name) and apply it in _prepare_entities_for_resolution(), after whitespace normalization and against a lowercased copy of the name, silently skipping matches (logged at debug level). Configured entity labels are deliberately "key:value" shaped (vectorize-ioGH-1558, e.g. tag-type label values like use:use-001), so _prepare_entities_for_resolution() now also accepts entity_labels and exempts any name that resolves as a configured label from the skip -- without this exemption, test_intrabatch_dedup_leaves_label_values_separate and test_entity_resolution_does_not_merge_distinct_label_values regressed. Tested by tests/test_entity_name_hygiene.py (attacking cases for URLs, Windows paths, times, and configured labels come first, per the false-positive risk).
Upstream merged the whitespace-normalization half of this PR as vectorize-io#3338 (44b597c) and went further, adding per-fact case-insensitive dedupe and an empty-candidate drop. Upstream's implementation is taken wholesale; only this branch's unique contribution is re-applied on top: skipping tag-shaped category-label candidates (domain:lens) that extraction was minting as entities. Check order is load-bearing: normalize -> empty-drop -> tag-shape -> dedupe. The tag-shape skip must precede dedupe so a skipped tag never enters seen_in_fact and every occurrence increments the counter; after dedupe a repeat tag would be silently deduped instead of counted. Configured entity labels (use:use-001) stay exempt via is_label_entity and still reach resolution. Our normalization tests are dropped as duplicates of upstream's; the tag-shape tests are kept.
verify-generated-files also runs ./scripts/hooks/lint.sh, and the re-applied is_label_entity call was wrapped by hand rather than by ruff. Formatting only; matches the +1/-3 the job reported.
JoshFunnell
changed the title
Contributor
Author
|
Heads-up on a scope change, since a body edit doesn't notify anyone. The whitespace-normalization half of this PR is removed. That work landed independently as #3338 ( What remains is only the tag-shape filter, which #3338 does not cover: extraction was minting category labels ( One ordering detail worth a look during review: the skip runs before the dedupe, so a skipped tag never enters |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extraction was minting category labels as entities —
domain:lens,domain:host,domain:memory— which behave like broad tags and measurably poison entity-based scoping: 61.5% vs 94.0% precision on one subject on a production bank. This skips tag-shaped candidate names at retain intake.What is left in this PR
_is_tag_shaped_name()+ a conservative shape^[a-z][a-z0-9_-]{1,15}:[a-z][a-z0-9_-]{1,24}$_prepare_entities_for_resolution, with askipped_tag_shapedcounter and a debug log lineentity_labelsthreaded through so configured labels can be exemptedCheck order is load-bearing
Tag-shape must run before dedupe: a skipped tag then never enters
seen_in_fact, so every tag-shaped occurrence incrementsskipped_tag_shapedand dedupe only tracks survivors. If it ran after dedupe, a second identical tag would be silently deduped instead of counted. Upstream's normalization, empty-drop and dedupe behaviour is unchanged.Configured labels stay exempt
A tag-shaped name that is a configured entity label (e.g.
use:use-001from a tag-type label group) is exempt viais_label_entityand still reaches entity resolution — label values are deliberatelykey:valueshaped (GH-1558 exact-match path).The shape deliberately does not match URLs, Windows paths, times (
12:30), names with spaces or dots (re: subject), or names with no colon.