diff --git a/_query-dsl/compound/hybrid.md b/_query-dsl/compound/hybrid.md index 9f22e4567de..bd67f3953e7 100644 --- a/_query-dsl/compound/hybrid.md +++ b/_query-dsl/compound/hybrid.md @@ -22,7 +22,7 @@ The following table lists all top-level parameters supported by `hybrid` queries Parameter | Description :--- | :--- `queries` | An array of one or more query clauses that are used to match documents. A document must match at least one query clause in order to be returned in the results. The documents' relevance scores from all query clauses are combined into one score by applying a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/). The maximum number of query clauses is 5. Required. -`filter` | A filter to apply to all the subqueries of the hybrid query. +`filter` | A filter to apply to all the subqueries of the hybrid query. The filter must be a single query object. To apply multiple filter conditions, combine them in a [Boolean query]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/). For more information, see [Hybrid search with pre-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/pre-filtering/). ### Rescoring hybrid queries Introduced 2.18 diff --git a/_vector-search/ai-search/hybrid-search/index.md b/_vector-search/ai-search/hybrid-search/index.md index 392b2a76030..3f30a177e05 100644 --- a/_vector-search/ai-search/hybrid-search/index.md +++ b/_vector-search/ai-search/hybrid-search/index.md @@ -349,6 +349,13 @@ The response contains the matching documents: ``` {% include copy-curl.html %} +## Filtering data + +Hybrid search supports two approaches to filtering: + +- **Pre-filtering** removes documents before they are scored. To use pre-filtering, add a top-level `filter` to the `hybrid` query. This is the most common approach for filtering hybrid search results. For more information, see [Hybrid search with pre-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/pre-filtering/) +- **Post-filtering** removes documents after all scoring is complete. To use post-filtering, add a `post_filter` to the search request. Use this approach for faceted search with aggregations when you want the facets to reflect the unfiltered query while filtering only the displayed hits. For more information, see [Hybrid search with post-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/post-filtering/). + ## Next steps - Explore our [tutorials]({{site.url}}{{site.baseurl}}/vector-search/tutorials/) to learn how to build AI search applications. \ No newline at end of file diff --git a/_vector-search/ai-search/hybrid-search/post-filtering.md b/_vector-search/ai-search/hybrid-search/post-filtering.md index 7436d2862a2..c9f8113187d 100644 --- a/_vector-search/ai-search/hybrid-search/post-filtering.md +++ b/_vector-search/ai-search/hybrid-search/post-filtering.md @@ -15,74 +15,106 @@ You can perform post-filtering on hybrid search results by providing the `post_f The `post_filter` clause is applied after the search results have been retrieved. Post-filtering is useful for applying additional filters to the search results without impacting the scoring or the order of the results. -Post-filtering does not impact document relevance scores or aggregation results. +Post-filtering does not impact aggregation results. {: .note} -## Example +To filter all subqueries during query execution instead of filtering the final results, use a common filter. For more information, see [Hybrid search with pre-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/pre-filtering/). -The following example request combines two query clauses---a `term` query and a `match` query---and contains a `post_filter`: +## Example: Faceted search with post-filtering + +Post-filtering is commonly used in faceted search, in which the UI displays aggregation counts (such as brand, color, and size filters) alongside search results. Using a `post_filter` keeps the aggregation counts based on the full unfiltered query while filtering only the displayed hits. + +Consider an index containing product documents: + +```json +{ + "name": "Nike Air Max", + "brand": "Nike", + "color": "Red", + "size": 10, + "price": 120, + "category": "Running Shoes" +} +``` + +A user searches for "running shoes", and the application constructs a query containing aggregations for brand, color, and size: ```json -GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline +POST /products/_search { "query": { - "hybrid":{ - "queries":[ - { - "match":{ - "passage_text": "hello" - } - }, - { - "term":{ - "passage_text":{ - "value":"planet" - } - } - } - ] + "match": { + "category": "running shoes" } - }, - "post_filter":{ - "match": { "passage_text": "world" } + "aggs": { + "brands": { + "terms": { "field": "brand.keyword" } + }, + "colors": { + "terms": { "field": "color.keyword" } + }, + "sizes": { + "terms": { "field": "size" } + } } } ``` {% include copy-curl.html %} -Compare the results to the results in the [example without post-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/#example-combining-a-match-query-and-a-term-query). In the example without post-filtering, the response contains two documents. In this example, the response contains one document because the second document is filtered out: +The response returns hits from all brands: + +``` +Nike Air Max +Nike Pegasus +Adidas Adizero +Puma Velocity +... +``` + +The response also returns aggregations that include counts for every brand, color, and size: + +``` +Brands: Nike (120), Adidas (80), Puma (45) +Colors: Black (90), White (70), Red (55) +Sizes: 8 (40), 9 (60), 10 (85) +``` + +The aggregations are typically displayed as facet filters in the UI. When a user selects a specific brand (for example, `Nike`) to filter results, using a pre-filter would exclude non-Nike documents before aggregations are computed, causing other brands to disappear from the facet counts. + +With `post_filter`, the query and aggregations run on the full result set. The filter is applied only to the displayed hits: ```json +POST /products/_search { - "took": 18, - "timed_out": false, - "_shards": { - "total": 2, - "successful": 2, - "skipped": 0, - "failed": 0 + "query": { + "match": { + "category": "running shoes" + } }, - "hits": { - "total": { - "value": 1, - "relation": "eq" + "aggs": { + "brands": { + "terms": { "field": "brand.keyword" } }, - "max_score": 0.3, - "hits": [ - { - "_index": "my-nlp-index", - "_id": "1", - "_score": 0.3, - "_source": { - "id": "s1", - "passage_text": "Hello world" - } - } - ] + "colors": { + "terms": { "field": "color.keyword" } + } + }, + "post_filter": { + "term": { "brand.keyword": "Nike" } } } ``` +{% include copy-curl.html %} + +The hits contain only Nike products, but the aggregations still reflect the full unfiltered query: + +``` +Brands: Nike (120), Adidas (80), Puma (45) +Colors: Black (90), White (70), Red (55) +``` + +All brand options remain visible in the facet, allowing the user to switch brands or compare counts without removing the filter. ## How post-filtering affects search results and scoring @@ -122,4 +154,4 @@ After applying a post-filter to the initial query results, the results are as fo Observe that: - Document `d2`'s score remains unchanged. -- Document `d4`'s score has changed. \ No newline at end of file +- Document `d4`'s score has changed. diff --git a/_vector-search/ai-search/hybrid-search/pre-filtering.md b/_vector-search/ai-search/hybrid-search/pre-filtering.md new file mode 100644 index 00000000000..81d3672446c --- /dev/null +++ b/_vector-search/ai-search/hybrid-search/pre-filtering.md @@ -0,0 +1,143 @@ +--- +layout: default +title: Hybrid search with pre-filtering +parent: Hybrid search +grand_parent: AI search +has_children: false +nav_order: 38 +--- + +# Hybrid search with pre-filtering +**Introduced 3.0** +{: .label .label-purple } + +You can perform pre-filtering on hybrid search results by providing a top-level `filter` parameter in the `hybrid` query. + +The `filter` is applied during query execution to each subquery, so only documents that match the filter are scored. Pre-filtering is useful for applying the same filter to all subqueries without duplicating it in each one. + +The `filter` must be a single query object. +{: .note} + +To filter the final results after they have been retrieved instead of filtering each subquery during execution, use a post-filter. For more information, see [Hybrid search with post-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/post-filtering/). + +## Example + +The following example request combines a `match` query and a `knn` query and applies a common `filter` that restricts both subqueries to the `shoes` category: + +```json +POST /products/_search?search_pipeline=nlp-search-pipeline +{ + "query": { + "hybrid": { + "filter": { + "term": { "category": "shoes" } + }, + "queries": [ + { + "match": { "description": "running shoes" } + }, + { + "knn": { + "embedding": { + "vector": [1.23, 0.45, 0.67, ...], + "k": 10 + } + } + } + ] + } + } +} +``` +{% include copy-curl.html %} + +OpenSearch applies the `category: shoes` filter to both the `match` and `knn` subqueries, which is equivalent to the following query that applies the filter to each subquery individually: + +```json +POST /products/_search?search_pipeline=nlp-search-pipeline +{ + "query": { + "hybrid": { + "queries": [ + { + "bool": { + "must": { + "match": { "description": "running shoes" } + }, + "filter": { + "term": { "category": "shoes" } + } + } + }, + { + "knn": { + "embedding": { + "vector": [1.23, 0.45, 0.67, ...], + "k": 10, + "filter": { + "term": { "category": "shoes" } + } + } + } + } + ] + } + } +} +``` +{% include copy-curl.html %} + +## Filtering on multiple conditions + +Because the `filter` must be a single query object, combine multiple conditions using a [Boolean query]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/). The following example filters results to only in-stock shoes: + +```json +"filter": { + "bool": { + "must": [ + { "term": { "category": "shoes" }}, + { "term": { "in_stock": true }} + ] + } +} +``` + +## Combining a common filter with subquery filters + +A subquery can define its own filter in addition to the common filter. In this case, OpenSearch combines the two using a logical `AND`, further narrowing that subquery's results. + +In the following example, the common filter restricts all subqueries to the `shoes` category, while the `match` subquery is additionally narrowed to the `nike` brand: + +```json +POST /products/_search?search_pipeline=nlp-search-pipeline +{ + "query": { + "hybrid": { + "filter": { + "term": { "category": "shoes" } + }, + "queries": [ + { + "bool": { + "must": { + "match": { "description": "running shoes" } + }, + "filter": { + "term": { "brand": "nike" } + } + } + }, + { + "knn": { + "embedding": { + "vector": [1.23, 0.45, 0.67, ...], + "k": 10 + } + } + } + ] + } + } +} +``` +{% include copy-curl.html %}