Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _query-dsl/compound/hybrid.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The following table lists all top-level parameters supported by `hybrid` queries
Parameter | Description
:--- | :---
`queries` | An array of one or more query clauses that are used to match documents. A document must match at least one query clause in order to be returned in the results. The documents' relevance scores from all query clauses are combined into one score by applying a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/). The maximum number of query clauses is 5. Required.
`filter` | A filter to apply to all the subqueries of the hybrid query.
`filter` | A filter to apply to all the subqueries of the hybrid query. The filter must be a single query object. To apply multiple filter conditions, combine them in a [Boolean query]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/). For more information, see [Hybrid search with pre-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/pre-filtering/).

### Rescoring hybrid queries
Introduced 2.18
Expand Down
7 changes: 7 additions & 0 deletions _vector-search/ai-search/hybrid-search/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,13 @@ The response contains the matching documents:
```
{% include copy-curl.html %}

## Filtering data

Hybrid search supports two approaches to filtering:

- **Pre-filtering** removes documents before they are scored. To use pre-filtering, add a top-level `filter` to the `hybrid` query. This is the most common approach for filtering hybrid search results. For more information, see [Hybrid search with pre-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/pre-filtering/)
- **Post-filtering** removes documents after all scoring is complete. To use post-filtering, add a `post_filter` to the search request. Use this approach for faceted search with aggregations when you want the facets to reflect the unfiltered query while filtering only the displayed hits. For more information, see [Hybrid search with post-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/post-filtering/).

## Next steps

- Explore our [tutorials]({{site.url}}{{site.baseurl}}/vector-search/tutorials/) to learn how to build AI search applications.
126 changes: 79 additions & 47 deletions _vector-search/ai-search/hybrid-search/post-filtering.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,74 +15,106 @@ You can perform post-filtering on hybrid search results by providing the `post_f

The `post_filter` clause is applied after the search results have been retrieved. Post-filtering is useful for applying additional filters to the search results without impacting the scoring or the order of the results.

Post-filtering does not impact document relevance scores or aggregation results.
Post-filtering does not impact aggregation results.
{: .note}

## Example
To filter all subqueries during query execution instead of filtering the final results, use a common filter. For more information, see [Hybrid search with pre-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/pre-filtering/).

The following example request combines two query clauses---a `term` query and a `match` query---and contains a `post_filter`:
## Example: Faceted search with post-filtering

Post-filtering is commonly used in faceted search, in which the UI displays aggregation counts (such as brand, color, and size filters) alongside search results. Using a `post_filter` keeps the aggregation counts based on the full unfiltered query while filtering only the displayed hits.

Consider an index containing product documents:

```json
{
"name": "Nike Air Max",
"brand": "Nike",
"color": "Red",
"size": 10,
"price": 120,
"category": "Running Shoes"
}
```

A user searches for "running shoes", and the application constructs a query containing aggregations for brand, color, and size:

```json
GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline
POST /products/_search
{
"query": {
"hybrid":{
"queries":[
{
"match":{
"passage_text": "hello"
}
},
{
"term":{
"passage_text":{
"value":"planet"
}
}
}
]
"match": {
"category": "running shoes"
}

},
"post_filter":{
"match": { "passage_text": "world" }
"aggs": {
"brands": {
"terms": { "field": "brand.keyword" }
},
"colors": {
"terms": { "field": "color.keyword" }
},
"sizes": {
"terms": { "field": "size" }
}
}
}
```
{% include copy-curl.html %}

Compare the results to the results in the [example without post-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/#example-combining-a-match-query-and-a-term-query). In the example without post-filtering, the response contains two documents. In this example, the response contains one document because the second document is filtered out:
The response returns hits from all brands:

```
Nike Air Max
Nike Pegasus
Adidas Adizero
Puma Velocity
...
```

The response also returns aggregations that include counts for every brand, color, and size:

```
Brands: Nike (120), Adidas (80), Puma (45)
Colors: Black (90), White (70), Red (55)
Sizes: 8 (40), 9 (60), 10 (85)
```

The aggregations are typically displayed as facet filters in the UI. When a user selects a specific brand (for example, `Nike`) to filter results, using a pre-filter would exclude non-Nike documents before aggregations are computed, causing other brands to disappear from the facet counts.

With `post_filter`, the query and aggregations run on the full result set. The filter is applied only to the displayed hits:

```json
POST /products/_search
{
"took": 18,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 2,
"skipped": 0,
"failed": 0
"query": {
"match": {
"category": "running shoes"
}
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
"aggs": {
"brands": {
"terms": { "field": "brand.keyword" }
},
"max_score": 0.3,
"hits": [
{
"_index": "my-nlp-index",
"_id": "1",
"_score": 0.3,
"_source": {
"id": "s1",
"passage_text": "Hello world"
}
}
]
"colors": {
"terms": { "field": "color.keyword" }
}
},
"post_filter": {
"term": { "brand.keyword": "Nike" }
}
}
```
{% include copy-curl.html %}

The hits contain only Nike products, but the aggregations still reflect the full unfiltered query:

```
Brands: Nike (120), Adidas (80), Puma (45)
Colors: Black (90), White (70), Red (55)
```

All brand options remain visible in the facet, allowing the user to switch brands or compare counts without removing the filter.

## How post-filtering affects search results and scoring

Expand Down Expand Up @@ -122,4 +154,4 @@ After applying a post-filter to the initial query results, the results are as fo

Observe that:
- Document `d2`'s score remains unchanged.
- Document `d4`'s score has changed.
- Document `d4`'s score has changed.
143 changes: 143 additions & 0 deletions _vector-search/ai-search/hybrid-search/pre-filtering.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
---
layout: default
title: Hybrid search with pre-filtering
parent: Hybrid search
grand_parent: AI search
has_children: false
nav_order: 38
---

# Hybrid search with pre-filtering
**Introduced 3.0**
{: .label .label-purple }

You can perform pre-filtering on hybrid search results by providing a top-level `filter` parameter in the `hybrid` query.

The `filter` is applied during query execution to each subquery, so only documents that match the filter are scored. Pre-filtering is useful for applying the same filter to all subqueries without duplicating it in each one.

The `filter` must be a single query object.
{: .note}

To filter the final results after they have been retrieved instead of filtering each subquery during execution, use a post-filter. For more information, see [Hybrid search with post-filtering]({{site.url}}{{site.baseurl}}/vector-search/ai-search/hybrid-search/post-filtering/).

## Example

The following example request combines a `match` query and a `knn` query and applies a common `filter` that restricts both subqueries to the `shoes` category:

```json
POST /products/_search?search_pipeline=nlp-search-pipeline
{
"query": {
"hybrid": {
"filter": {
"term": { "category": "shoes" }
},
"queries": [
{
"match": { "description": "running shoes" }
},
{
"knn": {
"embedding": {
"vector": [1.23, 0.45, 0.67, ...],
"k": 10
}
}
}
]
}
}
}
```
{% include copy-curl.html %}

OpenSearch applies the `category: shoes` filter to both the `match` and `knn` subqueries, which is equivalent to the following query that applies the filter to each subquery individually:

```json
POST /products/_search?search_pipeline=nlp-search-pipeline
{
"query": {
"hybrid": {
"queries": [
{
"bool": {
"must": {
"match": { "description": "running shoes" }
},
"filter": {
"term": { "category": "shoes" }
}
}
},
{
"knn": {
"embedding": {
"vector": [1.23, 0.45, 0.67, ...],
"k": 10,
"filter": {
"term": { "category": "shoes" }
}
}
}
}
]
}
}
}
```
{% include copy-curl.html %}

## Filtering on multiple conditions

Because the `filter` must be a single query object, combine multiple conditions using a [Boolean query]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/). The following example filters results to only in-stock shoes:

```json
"filter": {
"bool": {
"must": [
{ "term": { "category": "shoes" }},
{ "term": { "in_stock": true }}
]
}
}
```

## Combining a common filter with subquery filters

A subquery can define its own filter in addition to the common filter. In this case, OpenSearch combines the two using a logical `AND`, further narrowing that subquery's results.

In the following example, the common filter restricts all subqueries to the `shoes` category, while the `match` subquery is additionally narrowed to the `nike` brand:

```json
POST /products/_search?search_pipeline=nlp-search-pipeline
{
"query": {
"hybrid": {
"filter": {
"term": { "category": "shoes" }
},
"queries": [
{
"bool": {
"must": {
"match": { "description": "running shoes" }
},
"filter": {
"term": { "brand": "nike" }
}
}
},
{
"knn": {
"embedding": {
"vector": [1.23, 0.45, 0.67, ...],
"k": 10
}
}
}
]
}
}
}
```
{% include copy-curl.html %}
Loading