Skip to content

feat(search): add ngram support for Japanese text analysis in Elastic… #2914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
hachimitsu-pichi wants to merge 6 commits into
base: master
Choose a base branch
from
Draft

feat(search): add ngram support for Japanese text analysis in Elastic… #2914

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0b16030
feat(search): add ngram support for Japanese text analysis in Elastic…
hachimitsu-pichi Jan 21, 2026
be8445f
feat(search): add ICU normalization char filter to Elasticsearch sett…
hachimitsu-pichi Jan 21, 2026
01343b1
feat(search): add ngram field to Elasticsearch query for improved tex…
hachimitsu-pichi Jan 21, 2026
1366794
feat(search): refactor ja_ngram_tokenizer settings for improved reada…
hachimitsu-pichi Jan 21, 2026
e16a4ae
Merge branch 'master' into feat/ngram-elasticsearch
Eraxyso Jan 23, 2026
11adb62
Merge branch 'master' into feat/ngram-elasticsearch
howard12721 Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 59 additions & 16 deletions service/search/es.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,13 @@ var esMapping = m{
"text": m{
"type": "text",
"analyzer": "sudachi_analyzer",
"fields": m{
"ngram": m{
"type": "text",
"search_analyzer": "ja_ngram_search_analyzer",
"analyzer": "ja_ngram_index_analyzer",
},
},
Comment thread
howard12721 marked this conversation as resolved.
Comment thread
howard12721 marked this conversation as resolved.
},
"createdAt": m{
"type": "date",
Expand Down Expand Up @@ -144,28 +151,64 @@ var esMapping = m{
var esSetting = m{
"index": m{
"analysis": m{
"char_filter": m{
"normalize": m{
"type": "icu_normalizer",
"name": "nfkc",
"mode": "compose",
},
},
"tokenizer": m{
"sudachi_tokenizer": m{
"type": "sudachi_tokenizer",
},
"ja_ngram_tokenizer": m{
"type": "ngram",
"min_gram": 2,
"max_gram": 2,
"token_chars": []string{
"letter",
"digit",
},
},
},
},
"filter": m{
"sudachi_split_filter": m{
"type": "sudachi_split",
Comment thread
howard12721 marked this conversation as resolved.
"mode": "search",
},
},
"analyzer": m{
Comment on lines +175 to +182

@coderabbitai coderabbitai Bot Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== service/search/es.go around cited lines ==\n'
nl -ba service/search/es.go | sed -n '130,260p'

printf '\n== occurrences of analysis/filter/analyzer in file ==\n'
rg -n '"analysis"|"filter"|"analyzer"|text\.ngram|sudachi' service/search/es.go

Repository: traPtitech/traQ

Length of output: 240

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== service/search/es.go around cited lines ==\n'
awk 'NR>=130 && NR<=260 {printf "%d\t%s\n", NR, $0}' service/search/es.go

printf '\n== occurrences of analysis/filter/analyzer in file ==\n'
rg -n '"analysis"|"filter"|"analyzer"|text\.ngram|sudachi' service/search/es.go

Repository: traPtitech/traQ

Length of output: 4727

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== esSetting usage ==\n'
rg -n '\besSetting\b|Indices\.Create|settings' service/search/es.go

printf '\n== surrounding index creation code ==\n'
awk 'NR>=260 && NR<=420 {printf "%d\t%s\n", NR, $0}' service/search/es.go

Repository: traPtitech/traQ

Length of output: 5572

🌐 Web query:

Elasticsearch custom analyzer settings index.analysis filter analyzer char_filter tokenizer official docs create index

💡 Result:

To create a custom analyzer in Elasticsearch, you must define it within the settings of the index during index creation or update [1][2]. A custom analyzer is composed of three main components: a tokenizer (required), zero or more character filters, and zero or more token filters [1][3]. When defining a custom analyzer, you can use built-in components or create your own custom versions of tokenizers, filters, and character filters within the same analysis configuration block [1][4]. Example configuration for creating a custom analyzer: PUT /my-index { "settings": { "analysis": { "analyzer": { "my_custom_analyzer": { "type": "custom", "tokenizer": "my_tokenizer", "char_filter": ["my_char_filter"], "filter": ["my_token_filter", "lowercase"] } }, "tokenizer": { "my_tokenizer": { "type": "pattern", "pattern": "[.,!?]" } }, "char_filter": { "my_char_filter": { "type": "mapping", "mappings": [":) => happy"] } }, "filter": { "my_token_filter": { "type": "stop", "stopwords": "english" } } } } } Key considerations: 1. Analyzer Type: When defining a custom analyzer, set the type to custom or omit the type parameter entirely [1][5]. 2. Index Modification: New analyzers can generally only be defined on closed indices or during index creation [6]. If you need to add an analyzer to an existing index, you must close the index, update the settings, and then reopen it [6]. 3. Default Analyzer: You can set a custom analyzer as the default for an index by naming it default in the analyzer settings (e.g., analysis.analyzer.default) [7]. For more complex configurations, refer to the official Elasticsearch documentation for the specific tokenizer, character filter, or token filter types you are implementing, as each has its own set of configurable parameters [1][4][8].

Citations:

Keep filter and analyzer under analysis.
esSetting closes analysis before those blocks, so sudachi_split_filter and the custom analyzers are emitted as index.filter / index.analyzer instead of settings.analysis.*. Elasticsearch expects custom analysis components under settings.analysis, so this index body won’t register them correctly.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@service/search/es.go` around lines 175 - 182, The custom analysis settings
are being emitted outside the analysis block, so `sudachi_split_filter` and the
analyzer definitions are ending up under the wrong index settings keys. Update
`esSetting` in `search/es.go` so the `filter` and `analyzer` maps remain nested
under `analysis` alongside the other analysis components, using the existing
`esSetting` structure and its `analysis` builder.

"sudachi_analyzer": m{
"tokenizer": "sudachi_tokenizer",
"type": "custom",
"filter": []string{
"sudachi_split_filter",
"sudachi_normalizedform",
},
"discard_punctuation": true,
"resources_path": "/usr/share/elasticsearch/plugins/analysis-sudachi/",
"settings_path": "/usr/share/elasticsearch/plugins/analysis-sudachi/sudachi.json",
},
"filter": m{
"sudachi_split_filter": m{
"type": "sudachi_split",
"mode": "search",
"ja_ngram_index_analyzer": m{
"type": "custom",
"char_filter": []string{
"normalize",
},
"tokenizer": "ja_ngram_tokenizer",
Comment thread
howard12721 marked this conversation as resolved.
"filter": []string{
"lowercase",
},
},
"analyzer": m{
"sudachi_analyzer": m{
"tokenizer": "sudachi_tokenizer",
"type": "custom",
"filter": []string{
"sudachi_split_filter",
"sudachi_normalizedform",
},
"discard_punctuation": true,
"resources_path": "/usr/share/elasticsearch/plugins/analysis-sudachi/",
"settings_path": "/usr/share/elasticsearch/plugins/analysis-sudachi/sudachi.json",
"ja_ngram_search_analyzer": m{
"type": "custom",
"char_filter": []string{
"normalize",
},
"tokenizer": "ja_ngram_tokenizer",
"filter": []string{
"lowercase",
},
},
},
Expand Down Expand Up @@ -305,7 +348,7 @@ func (e *esEngine) Do(q *Query) (Result, error) {
if q.Word.Valid {
body := simpleQueryString{
Query: q.Word.V,
Fields: []string{"text"},
Fields: []string{"text", "text.ngram"},
DefaultOperator: "AND",
}

Expand Down