Elasticsearch:Use the IK analyzer plugin (analysis-ik)

Tokenization rules

The IK analyzer supports two tokenization modes:

• ik_max_word: Splits text at the finest granularity. Ideal for term-based queries.

• ik_smart: Splits text at a coarser granularity. Ideal for phrase-based queries.

Dictionary types

The IK analyzer supports the following dictionary types.

Dictionary update methods

Update dictionaries when the defaults do not meet your needs. The IK analyzer supports the following methods.

Cold update

To perform a cold update:

1. Go to the instance details page.

   1. Log on to the Alibaba Cloud Elasticsearch console.

   2. In the navigation pane on the left, click Elasticsearch Clusters.

   3. In the top menu bar, select a resource group and region.

   4. In the Elasticsearch instance list, click the target instance ID to go to its details page.

2. Go to the cold update page for the analysis-ik plugin.

   1. In the navigation pane on the left, choose Configuration and Management > Plug-ins.

   2. On the Built-in Plug-ins tab, find the analysis-ik plugin and click Standard Update in the Actions column.

3. Perform the cold update.

   1. In the Configure IK Dictionaries - Standard Update dialog box, click Edit next to the target dictionary, upload the required dictionary file as instructed, then click Save.

      You can upload dictionary files using one of the following methods:

      • Upload On-premises File: Click the icon or drag and drop your local file as instructed.

      • Upload OSS File: Enter the bucket name and dictionary file name, then click Add.

        • The bucket and Alibaba Cloud ES instance must be in the same region.

        • OSS dictionary files are not automatically synchronized. If the source file changes, perform a dictionary update to apply changes.

      Note

      • Each dictionary type supports only one DIC-format file. The uploaded file replaces the original dictionary.

      • Dictionary file names must end with .dic. File names can contain letters, digits, and underscores, and must not exceed 30 characters.

      • To restore a dictionary to its default file, download the default file and re-upload it. Obtain the default dictionary files from the following links:

        • Main dictionary (main.dic)

        • Stop word dictionary (stopword.dic)

        • Preposition dictionary (preposition.dic)

        • Quantifier dictionary (quantifier.dic)

   2. Select the risk acknowledgment and click OK to restart the instance.

      After the instance restarts, the dictionary update is complete.

4. (Optional) Test whether the dictionary update took effect.

   1. Log on to the Kibana console.

   2. Click the icon in the upper-left corner and choose Management > Developer Tools to open the code editor.

      Run the following code to perform coarse-grained tokenization on the input text Chinese character input method for computers.

      Note

      In practice, replace the text value with a word from your dictionary.

      GET _analyze
      {
        "analyzer": "ik_smart",
        "text": "Chinese character input method for computers"
      }

      The expected response is as follows.

      {
        "tokens" : [
          {
            "token" : "computer",
            "start_offset" : 0,
            "end_offset" : 3,
            "type" : "CN_WORD",
            "position" : 0
          },
          {
            "token" : "Chinese character input",
            "start_offset" : 3,
            "end_offset" : 7,
            "type" : "CN_WORD",
            "position" : 1
          },
          {
            "token" : "method",
            "start_offset" : 7,
            "end_offset" : 9,
            "type" : "CN_WORD",
            "position" : 2
          }
        ]
      }

Hot update

To perform a hot update:

1. Go to the instance details page.

   1. Log on to the Alibaba Cloud Elasticsearch console.

   2. In the navigation pane on the left, click Elasticsearch Clusters.

   3. In the top menu bar, select a resource group and region.

   4. In the Elasticsearch instance list, click the target instance ID to go to its details page.

2. Go to the hot update page for the analysis-ik plugin.

   1. In the navigation pane on the left, choose Configuration and Management > Plug-ins.

   2. On the Built-in Plug-ins tab, find the analysis-ik plugin and click Rolling Update in the Actions column.

3. Perform the hot update.

   1. In the Configure IK Dictionaries - Rolling Update dialog box, click Edit next to the target dictionary, upload the required dictionary file as instructed, then click Save.

      You can upload dictionary files using one of the following methods:

      • Upload On-premises File: Click the icon or drag and drop your local file as instructed.

      • Upload OSS File: Enter the bucket name and dictionary file name, then click Add.

        • The bucket and Alibaba Cloud ES instance must be in the same region.

        • OSS dictionary files are not automatically synchronized. If the source file changes, perform a dictionary update to apply changes.

      Note

      • You can upload multiple dictionary files. File names must end with .dic. File names can contain letters, digits, and underscores, and must not exceed 30 characters.

      • To modify an uploaded dictionary file, click the icon next to it to download and edit the file. Then delete the original file and re-upload the updated one. After deleting the original file, click Save; otherwise, the system reports that a file with the same name already exists.

   2. Click OK and wait for ES nodes to finish loading the dictionary.

      ES nodes load dictionary files automatically. Loading times vary across nodes — wait until all nodes finish.

4. (Optional) Test whether the dictionary update took effect.

   1. Log on to the Kibana console.

   2. Click the icon in the upper-left corner and choose Management > Developer Tools to open the code editor.

      Run the following code to perform coarse-grained tokenization on the input text Chinese character input method for computers.

      Note

      In practice, replace the text value with a word from your dictionary.

      GET _analyze
      {
        "analyzer": "ik_smart",
        "text": "Chinese character input method for computers"
      }

      The expected response is as follows.

      {
        "tokens" : [
          {
            "token" : "computer",
            "start_offset" : 0,
            "end_offset" : 3,
            "type" : "CN_WORD",
            "position" : 0
          },
          {
            "token" : "Chinese character input",
            "start_offset" : 3,
            "end_offset" : 7,
            "type" : "CN_WORD",
            "position" : 1
          },
          {
            "token" : "method",
            "start_offset" : 7,
            "end_offset" : 9,
            "type" : "CN_WORD",
            "position" : 2
          }
        ]
      }

Optimize tokenization for alphanumeric mixed words

When you use the ik_max_word tokenizer, alphanumeric mixed strings such as fawjh6bcm may not be tokenized effectively for search. To improve recall, set enable_single_word to true. This splits mixed words into single-character tokens, enabling character-level matching.

The following example creates a custom analyzer with enable_single_word enabled:

PUT my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "ik_single_char_analyzer": {
          "type": "custom",
          "tokenizer": "my_ik_tokenizer"
        }
      },
      "tokenizer": {
        "my_ik_tokenizer": {
          "type": "ik_max_word",
          "enable_single_word": true
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "content": {
        "type": "text",
        "analyzer": "ik_single_char_analyzer"
      }
    }
  }
}

Use the ngram tokenizer for mixed Chinese-English search

When searching for concatenated English model numbers (such as jh6bcm) mixed with Chinese text, the IK analyzer alone may not produce effective tokens for partial matching. In this case, use the ngram tokenizer to complement or replace the IK analyzer.

The ngram tokenizer breaks text into contiguous character sequences of configurable length, enabling substring matching.

The following example creates an index that uses the ngram tokenizer:

PUT my_ngram_index
{
  "settings": {
    "analysis": {
      "tokenizer": {
        "my_ngram_tokenizer": {
          "type": "ngram",
          "min_gram": 2,
          "max_gram": 3,
          "token_chars": ["letter", "digit"]
        }
      },
      "analyzer": {
        "ngram_analyzer": {
          "type": "custom",
          "tokenizer": "my_ngram_tokenizer"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "model_number": {
        "type": "text",
        "analyzer": "ngram_analyzer"
      }
    }
  }
}

For more information about the ngram tokenizer configuration, see Ngram tokenizer.

Uppercase words in synonym files cause errors. What do I do?

The IK analyzer is case-sensitive when processing synonym files. If a synonym file contains uppercase words (such as IT), parsing errors may occur during indexing.

You can resolve this issue using either of the following methods.

• Convert all synonyms to lowercase: Edit your synonym file and convert all entries to lowercase before uploading.

• Add a lowercase filter to the analyzer: Add a lowercase filter before the synonym filter in your analyzer configuration to normalize text before synonym matching. Example:

  PUT my_index
  {
    "settings": {
      "analysis": {
        "filter": {
          "my_synonym_filter": {
            "type": "synonym",
            "synonyms_path": "your_synonyms.txt"
          }
        },
        "analyzer": {
          "ik_synonym_analyzer": {
            "type": "custom",
            "tokenizer": "ik_max_word",
            "filter": ["lowercase", "my_synonym_filter"]
          }
        }
      }
    }
  }

  Important

  The lowercase filter must appear before the synonym filter in the filter chain to ensure text is converted to lowercase before synonym matching occurs.