Overview
You can specify tokenization methods for fields in the analyzer clause. The system converts texts in the specified fields to terms based on the tokenization methods that you specified. You can specify a tokenization method for all fields or specific fields. You can also specify fields whose values you want the system to not convert to terms.
Syntax
analyzer=analyzer_description[{',' analyzer_description}]Multiple analyzer_description parameters are separated by commas (,). The following table describes the available parameters.
Parameter | Valid value | Default value | Description |
no_tokenize_indexes | index_name1';'index_name2 | None | Specifies the fields whose values you want the system to not convert to terms. You must separate multiple fields with semicolons (;). This parameter does not affect other operations such as normalization and stop word removal that are performed on the values of the specified fields. |
specific_index_analyzer | index_name'#'analyzer_name | None | Specifies an analyzer for a specified field. The analyzer that is specified by this parameter is assigned a higher priority than the analyzer that is specified by the global_analyzer parameter and the analyzer that is specified in the schema. Separate multiple fields with semicolons (;). Example: index1'#'analyzer1;index2'#'analyzer2. The analyzers that you specify in this parameter must be specified in the analyzer.json file. |
global_analyzer | IDENTIFIER | None | Specifies a global analyzer that can be used for all fields. The analyzer that is specified by this parameter is assigned a higher priority than the analyzer that is specified in the schema. The analyzers that you specify in this parameter must be specified in the analyzer.json file. If you do not set the global_analyzer parameter or do not specify an analysis method for a field by using the specific_index_analyzer and no_tokenize_indexes parameters, the system uses the analyzer that is specified in the schema.json file to convert the text in the field to terms. |
Example:
Disable tokenization for specific fields (
no_tokenize_indexes):analyzer=no_tokenize_indexes:defaultNote: In the query, the
defaultfield is not tokenized. However, other analysis processes like normalization and stop word removal still apply. To specify multiple fields, separate them with a semicolon (;), for example,no_tokenize_indexes:index1;index2.Specify an analyzer for a specific field (
specific_index_analyzer):analyzer=specific_index_analyzer:index2#aliwsNote: The
index2field uses thealiwsanalyzer. This setting overrides any analyzer defined byglobal_analyzeror in the schema. To specify multiple fields, separate them with a semicolon (;), for example,specific_index_analyzer:index1#extend_analyzer;index2#aliws. The analyzer must be defined in youranalyzer.jsonconfiguration file.Specify a global analyzer (
global_analyzer):analyzer=global_analyzer:singlews_analyzerNote: This sets
singlews_analyzeras the global analyzer, which overrides the analyzer defined in the schema. However, this setting does not affect fields specified inno_tokenize_indexesandspecific_index_analyzer. The analyzer must be defined in youranalyzer.jsonconfiguration file.Combine multiple parameters (separated by commas):
analyzer=global_analyzer:singlews_analyzer,no_tokenize_indexes:default,specific_index_analyzer:default#extend_analyzer;index2#aliws Note: In this query, tokenization is disabled for the `default` field. The `aliws` analyzer is used for the `index2` field. Any other field uses the global `singlews_analyzer`.
Usage notes
The analyzer clause is optional.
Field analysis settings are determined by the following order of precedence:
no_tokenize_indexes>specific_index_analyzer>global_analyzer> the analyzer specified in the schema. For example, if theindex1field is specified in bothno_tokenize_indexesandspecific_index_analyzer, tokenization is disabled for that field becauseno_tokenize_indexeshas higher precedence.If you do not include the analyzer clause in a query statement, the system uses the analyzer that is specified in the schema.