Tablestore:Tokenization

Single-word tokenization (SingleWord)

The single-word tokenization type applies to all languages, including Chinese, English, and Japanese. It is the default tokenizer type for Text fields.

After you set the tokenizer type to single-word tokenization, tokenization works as follows:

• Chinese text is split character by character. Chinese examples in this topic are romanized in pinyin so that the token boundaries stay visible. For example, the two-character city name "hangzhou" (Hangzhou) is split into "hang" and "zhou". A match query or match phrase query for "hang" returns rows whose content contains "hangzhou".

• English letters and digits are split by spaces or punctuation marks.

  • If caseSensitive is false, tokenization is case-insensitive, and Tablestore converts all tokenized English letters to lowercase before storing them. For example, "Hang Zhou" is split into "hang" and "zhou", and a match query or match phrase query for "hang", "HANG", or "Hang" returns this row.

  • If caseSensitive is true, tokenization is case-sensitive, and Tablestore preserves the original case of the tokenized English letters when storing them. For example, "Hang Zhou" is split into "Hang" and "Zhou", and a match query or match phrase query for "Hang" or "Zhou" returns this row.

• Terms in which digits and English letters are joined, such as product model numbers, are also split by spaces or punctuation marks. However, by default, the digits and letters within such a term are not split apart. For example, "iphone6" remains the single token "iphone6". A match query or match phrase query returns the row only if you specify the complete term "iphone6", and a query for "iphone" returns nothing. To split the digits from the letters, set the delimitWord parameter to true. In this case, "iphone6" is split into "iphone" and "6", and a match query or match phrase query for either "iphone" or "6" returns this row.

  The following table describes the parameters for single-word tokenization.

Delimiter tokenization (Split)

Tablestore provides tokenization based on a general-purpose dictionary. However, some specialized industries need custom dictionaries for tokenization. To address this, Tablestore provides delimiter tokenization, which is also called custom tokenization: you tokenize the content yourself, join the tokens with a specific delimiter, and then write the result to Tablestore.

The delimiter tokenization type applies to all languages, including Chinese, English, and Japanese.

After you configure delimiter tokenization for a Text field, the field value is split at each occurrence of the value that the delimiter parameter specifies. For example, if the field value is "badminton,table tennis,rap" and the delimiter is a comma (,), the value is split into "badminton", "table tennis", and "rap", and these tokens are indexed. A match query or match phrase query for "badminton", "table tennis", "rap", or "badminton,table tennis" returns this row.

The following table describes the parameters for delimiter tokenization.

Minimum semantic tokenization (MinWord)

The minimum semantic tokenization type applies to Chinese and is generally used in full-text search scenarios.

After you set the tokenizer type to minimum semantic tokenization, the system splits the content of the Text field into the smallest possible number of semantic tokens. For example, the three-character term "lihuacha" (pear blossom tea) is split into "li" and "huacha", and the resulting tokens do not overlap. As another example, the seven-character term "zhonghuarenmingongheguo" is kept as the single token "zhonghuarenmingongheguo".

Maximum semantic tokenization (MaxWord)

The maximum semantic tokenization type applies to Chinese and is generally used in full-text search scenarios.

After you set the tokenizer type to maximum semantic tokenization, the system splits out as many semantic tokens as possible. The tokens overlap one another, their accumulated length is greater than the length of the original text, and the index size also expands. For example, the three-character term "lihuacha" (pear blossom tea) is split into "lihua" and "huacha", which overlap at "hua". As another example, the seven-character term "zhonghuarenmingongheguo" is split into "zhonghuarenmingongheguo", "zhonghuarenmin", "zhonghua", "huaren", "renmingongheguo", "renmin", "gongheguo", "gonghe", and "guo".

This tokenizer type produces more tokens and therefore a higher probability of a match at query time, but the index size expands considerably. It is suitable for match query rather than match phrase query. If you use match phrase query, the query keyword is tokenized in the same way, so the position information overlaps, which may prevent the data from being found.

Fuzzy tokenization (Fuzzy)

The fuzzy tokenization type applies to all languages, including Chinese, English, and Japanese. It is generally used in scenarios that involve short text, such as titles, movie names, book names, file names, and directory names.

The fuzzy tokenization type returns results with very low latency and performs better than wildcard query, but the index size expands to a certain extent.

After you set the tokenizer type to fuzzy tokenization, the system performs N-gram tokenization on the text content, and the length of each resulting token is between minChars and maxChars. This is useful for features such as drop-down suggestions.

To implement a fuzzy query, you must use match phrase query on the column that uses fuzzy tokenization; no other query type can be used. If you have multiple types of query requirements for that column, use the virtual column feature. For more information about how to use virtual columns, see Virtual columns.