Hologres:Full-text inverted index

Create an index

CREATE INDEX [ IF NOT EXISTS ] idx_name ON table_name
       USING FULLTEXT (column_name [ , ... ])
       [ WITH ( storage_parameter [ = value ] [ , ... ] ) ];

Modify an index

-- Modify the index configuration
ALTER INDEX [ IF EXISTS ] <idx_name> SET ( <storage_parameter> = '<storage_value>' [ , ... ] );

-- Reset to the default configuration
ALTER INDEX [ IF EXISTS ] <idx_name> RESET ( <storage_parameter> [ , ... ] );

Delete an index

DROP INDEX [ IF EXISTS ] <idx_name> [ RESTRICT ];

View indexes

Hologres provides the hologres.hg_index_properties system table to view created full-text inverted indexes and their locations.

SELECT * FROM hologres.hg_index_properties;

You can run the following SQL statement to view the table and column that correspond to an index.

SELECT 
    t.relname AS table_name, 
    a.attname AS column_name
FROM pg_class t
    JOIN pg_index i ON t.oid = i.indrelid
    JOIN pg_class idx ON i.indexrelid = idx.oid
    JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = ANY(i.indkey)
WHERE t.relnamespace = (SELECT oid FROM pg_namespace WHERE nspname = '<namespace>')
    AND idx.relname = '<indexname>'
LIMIT 1;

Parameters:

• namespace: The value of the table_namespace field in the result returned by the SELECT * FROM hologres.hg_index_properties; command.

• indexname: The name of the index.

TEXT_SEARCH search function

The `TEXT_SEARCH` function calculates the BM25 relevance score for a source text based on a search query.

TEXT_SEARCH (
  <search_data> TEXT/VARCHAR/CHAR
  ,<search_expression> TEXT
  [ ,<mode> TEXT DEFAULT 'match'
  ,<operator> TEXT DEFAULT 'OR'
  ,<tokenizer> TEXT DEFAULT ''
  ,<analyzer_params> TEXT DEFAULT ''
  ,<options> TEXT DEFAULT '']
)

TOKENIZE function

The `TOKENIZE` function outputs tokenization results based on the tokenizer configuration. You can use it to debug the tokenization effect of a full-text inverted index.

TOKENIZE (
  <search_data> TEXT
  [ ,<tokenizer> TEXT DEFAULT ''
  ,<analyzer_params> TEXT DEFAULT '']
)

Verify index usage

You can use an execution plan to determine whether an SQL statement uses a full-text inverted index. If the execution plan includes Fulltext Filter, the full-text inverted index is used. For more information about execution plans, see EXPLAIN and EXPLAIN ANALYZE.

Example SQL statement:

EXPLAIN ANALYZE SELECT * FROM wiki_articles WHERE text_search(content, 'Yangtze River') > 0;

The following execution plan contains the Fulltext Filter field. This indicates that the SQL statement uses a full-text inverted index.

QUERY PLAN
Gather  (cost=0.00..1.00 rows=1 width=12)
  ->  Local Gather  (cost=0.00..1.00 rows=1 width=12)
        ->  Index Scan using Clustering_index on wiki_articles  (cost=0.00..1.00 rows=1 width=12)
              Fulltext Filter: (text_search(content, search_expression => 'Yangtze River'::text, mode => match, operator => OR, tokenizer => jieba, analyzer_params => {"filter":["removepunct","lowercase",{"stop_words":["_english_"],"type":"stop"},{"language":"english","type":"stemmer"}],"tokenizer":{"hmm":true,"mode":"search","type":"jieba"}}, options => ) > '0'::double precision)
Query Queue: init_warehouse.default_queue
Optimizer: HQO version 4.0.0

Data preparation

Run the following SQL statements to create a test table and insert data into it.

-- Create a table.
CREATE TABLE wiki_articles (id int, content text);

-- Create an index.
CREATE INDEX ft_idx_1 ON wiki_articles
       USING FULLTEXT (content)
       WITH (tokenizer = 'jieba');

-- Write data.
INSERT INTO wiki_articles VALUES
  (1, 'The Yangtze River is the longest river in China and the third longest in the world, with a total length of about 6,300 kilometers.'),
  (2, 'Li was born in 1962 in Wendeng County, Shandong.'),
  (3, 'He graduated from the department of physics at Shandong University.'),
  (4, 'The Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.'),
  (5, 'The Spring Festival usually falls between late January and mid-February in the Gregorian calendar. Main customs during the Spring Festival include pasting spring couplets, setting off firecrackers, having a New Year''s Eve dinner, and making New Year visits.'),
  (6, 'In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.'),
  (7, 'Shandong has dozens of universities.'),
  (8, 'ShanDa is a famous university of Shandong.');

-- Compaction
VACUUM wiki_articles;

-- Query table data.
SELECT * FROM wiki_articles limit 1;

The following result is returned:

id |                       content                       
---+---------------------------------------------------
 1 | The Yangtze River is the longest river in China and the third longest in the world, with a total length of about 6,300 kilometers.

Different search examples

• Keyword match.

  -- (K1) Keyword match (default operator=OR). Documents containing 'shandong' or 'university' are matched.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university') > 0;
  
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    2 | Li was born in 1962 in Wendeng County, Shandong.
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.
  
  -- (K2) Keyword match (operator=AND). Documents must contain both 'shandong' and 'university' to be matched.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university', operator => 'AND') > 0;
  
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.

• Phrase search.

  -- (P1) Phrase search (default slop = 0). A match occurs only if 'shandong' is immediately followed by 'university'.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase') > 0;
  
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
  (1 row)
  
  -- (P2) Phrase search with slop = 14. The distance between 'shandong' and 'university' cannot exceed 14 characters. This matches "Shandong has dozens of universities."
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase', options => 'slop=14;') > 0;
          
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
  (2 rows)
  
  -- (P3) Phrase search supports unordered phrases, but the slop calculation is different and requires a larger value than for ordered phrases.
  --      Therefore, 'university of Shandong' can also match the following query, but it will not be matched if slop=22.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase', options => 'slop=23;') > 0;
          
  -- Result   
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.
  (3 rows)
  
  -- (P4) Behavior with a standard tokenizer index. (ALTER INDEX ft_idx_1 SET (tokenizer = 'standard');)
  --      For the standard tokenizer, slop is calculated in tokens.
  --      As long as there are 0 tokens in between, regardless of the number of spaces, it is considered a phrase match.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase') > 0;
  
  -- Result
   id |        content         
  ----+------------------------
    1 | shandong university
    2 | shandong  university
    3 | shandong   university
    4 | shandong    university
  (4 rows)
  
  -- (P5) Punctuation is ignored. (Example with Jieba tokenizer)
  --      Even though there is a comma between 'river' and 'with' in the text, and the query string has a period.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'river. with', mode => 'phrase') > 0;
  
  -- Result
   id |                       content                       
  ----+-----------------------------------------------------
    1 | The Yangtze River is the longest river in China and the third longest in the world, with a total length of about 6,300 kilometers.
  (1 row)

• Natural language search.

  -- (N1) Natural language query: Without any symbols, this is equivalent to a keyword match. Same as (K1).
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong university', 'natural_language') > 0;
   id |                               content                               
  ----+---------------------------------------------------------------------
    7 | Shandong has dozens of universities.
    2 | Li was born in 1962 in Wendeng County, Shandong.
    3 | He graduated from the department of physics at Shandong University.
    8 | ShanDa is a famous university of Shandong.
  
  -- (N2) Natural language query: Keyword match. Must contain ('shandong' AND 'university') OR 'heritage'. The AND operator has higher precedence than OR.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '(shandong AND university) OR heritage', 'natural_language') > 0;
  -- Equivalent to
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong AND university OR heritage', 'natural_language') > 0;
  -- Equivalent to
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '(+shandong +university) heritage', 'natural_language') > 0;
  
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    8 | ShanDa is a famous university of Shandong.
    7 | Shandong has dozens of universities.
    3 | He graduated from the department of physics at Shandong University.
    6 | In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.
  
  -- (N3) Natural language query: Keyword match. Must contain 'shandong', must not contain 'university', and may contain 'heritage'.
  --      In this query, the 'heritage' keyword does not have a + or - sign, so it does not affect which rows are matched. However, it does affect the match score. Rows with 'heritage' will have a higher score.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '+shandong -university heritage', 'natural_language') > 0;
   id |                     content                      
  ----+--------------------------------------------------
    2 | Li was born in 1962 in Wendeng County, Shandong.
  
  -- Must contain 'shandong', must not contain 'physics', and may contain 'famous'. The relevance score is higher if it contains 'famous'.
  -- Note: This query shows the score calculation result on a single shard. The calculated BM25 score may vary with different numbers of shards and different file organizations.
  SELECT id,
         content,
         TEXT_SEARCH(content, '+shandong -physics famous', 'natural_language') as score
  FROM wiki_articles
  WHERE TEXT_SEARCH(content, '+shandong -physics famous', 'natural_language') > 0
  ORDER BY score DESC;
  
  -- Result
   id |                     content                      |  score   
  ----+--------------------------------------------------+----------
    8 | ShanDa is a famous university of Shandong.       |  2.92376
    7 | Shandong has dozens of universities.             | 0.863399
    2 | Li was born in 1962 in Wendeng County, Shandong. | 0.716338
  
  -- (N4) Natural language query: Phrase search. Equivalent to (P1). The phrase must be enclosed in double quotation marks (""). If the phrase contains a double quotation mark, it must be escaped with a backslash (\).
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '"shandong university"', 'natural_language') > 0;
          
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
  
  -- (N5) Natural language query: Phrase search. Equivalent to (P2). Supports setting slop with the ~ syntax.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '"shandong university"~23', 'natural_language') > 0;
          
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    8 | ShanDa is a famous university of Shandong.
    7 | Shandong has dozens of universities.
    3 | He graduated from the department of physics at Shandong University.
  
  -- (N6) Natural language query: Match all documents.
  SELECT * FROM wiki_articles                                                                                  
          WHERE TEXT_SEARCH(content, '*', 'natural_language') > 0;
          
  -- Result
   id |                                           content                                            
  ----+----------------------------------------------------------------------------------------------
    1 | The Yangtze River is the longest river in China and the third longest in the world, with a total length of about 6,300 kilometers.
    2 | Li was born in 1962 in Wendeng County, Shandong.
    3 | He graduated from the department of physics at Shandong University.
    4 | The Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.
    5 | The Spring Festival usually falls between late January and mid-February in the Gregorian calendar. Main customs during the Spring Festival include pasting spring couplets, setting off firecrackers, having a New Year's Eve dinner, and making New Year visits.
    6 | In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.

• Term search.

  -- (T1) Term query: The tokenization result explicitly contains "Spring Festival", so it can be retrieved.
  SELECT * FROM wiki_articles 
           WHERE TEXT_SEARCH(content, 'Spring Festival', 'term') > 0;
  
  -- Result         
   id |                                           content                                            
  ----+----------------------------------------------------------------------------------------------
    4 | The Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.
    5 | The Spring Festival usually falls between late January and mid-February in the Gregorian calendar. Main customs during the Spring Festival include pasting spring couplets, setting off firecrackers, having a New Year's Eve dinner, and making New Year visits.
    6 | In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.
  
  
  -- (T2) Term query: Example that returns no results.
  -- Because the content column of the wiki_articles source table uses the Jieba tokenizer by default, which splits "shandong university".
  -- We recommend using term queries with the keyword tokenizer. In that case, the target data can be retrieved.
  SELECT * FROM wiki_articles 
           WHERE TEXT_SEARCH(content, 'shandong university', 'term') > 0;
  
  -- Result         
   id | content 
  ----+---------

Complex query examples

• Query with a primary key.

  -- Retrieve text that contains 'shandong' or 'university' and has an id of 3.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university') > 0 and id = 3;
  
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    
  
  -- Retrieve text that contains 'shandong' or 'university', or has an id less than 2.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university') > 0 OR id < 2;
  
  -- Result
   id |                               content                               
  ----+---------------------------------------------------------------------
    2 | Li was born in 1962 in Wendeng County, Shandong.
    8 | ShanDa is a famous university of Shandong.
    1 | The Yangtze River is the longest river in China and the third longest in the world, with a total length of about 6,300 kilometers.
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.

• Obtain the score and retrieve the top three results.

  SELECT id,
         content,
         TEXT_SEARCH(content, 'shandong university') AS score,
         TOKENIZE(content, 'jieba')
    FROM wiki_articles
  ORDER BY score DESC
  LIMIT 3;
  
  -- Result
  id  |                               content                               |  score  |                     tokenize                     
  ----+---------------------------------------------------------------------+---------+--------------------------------------------------
    8 | ShanDa is a famous university of Shandong.                          | 2.74634 | {shanda,famous,univers,shandong}
    7 | Shandong has dozens of universities.                                | 2.74634 | {shandong,has,dozen,univers}
    3 | He graduated from the department of physics at Shandong University. | 2.38178 | {he,graduat,from,depart,physic,shandong,univers}

• Use the TEXT_SEARCH function in both the SELECT list and the WHERE clause.

  SELECT id,
         content,
         TEXT_SEARCH(content, 'shandong university') AS score,
         TOKENIZE(content, 'jieba')
    FROM wiki_articles
   WHERE TEXT_SEARCH(content, 'shandong university') > 0
  ORDER BY score DESC;
  
  -- Result
  id  |                               content                               |  score  |                     tokenize                     
  ----+---------------------------------------------------------------------+---------+--------------------------------------------------
    7 | Shandong has dozens of universities.                                | 2.74634 | {shandong,has,dozen,univers}
    8 | ShanDa is a famous university of Shandong.                          | 2.74634 | {shanda,famous,univers,shandong}
    3 | He graduated from the department of physics at Shandong University. | 2.38178 | {he,graduat,from,depart,physic,shandong,univers}
    2 | Li was born in 1962 in Wendeng County, Shandong.                    | 1.09244 | {li,born,1962,wendeng,counti,shandong}

• Retrieve the documents from the 'wiki' source that are most relevant to 'shandong university'.

  -- Source table for JOIN.
  CREATE TABLE article_source (id int primary key, source text);
  INSERT INTO article_source VALUES (1, 'baike'), (2, 'wiki'), (3, 'wiki'), (4, 'baike'),
                                    (5, 'baike'), (6, 'baike'), (7, 'wiki'), (8, 'paper'),
                                    (9, 'http_log'), (10, 'http_log'), (11, 'http_log');
                                    
  SELECT a.id,
         source, content,
         TEXT_SEARCH(content, 'shandong university') AS score,
         TOKENIZE(a.content, 'jieba')
    FROM wiki_articles a
    JOIN article_source b
      ON (a.id = b.id)
   WHERE TEXT_SEARCH(a.content, 'shandong university') > 0
     AND b.source = 'wiki'
  ORDER BY score DESC;
  
  -- Result
  id  | source |                               content                               |  score  |                     tokenize                     
  ----+--------+---------------------------------------------------------------------+---------+--------------------------------------------------
    7 | wiki   | Shandong has dozens of universities.                                | 2.74634 | {shandong,has,dozen,univers}
    3 | wiki   | He graduated from the department of physics at Shandong University. | 2.38178 | {he,graduat,from,depart,physic,shandong,univers}
    2 | wiki   | Li was born in 1962 in Wendeng County, Shandong.                    | 1.09244 | {li,born,1962,wendeng,counti,shandong}

Reindex using Serverless resources

If you modify table properties, compaction may be triggered and the index may be rebuilt, which consumes a large amount of CPU resources. To modify the following table properties, perform the steps below:

• Modifying `bitmap_columns`, `dictionary_encoding_columns`, or vector indexes triggers compaction and reindexing. Therefore, do not use the `ALTER TABLE xxx SET` syntax. Instead, run the following command to use the `REBUILD` syntax with Serverless Computing resources. For more information, see REBUILD.

ASYNC REBUILD TABLE <table_name> 
WITH (
    rebuild_guc_hg_computing_resource = 'serverless'
)
SET (
    bitmap_columns = '<col1>,<col2>',
    dictionary_encoding_columns = '<col1>:on,<col2>:off',
    vectors = '{
    "<col_vector>": {
        "algorithm": "HGraph",
        "distance_method": "Cosine",
        "builder_params": {
            "base_quantization_type": "rabitq",
            "graph_storage_type": "compressed",
            "max_degree": 64,
            "ef_construction": 400,
            "precise_quantization_type": "fp32",
            "use_reorder": true,
            "max_total_size_to_merge_mb" : 4096
        }
    }
    }'
);

• Modifying column-oriented JSONB columns or full-text index columns also triggers compaction and reindexing. The `REBUILD` syntax is not supported for these modifications. Instead, create a temporary table by following these steps:

BEGIN ;
-- Clean up potential temporary tables.
DROP TABLE IF EXISTS <table_new>;
-- Create a temporary table.
SET hg_experimental_enable_create_table_like_properties=on;
CALL HG_CREATE_TABLE_LIKE ('<table_new>', 'select * from <table>');
COMMIT ;

-- Enable column-oriented storage for JSON-formatted data for the corresponding column.
ALTER TABLE <table_new> ALTER COLUMN <column_name> SET (enable_columnar_type = ON);
-- Create a full-text index on the corresponding column.
CREATE INDEX <idx_name> ON <table_new> USING FULLTEXT (column_name);

-- Insert data into the temporary table, use Serverless resources to run the job, and synchronously complete index building.
SET hg_computing_resource = 'serverless';
INSERT INTO <table_new> SELECT * FROM <table>;
ANALYZE <table_new>;

BEGIN ;
-- Delete the old table.
DROP TABLE IF EXISTS  <table>;
-- Rename the temporary table.
ALTER TABLE <table_new> RENAME TO <table>;
COMMIT ;

• To modify other properties, such as `distribution_key`, `clustering_key`, `segment_key`, and storage format, use the `REBUILD` syntax with Serverless Computing resources.

analyzer_params configuration requirements

The `analyzer_params` parameter has the following configuration requirements:

• Only JSON-formatted strings are supported.

• The top level of the JSON supports two keys: `tokenizer` and `filter`.

  • filter: Optional. The value is a JSON array used to configure token filter properties. If you configure multiple token filter properties, they are applied to each token in the specified order.

  • tokenizer: Required. The value is a JSON object used to configure tokenizer properties. The JSON object supports the following keys:

    • type: Required. The tokenizer name.

    • Other parameters for the tokenizer object. Different tokenizers support different parameters, as shown in the following table:

      Tokenizer	Other parameters for the tokenizer object	Parameter description	Parameter values
      jieba	mode	Defines the tokenization mode.	search (default): Lists multiple possible combinations during tokenization, which allows for redundancy. For example, the tokenization result for "traditional festival" is "traditional", "festival", and "traditional festival". exact: Prevents redundant splitting during tokenization. For example, the tokenization result for "traditional festival" is only "traditional festival".
      	hmm	Specifies whether to use a Hidden Markov Model (HMM) to identify words that are not in the dictionary. This improves new word recognition.	true (default): Enabled false: Disabled.
      standard	max_token_length	The maximum token length.	A positive integer. The default value is 255. If a token exceeds this length, it is split at intervals of `max_token_length`.
      ik	mode	Defines the tokenization mode.	ik_max_word (default): Performs fine-grained tokenization that outputs all possible short words. For example, the tokenization result for "Nanjing City" is "Nanjing" and "City". ik_smart: Performs coarse-grained tokenization that prioritizes long words and reduces the number of splits. The output tokens do not overlap. This mode attempts to combine numbers and quantifiers into a single token. For example, "Nanjing City" and "1 kilometer" are not further tokenized.
      	enable_lowercase	Specifies whether to convert tokens to lowercase.	true (default) false
      ngram	min_ngram	The minimum character length of a token.	A positive number. The default value is 1. The maximum difference between `min_ngram` and `max_ngram` is 3. Note: You can adjust the maximum difference by setting the following GUC. For example, SET hg_fulltext_index_max_ngram_diff = 5.
      	max_ngram	The maximum character length of a token.	The default value is 2. The value must be in the range of [1, 255]. The maximum difference between `min_ngram` and `max_ngram` is 3. Note If the difference between `max_ngram` and `min_ngram` is too large, the ngram tokenizer generates many tokens. This can increase resource consumption, require more index storage, and lengthen index building times.
      	prefix_only	Specifies whether to consider only prefixes.	true false (default)
      pinyin	keep_first_letter	Specifies whether to keep the first letter of each Chinese character as a token.	true (default): Keeps the first letter. For example, the tokenization result for "Li Ming" includes "lm". false: Do not retain.
      	keep_separate_first_letter	Specifies whether to keep the first letter of each Chinese character as a separate token.	true: Keeps the first letters separately. For example, the tokenization result for "Li Ming" includes "l,m". false (default): The setting does not retain the value.
      	limit_first_letter_length	The maximum length of the first-letter token.	An integer that specifies the maximum length of the first-letter string. The default value is 16.
      	keep_full_pinyin	Specifies whether to keep the full Pinyin of each Chinese character as separate tokens.	true (default): Keeps the full Pinyin. For example, the tokenization result for "Li Ming" includes "li,ming". false: The item is not retained.
      	keep_joined_full_pinyin	Specifies whether to join the full Pinyin of each Chinese character into a single token.	true: Joins the full Pinyin. For example, the tokenization result for "Li Ming" includes "liming". false (default): The system does not establish a connection.
      	keep_none_chinese	Specifies whether to keep non-Chinese letters or numbers in the tokenization result.	true (default): Retain. false: Not retained.
      	keep_none_chinese_together	Specifies whether to keep non-Chinese letters or numbers together.	true (default): Keeps non-Chinese characters together. For example, the tokenization result for "DJ Li Ming" is "DJ,li,ming", where "DJ" stays together. false: Does not keep non-Chinese characters together. For example, the tokenization result for "DJ Li Ming" is "D,J,li,ming". Note This parameter takes effect only when `keep_none_chinese` is set to `true`.
      	keep_none_chinese_in_first_letter	Specifies whether to keep non-Chinese letters or numbers in the first-letter token.	true (default): Retain. For example, the tokenization result of 李明AT2025 includes lmat2025. false: The item is not retained.
      	keep_none_chinese_in_joined_full_pinyin	Specifies whether to keep non-Chinese letters or numbers in the joined full Pinyin token.	true: Keeps non-Chinese characters. For example, the tokenization result for "Li Ming AT2025" includes "limingat2025". false (default): The resource is not retained.
      	none_chinese_pinyin_tokenize	If non-Chinese letters are valid Pinyin, specifies whether to split them into separate Pinyin terms.	true (default): Splits the terms. For example, the tokenization result for "limingalibaba2025" includes "li,ming,a,li,ba,ba,2025". false: Does not split the terms. Note This parameter takes effect only when `keep_none_chinese` and `keep_none_chinese_together` are set to `true`.
      	keep_original	Specifies whether to keep the original input.	true: Keeps the original input. false (default): Does not keep the original input.
      	lowercase	Specifies whether to convert non-Chinese letters to lowercase.	true (default) false
      	trim_whitespace	Specifies whether to trim whitespace characters.	true (default) false
      	remove_duplicated_term	Specifies whether to remove duplicate terms.	If this parameter is set to true, certain tokens are removed. For example, "de的" is tokenized as "de", which affects the results of phrase queries for "de的". false (default): Does not remove duplicate terms.
      	keep_separate_chinese	Specifies whether to keep individual Chinese characters as separate tokens.	true: Keeps individual characters. For example, the tokenization result for "Li Ming" includes the two tokens "Li" and "Ming". false (default): The item is not retained.

Default analyzer_params configurations

The default `analyzer_params` configurations for different tokenizers are as follows:

{
  "tokenizer": {
    "type": "jieba", 
    "mode": "search",
    "hmm": true
  }, 
  "filter": [
    "removepunct",
    "lowercase",
    {"type": "stop", "stop_words": ["_english_"]},
    {"type": "stemmer", "language": "english"}
  ]
}

 Spring Festival
 ,
 that is
 ,
 the Lunar New Year
 ,
 is
 China
 's
 most
 important
 traditional
 festival
 traditional festival
 .

{
  "tokenizer": {
    "type": "whitespace"
  }
}

Spring Festival, that is the Lunar New Year, is China's most important traditional festival.

{
  "tokenizer": {
    "type": "keyword"
  }
}

Spring Festival, that is the Lunar New Year, is China's most important traditional festival.

{
  "tokenizer": {
    "type": "simple"
  }, 
  "filter": [
    "lowercase"
  ]
}

Spring Festival
that is the Lunar New Year
is China's most important traditional festival

{
  "tokenizer": {
    "type": "standard",
    "max_token_length": 255
  }, 
  "filter": [
    "lowercase"
  ]
}

Spring
Festival
that
is
the
Lunar
New
Year
is
China
's
most
important
traditional
festival

{
  "tokenizer": {
    "type": "icu"
  }, 
  "filter": [
    "removepunct",
    "lowercase"
  ]
}

Spring Festival
,
 that is
 the Lunar New Year
 ,
 is
 China
 's most
 important
 traditional festival
 .

{
  "tokenizer": {
    "type": "ik",
    "mode": "ik_max_word",
    "enable_lowercase": true
  },
  "filter": [
    {"type": "stop", "stop_words": ["_english_"]},
    {"type": "stemmer", "language": "english"}
  ]
}

Spring Festival
that
is
the Lunar New Year
is
China
's most
most important
important
traditional festival
traditional
festival

{
  "tokenizer": {
    "type": "ngram",
    "min_gram": 1,
    "max_gram": 2,
    "prefix_only": false
  }
}

Spring
Spring Festival
Festival
Festival,
,
, that
that
that is
is
is the
the
the Lunar
Lunar
Lunar New
New
New Year
Year
Year,
,
, is
is
is China
China
China's
's
's most
most
most important
important
important
important
important
important
traditional
traditional
traditional festival
festival
festival
.

{
  "tokenizer": {
    "type": "pinyin",
    "keep_first_letter": true,
    "keep_separate_first_letter": false,
    "keep_full_pinyin": true,
    "keep_joined_full_pinyin": false,
    "keep_none_chinese": true,
    "keep_none_chinese_together": true,
    "none_chinese_pinyin_tokenize": true,
    "keep_original": false,
    "limit_first_letter_length": 16,
    "lowercase": true,
    "trim_whitespace": true,
    "keep_none_chinese_in_first_letter": true,
    "keep_none_chinese_in_joined_full_pinyin": false,
    "remove_duplicated_term": false,
    "ignore_pinyin_offset": true,
    "fixed_pinyin_offset": false,
    "keep_separate_chinese": false
  }
}

chun
cjjnlxnszgzzydct
jie
ji
nong
li
xin
nian
shi
zhong
guo
zui
zhong
yao
de
chuan
tong
jie
ri

Filter configuration in analyzer_params

Hologres supports the following filters, also known as token filter properties, in `analyzer_params`.

"lowercase"

"_english_"
"_danish_"
"_dutch_"
"_finnish_"
"_french_"
"_german_"
"_hungarian_"
"_italian_"
"_norwegian_"
"_portuguese_"
"_russian_"
"_spanish_"
"_swedish_"

"arabic",
"danish",
"dutch",
"english",
"finnish",
"french",
"german",
"greek",
"hungarian",
"italian",
"norwegian",
"portuguese",
"romanian",
"russian",
"spanish",
"swedish",
"tamil",
"turkish"

{"type": "length", "max": 10}

"removepunct"

{
  "type": "pinyin",
  "keep_first_letter": true,
  "keep_separate_first_letter": false,
  "keep_full_pinyin": true,
  "keep_joined_full_pinyin": false,
  "keep_none_chinese": true,
  "keep_none_chinese_together": true,
  "none_chinese_pinyin_tokenize": true,
  "keep_original": false,
  "limit_first_letter_length": 16,
  "lowercase": true,
  "trim_whitespace": true,
  "keep_none_chinese_in_first_letter": true,
  "keep_none_chinese_in_joined_full_pinyin": false,
  "remove_duplicated_term": false,
  "ignore_pinyin_offset": true,
  "fixed_pinyin_offset": false,
  "keep_separate_chinese": false
}