Hologres：Indeks terbalik teks penuh

Buat indeks

CREATE INDEX [ IF NOT EXISTS ] idx_name ON table_name
       USING FULLTEXT (column_name [ , ... ])
       [ WITH ( storage_parameter [ = value ] [ , ... ] ) ];

Ubah indeks

-- Ubah konfigurasi indeks
ALTER INDEX [ IF EXISTS ] <index_name> SET ( <storage_parameter> = '<storage_value>' [ , ... ] );

-- Atur ulang ke konfigurasi default
ALTER INDEX [ IF EXISTS ] <index_name> RESET ( <storage_parameter> [ , ... ] );

Hapus indeks

DROP INDEX [ IF EXISTS ] <index_name> [ RESTRICT ];

Lihat indeks

Hologres menyediakan tabel sistem hologres.hg_index_properties, yang dapat Anda gunakan untuk melihat indeks terbalik teks penuh yang telah dibuat dan lokasi yang sesuai.

SELECT * FROM hologres.hg_index_properties;

Anda dapat menjalankan pernyataan SQL berikut untuk melihat tabel dan kolom yang sesuai dengan suatu indeks.

SELECT 
    t.relname AS table_name, 
    a.attname AS column_name
FROM pg_class t
    JOIN pg_index i ON t.oid = i.indrelid
    JOIN pg_class idx ON i.indexrelid = idx.oid
    JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = ANY(i.indkey)
WHERE t.relnamespace = (SELECT oid FROM pg_namespace WHERE nspname = '<namespace>')
    AND idx.relname = '<indexname>'
LIMIT 1;

Parameter:

• namespace: Nilai field table_namespace dalam hasil perintah SELECT * FROM hologres.hg_index_properties;.

• indexname: Nama indeks yang telah dibuat.

Fungsi pencarian TEXT_SEARCH

Fungsi TEXT_SEARCH menghitung skor relevansi BM25 untuk teks sumber berdasarkan kueri pencarian.

TEXT_SEARCH (
  <search_data> TEXT/VARCHAR/CHAR
  ,<search_expression> TEXT
  [ ,<mode> TEXT DEFAULT 'match'
  ,<operator> TEXT DEFAULT 'OR'
  ,<tokenizer> TEXT DEFAULT ''
  ,<analyzer_params> TEXT DEFAULT ''
  ,<options> TEXT DEFAULT '']
)

Fungsi tokenisasi TOKENIZE

Fungsi TOKENIZE menghasilkan hasil tokenisasi berdasarkan konfigurasi tokenizer. Anda dapat menggunakan fungsi ini untuk menguji efek tokenisasi indeks terbalik teks penuh.

TOKENIZE (
  <search_data> TEXT
  [ ,<tokenizer> TEXT DEFAULT ''
  ,<analyzer_params> TEXT DEFAULT '']
)

Verifikasi penggunaan indeks

Anda dapat memeriksa rencana eksekusi untuk melihat apakah pernyataan SQL menggunakan indeks terbalik teks penuh. Jika rencana tersebut berisi Fulltext Filter, berarti indeks berhasil digunakan. Untuk informasi lebih lanjut tentang rencana eksekusi, lihat EXPLAIN dan EXPLAIN ANALYZE.

Contoh pernyataan SQL:

EXPLAIN ANALYZE SELECT * FROM wiki_articles WHERE text_search(content, 'Yangtze River') > 0;

Rencana eksekusi berikut berisi field Fulltext Filter, yang menunjukkan bahwa pernyataan SQL berhasil menggunakan indeks terbalik teks penuh.

QUERY PLAN
Gather  (cost=0.00..1.00 rows=1 width=12)
  ->  Local Gather  (cost=0.00..1.00 rows=1 width=12)
        ->  Index Scan using Clustering_index on wiki_articles  (cost=0.00..1.00 rows=1 width=12)
              Fulltext Filter: (text_search(content, search_expression => 'Yangtze River'::text, mode => match, operator => OR, tokenizer => jieba, analyzer_params => {"filter":["removepunct","lowercase",{"stop_words":["_english_"],"type":"stop"},{"language":"english","type":"stemmer"}],"tokenizer":{"hmm":true,"mode":"search","type":"jieba"}}, options => ) > '0'::double precision)
Query Queue: init_warehouse.default_queue
Optimizer: HQO version 4.0.0

Persiapkan data

Anda dapat menjalankan pernyataan SQL berikut untuk membuat tabel uji dan menulis data ke dalamnya.

-- Buat tabel.
CREATE TABLE wiki_articles (id int, content text);

-- Buat indeks.
CREATE INDEX ft_idx_1 ON wiki_articles
       USING FULLTEXT (content)
       WITH (tokenizer = 'jieba');

-- Tulis data.
INSERT INTO wiki_articles VALUES
  (1, 'The Yangtze River is China''s largest river and the third longest river in the world, with a total length of about 6,300 kilometers.'),
  (2, 'Li was born in 1962 in Wendeng County, Shandong.'),
  (3, 'He graduated from the department of physics at Shandong University.'),
  (4, 'The Spring Festival, the Lunar New Year, is the most important traditional festival in China.'),
  (5, 'The Spring Festival usually falls between late January and mid-February in the Gregorian calendar. Main customs during the Spring Festival include pasting spring couplets, setting off firecrackers, having a New Year''s Eve dinner, and making New Year visits.'),
  (6, 'In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.'),
  (7, 'Shandong has dozens of universities.'),
  (8, 'ShanDa is a famous university of Shandong.');

-- Compaction
VACUUM wiki_articles;

-- Kueri data tabel.
SELECT * FROM wiki_articles limit 1;

Berikut adalah contoh hasilnya:

id |                       content                       
---+---------------------------------------------------
 1 | The Yangtze River is China's largest river and the third longest river in the world, with a total length of about 6,300 kilometers.

Contoh pencarian berbeda

• Pencocokan kata kunci

  -- (K1) Pencocokan kata kunci (operator default=OR). Dokumen yang berisi 'shandong' atau 'university' akan dicocokkan.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university') > 0;
  
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    2 | Li was born in 1962 in Wendeng County, Shandong.
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.
  
  -- (K2) Pencocokan kata kunci (operator=AND). Pencocokan hanya terjadi jika dokumen berisi kedua kata 'shandong' dan 'university'.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university', operator => 'AND') > 0;
  
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.

• Pencarian frasa

  -- (P1) Pencarian frasa (slop default = 0). Pencocokan hanya terjadi jika 'shandong' diikuti langsung oleh 'university'.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase') > 0;
  
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
  (1 row)
  
  -- (P2) Pencarian frasa dengan slop = 14. Jika jarak antara 'shandong' dan 'university' tidak lebih dari 14 karakter, "Shandong has dozens of universities." dapat dicocokkan.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase', options => 'slop=14;') > 0;
          
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
  (2 rows)
  
  -- (P3) Pencarian frasa mendukung pencocokan frasa tak terurut, tetapi perhitungan slop berbeda dan memerlukan nilai yang lebih besar daripada frasa terurut.
  --      Oleh karena itu, 'university of Shandong' juga dapat mencocokkan kueri berikut, tetapi tidak akan dicocokkan jika slop = 22.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase', options => 'slop=23;') > 0;
          
  -- Hasil   
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.
  (3 rows)
  
  -- (P4) Perilaku dengan indeks tokenizer standard. (ALTER INDEX ft_idx_1 SET (tokenizer = 'standard');)
  --      Untuk tokenizer standard, slop dihitung dalam token.
  --      Selama tidak ada token di antaranya, dianggap sebagai pencocokan frasa, terlepas dari jumlah spasi.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university', mode => 'phrase') > 0;
  
  -- Hasil
   id |        content         
  ----+------------------------
    1 | shandong university
    2 | shandong  university
    3 | shandong   university
    4 | shandong    university
  (4 rows)
  
  -- (P5) Tanda baca diabaikan. (Contoh dengan tokenizer Jieba)
  --      Bahkan jika teks memiliki koma antara 'long river' dan 'total length', dan string kueri memiliki titik.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'long river.total length', mode => 'phrase') > 0;
  
  -- Hasil
   id |                       content                       
  ----+-----------------------------------------------------
    1 | The Yangtze River is China's largest river and the third longest river in the world, with a total length of about 6,300 kilometers.
  (1 row)

• Kueri bahasa alami

  -- (N1) Kueri bahasa alami: Tanpa simbol, ini setara dengan pencocokan kata kunci secara default. Sama seperti (K1).
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong university', 'natural_language') > 0;
   id |                               content                               
  ----+---------------------------------------------------------------------
    7 | Shandong has dozens of universities.
    2 | Li was born in 1962 in Wendeng County, Shandong.
    3 | He graduated from the department of physics at Shandong University.
    8 | ShanDa is a famous university of Shandong.
  
  -- (N2) Kueri bahasa alami: Pencocokan kata kunci. Pencocokan terjadi jika dokumen berisi kedua kata 'shandong' dan 'university', atau berisi 'heritage'. Operator AND memiliki prioritas lebih tinggi daripada operator OR.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '(shandong AND university) OR heritage', 'natural_language') > 0;
  -- Setara dengan
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, 'shandong AND university OR heritage', 'natural_language') > 0;
  -- Setara dengan
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '(+shandong +university) heritage', 'natural_language') > 0;
  
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    8 | ShanDa is a famous university of Shandong.
    7 | Shandong has dozens of universities.
    3 | He graduated from the department of physics at Shandong University.
    6 | In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.
  
  -- (N3) Kueri bahasa alami: Pencocokan kata kunci. Harus berisi 'shandong', tidak boleh berisi 'university', dan boleh berisi 'heritage'.
  --      Dalam kueri ini, kata kunci 'heritage' tidak diawali simbol + atau -. Ini tidak memengaruhi baris mana yang dicocokkan, tetapi memengaruhi skor pencocokan. Dokumen yang berisi 'heritage' akan memiliki skor lebih tinggi.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '+shandong -university heritage', 'natural_language') > 0;
   id |                     content                      
  ----+--------------------------------------------------
    2 | Li was born in 1962 in Wendeng County, Shandong.
  
  -- Harus berisi 'shandong', tidak boleh berisi 'physics', dan boleh berisi 'famous'. Dokumen yang berisi 'famous' memiliki skor relevansi lebih tinggi.
  -- Catatan: Kueri ini menunjukkan skor yang dihitung pada satu shard. Skor BM25 yang dihitung dapat berbeda dengan jumlah shard atau organisasi file yang berbeda.
  SELECT id,
         content,
         TEXT_SEARCH(content, '+shandong -physics famous', 'natural_language') as score
  FROM wiki_articles
  WHERE TEXT_SEARCH(content, '+shandong -physics famous', 'natural_language') > 0
  ORDER BY score DESC;
  
  -- Hasil
   id |                     content                      |  score   
  ----+--------------------------------------------------+----------
    8 | ShanDa is a famous university of Shandong.       |  2.92376
    7 | Shandong has dozens of universities.             | 0.863399
    2 | Li was born in 1962 in Wendeng County, Shandong. | 0.716338
  
  -- (N4) Kueri bahasa alami: Pencarian frasa. Ini setara dengan (P1). Frasa harus diapit tanda kutip ganda (""). Jika frasa berisi tanda kutip ganda, escape dengan backslash (\).
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '"shandong university"', 'natural_language') > 0;
          
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
  
  -- (N5) Kueri bahasa alami: Pencarian frasa. Ini setara dengan (P2). Sintaksis ~ didukung untuk mengatur slop.
  SELECT * FROM wiki_articles
          WHERE TEXT_SEARCH(content, '"shandong university"~23', 'natural_language') > 0;
          
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    8 | ShanDa is a famous university of Shandong.
    7 | Shandong has dozens of universities.
    3 | He graduated from the department of physics at Shandong University.
  
  -- (N6) Kueri bahasa alami: Cocokkan semua dokumen.
  SELECT * FROM wiki_articles                                                                                  
          WHERE TEXT_SEARCH(content, '*', 'natural_language') > 0;
          
  -- Hasil
   id |                                           content                                            
  ----+----------------------------------------------------------------------------------------------
    1 | The Yangtze River is China's largest river and the third longest river in the world, with a total length of about 6,300 kilometers.
    2 | Li was born in 1962 in Wendeng County, Shandong.
    3 | He graduated from the department of physics at Shandong University.
    4 | The Spring Festival, the Lunar New Year, is the most important traditional festival in China.
    5 | The Spring Festival usually falls between late January and mid-February in the Gregorian calendar. Main customs during the Spring Festival include pasting spring couplets, setting off firecrackers, having a New Year's Eve dinner, and making New Year visits.
    6 | In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.
    7 | Shandong has dozens of universities.
    8 | ShanDa is a famous university of Shandong.

• Kueri term

  -- (T1) Kueri term: Hasil tokenisasi secara eksplisit berisi "Spring Festival", sehingga dapat diambil.
  SELECT * FROM wiki_articles 
           WHERE TEXT_SEARCH(content, 'Spring Festival', 'term') > 0;
  
  -- Hasil         
   id |                                           content                                            
  ----+----------------------------------------------------------------------------------------------
    4 | The Spring Festival, the Lunar New Year, is the most important traditional festival in China.
    5 | The Spring Festival usually falls between late January and mid-February in the Gregorian calendar. Main customs during the Spring Festival include pasting spring couplets, setting off firecrackers, having a New Year's Eve dinner, and making New Year visits.
    6 | In 2006, the Spring Festival was approved by the State Council as one of the first national intangible cultural heritages.
  
  
  -- (T2) Kueri term: Contoh kueri yang tidak menghasilkan data.
  -- Karena kolom content pada tabel sumber wiki_articles menggunakan tokenizer Jieba secara default, "shandong university" dipisah.
  -- Untuk skenario ini, gunakan kueri term dengan tokenizer keyword untuk mengambil data target.
  SELECT * FROM wiki_articles 
           WHERE TEXT_SEARCH(content, 'shandong university', 'term') > 0;
  
  -- Hasil         
   id | content 
  ----+---------

Contoh kueri kompleks

• Gabungkan kueri dengan primary key.

  -- Ambil teks yang berisi 'shandong' atau 'university' dan memiliki ID 3.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university') > 0 and id = 3;
  
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    3 | He graduated from the department of physics at Shandong University.
    
  
  -- Ambil teks yang berisi 'shandong' atau 'university', atau memiliki ID kurang dari 2.
  SELECT * FROM wiki_articles WHERE TEXT_SEARCH(content, 'shandong university') > 0 OR id < 2;
  
  -- Hasil
   id |                               content                               
  ----+---------------------------------------------------------------------
    2 | Li was born in 1962 in Wendeng County, Shandong.
    8 | ShanDa is a famous university of Shandong.
    1 | The Yangtze River is China's largest river and the third longest river in the world, with a total length of about 6,300 kilometers.
    3 | He graduated from the department of physics at Shandong University.
    7 | Shandong has dozens of universities.

• Ambil skor dan kembalikan 3 hasil teratas.

  SELECT id,
         content,
         TEXT_SEARCH(content, 'shandong university') AS score,
         TOKENIZE(content, 'jieba')
    FROM wiki_articles
  ORDER BY score DESC
  LIMIT 3;
  
  -- Hasil
  id  |                               content                               |  score  |                     tokenize                     
  ----+---------------------------------------------------------------------+---------+--------------------------------------------------
    8 | ShanDa is a famous university of Shandong.                          | 2.74634 | {shanda,famous,univers,shandong}
    7 | Shandong has dozens of universities.                                | 2.74634 | {shandong,has,dozen,univers}
    3 | He graduated from the department of physics at Shandong University. | 2.38178 | {he,graduat,from,depart,physic,shandong,univers}

• Gunakan fungsi TEXT_SEARCH baik di klausa output maupun WHERE.

  SELECT id,
         content,
         TEXT_SEARCH(content, 'shandong university') AS score,
         TOKENIZE(content, 'jieba')
    FROM wiki_articles
   WHERE TEXT_SEARCH(content, 'shandong university') > 0
  ORDER BY score DESC;
  
  -- Hasil
  id  |                               content                               |  score  |                     tokenize                     
  ----+---------------------------------------------------------------------+---------+--------------------------------------------------
    7 | Shandong has dozens of universities.                                | 2.74634 | {shandong,has,dozen,univers}
    8 | ShanDa is a famous university of Shandong.                          | 2.74634 | {shanda,famous,univers,shandong}
    3 | He graduated from the department of physics at Shandong University. | 2.38178 | {he,graduat,from,depart,physic,shandong,univers}
    2 | Li was born in 1962 in Wendeng County, Shandong.                    | 1.09244 | {li,born,1962,wendeng,counti,shandong}

• Ambil dokumen dari sumber 'wiki' yang paling relevan dengan 'shandong university'.

  -- Tabel sumber untuk JOIN.
  CREATE TABLE article_source (id int primary key, source text);
  INSERT INTO article_source VALUES (1, 'baike'), (2, 'wiki'), (3, 'wiki'), (4, 'baike'),
                                    (5, 'baike'), (6, 'baike'), (7, 'wiki'), (8, 'paper'),
                                    (9, 'http_log'), (10, 'http_log'), (11, 'http_log');
                                    
  SELECT a.id,
         source, content,
         TEXT_SEARCH(content, 'shandong university') AS score,
         TOKENIZE(a.content, 'jieba')
    FROM wiki_articles a
    JOIN article_source b
      ON (a.id = b.id)
   WHERE TEXT_SEARCH(a.content, 'shandong university') > 0
     AND b.source = 'wiki'
  ORDER BY score DESC;
  
  -- Hasil
  id  | source |                               content                               |  score  |                     tokenize                     
  ----+--------+---------------------------------------------------------------------+---------+--------------------------------------------------
    7 | wiki   | Shandong has dozens of universities.                                | 2.74634 | {shandong,has,dozen,univers}
    3 | wiki   | He graduated from the department of physics at Shandong University. | 2.38178 | {he,graduat,from,depart,physic,shandong,univers}
    2 | wiki   | Li was born in 1962 in Wendeng County, Shandong.                    | 1.09244 | {li,born,1962,wendeng,counti,shandong}

Contoh tokenizer berbeda

• Gunakan tokenizer Jieba default. Mode pencarian default menghasilkan lebih banyak bentuk kata untuk meningkatkan hasil pencarian.

  SELECT TOKENIZE('He came to Beijing Tsinghua University', 'jieba');
  
  -- Hasil
                  tokenize                
  --------------------------------------
  {he,came,to,beijing,tsinghua,hua,university,tsinghua university}

• Gunakan tokenizer Jieba kustom dalam mode exact, yang tidak menghasilkan bentuk kata tambahan.

  SELECT TOKENIZE('He came to Beijing Tsinghua University', 'jieba', '{"tokenizer": {"type": "jieba", "mode": "exact"}}');
  
  -- Hasil
          tokenize         
  -----------------------
  {he,came,to,beijing,tsinghua university}

• Perbandingan tokenizer (masing-masing dengan konfigurasi default-nya).

  SELECT TOKENIZE('He came to Beijing Tsinghua University', 'jieba') as jieba,
         TOKENIZE('He came to Beijing Tsinghua University', 'ik') as ik,
         TOKENIZE('He came to Beijing Tsinghua University', 'keyword') as keyword,
         TOKENIZE('He came to Beijing Tsinghua University', 'whitespace') as whitespace,
         TOKENIZE('He came to Beijing Tsinghua University', 'simple') as simple,
         TOKENIZE('He came to Beijing Tsinghua University', 'standard') as standard,
         TOKENIZE('He came to Beijing Tsinghua University', 'icu') as icu,
         TOKENIZE('He came to Beijing Tsinghua University', 'ngram') as ngram,
         TOKENIZE('He came to Beijing Tsinghua University', 'pinyin') as pinyin;
           
  -- Hasil
  -[ RECORD 1 ]--------------------------------------------------------------------
  jieba      | {he,came,to,beijing,tsinghua,hua,university,tsinghua university}
  ik         | {he,came,to,beijing,tsinghua university,tsinghua,university}
  keyword    | {He came to Beijing Tsinghua University}
  whitespace | {He,came,to,Beijing,Tsinghua,University}
  simple     | {he,came,to,beijing,tsinghua,university}
  standard   | {he,came,to,beijing,tsinghua,university}
  icu        | {he,came,to,beijing,tsinghua,university}
  ngram      | {h,he,e,"e ",c,ca,a,am,m,me,e,"e ",t,to,o,"o ",b,be,e,ei,i,ij,j,ji,i,in,n,ng,g,"g ",t,ts,s,si,i,in,n,ng,g,gh,h,hu,u,ua,a,"a ",u,un,n,ni,i,iv,v,ve,e,er,r,rs,s,si,i,it,t,ty,y}
  pinyin     | {ta,tldbjqhdx,lai,dao,bei,jing,qing,hua,da,xue}

• Perbandingan efek tokenisasi pada http_logs.

  SELECT TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'jieba') as jieba,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'ik') as ik,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'keyword') as keyword,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'whitespace') as whitespace,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'simple') as simple,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'standard') as standard,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'icu') as icu,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'ngram') as ngram,
         TOKENIZE('211.11.X.X - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0', 'pinyin') as pinyin;
         
  -- Hasil yang dikembalikan       
  -[ RECORD 1 ]-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  jieba      | {211.11,9.0,1998-06,21t15,00,01-05,00,get,english,index,html,http,1.0,304,0}
  ik         | {211.11.X.X,1998-06-21t15,1998,06,21,t,15,00,01-05,01,05,00,get,english,index.html,index,html,http,1.0,304,0}
  keyword    | {"211.11.X.X - - [1998-06-21T15:00:01-05:00] \\\"GET /english/index.html HTTP/1.0\\\" 304 0"}
  whitespace | {211.11.X.X,-,-,[1998-06-21T15:00:01-05:00],"\\\"GET",/english/index.html,"HTTP/1.0\\\"",304,0}
  simple     | {211,11,9,0,1998,06,21t15,00,01,05,00,get,english,index,html,http,1,0,304,0}
  standard   | {211.11.X.X,1998,06,21t15,00,01,05,00,get,english,index.html,http,1.0,304,0}
  icu        | {211.11.X.X,1998,06,21t15,00,01,05,00,get,english,index.html,http,1.0,304,0}
  ngram      | {2,21,1,11,1,1.,.,.1,1,11,1,1.,.,.9,9,9.,.,.0,0,"0 "," "," -",-,"- "," "," -",-,"- "," "," [",[,[1,1,19,9,99,9,98,8,8-,-,-0,0,06,6,6-,-,-2,2,21,1,1T,T,T1,1,15,5,5:,:,:0,0,00,0,0:,:,:0,0,01,1,1-,-,-0,0,05,5,5:,:,:0,0,00,0,0],],"] "," "," \\","\\","\\\"","\"","\"G",G,GE,E,ET,T,"T "," "," /",/,/e,e,en,n,ng,g,gl,l,li,i,is,s,sh,h,h/,/,/i,i,in,n,nd,d,de,e,ex,x,x.,.,.h,h,ht,t,tm,m,ml,l,"l "," "," H",H,HT,T,TT,T,TP,P,P/,/,/1,1,1.,.,.0,0,"0\\","\\","\\\"","\"","\" "," "," 3",3,30,0,04,4,"4 "," "," 0",0}
  pinyin     | {211119019980621,211119019980621t,t,1500010500,ge,teng,li,shi,n,de,x,h,t,m,l,h,t,t,p,103040}

Persyaratan konfigurasi analyzer_params

Persyaratan konfigurasi untuk parameter analyzer_params adalah sebagai berikut:

• Hanya string berformat JSON yang didukung.

• Tingkat atas JSON mendukung dua kunci: tokenizer dan filter. Nilainya dijelaskan sebagai berikut:

  • filter: Opsional. Nilainya adalah array JSON yang digunakan untuk mengonfigurasi properti penyaringan token. Jika Anda mengonfigurasi beberapa properti penyaringan token, properti tersebut diterapkan pada setiap token sesuai urutan yang ditentukan.

  • tokenizer: Wajib. Nilainya adalah objek JSON yang digunakan untuk mengonfigurasi properti tokenizer. Objek JSON mendukung kunci berikut:

    • type: Wajib. Nama tokenizer.

    • Parameter lain dari objek tokenizer. Parameter yang didukung bervariasi tergantung tokenizer, seperti yang ditunjukkan dalam tabel berikut:

      Tokenizer	Parameter lain dari objek tokenizer	Deskripsi parameter	Nilai parameter
      jieba	mode	Menentukan mode tokenisasi.	search (default): Menampilkan beberapa kombinasi yang mungkin selama tokenisasi, yang memungkinkan redundansi. Misalnya, hasil tokenisasi untuk "traditional festival" adalah "traditional", "festival", dan "traditional festival". exact: Tidak melakukan pemisahan redundan selama tokenisasi. Misalnya, hasil tokenisasi untuk "traditional festival" adalah "traditional festival".
      	hmm	Menentukan apakah akan menggunakan Model Markov Tersembunyi (HMM) untuk mengenali kata yang tidak ada di kamus. Ini meningkatkan pengenalan kata baru.	true: Fitur diaktifkan. Ini adalah nilai default. false: Pengaturan ini dinonaktifkan.
      standard	max_token_length	Panjang maksimum token.	Bilangan bulat positif. Nilai default adalah 255. Jika token melebihi panjang ini, token tersebut dipisah pada interval nilai max_token_length.
      ik	mode	Menentukan mode tokenisasi.	ik_max_word (default): Melakukan tokenisasi detail halus dan menghasilkan semua kata pendek yang mungkin. Misalnya, hasil tokenisasi untuk "Nanjing City" adalah "Nanjing" dan "City". ik_smart: Melakukan tokenisasi kasar. Mode ini memprioritaskan kata panjang untuk mengurangi jumlah pemisahan. Token output tidak tumpang tindih. Mode ini mencoba menggabungkan angka dan kuantifier menjadi satu token. Misalnya, "Nanjing City" dan "1 kilometer" tidak ditokenisasi lebih lanjut.
      	enable_lowercase	Menentukan apakah akan mengonversi token menjadi huruf kecil.	true (default) false
      ngram	min_ngram	Panjang minimum token dalam karakter.	Bilangan bulat positif. Nilai default adalah 1. Selisih maksimum antara nilai ini dan max_ngram adalah 3. Catatan: Anda dapat menyesuaikan nilai selisih maksimum dengan mengatur GUC berikut, misalnya, SET hg_fulltext_index_max_ngram_diff = 5
      	max_ngram	Panjang maksimum token dalam karakter.	Nilai default adalah 2. Nilai harus berada dalam rentang [1, 255]. Selisih maksimum antara nilai ini dan min_ngram adalah 3. Catatan Jika selisih antara max_ngram dan min_ngram terlalu besar, tokenizer ngram menghasilkan banyak token. Hal ini meningkatkan konsumsi resource, memerlukan lebih banyak penyimpanan indeks, dan memperpanjang waktu pembuatan indeks.
      	prefix_only	Menentukan apakah hanya mempertimbangkan awalan.	true false (default)
      pinyin	keep_first_letter	Menentukan apakah akan menyimpan huruf awal Pinyin setiap karakter Mandarin.	true (default): Menyimpan huruf awal. Misalnya, hasil tokenisasi untuk "李明" (Li Ming) mencakup "lm". false: Tidak akan disimpan.
      	keep_separate_first_letter	Menentukan apakah akan menyimpan huruf awal Pinyin setiap karakter Mandarin sebagai token terpisah.	true: Menyimpan huruf awal sebagai token terpisah. Misalnya, hasil tokenisasi untuk "李明" (Li Ming) mencakup "l" dan "m". false (default): Item tidak disimpan.
      	limit_first_letter_length	Panjang maksimum token yang berisi huruf awal.	Panjang maksimum string huruf awal yang disimpan. Bilangan bulat. Default-nya adalah 16.
      	keep_full_pinyin	Menentukan apakah akan menyimpan Pinyin lengkap setiap karakter Mandarin sebagai token terpisah.	true (default): Menyimpan suku kata Pinyin sebagai token terpisah. Misalnya, hasil tokenisasi untuk "李明" (Li Ming) mencakup "li" dan "ming". false: Item tidak disimpan.
      	keep_joined_full_pinyin	Menentukan apakah akan menggabungkan Pinyin lengkap semua karakter Mandarin menjadi satu token.	true: Menggabungkan suku kata Pinyin menjadi satu token. Misalnya, hasil tokenisasi untuk "李明" (Li Ming) mencakup "liming". false (bawaan): Koneksi tidak dibuat.
      	keep_none_chinese	Menentukan apakah akan menyimpan huruf atau angka non-Mandarin dalam hasil tokenisasi.	true (default): Menyimpannya. Salah: Jangan simpan.
      	keep_none_chinese_together	Menentukan apakah akan menyimpan huruf atau angka non-Mandarin yang berurutan bersama sebagai satu token.	true (default): Menyimpan teks non-Mandarin yang berurutan bersama. Misalnya, hasil tokenisasi untuk "DJ李明" (DJ Li Ming) mencakup "DJ", "li", dan "ming", di mana "DJ" adalah satu token. false: Memisahkan teks non-Mandarin yang berurutan. Misalnya, hasil tokenisasi untuk "DJ李明" (DJ Li Ming) mencakup "D", "J", "li", dan "ming". Catatan Parameter ini hanya berlaku ketika keep_none_chinese diatur ke true.
      	keep_none_chinese_in_first_letter	Menentukan apakah akan menyertakan huruf atau angka non-Mandarin dalam token yang berisi huruf awal.	true (default): Menggabungkan huruf awal Pinyin dengan teks non-Mandarin. Misalnya, hasil tokenisasi untuk "李明AT2025" (Li Ming AT2025) mencakup token "lmat2025". false: Jangan pertahankan.
      	keep_none_chinese_in_joined_full_pinyin	Menentukan apakah akan menyertakan huruf atau angka non-Mandarin dalam token Pinyin lengkap yang digabung.	true: Menggabungkan Pinyin lengkap dengan teks non-Mandarin. Misalnya, hasil tokenisasi untuk "李明AT2025" (Li Ming AT2025) mencakup token "limingat2025". false (default): Tidak menyimpannya.
      	none_chinese_pinyin_tokenize	Jika huruf non-Mandarin membentuk suku kata Pinyin, tentukan apakah akan memisahkannya menjadi istilah Pinyin terpisah.	true (default): Memisahkannya. Misalnya, hasil tokenisasi untuk "limingalibaba2025" mencakup "li", "ming", "a", "li", "ba", "ba", dan "2025". false: Tidak memisahkannya. Catatan Parameter ini hanya berlaku ketika keep_none_chinese dan keep_none_chinese_together diatur ke true.
      	keep_original	Menentukan apakah akan menyimpan input asli.	true: Menyimpan input asli. false (default): Tidak menyimpan input asli.
      	lowercase	Menentukan apakah huruf non-Tiongkok akan dikonversi menjadi huruf kecil.	true (default) false
      	trim_whitespace	Menentukan apakah akan menghapus karakter spasi.	true (default) false
      	remove_duplicated_term	Menentukan apakah akan menghapus token duplikat.	true: Menghapus token duplikat. Misalnya, hasil tokenisasi untuk "de的" adalah "de". Namun, ini memengaruhi hasil kueri frasa untuk "de的". false (default): Tidak menghapus duplikat.
      	keep_separate_chinese	Menentukan apakah akan menyimpan karakter Mandarin individual sebagai token terpisah.	true: Menyimpan karakter tunggal sebagai token terpisah. Misalnya, hasil tokenisasi untuk "李明" (Li Ming) mencakup dua token "李" dan "明". false (default): Tidak menyimpannya.

Konfigurasi analyzer_params default

Konfigurasi analyzer_params default untuk tokenizer berbeda adalah sebagai berikut:

{
  "tokenizer": {
    "type": "jieba", 
    "mode": "search",
    "hmm": true
  }, 
  "filter": [
    "removepunct",
    "lowercase",
    {"type": "stop", "stop_words": ["_english_"]},
    {"type": "stemmer", "language": "english"}
  ]
}

Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.

{
  "tokenizer": {
    "type": "whitespace"
  }
}

The Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.

{
  "tokenizer": {
    "type": "keyword"
  }
}

The Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.

{
  "tokenizer": {
    "type": "simple"
  }, 
  "filter": [
    "lowercase"
  ]
}

Spring Festival
Also known as the Lunar New Year
The most important traditional festival in China

{
  "tokenizer": {
    "type": "standard",
    "max_token_length": 255
  }, 
  "filter": [
    "lowercase"
  ]
}

Spring
Festival,
also
known
as
Lunar
New
Year,
is
the
most
important
traditional
festival
in
China.

{
  "tokenizer": {
    "type": "icu"
  }, 
  "filter": [
    "removepunct",
    "lowercase"
  ]
}

The Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.

{
  "tokenizer": {
    "type": "ik",
    "mode": "ik_max_word",
    "enable_lowercase": true
  },
  "filter": [
    {"type": "stop", "stop_words": ["_english_"]},
    {"type": "stemmer", "language": "english"}
  ]
}

Spring Festival, i.e., the Lunar New Year, is China's most important traditional festival.

{
  "tokenizer": {
    "type": "ngram",
    "min_gram": 1,
    "max_gram": 2,
    "prefix_only": false
  }
}

Spring Festival, also known as the Lunar New Year, is the most important traditional festival in China.

{
  "tokenizer": {
    "type": "pinyin",
    "keep_first_letter": true,
    "keep_separate_first_letter": false,
    "keep_full_pinyin": true,
    "keep_joined_full_pinyin": false,
    "keep_none_chinese": true,
    "keep_none_chinese_together": true,
    "none_chinese_pinyin_tokenize": true,
    "keep_original": false,
    "limit_first_letter_length": 16,
    "lowercase": true,
    "trim_whitespace": true,
    "keep_none_chinese_in_first_letter": true,
    "keep_none_chinese_in_joined_full_pinyin": false,
    "remove_duplicated_term": false,
    "ignore_pinyin_offset": true,
    "fixed_pinyin_offset": false,
    "keep_separate_chinese": false
  }
}

chun
cjjnlxnszgzzydct
jie
ji
nong
li
xin
nian
shi
zhong
guo
zui
zhong
yao
de
chuan
tong
jie
ri

Konfigurasi filter dalam analyzer_params

Hologres mendukung konfigurasi properti filter berikut (properti penyaringan token) dalam analyzer_params.

"lowercase"

"_english_"
"_danish_"
"_dutch_"
"_finnish_"
"_french_"
"_german_"
"_hungarian_"
"_italian_"
"_norwegian_"
"_portuguese_"
"_russian_"
"_spanish_"
"_swedish_"

"arabic",
"danish",
"dutch",
"english",
"finnish",
"french",
"german",
"greek",
"hungarian",
"italian",
"norwegian",
"portuguese",
"romanian",
"russian",
"spanish",
"swedish",
"tamil",
"turkish"

{"type": "length", "max": 10}

"removepunct"

{
  "type": "pinyin",
  "keep_first_letter": true,
  "keep_separate_first_letter": false,
  "keep_full_pinyin": true,
  "keep_joined_full_pinyin": false,
  "keep_none_chinese": true,
  "keep_none_chinese_together": true,
  "none_chinese_pinyin_tokenize": true,
  "keep_original": false,
  "limit_first_letter_length": 16,
  "lowercase": true,
  "trim_whitespace": true,
  "keep_none_chinese_in_first_letter": true,
  "keep_none_chinese_in_joined_full_pinyin": false,
  "remove_duplicated_term": false,
  "ignore_pinyin_offset": true,
  "fixed_pinyin_offset": false,
  "keep_separate_chinese": false
}