AI Search Open Platform menyediakan layanan pemotongan dokumen yang dapat dipanggil melalui API. Anda dapat mengintegrasikan layanan ini ke dalam alur kerja bisnis untuk meningkatkan efisiensi pengambilan dan pemrosesan data.
|
Service name |
Service ID |
Description |
API QPS limit |
|
Document Chunking Service-001 |
ops-document-split-001 |
Menyediakan strategi pemotongan teks umum untuk membagi data terstruktur dalam format HTML, Markdown, dan plain text berdasarkan format paragraf, semantik teks, atau aturan tertentu. Layanan ini juga mendukung ekstraksi blok kode, gambar, dan tabel dari teks kaya (rich text). |
2 Catatan
Untuk meminta batas API QPS yang lebih tinggi, kirim tiket ke dukungan teknis. |
Dalam pipeline Retrieval-Augmented Generation (RAG), dokumen biasanya dikonversi menjadi vektor dan disimpan di database vektor untuk pengambilan. Layanan pemotongan dokumen membagi dokumen panjang menjadi chunk yang lebih kecil sesuai dengan persyaratan panjang model penyematan teks (text embedding). Proses ini memungkinkan vektorisasi dan pengambilan konten dari dokumen yang sangat panjang.
Penggunaan dasar
API pemotongan menerima string teks biasa dan opsi konfigurasi tambahan sebagai input, lalu mengembalikan teks yang telah dipotong serta elemen rich text (jika ada). Respons API berisi empat daftar: chunks, nodes, rich_texts, dan sentences. Untuk langkah penyematan berikutnya, ekstrak konten dari daftar chunks dan dari item dalam daftar rich_texts yang tipe-nya bukan image. Anda dapat merujuk ke templat kode dari pusat skenario. Contoh kode Python berikut menunjukkan cara melakukannya:
# Ekstrak hasil pemotongan. Perhatikan bahwa hanya ["chunks"] dan ["rich_texts"] (kecuali gambar) yang digunakan.
doc_list = []
for chunk in document_split_result.body.result.chunks:
doc_list.append({"id": chunk.meta.get("id"), "content": chunk.content})
for rich_text in document_split_result.body.result.rich_texts:
if rich_text.meta.get("type") != "image":
doc_list.append({"id": rich_text.meta.get("id"), "content": rich_text.content})
Penggunaan lanjutan
Layanan pemotongan dokumen dapat membagi dokumen kompleks menjadi chunk berdasarkan batas token tertentu, sehingga membentuk struktur pohon multi-node. Struktur pohon ini dapat digunakan selama fase pengambilan dalam pipeline RAG untuk menambahkan konteks ke chunk yang di-recall guna meningkatkan akurasi respons akhir.
Logika layanan membagi teks pada tingkat struktural tertinggi yang memungkinkan. Jika chunk hasil pemotongan melebihi panjang yang ditentukan, layanan akan secara rekursif membaginya hingga semua chunk memenuhi persyaratan panjang. Proses rekursif ini membentuk pohon chunk, di mana setiap node daun (leaf node) merepresentasikan chunk akhir, juga dikenal sebagai final node.
Dalam proses recall vektor berikutnya, Anda dapat menggunakan informasi dari pohon chunk untuk pelengkapan konteks. Misalnya, dalam batas token model Anda, Anda dapat menyertakan chunk saudara (sibling chunks) dari level yang sama dengan chunk yang di-recall agar informasi konteks lebih lengkap.
Sebagai contoh, diberikan teks berikut:
After you successfully enable the AI Search Open Platform service for the first time, the system automatically creates a default workspace: Default.
Click Create Workspace. Enter a custom workspace name, and then click Confirm. After you click Create new API key, the system generates an API key. You can then click the Copy button to copy and save the API key.
Pohon chunk yang mungkin dihasilkan adalah sebagai berikut:
root (6b15)
|
+-- paragraph_node (557b)
|
+-- newline_node (ef4d)[After you successfully enable the AI Search Open Platform...Default.]
|
+-- newline_node (c618)
|
+-- sentence_node (98ce)[Click Create Workspace...and then click Confirm.]
|
+-- sentence_node (922a)[After you click Create new API key...to copy and save the API key.]
Dengan ukuran maksimum chunk tertentu, pohon chunk lengkap berisi dua jenis node: final nodes (node yang memiliki konten chunk) dan intermediate nodes (node logis tanpa konten). Layanan mengembalikan seluruh pohon sebagai daftar semua node (nodes) dan final nodes dalam daftar terpisah (chunks). Berikut adalah beberapa tipe node yang mungkin muncul:
-
root: Node akar.
-
paragraph_node: Node paragraf yang merepresentasikan pemisahan berdasarkan delimiter
"\n\n"dan mengidentifikasi posisi paragraf. Karena contoh tidak mengandung"\n\n", hanya ada satu node perantara seperti ini. -
newline_node: Node baris baru yang merepresentasikan pemisahan berdasarkan delimiter
"\n". Dalam contoh,newline_node (ef4d)memenuhi persyaratan ukuran chunk sehingga merupakan final node, sedangkannewline_node (c618)memerlukan pemisahan lebih lanjut dan merupakan intermediate node. -
sentence_node: Node kalimat yang merepresentasikan pemisahan berdasarkan delimiter kalimat, seperti titik (
.). -
subsentence_node: Node subkalimat yang merepresentasikan pemisahan berdasarkan delimiter klausa, seperti koma (
,). Tipe ini tidak muncul dalam contoh.
Untuk konten dalam format Markdown atau HTML, layanan juga mengekstraksi elemen rich text ke dalam daftar terpisah rich_texts. Contoh elemen tersebut termasuk tag <img>, <table>, dan <code>. Dalam teks yang telah dipotong, layanan mengganti posisi elemen-elemen ini dengan placeholder seperti [image_0], <table>table_0</table>, dan <code>code_0</code>. Desain ini memungkinkan recall independen terhadap blok rich text dan memungkinkan penyisipan kembali ke konteks aslinya saat diperlukan. Setiap blok rich text termasuk dalam satu final node chunk yang unik.
Selain itu, untuk meningkatkan tingkat recall pada kueri pendek, Anda dapat mengatur parameter strategy.need_sentence ke true. Ini menginstruksikan layanan untuk membagi teks asli berdasarkan kalimat dan mengembalikan hasilnya dalam daftar terpisah sentences, yang dapat digunakan sebagai jalur recall independen. Untuk memfasilitasi perluasan kalimat, setiap blok kalimat termasuk dalam satu final node chunk yang unik. Perhatikan bahwa daftar sentences ini tidak terkait dengan tipe sentence_node yang dijelaskan sebelumnya.
Bidang-bidang yang dicetak tebal di atas—chunks, nodes, rich_texts, dan sentences—adalah bidang yang dikembalikan oleh API. Informasi penggunaan detail tersedia dalam deskripsi parameter berikut. Untuk kesederhanaan, output setiap chunk menggunakan sintaksis HTML yang disederhanakan.
Prasyarat
-
Dapatkan kredensial otentikasi
Platform terbuka AI Search memerlukan API key untuk otentikasi. Untuk instruksi, lihat Dapatkan API key.
-
Dapatkan titik akhir layanan
Anda dapat memanggil layanan melalui jaringan publik atau VPC. Untuk detailnya, lihat Dapatkan titik akhir layanan.
Spesifikasi permintaan
Catatan umum
-
Ukuran maksimum body permintaan adalah 8 MB.
Metode permintaan
POST
URL
{host}/v3/openapi/workspaces/{workspace_name}/document-split/{service_id}
-
host: Titik akhir layanan. Anda dapat memanggil layanan melalui Internet atau melalui VPC. Untuk informasi lebih lanjut, lihat Dapatkan titik akhir layanan. Placeholder{host}adalah alamat titik akhir API. Anda dapat memperolehnya dari halaman API key Management di panel navigasi kiri AI Search Development Workbench. Bagian API Endpoint menyediakan domain API publik untuk akses Internet dan domain API internal untuk akses dari VPC di wilayah yang sama. Kedua domain mendukung HTTPS. -
workspace_name: Nama ruang kerja. Contohnya,default. -
service_id: ID layanan bawaan. Contohnya,ops-document-split-001.
Parameter permintaan
Parameter header
Otentikasi API key
|
Parameter |
Type |
Required |
Description |
Example |
|
Content-Type |
String |
Yes |
Jenis permintaan: |
application/json |
|
Authorization |
String |
Yes |
API key. |
Bearer OS-d1**2a |
Parameter badan
|
Parameter |
Type |
Required |
Description |
Example |
|
document.content |
String |
Yes |
Konten teks biasa yang akan dipotong. Menurut standar JSON, karakter khusus berikut dalam field string harus di-escape: |
"Title\nFirst line\nSecond line" |
|
document.content_encoding |
String |
No |
Pengkodean karakter konten.
|
utf8 |
|
document.content_type |
String |
No |
Format konten.
|
html |
|
strategy.type |
String |
No |
Strategi pemotongan paragraf.
|
default |
|
strategy.max_chunk_size |
Int |
No |
Panjang maksimum chunk. Nilai default: 300. |
300 |
|
strategy.compute_type |
String |
No |
Metode perhitungan panjang chunk.
|
token |
|
strategy.need_sentence |
Boolean |
No |
Menentukan apakah chunk tingkat kalimat juga dikembalikan untuk mengoptimalkan pencarian kueri pendek.
|
false |
|
strategy.custom_split_label |
String |
No |
String pemisah kustom. Saat parameter ini tidak kosong, hanya string kustom yang digunakan untuk pemisahan; strategi default diabaikan. Beberapa string pemisah kustom dapat ditentukan, dipisahkan oleh koma (
|
|
Informasi tambahan:
-
Parameter
strategy.need_sentence: Pemotongan tingkat kalimat adalah strategi yang independen dari pemotongan tingkat paragraf. Setiap kalimat dikembalikan sebagai chunk individual. Saat pemotongan tingkat kalimat diaktifkan, chunk pendek (kalimat) dan panjang (paragraf) dapat di-recall secara bersamaan, saling melengkapi untuk meningkatkan tingkat recall secara keseluruhan.
Parameter respons
|
Parameter |
Type |
Description |
Example |
|
request_id |
String |
ID unik yang diberikan sistem untuk panggilan API. |
B4AB89C8-B135-****-A6F8-2BAB801A2CE4 |
|
latency |
Float/Int |
Waktu yang dibutuhkan untuk memproses permintaan, dalam milidetik (ms). |
10 |
|
usage |
Object |
Informasi penagihan untuk panggilan. |
"usage": { "token_count": 3072 } |
|
usage.token_count |
Int |
Jumlah token. |
3072 |
|
result.chunks |
List(Chunk) |
Daftar hasil pemotongan (final nodes), berisi konten chunk dan metadata. |
[{ "content" : "xxx", "meta":{'parent_id':x, 'id': x, 'type': 'text'} }] |
|
result.chunks[].content |
String |
Konten chunk. |
"xxx" |
|
result.chunks[].meta |
Map |
Metadata untuk chunk. Semua field berikut bertipe string:
|
{ 'parent_id': '3b94a18555c44b67b193c6ab4f****', 'id': 'c9edcb38fdf34add90d62f6bf5c6****, 'type': 'text' 'token': 10, } |
|
result.rich_texts |
List(RichText) |
Output rich text. Saat Catatan
Format ini tidak didukung saat |
[{ "content" : "xxx", "meta":{'belonged_chunk_id':x, 'id': x, 'type': 'table'} }] |
|
result.rich_texts[].content |
String |
Konten chunk rich text. |
"<table><tr>\n<th>Action</th>\n<th>Description</th>\n</tr><tr>\n<td>Hide component</td>\n<td>Hides the component. No parameters are required.</td>\n</tr></table>" |
|
result.rich_texts[].meta |
Map |
Metadata untuk chunk rich text. Semua field berikut bertipe string:
|
{ 'type': 'table', 'belonged_chunk_id': 'f0254cb7a5144a1fb3e5e024a3****b', 'id': 'table_2-1' 'token': 10 } |
|
result.nodes |
List(Node) |
Daftar semua node dalam pohon chunk. |
[{'parent_id':x, 'id': x, 'type': 'text'}] |
|
result.nodes[] |
Map |
Informasi tentang node dalam pohon chunk. Semua field berikut bertipe string:
|
{ 'id': 'f0254cb7a5144a1fb3e5e024a3****b', 'type': 'paragraph_node', 'parent_id': 'f0254cb7a5144a1fb3e5e024a3****b' } |
|
result.sentences (Optional) |
List(sentence) |
Daftar kalimat dari setiap chunk hanya dikembalikan jika |
[{ "content" : "xxx", "meta":{'belonged_chunk_id':x, 'id': x, 'type': 'sentence'} }] |
|
result.sentences[].content (Optional) |
String |
Konten kalimat. |
"123" |
|
result.sentences[].meta (Optional) |
Map |
Metadata kalimat:
|
{ 'id': 'f0254cb7a5144a1fb3e5e024a3****b1-1', 'type': 'sentence', 'belonged_chunk_id': 'f0254cb7a5144a1fb3e5e024a3****b', 'token': 10 } |
Contoh permintaan cURL
curl -XPOST -H"Content-Type: application/json"
"http://***-hangzhou.opensearch.aliyuncs.com/v3/openapi/workspaces/default/document-split/ops-document-split-001"
-H "Authorization: Bearer YOUR_API_KEY"
-d "{
\"document\":{
\"content\":\"Product Advantages\\nIndustry Algorithm Edition\\nIntelligent\\nFeatures a rich set of customizable algorithm models and industry-specific recall and ranking algorithms to ensure superior search results.\\n\\nFlexible and Customizable\\nDevelopers can customize algorithm models, application structures, data processing, query analysis, and ranking configurations based on their business characteristics and data. This meets personalized search needs, increases click-through rates, enables rapid business iteration, and significantly shortens the time-to-market for new features.\\n\\nSecure and Stable\\nProvides 24/7 operational maintenance and technical support through online tickets and phone support. A comprehensive incident response mechanism includes fault monitoring, automatic alerts, and rapid issue identification. Access control and isolation are enforced at the API level by using Alibaba Cloud AccessKeyId and AccessKeySecret security pairs, ensuring user-level data segregation and security. Data is redundantly backed up to prevent loss.\\n\\nElastic Scaling\\nResources can be elastically scaled up or down as needed.\\n\\nRich Peripheral Features\\nSupports a range of peripheral search features such as hot searches, search suggestions, and statistical reports for convenient display and analysis.\\n\\nOut-of-the-Box\\nNo need to deploy or maintain clusters. Quickly access a one-stop search service.\\n\\nHigh-Performance Retrieval Edition\\nHigh Throughput\\nSupports tens of thousands of write TPS for a single table, with updates in seconds.\\n\\nSecure and Stable\\nProvides 24/7 operational maintenance and technical support through online tickets and phone support. A comprehensive incident response mechanism includes fault monitoring, automatic alerts, and rapid issue identification. Access control and isolation are enforced at the API level by using Alibaba Cloud AccessKeyId and AccessKeySecret security pairs, ensuring user-level data segregation and security. Data is redundantly backed up to prevent loss.\\n\\nElastic Scaling\\nResources can be elastically scaled up or down as needed.\\n\\nOut-of-the-Box\\nNo need to deploy or maintain clusters. Quickly access a one-stop search service.\\n\\nVector Retrieval Edition\\nStable\\nThe underlying implementation in C++ has been developed for over a decade and supports multiple core businesses, making it highly stable and suitable for mission-critical search scenarios.\\n\\nEfficient\\nA distributed search engine that efficiently supports massive data retrieval and real-time data updates (effective in seconds), making it ideal for search scenarios that are sensitive to query latency and timeliness.\\n\\nCost-Effective\\nSupports multiple index compression strategies and multi-value index loading tests to meet user query needs at a lower cost.\\n\\nVector Algorithms\\nSupports vector retrieval for various unstructured data such as voice, images, videos, text, and behaviors.\\n\\nSQL Queries\\nSupports SQL syntax and online multi-table joins. Provides a rich set of built-in UDFs and a UDF customization mechanism to meet diverse retrieval needs. SQL Studio is integrated into the operations system for easy SQL development and testing.\\n\\nRecall Engine Edition\\nStable\\nThe underlying implementation in C++ has been developed for over a decade and supports multiple core businesses, making it highly stable and suitable for mission-critical search scenarios.\\n\\nEfficient\\nThe Wentian Engine is a distributed search engine that efficiently supports massive data retrieval and real-time data updates (effective in seconds), making it ideal for search scenarios that are sensitive to query latency and timeliness.\\n\\nCost-Effective\\nThe Wentian Engine supports multiple index compression strategies and multi-value index loading tests to meet user query needs at a lower cost.\\n\\nFeature-Rich\\nThe Wentian Engine supports various analyzer types, index types, and powerful query syntax to meet user retrieval needs. A plugin mechanism is also provided for customizing business logic.\\n\\nSQL Queries\\nThe Wentian Engine supports SQL syntax and online multi-table joins. Provides a rich set of built-in UDFs and a UDF customization mechanism to meet diverse retrieval needs. SQL Studio will soon be integrated into the operations system for easy SQL development and testing.\",
\"content_encoding\":\"utf8\",\"content_type\":\"text\"
},
\"strategy\":{
\"type\":\"default\",
\"max_chunk_size\":300,
\"compute_type\":\"token\",
\"need_sentence\":false
}
}"
Contoh respons
Respons sukses
{
"request_id": "47EA146B-****-448C-A1D5-50B89D7EA434",
"latency": 161,
"usage": {
"token_count": 800
},
"result": {
"chunks": [
{
"content": "Product Advantages\\nIndustry Algorithm Edition\\nIntelligent\\nFeatures a rich set of customizable algorithm models and industry-specific recall and ranking algorithms to ensure superior search results.\\n\\nFlexible and Customizable\\nDevelopers can customize algorithm models, application structures, data processing, query analysis, and ranking configurations based on their business characteristics and data. This meets personalized search needs, increases click-through rates, enables rapid business iteration, and significantly shortens the time-to-market for new features.\\n\\nSecure and Stable\\nProvides 24/7 operational maintenance and technical support through online tickets and phone support. A comprehensive incident response mechanism includes fault monitoring, automatic alerts, and rapid issue identification. Access control and isolation are enforced at the API level by using Alibaba Cloud AccessKeyId and AccessKeySecret security pairs, ensuring user-level data segregation and security. Data is redundantly backed up to prevent loss.\\n\\nElastic Scaling\\nResources can be elastically scaled up or down as needed.\\n\\nRich Peripheral Features\\nSupports a range of peripheral search features such as hot searches, search suggestions, and statistical reports for convenient display and analysis.\\n\\nOut-of-the-Box\\nNo need to deploy or maintain clusters. Quickly access a one-stop search service.\\n\\nHigh-Performance Retrieval Edition\\nHigh Throughput\\nSupports tens of thousands of write TPS for a single table, with updates in seconds",
"meta": {
"parent_id": "dee776dda3ff4b078bccf989a6bd****",
"id": "27eea7c6b2874cb7a5bf6c71afbf****",
"type": "text"
}
},
{
"content": ".\\n\\nSecure and Stable\\nProvides 24/7 operational maintenance and technical support through online tickets and phone support. A comprehensive incident response mechanism includes fault monitoring, automatic alerts, and rapid issue identification. Access control and isolation are enforced at the API level by using Alibaba Cloud AccessKeyId and AccessKeySecret security pairs, ensuring user-level data segregation and security. Data is redundantly backed up to prevent loss.\\n\\nElastic Scaling\\nResources can be elastically scaled up or down as needed.\\n\\nOut-of-the-Box\\nNo need to deploy or maintain clusters. Quickly access a one-stop search service.\\n\\nVector Retrieval Edition\\nStable\\nThe underlying implementation in C++ has been developed for over a decade and supports multiple core businesses, making it highly stable and suitable for mission-critical search scenarios.\\n\\nEfficient\\nA distributed search engine that efficiently supports massive data retrieval and real-time data updates (effective in seconds), making it ideal for search scenarios that are sensitive to query latency and timeliness.\\n\\nCost-Effective\\nSupports multiple index compression strategies and multi-value index loading tests to meet user query needs at a lower cost.\\n\\nVector Algorithms\\nSupports vector retrieval for various unstructured data such as voice, images, videos, text, and behaviors.\\n\\nSQL Queries\\nSupports SQL syntax and online multi-table joins. Provides a rich set of built-in UDFs and a UDF customization mechanism to meet diverse retrieval needs",
"meta": {
"parent_id": "dee776dda3ff4b078bccf989a6bd****",
"id": "bf9fcfb47fcf410aa05216e268df****",
"type": "text"
}
},
{
"content": ". SQL Studio is integrated into the operations system for easy SQL development and testing.\\n\\nRecall Engine Edition\\nStable\\nThe underlying implementation in C++ has been developed for over a decade and supports multiple core businesses, making it highly stable and suitable for mission-critical search scenarios.\\n\\nEfficient\\nThe Wentian Engine is a distributed search engine that efficiently supports massive data retrieval and real-time data updates (effective in seconds), making it ideal for search scenarios that are sensitive to query latency and timeliness.\\n\\nCost-Effective\\nThe Wentian Engine supports multiple index compression strategies and multi-value index loading tests to meet user query needs at a lower cost.\\n\\nFeature-Rich\\nThe Wentian Engine supports various analyzer types, index types, and powerful query syntax to meet user retrieval needs. A plugin mechanism is also provided for customizing business logic.\\n\\nSQL Queries\\nThe Wentian Engine supports SQL syntax and online multi-table joins. Provides a rich set of built-in UDFs and a UDF customization mechanism to meet diverse retrieval needs. SQL Studio will soon be integrated into the operations system for easy SQL development and testing.",
"meta": {
"parent_id": "dee776dda3ff4b078bccf989a6bd****",
"id": "26ab0e4f7665487bb0a82c5a226a****",
"type": "text"
}
}
],
"nodes": [
{
"id": "dee776dda3ff4b078bccf989a6bd****",
"type": "root",
"parent_id": "dee776dda3ff4b078bccf989a6bd****"
},
{
"id": "27eea7c6b2874cb7a5bf6c71afbf****",
"type": "sentence",
"parent_id": "dee776dda3ff4b078bccf989a6bd****"
},
{
"id": "bf9fcfb47fcf410aa05216e268df****",
"type": "sentence",
"parent_id": "dee776dda3ff4b078bccf989a6bd****"
},
{
"id": "26ab0e4f7665487bb0a82c5a226a****",
"type": "sentence",
"parent_id": "dee776dda3ff4b078bccf989a6bd****"
}
],
"rich_texts": []
}
}
Respons error
Jika terjadi error, respons mencakup field code dan message yang menunjukkan penyebabnya.
{
"request_id": "817964CD-1B84-4AE1-9B63-4FB99734****",
"latency": 0,
"code": "InvalidParameter",
"message": "JSON parse error: Invalid UTF-8 start byte 0xbc; nested exception is com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xbc\n at line: 2, column: 19]"
}
Kode status
Untuk informasi lebih lanjut, lihat Kode status untuk AI Search Open Platform.