Platform For AI:Knowledge base management

How it works

A LangStudio Knowledge Base transforms files from an OSS data source into a format retrievable by large language models (LLMs). The workflow consists of three core steps:

1. Data reading and chunking: The system reads source files from the OSS location you specify.

   • Unstructured documents are parsed and split into smaller, semantically complete text blocks (chunks).

   • Structured data is chunked by row.

   • Images are processed as a whole without being chunked.

2. Vectorization (Embedding): The system calls an embedding model to convert each data chunk or image into a numerical vector that represents its semantic meaning.

3. Storage and indexing: The system stores the vector data in a vector database and creates an index for efficient retrieval.

Getting Started: Create and use a knowledge base

This section shows you how to quickly create a Document-type knowledge base and use it in an Application Flow.

1. Create a knowledge base. Navigate to LangStudio and select a workspace. On the Knowledge Base tab, click Create Knowledge Base. Configure the following key parameters, leave the rest as default, and then click OK.

   Parameter		Description
   Basic Configuration	Name	Enter a custom name for the knowledge base, such as test_kg.
   	Data Source OSS Path	The location where the knowledge base source files are stored. For example: oss://examplebucket.oss-cn-hangzhou-internal.aliyuncs.com/test/original/.
   	Output OSS Path	Stores the intermediate results and index information generated from document parsing. The final output depends on the selected vector database type. For example: oss://examplebucket.oss-cn-hangzhou-internal.aliyuncs.com/test/output/. Important If the Instance RAM Role set for the runtime is the PAI default role, we recommend that you set this parameter to a directory within the current workspace default storage path's OSS Bucket.
   	Type	Select Document.
   Embedding Model and Database	Embedding Type	Select Alibaba Cloud Model Studio Service (you must create a connection in advance, see Connection Configuration), and then select the created connection and model.
   	Vector Database Type	Select FAISS for quick testing.

2. Upload files.

   1. On the Knowledge Base tab, click the knowledge base you just created. On the Overview page, switch to the Documents tab. This tab displays the documents from the OSS path you configured as the data source.

   2. You can add or update files using the Upload button on the page, or you can upload files directly to the data source OSS path. For example, upload rag_test_doc.txt through the page. For more information on supported file formats, see Knowledge Base Types.

      

3. Update the index. After uploading files, click Update Index in the upper-right corner. In the dialog box that appears, configure the computing resources and network. After the index update task succeeds, the file status changes to Indexed. You can click a file to preview its chunks. For an image knowledge base, the system returns a list of images.

   Note

   For document chunks stored in Milvus, you can set their status to Enabled or Disabled individually. Disabled chunks will not be retrieved during a search.

   

4. Run a retrieval test. After the index is updated, switch to the Recall Test tab. Enter a query and tune the retrieval parameters to test the retrieval performance.

   

5. Use the knowledge base in an Application Flow. After testing, you can retrieve information from the knowledge base in an application flow. In the knowledge base node, you can enable the query rewriting and result reranking features and view the rewritten query in the execution trace.

   

   The result is a List[Dict]. Each Dict has content and score keys, which represent the retrieved document chunk and its similarity score with the input query.

   [
     {
       "score": 0.8057173490524292,
       "content": "Due to the uncertainty caused by the pandemic, XX Bank proactively increased provisions for impairment losses on loans, advances, and non-credit assets based on economic trends and forecasts for China or the Chinese mainland. The bank also increased the write-off and disposal of non-performing assets to improve the provision coverage ratio. In 2020, the net profit reached 28.928 billion CNY, a year-on-year increase of 2.6%, and profitability gradually improved.\n(CNY in millions) 2020 2019 Change (%)\nOperating Results and Profitability\nOperating Income 153,542 137,958 11.3\nOperating Profit Before Impairment Losses 107,327 95,816 12.0\nNet Profit 28,928 28,195 2.6\nCost-to-Income Ratio(1)(%) 29.11 29.61 down 0.50 percentage points\nAverage Return on Total Assets (%) 0.69 0.77 down 0.08 percentage points\nWeighted Average Return on Equity (%) 9.58 11.30 down 1.72 percentage points\nNet Interest Margin(2)(%) 2.53 2.62 down 0.09 percentage points\nNote: (1) Cost-to-Income Ratio = Business and management fees / Operating income.",
       "id": "49f04c4cb1d48cbad130647bd0d75f***1cf07c4aeb7a5d9a1f3bda950a6b86e",
       "metadata": {
         "page_label": "40",
         "file_name": "2021-02-04_XX_Insurance_Group_Co_Ltd_XX_China_XX_2020_Annual_Report.pdf",
         "file_path": "oss://my-bucket-name/datasets/chatglm-fintech/2021-02-04__XX_Insurance_Group_Co_Ltd__601318__China_XX__2020_Annual_Report.pdf",
         "file_type": "application/pdf",
         "file_size": 7982999,
         "creation_date": "2024-10-10",
         "last_modified_date": "2024-10-10"
       }
     },
     {
       "score": 0.7708036303520203,
       "content": "7.2 billion CNY, a year-on-year increase of 5.2%.\n2020\n(CNY in millions) Life and Health Insurance Business, Property and Casualty Insurance Business, Banking Business, Trust Business, Securities Business, Other Asset Management Business, Technology Business, Other Business and Consolidation Elimination, Group Consolidated\nNet profit attributable to shareholders of the parent company 95,018 16,083 16,766 2,476 2,959 5,737 7,936 (3,876) 143,099\nMinority interest 1,054 76 12,162 3 143 974 1,567 281 16,260\nNet profit (A) 96,072 16,159 28,928 2,479 3,102 6,711 9,503 (3,595) 159,359\nItems excluded:\n Short-term investment fluctuation(1)(B) 10,308 – – – – – – – 10,308\n Impact of discount rate change (C) (7,902) – – – – – – – (7,902)\n One-time significant items and others excluded by management as not part of daily operating income and expenditure (D) – – – – – – 1,282 – 1,282\nOperating profit (E=A-B-C-D) 93,666 16,159 28,928 2,479 3,102 6,711 8,221 (3,595) 155,670\nOperating profit attributable to shareholders of the parent company 92,672 16,",
       "id": "8066c16048bd722d030a85ee8b1***36d5f31624b28f1c0c15943855c5ae5c9f",
       "metadata": {
         "page_label": "19",
         "file_name": "2021-02-04_XX_Insurance_Group_Co_Ltd_XXX_China_XX_2020_Annual_Report.pdf",
         "file_path": "oss://my-bucket-name/datasets/chatglm-fintech/2021-02-04__XX_Insurance_Group_Co_Ltd__601318__China_XX__2020_Annual_Report.pdf",
         "file_type": "application/pdf",
         "file_size": 7982999,
         "creation_date": "2024-10-10",
         "last_modified_date": "2024-10-10"
       }
     }
   ]

Detailed features

Configure retrieval parameters

• Top K: The maximum number of relevant text chunks to retrieve from the knowledge base. The value range is 1 to 100.

• Score threshold: Sets a similarity score threshold (0 to 1). Only chunks with a score above this threshold are returned. A higher value means a higher required similarity between the text and the query.

• Retrieval Pattern: The default is Dense (vector) retrieval. If you need to use Hybrid retrieval (vector + keyword), the vector database must be Milvus 2.4.x or later, or Elasticsearch. For guidance on choosing a retrieval mode, see Select a retrieval mode.

• Metadata filter condition: Filters the search scope using metadata to improve accuracy. For more information, see Use metadata.

• Query rewrite: Uses a large language model (LLM) to optimize a user's vague, colloquial, or context-dependent original query, making it clearer, more complete, and more intentional to improve retrieval accuracy. For more information about usage scenarios, see Query rewrite.

• Result Reranking: Uses a reranking model to reorder the initial retrieval results, placing the most relevant results at the top. For more information about usage scenarios, see Result Reranking.

  Note

  Result reranking requires a reranking model. Supported model service connection types include: Model Studio, AI Search Open Platform Model Service, and General Reranker Model Service.

Troubleshooting

When an index update or version cloning task fails, follow these steps to troubleshoot:

1. View operation records: On the knowledge base details page, find the failed task in the Operation Records and click View Task.

2. Check task logs: The system redirects you to the PAI workflow page. Check the logs of the failed node.For example, the workflow task for updating the index of a Document-type knowledge base includes the following three nodes. Except for the read-oss-file node, each node creates a PAI-DLC task. You can also view DLC task details through the Job URL in the logs.

   • read-oss-file: Reads OSS files.

   • rag-parse-chunk: Responsible for document preprocessing and chunking.

   • rag-sync-index: Responsible for embedding text chunks and synchronizing them to the vector database.

Optimize retrieval performance