Alibaba Cloud Model Studio:Optimize RAG performance

2.2.1 Intelligent chunking

Choosing the right text chunk length for your knowledge base is not easy because you must consider multiple factors, such as:

• Document type: For example, for professional literature, increasing the length usually helps to retain more contextual information. For social media posts, shortening the length can capture semantics more accurately.

• Prompt complexity: Generally, if the user's prompt is complex and specific, you may need to increase the length. Otherwise, shortening the length is more appropriate.

These conclusions do not necessarily apply to all situations. You need to choose the right tools and experiment repeatedly to find the right text chunk length. For example, LlamaIndex provides evaluation functions for different chunking methods. However, this process can be difficult.

If you want to achieve good results quickly, we recommend that you select Intelligent Splitting for Document Splitting when you create a knowledge base.

When this strategy is applied, the knowledge base performs the following steps:

1. First, use the system's built-in sentence delimiters to divide the document into several paragraphs.

2. Based on the divided paragraphs, adaptively select chunking points for chunking based on semantic relevance (semantic chunking), rather than chunking based on a fixed length.

During this process, the knowledge base strives to ensure the semantic integrity of each part of the document and tries to avoid unnecessary divisions and chunking. This strategy is applied to all documents in this knowledge base, including subsequently imported documents.

2.2.2 Correcting text chunk content

In the actual chunking process, unexpected chunking or other problems can still occur. For example, spaces in the text are sometimes parsed as %20 after chunking.

Therefore, Model Studio recommends that you perform a manual check after you import the document into the knowledge base to confirm the semantic integrity and correctness of the text chunk content. If you find unexpected chunking or other parsing errors, you can edit the text chunk directly to correct it. After you save your changes, the original content of the text chunk becomes invalid, and the new content is used for knowledge base retrieval.

Note that this action modifies only the text chunks in the knowledge base, not the original document or data table in your data management (temporary storage). Therefore, if you import it into the knowledge base again later, you must perform another manual check and correction.

2.3.1 Multi-round conversation rewriting

In a multi-round conversation, a user might ask a question with a short prompt, such as "Bailian Phone X1". This prompt may cause the RAG system to lack the necessary context during retrieval for the following reasons:

• A mobile phone product usually has multiple generations available for sale at the same time.

• For the same generation of a product, the manufacturer usually offers multiple storage capacity options, such as 128 GB and 256 GB.

  ...

The user may have already provided this key information in previous turns of the conversation. Using this information effectively can help RAG retrieve more accurate information.

To address this situation, you can use the Multi-round Conversation Rewriting feature in Model Studio. The system automatically rewrites the user's prompt into a more complete form based on the conversation history.

For example, a user asks:

Bailian Phone X1.

When the multi-round conversation rewriting feature is enabled, the system rewrites the user's prompt based on their conversation history before retrieval, as shown in the following example:

Provide all available versions of Bailian Phone X1 in the product library and their specific parameters.

This rewritten prompt helps RAG better understand the user's intent and provide a more accurate response.

The figure below shows how to enable the multi-round conversation rewriting feature. This feature is also enabled when you select Recommended.

Note that the multi-round conversation rewriting feature is attached to the knowledge base. Once enabled, it applies only to queries related to the current knowledge base. If you do not enable this configuration when you create the knowledge base, you cannot enable it later unless you recreate the knowledge base.

2.3.2 Tag filtering

The content of this section applies only to document search knowledge bases.

When you use a music app, you might filter songs by the artist's name to quickly find songs from that artist.

Similarly, adding tags to your unstructured documents introduces additional structured information. This allows the application to filter documents based on tags when retrieving from the knowledge base, which improves retrieval accuracy and efficiency.

Model Studio supports two methods for setting tags:

• Set tags when uploading documents: For the steps in the console, see Import data.

• Edit tags on the Data Management page: For uploaded documents, you can click Tag on the right of the document to edit.

  

Model Studio supports two methods for using tags:

• When you call an application through an API, you can specify tags in the tags request parameter.

• Set tags when editing an application in the console. However, this method is applicable only to agent applications.

  Note that this setting applies to all subsequent questions and answers for this agent application.

  

2.3.3 Extracting metadata

The content of this section applies only to document search knowledge bases.

Embedding metadata into text chunks can effectively enhance the contextual information of each chunk. In specific scenarios, this method can significantly improve the RAG performance of document search knowledge bases.

Consider the following scenario:

A knowledge base contains many mobile phone product description documents. The document names are the phone models (such as Bailian X1, Bailian Zephyr Z9, etc.), and all documents include a "Function Overview" chapter.

If metadata is not enabled for this knowledge base, a user might enter the following prompt for retrieval:

Function overview of Bailian Phone X1.

A retrieval test shows which chunks were recalled, as shown in the following figure. Because all documents contain a "Function Overview" section, the knowledge base recalls some text chunks that are unrelated to the query entity (Bailian Phone X1) but are similar to the prompt, such as Chunk 1 and Chunk 2 in the figure. Their rankings are even higher than the text chunk that is actually needed. This clearly affects RAG performance.

The recall results of the retrieval test only guarantee the ranking. The absolute value of the similarity score is for reference only. When the difference in absolute values is small (within 5%), the recall probability can be considered the same.

Next, set the phone name as metadata. This is equivalent to adding the corresponding phone name information to the text chunks of each document. You can follow the steps in metadata extraction and then run the same test for comparison.

At this point, the knowledge base adds a layer of structured search before the vector search. The process is as follows:

1. Extract metadata {"key": "name", "value": "Bailian Phone X1"} from the prompt.

2. Based on the extracted metadata, find all text chunks that contain the "Bailian Phone X1" metadata.

3. Then, perform a vector (semantic) search to find the most relevant text chunks.

The actual recall effect of the retrieval test after enabling metadata is shown in the following figure. As you can see, the knowledge base can now accurately find the text chunk that is related to "Bailian Phone X1" and contains "Function Overview".

Another common application of metadata is to embed date information in text chunks to filter for recent content. For more information about metadata usage, see metadata extraction.

2.3.4 Adjusting the similarity threshold

When the knowledge base finds text chunks related to the user's prompt, it first sends them to the Rank model for reordering. You can configure this in the Custom parameter settings when you create the knowledge base. The similarity threshold is then used to filter the reordered text chunks. Only text chunks with a similarity score that exceeds this threshold are likely to be provided to the model.

Lowering this threshold may recall more text chunks, but it may also cause some less relevant text chunks to be recalled. Increasing this threshold may reduce the number of recalled text chunks.

If it is set too high, as shown in the following figure, it may cause the knowledge base to discard all relevant text chunks. This limits the model's ability to obtain sufficient background information to generate an answer.

There is no single best threshold, only the one that is most suitable for your scenario. You need to experiment with different similarity thresholds through retrieval tests, observe the recall results, and find the solution that best suits your needs.

2.3.5 Increase the number of recalled chunks

The number of recalled chunks is the K value in the multi-channel recall strategy. After the similarity threshold filtering, if the number of text chunks exceeds K, the system selects the K text chunks with the highest similarity scores to provide to the model. Because of this, an inappropriate K value may cause RAG to miss the correct text chunks, which affects the model's ability to generate a complete answer.

For example, in the following case, the user retrieves with the following prompt:

What are the advantages of the Bailian X1 phone?

As you can see in the following diagram, there are a total of 7 text chunks in the target knowledge base that are related to the user's prompt and should be returned. These are marked in green on the left. However, because this number exceeds the currently set maximum number of recalled chunks (K), the text chunks containing advantage 5 (ultra-long standby) and advantage 6 (clear photos) are discarded and not provided to the model.

Because RAG cannot determine how many text chunks are needed to provide a complete answer, the model will generate an answer based on the provided chunks, even if they are incomplete.

Many experiments show that in scenarios that require lists, summaries, or comparisons, providing more high-quality text chunks (for example, K=20) to the model is more effective than providing only the top 10 or top 5. Although this may introduce noise, if the text chunk quality is high, the model can usually offset the impact of the noise.

You can adjust the Number Of Recalled Chunks when you edit an application.

Note that a larger number of recalled chunks is not always better. Sometimes, after the recalled text chunks are assembled, their total length may exceed the input length limit of the model. This causes some text chunks to be truncated, which in turn affects RAG performance.

Therefore, we recommend that you select Intelligent Assembly. This strategy recalls as many relevant text chunks as possible without exceeding the maximum input length of the model.

Method 1: Constrain the output content

You can provide contextual information, instructions, and the expected output format in the prompt template to instruct the model. For example, you can add the following output instruction to require the model to:

If the information provided is not sufficient to answer the question, please state clearly, "Based on the existing information, I cannot answer this question." Do not invent an answer.

This reduces the chance of the model generating hallucinations.

Method 2: Add examples

Use the Few-Shot Prompting method to add question-and-answer examples to the prompt for the model to imitate. This guides the model to correctly use the retrieved knowledge. The following example uses Qwen-Plus.

# Requirement
Please extract the technical specifications from the text below and display them in JSON format.
${documents}

# Requirement
Please extract the technical specifications from the text below and display them in JSON format. Please strictly follow the fields given in the example.
${documents}

# Example
## Input: Stardust S9 Pro, a groundbreaking 6.9-inch 1440 x 3088 pixel under-screen camera design, brings a boundless visual experience. The top configuration of 512 GB storage and 16 GB RAM, combined with a 6000 mAh battery and 100 W fast charging technology, allows performance and battery life to go hand in hand, leading the technological trend. Reference price: 5999 - 6499.
## Output: { "product":"Stardust S9 Pro", "screen_size":"6.9inch", "ram_size": "16GB", "battery":"6000mAh" }

Method 3: Add content delimiters

If the retrieved text chunks are randomly mixed in the prompt template, it is difficult for the model to understand the structure of the entire prompt. Therefore, we recommend that you clearly separate the prompt and the ${documents} variable.

In addition, to ensure the best effect, make sure that the ${documents} variable appears only once in your prompt template. For more information, see the correct example on the left in the following table.

# Role
You are a customer service representative, focusing on analyzing and solving user problems, and providing accurate solutions by retrieving the knowledge base.

# Requirements
**Return the result directly**: Please provide a direct result based on the user's prompt and the content in the knowledge base, without
inference.
**Do not include specific contact information in the returned result**: The returned result should only include a summary of the user's prompt, the
names of relevant on-duty personnel, and their responsibilities.
**Default contact**: If no relevant on-duty personnel can be found, please return "On-duty representative today: Model Studio Customer Service 01".

# Knowledge Base
Please remember the following materials, they may be helpful in answering the question.
${documents}

# Role
You are a customer service representative, focusing on analyzing and solving user problems, and providing accurate solutions by retrieving the knowledge base.
Please use the information in ${documents} to assist in answering.

# Requirements
**Return the result directly**: Please provide a direct result based on the user's prompt and the content in the knowledge base, without
inference.
**Do not include specific contact information in the returned result**: The returned result should only include a summary of the user's prompt, the
names of relevant on-duty personnel, and their responsibilities.
**Default contact**: If no relevant on-duty personnel can be found, please return "On-duty representative today: Model Studio Customer Service 01".

# Knowledge Base
Please remember the following materials, they may be helpful in answering the question.
${documents}