RAG performance optimization

2.2.1 Intelligent chunking

Choosing the optimal text chunk length for your knowledge base can be challenging because it depends on multiple factors, such as:

• Document type: For professional literature, longer chunks often help retain more context. For social media posts, shorter chunks can capture semantics more precisely.

• Prompt complexity: Generally, if a user's prompt is complex and specific, you may need longer chunks. Otherwise, shorter chunks might be more appropriate.

These conclusions do not necessarily apply to all situations. You need to choose the right tools and experiment repeatedly to find the right text chunk length. For example, LlamaIndex provides evaluation functions for different chunking methods. However, this process can be complex and time-consuming.

For a quick and effective solution, set Document Chunking to Intelligent Splitting when you create a knowledge base.

When this strategy is applied, the knowledge base:

1. First uses built-in sentence delimiters to divide the document into paragraphs.

2. Based on the divided paragraphs, adaptively selects chunk boundaries based on semantic relevance (semantic chunking), instead of using a fixed length.

This process ensures the semantic integrity of each document part and avoids unnecessary divisions. This strategy applies to all documents in this knowledge base, including subsequently imported documents.

2.2.2 Correct chunk content

Of course, during the actual chunking process, unexpected splits or other problems can still occur (for example, spaces in the text are sometimes parsed as %20 after chunking).

Therefore, Model Studio recommends manually checking the chunk content for semantic integrity and correctness after importing a document. If you find unexpected chunks or other parsing errors, you can edit the text chunks directly to correct them. After you save, the original content of the text chunk becomes invalid, and the new content is used for knowledge base retrieval.

Note that this action modifies only the text chunks in the knowledge base, not the original document or data table in your data management (temporary storage). Therefore, if you re-import the document, you must perform the manual check and correction again.

2.3.1 Multi-turn conversation rewriting

In a multi-turn conversation, a user might ask a question with a short prompt, such as "Model Studio Phone X1." This can cause the RAG system to lack the necessary context during retrieval for the following reasons:

• A phone product often has multiple generations for sale at the same time.

• For the same generation of a product, the manufacturer usually offers multiple storage options, such as 128 GB and 256 GB.

  ...

This key information may have been provided in previous conversational turns. Using it effectively helps RAG retrieve more accurate information.

To address this, you can use the Multi-round Conversation Rewriting feature in Model Studio. The system automatically rewrites the user's prompt into a more complete form based on the conversation history.

For example, the user asks:

Model Studio Phone X1.

With multi-turn conversation rewriting enabled, the system rewrites the user's prompt based on their conversation history before retrieval (example only):

Provide all available versions of Model Studio Phone X1 in the product library and their specific parameters.

This rewritten prompt helps RAG better understand the user's intent and provide a more accurate response.

The following figure shows how to enable the multi-turn conversation rewriting feature. This feature is also enabled when you select the Recommended Configuration.

Note that the multi-turn conversation rewriting feature is bound to the knowledge base. Once enabled, it applies only to queries related to the current knowledge base. This setting cannot be changed later; you must recreate the knowledge base to enable it.

2.3.2 Tag filtering

This section applies only to document search knowledge bases.

When you use a music app, you might filter songs by artist to quickly find all songs by that artist.

Similarly, adding tags to your unstructured documents introduces additional structured information. When retrieving from the knowledge base, the application can first filter documents based on tags, which improves retrieval accuracy and efficiency.

Model Studio supports the following two methods for setting tags:

• Set tags when uploading documents: For console steps, see Import data.

• Edit tags on the Data Management page: For uploaded documents, click Tag to the right of the document to edit its tags.

  

Model Studio supports the following two methods for using tags:

• When you call a Model Studio application by using an API, you can specify tags in the tags request parameter.

• Set tags when editing an application in the console. However, this method is applicable only to agent applications.

  Note that this setting applies to all subsequent user questions and answers for this agent application.

  

2.3.3 Metadata extraction

This section applies only to document search knowledge bases.

Embedding metadata into text chunks can effectively enhance the context of each chunk. In specific scenarios, this method can significantly improve the RAG performance of document search knowledge bases.

Consider the following scenario:

A knowledge base contains many phone product manuals. The document names are the phone models (such as Model Studio X1 and Model Studio Zephyr Z9), and all documents include a "Feature Overview" chapter.

If metadata is not enabled for this knowledge base, a user might enter the following prompt for retrieval:

Feature overview of Model Studio Phone X1.

A retrieval test reveals the recalled chunks. Because all documents contain "Feature Overview," the knowledge base recalls some text chunks that are unrelated to the query entity (Model Studio Phone X1) but are similar to the prompt, such as Chunk 1 and Chunk 2 in the figure. Their rankings are even higher than that of the required text chunk, which negatively impacts RAG performance.

The retrieval test results guarantee the ranking, but the absolute similarity score is for reference only. When the difference in absolute values is small (within 5%), the recall probability can be considered the same.

Next, set the phone name as metadata by following the steps in metadata extraction. This attaches the corresponding phone name information to the text chunks of each document. Then, run the same test for comparison.

At this point, the knowledge base adds a layer of structured search before the vector search. The complete process is as follows:

1. Extract metadata {"key": "name", "value": "Model Studio Phone X1"} from the prompt.

2. Based on the extracted metadata, find all text chunks that contain the "Model Studio Phone X1" metadata.

3. Then, perform a vector (semantic) search to find the most relevant text chunks.

After enabling metadata, the knowledge base can now accurately find the text chunk that is related to "Model Studio Phone X1" and contains "Feature Overview."

Another common application of metadata is to embed date information in text chunks to filter for recent content. See metadata extraction.

2.3.4 Similarity threshold

When the knowledge base finds text chunks related to the user's prompt, it first sends them to the Rank model (configured in Custom parameter settings when you create the knowledge base) for reordering. The similarity threshold is then used to filter the reordered text chunks. Only text chunks with a similarity score that exceeds this threshold can be provided to the model.

Lowering this threshold may recall more text chunks, but it may also cause some less relevant text chunks to be recalled. Increasing this threshold may reduce the number of recalled text chunks.

If the threshold is set too high, it may cause the knowledge base to discard all relevant text chunks. This limits the model's ability to get sufficient background information to generate an answer.

The optimal threshold depends on your scenario. You need to experiment with different similarity thresholds through retrieval tests, observe the recall results, and find the solution that best suits your needs.

2.3.5 Number of recalled chunks

The number of recalled chunks is the K value in the multi-channel recall strategy. After similarity threshold filtering, if the number of text chunks exceeds K, the system selects the K text chunks with the highest similarity scores to provide to the model. Because of this, an inappropriate K value may cause RAG to miss correct text chunks, which affects the model's ability to generate a complete answer.

For example, a user retrieves information with the following prompt:

What are the advantages of the Model Studio X1 phone?

There are 7 text chunks in the target knowledge base that are relevant to the user's prompt and should be returned (marked in green on the left). However, because this number exceeds the currently set maximum number of recalled chunks (K), the text chunks containing advantage 5 (ultra-long standby) and advantage 6 (clear photos) are discarded and not provided to the model.

Because RAG cannot determine how many text chunks are needed to provide a "complete" answer, the model will generate an answer based on the provided chunks, even if they are incomplete.

Many experiments show that in scenarios such as "List...," "Summarize...," and "Compare X and Y...," providing more high-quality text chunks (for example, K=20) to the model is more effective than providing only the top 10 or top 5. Although this may introduce noise, if the text chunk quality is high, a capable model can typically handle it.

You can adjust the Number of Recalled Chunks when you edit an application in Model Studio.

However, a larger number of recalled chunks is not always better. Sometimes, after the recalled text chunks are assembled, their total length may exceed the input length limit of the model, causing truncation and negatively affecting RAG performance.

Select Intelligent Assembly. This strategy recalls as many relevant text chunks as possible without exceeding the maximum input length of the model.

Method 1: Constrain the output content

You can provide contextual information, instructions, and the expected output format in the prompt template to instruct the model. For example, you can add the following output instruction:

If the information provided is not sufficient to answer the question, state clearly, "Based on the existing information, I cannot answer this question." Do not invent an answer.

This reduces the likelihood of model hallucinations.

Method 2: Add examples

Use the few-shot prompting method to add question-and-answer examples to the prompt for the model to imitate. This guides the model to correctly use the retrieved knowledge. The following example uses Qwen-Plus.

# Requirement
Please extract the technical specifications from the text below and display them in JSON format.
${documents}

# Requirement
Please extract the technical specifications from the text below and display them in JSON format. Please strictly follow the fields given in the example.
${documents}

# Example
## Input: Stardust S9 Pro, a groundbreaking 6.9-inch 1440 x 3088 pixel under-display camera design, brings a boundless visual experience. The top configuration of 512 GB storage and 16 GB RAM, combined with a 6000 mAh battery and 100 W fast charging technology, allows performance and battery life to go hand in hand, leading the technological trend. Reference price: 5999 - 6499.
## Output: { "product":"Stardust S9 Pro", "screen_size":"6.9inch", "ram_size": "16GB", "battery":"6000mAh" }

Method 3: Add content delimiters

If retrieved text chunks are randomly mixed in a prompt template, it is difficult for a large model to understand the overall prompt structure. Clearly separate the prompt from the ${documents} variable.

In addition, to ensure the best results, make sure that the variable ${documents} appears only once in your prompt template. For reference, see the correct example on the left below.

# Role
You are a customer service representative, focusing on analyzing and solving user problems and providing accurate solutions by retrieving from the knowledge base.

# Requirements
**Return the result directly**: Please provide a direct result based on the user's prompt and the content in the knowledge base, without
inference.
**Do not include specific contact information in the returned result**: The returned result should only include a summary of the user's prompt, the
names of relevant on-duty personnel, and their responsibilities.
**Default contact**: If no relevant on-duty personnel can be found, please return "On-duty representative today: Model Studio Customer Service 01".

# Knowledge base
Please remember the following materials, they may be helpful in answering the question.
${documents}

# Role
You are a customer service representative, focusing on analyzing and solving user problems and providing accurate solutions by retrieving from the knowledge base.
Please use the information in ${documents} to assist in answering.

# Requirements
**Return the result directly**: Please provide a direct result based on the user's prompt and the content in the knowledge base, without
inference.
**Do not include specific contact information in the returned result**: The returned result should only include a summary of the user's prompt, the
names of relevant on-duty personnel, and their responsibilities.
**Default contact**: If no relevant on-duty personnel can be found, please return "On-duty representative today: Model Studio Customer Service 01".

# Knowledge base
Please remember the following materials, they may be helpful in answering the question.
${documents}