Hologres:Use Hologres, DeepSeek, and PAI EAS to deploy a RAG-based chatbot

Limits of LLMs

• Field knowledge limits: In most cases, large language models (LLMs) are trained by using large-scale general datasets. In this case, LLMs struggle to provide in-depth and targeted processing for specialized vertical fields.

• Information update delay: The static nature of the training datasets prevents LLMs from accessing and incorporating real-time information and knowledge updates.

• Misleading outputs: LLMs are prone to hallucinations, producing outputs that appear plausible but are factually incorrect. This is attributed to factors such as data bias and inherent model limits.

RAG technology

With the rapid development of AI technology, generative AI has made remarkable achievements in various fields such as text generation and image generation. To address the limits of LLMs and enhance the capabilities and accuracy of LLMs, the RAG technology emerges.

Technology characteristics: By integrating external knowledge bases, RAG significantly reduces the issue of generating fictional content in LLMs and enhances its ability to access and apply the latest knowledge. This results in more personalized and precise customization of LLMs.

Core components of a RAG-based chatbot

Step 1: Prepare a Hologres vector database

1. Purchase a Hologres instance and create a database. For more information, see Purchase a Hologres instance and Create a database.

   Note

   • After you create an account, you must grant database-related permissions to the account. For more information, see Hologres permission models. You can check whether the permissions are granted by following the instructions in Connect to HoloWeb and perform queries.

   • We recommend that you use the simple permission model (SPM) to grant the permissions at or above the developer level to the account.

2. Obtain the Hologres instance endpoint, which is required in Step 2.

   1. Log on to the Hologres console.

   2. In the left-side navigation pane, click Instances.

   3. On the page that appears, find the desired instance and click the name of the instance. In the Network Information section of the Instance Details page, find Select VPC and click Copy in the Endpoint column to copy the Hologres instance endpoint.

Step 2: Deploy a RAG-based chatbot that uses DeepSeek

1. Log on to the PAI console.

2. In the left-side navigation pane, click Workspaces.

3. On the Workspaces page, click Create Workspace. On the Create Workspace page, configure parameters to create a workspace. For more information, see Create and manage a workspace.

4. In the left-side navigation pane, choose Model Deployment > Elastic Algorithm Service (EAS) to go to the Elastic Algorithm Service (EAS) page.

5. On the Inference Service tab, click Deploy Service. On the Deploy Service page, click RAG-based Smart Dialogue Deployment in the Scenario-based Model Deployment section.

6. On the RAG-based LLM Chatbot Deployment page, configure the parameters. The following table describes the parameters.

   Parameter		Description
   Basic Information	Version	The deployment version. Valid values: LLM-Integrated Deployment: Deploy a RAG service that is integrated with an LLM. LLM-Separated Deployment: Deploy only a RAG service. The RAG service allows for easy connection to and flexible replacement of an LLM. This deployment version provides a higher flexibility. In this example, LLM-Integrated Deployment is selected. You can select a deployment version based on your business requirements.
   	Model Type	In this example, a DeepSeek series model is used. You can select an open source model based on your business scenario.
   Resource Deployment	Resource configurations	The system recommends the appropriate resource specifications based on the selected model type. If you use other resource specifications, the model service may fail to start.
   Vector Database Settings	Vector Database Type	Select Hologres.
   	Invocation Information	The host information of the specified VPC. Go to the Instance Details page in the Hologres console. In the Network Information section, find Select VPC and click Copy in the Endpoint column to copy the instance endpoint. The host information refers to the instance endpoint with :80 excluded from the end.
   	Database Name	The name of the database in the Hologres instance. For more information about how to create a database, see Create a database.
   	Account	The custom account that you created. For more information, see User management.
   	Password	The password of the custom account that you created.
   	Table Name	The name of the table. You can enter a new table name or an existing table name. A test table named feature_tb is used in this example. We recommend that you enter a new table name. This way, PAI-RAG automatically creates the corresponding table in Hologres. If you enter a table name that does not exist, PAI-RAG automatically creates the corresponding vector index table. If you enter an existing table name, the table schema must meet the requirements of PAI-RAG. For example, you can enter the name of a Hologres table that is automatically created when you previously deploy a RAG-based chatbot by using EAS.
   VPC	VPC (VPC)	We recommend that you use a private network for access. To use a private network for access, you must make sure that the VPC configured in EAS is the same as the VPC where the Hologres instance is deployed. For more information about how to create a VPC, a vSwitch, and a security group, see Create and manage a VPC and Create a security group.
   	vSwitch
   	Security Group Name

7. After you configure the parameters, click Deploy. When the service status changes to Running, the RAG-based chatbot is deployed.

   

Step 3: Perform model inference on the web UI

After the RAG-based chatbot is deployed, click the Inference Service tab on the Elastic Algorithm Service (EAS) page. Find the service and click View Web App in the Service Type column to start the web UI.

Perform the following operations to debug the service on the web UI.

1. Configure parameters related to the vector database and LLMs.

   On the Settings tab, you can modify embedding-related parameters and LLM-related parameters. After you configure the parameters, click Update Index to save the configurations.

   • Embedding-related parameters

     Parameter	Description
     Index Name	The system can update existing indexes. You can select New from the Index Name drop-down list and specify an index name in the New Index Name field to isolate data of different knowledge bases. For more information, see How to Isolate Knowledge Base Data Using the RAG Service?
     Embedding Type	The huggingface and dashscope models are supported. In this example, the huggingface model is selected. huggingface: The system provides built-in embedding models for you to choose. dashscope: Models supported by Alibaba Cloud Model Studio are available. By default, the text-embedding-v2 model is used. For more information, see Embedding. Note If you select dashscope, you must configure a public network connection for EAS and configure an API key for Model Studio. You are separately billed for calling models supported by Model Studio. For more information, see Billable items.
     Embedding Dimension	The output vector dimension. The dimension setting directly affects the model performance. After you select an embedding model, the system automatically configures this parameter. No manual operation is required.
     Embedding Batch Size	The batch processing size.

   • LLM-related parameters

     • If you select LLM-Integrated Deployment for Version when you deploy the RAG-based chatbot, an LLM is integrated in the RAG service. In this case, you do not need to configure LLM-related parameters.

     • If you select LLM-Separated Deployment for Version, perform the following steps to obtain the endpoint and token. Then, configure the required parameters.

       1. On the Inference Service tab of the Elastic Algorithm Service (EAS) page, click the name of the service that you deployed.

       2. On the Overview tab, click View Endpoint Information in the Basic Information section.

       3. In the Invocation Method dialog box, obtain the endpoint and token based on the connection method.

2. Upload a business data file.

   On the Upload tab, you can configure the semantic-based chunking parameters and upload your business data file. The system automatically completes embedding and stores the embedded data in the Hologres vector database.

   1. Configure semantic-based chunking parameters.

      You can configure the following parameters to specify the chunk size and extract Q&A information.

      Parameter	Description
      Chunk Size	The size of each chunk. Unit: bytes. Default value: 500.
      Chunk Overlap	The portion of overlap between adjacent chunks. Default value: 10.
      Process with MultiModal	Specifies whether to use multimodal model processing. If you select the option, you can process images in PDF, Word, and Markdown files. A file in the TXT format is used in this example. Therefore, you do not need to select this option.
      Process PDF with OCR	Specifies whether to parse PDF files in OCR mode.

   2. Upload a business data file.

      In this example, the test file rag_hologres.txt is used.

      • The following file formats are supported: TXT, PDF, XLSX, XLS, CSV, DOCX, DOC, MD, and HTML.

      • You can upload local files, local directories, or objects from OSS.

      After you upload the business data file, the system performs data cleansing and semantic-based chunking, and then stores the data to the Hologres vector database. Data cleansing includes text extraction and hyperlink replacement.

      The following figure shows the data in the Hologres vector database. You can log on to the HoloWeb console to query the data. For more information, see Connect to HoloWeb and perform queries.

      

3. Configure model inference parameters.

   On the Chat tab, you can configure parameters for vector retrieval and model inference.

   • Policy parameters

     Parameter	Description
     Retrieval	Returns the top K relevant results from the vector database.
     LLM	Uses the answer from the LLM.
     Chat (Web Search)	Determines whether Internet search is needed based on the user question. If Internet search is required, the chatbot imports both the search results and the user question into the LLM. To support Internet search, you must configure a public network connection for EAS.
     Chat (Knowledge Base)	Merges the results retrieved from the vector database with the user question, populates them into the selected prompt template, and then submits the prompt to the LLM to generate an answer.

   • General parameters

     Parameter	Description
     Streaming Output	Specifies whether to generate results in streaming mode. Streaming Output is selected by default.
     Need Citation	Specifies whether to include citations in answers.
     Inference with multi-modal LLM	Specifies whether to display images when you use a multimodal LLM.

   • Vector retrieval parameters. The following vector retrieval methods are supported:

     • Embedding Only: Uses vector database-based retrieval.

     • Keyword Only: Uses keyword-based retrieval.

     • Hybrid: Uses the multi-channel recall fusion of vector database-based retrieval and keyword-based retrieval.

   • LLM-related parameters

     Temperature: Controls the randomness of the generated content. Lower temperature values result in more deterministic and fixed outputs, while higher temperatures lead to more diverse and creative results.

4. Perform model inference.

   On the Chat tab, select a knowledge base index by specifying the Index Name parameter and configure a Q&A policy. You can use multiple methods to perform inference verification on the LLM and optimize the chatbot.

   • Retrieval: Enter the keyword Serverless Computing as a question.

     

   • LLM: Enter Can I use Hologres Serverless Computing to accelerate queries when using a Fixed Plan? as a question.

     

   • Chat (Knowledge Base): Enter Can I use Hologres Serverless Computing to accelerate queries when using a Fixed Plan? as a question.

     

Step 4: Call an API operation to perform model inference

After you test the Q&A performance of the RAG-based chatbot on the web UI, you can perform the following steps to call an API operation provided by PAI to apply the RAG-based chatbot to your business system.

1. Obtain the invocation information of the RAG-based chatbot.

   1. Click the name of the RAG-based chatbot to go to the Overview tab.

   2. In the Basic Information section, click View Endpoint Information.

   3. In the Invocation Method dialog box, obtain the endpoint and token of the model service based on your network environment.

2. Call an API operation. For more information, see Call the API of a RAG-based LLM chatbot.