Hologres:Build a RAG-based chatbot using Hologres, DeepSeek, and PAI-EAS

Limitations of large language models (LLMs)

• Domain knowledge limitations: Large language models are typically trained on vast, general-purpose datasets and struggle to provide in-depth, targeted responses for specialized domains.

• Information latency: The static nature of their training data prevents LLMs from accessing the latest information in real time.

• Misleading outputs: Due to data biases and inherent model flaws, LLMs can produce responses that seem plausible but are factually incorrect, a phenomenon known as "hallucination".

Retrieval-Augmented Generation (RAG)

With the rapid development of AI technology, generative AI has achieved remarkable success in fields like text and image generation. Retrieval-Augmented Generation (RAG) was developed to overcome the inherent limitations of large language models (LLMs), enhancing their functionality and accuracy.

Key features: RAG integrates with external knowledge bases to significantly reduce the likelihood of an LLM generating fabricated content. This improves the model's ability to access and apply the latest information, enabling more personalized and accurate LLM customization.

Core RAG chatbot components

Step 1: Prepare a Hologres vector database

1. Purchase a Hologres instance and create a database. For more information, see Purchase a Hologres instance and Create a database.

   Note

   • After you create an account, you must grant it database permissions. For more information, see Hologres permission models. You can connect to HoloWeb and run queries to verify the permissions.

   • We recommend using the simple permission model to grant the account developer-level (or higher) permissions.

2. Obtain the instance endpoint. You will need this in Step 2.

   1. Log on to the Hologres console.

   2. In the left-side navigation pane, click Instances to go to the Instances page.

   3. Click the name of your target instance. On the Instance Details page, view the Network Information section. The Domain Name listed for the Dedicated VPC is your Hologres instance endpoint.

Step 2: Deploy a RAG service with DeepSeek

1. Log on to the PAI console.

2. In the left-side navigation pane, click Workspaces to go to the Workspaces page.

3. Click Create Workspace to create a workspace. For more information, see Create and manage a workspace.

4. In the left-side navigation pane, choose Model Deployment > Elastic Algorithm Service (EAS).

5. On the Inference Service tab, click Deploy Service. On the deployment page, in the Scenario-based Model Deployment section, select RAG-based Smart Dialogue Deployment.

6. On the RAG-based LLM Chatbot Deployment page, configure the parameters.

   Parameter		Description
   Basic information	Version selection	The following two deployment versions are supported: LLM-integrated deployment: Deploys the LLM and RAG services as a single service. LLM-separated deployment: Deploys only the RAG service. This provides more flexibility, as you can freely connect to and switch between different LLM services. This tutorial uses LLM-Integrated Deployment. You can choose the version that best fits your business needs.
   	Model Type	This tutorial uses a model from the DeepSeek series as an example. You can select an appropriate open-source model for your specific use case.
   Resource deployment	Resource configuration selection	The system automatically recommends suitable resource specifications based on the selected model type. Using a different resource specification might cause the model service to fail at startup.
   Vector database settings	Vector Database Type	Select Hologres.
   	Invocation Information	Enter the host for the specified VPC. To obtain it, go to the instance details page in the Hologres console. In the Network Information section, click Copy next to the VPC endpoint. Use the host information before the :80 part.
   	Database Name	The name of the database in your Hologres instance. For information about how to create a database, see Create a database.
   	Account	The custom user account that you created. For more information, see Manage users.
   	Password	The password for the custom user account.
   	Table Name	Enter a name for a new or existing table. This tutorial uses the test table name feature_tb. We recommend that you enter a new table name, which allows the RAG service to automatically create the table in Hologres for you. If the table does not exist, the RAG service automatically creates a corresponding vector index table. If the table already exists, its structure must meet the requirements of the RAG service. For example, you can use a Hologres table that was automatically created by a previous EAS RAG service deployment.
   VPC	VPC	We recommend accessing Hologres over a private network. This requires the VPC configured in EAS to be the same as the VPC of the Hologres instance. To create a VPC, vSwitch, and security group, see Create and manage a VPC and Create a security group.
   	vSwitch
   	Security Group Name

7. After you configure the parameters, click Deploy. When the service status changes to Running, the service is successfully deployed.

Step 3: Verify inference on the web UI

After the RAG service is deployed, on the Elastic Algorithm Service (EAS) page, go to the Inference Service tab. Find your service and click View Web App in the Service Type column to open the web UI.

Follow these steps to debug your service on the web UI.

1. Configure the vector retrieval database and large language model settings.

   On the Settings tab, you can modify Embedding-related parameters and select the LLM you want to use. After you complete the configuration, click Update Index to save your changes.

   • Index parameters:

     Parameter	Description
     Index name	The system supports updating an existing index. You can select New from the drop-down list to create a new index. By specifying different index names, you can isolate data from different knowledge bases. For more information, see How do I use the RAG service to isolate knowledge base data?.
     Embedding type	Supports huggingface and dashscope models. This tutorial uses a huggingface model. huggingface: The system provides built-in Embedding models for you to choose from. dashscope: Uses a Model Studio model. By default, the text-embedding-v2 model is used. For more information, see Text and multi-modal vectorization. Note If you select dashscope, you must configure a public network connection for EAS and provide a Model Studio API key. Calls to Model Studio models are billed separately. For more information, see Model Studio billable items.
     Embedding dimension	The output vector dimension. This setting directly affects model performance. After you select an Embedding model, the system automatically configures the embedding dimension.
     Embedding batch size	The batch processing size.

   • Large Language Model parameters:

     • If you chose the LLM-Integrated Deployment version for the RAG-based chatbot service, the LLM service and RAG service are deployed together, and you can use the default settings.

     • If you chose the LLM-Separated Deployment version, follow these steps to obtain and enter the service address and token.

       1. Go to the Inference Service tab on the EAS page and click the name of your service.

       2. On the Overview page, click Invocation Information in the Basic Information section.

       3. In the Invocation Information dialog box, obtain the service endpoint and Token based on your connection method.

2. Upload your business data files.

   On the Upload tab, set the semantic chunking parameters for your data and upload the file. The system automatically performs embedding and stores the data in the Hologres vector database.

   1. Set semantic chunking parameters.

      Configure the following parameters to control the document chunking granularity and extract Question and Answer information.

      Parameter	Description
      Chunk size	Specifies the size of each chunk in bytes. The default value is 500.
      Chunk overlap	The amount of overlap between adjacent chunks. The default value is 10.
      Process with multimodal	Uses a multi-modal model to process images in PDF, Word, and MD files. This tutorial uses a text file, so do not select this option.
      Process PDF with OCR	Uses OCR mode to parse PDF files.

   2. Upload the business data file.

      This tutorial uses the test file rag_hologres.txt.

      • Supported file types are TXT, PDF, XLSX, XLS, CSV, DOCX, DOC, MD, and HTML.

      • You can upload a local file, a local directory, or a file from OSS.

      After the file is successfully uploaded, the system cleans the data (text extraction, hyperlink replacement, and more) and performs semantic chunking. Then, it stores the data in the Hologres vector database.

      The following content shows the data in the Hologres table. You can log on to HoloWeb to query the data. For more information, see Connect to HoloWeb and run queries.

      In HoloWeb, run the select * from feature_tb; query. The data table contains four columns: id (document hash identifier), vector (text embedding vector, stored as a floating-point number array), metadata (metadata in JSON format, which contains fields such as file_path), and document (the original document text).

3. Configure model inference parameters.

   On the Chat tab, you can configure parameters for vector retrieval and model inference verification.

   • Policy parameters

     Parameter	Description
     Retrieval	Directly retrieves and returns the top K most similar results from the vector database.
     LLM	Directly uses the LLM to generate a response.
     Chat(Web Search)	Automatically determines whether a web search is needed based on the user's query. If a search is required, the search results and the user query are sent to the LLM service. To use web search, you must enable EAS to access public or internal networks.
     Chat(Knowledge Base)	Merges the results retrieved from the vector database with the user query, populates them into the selected prompt template, and sends the combined prompt to the LLM service to generate a final answer.

   • General parameters

     Parameter	Description
     Streaming output	By default, Streaming Output is enabled, and the system outputs results in a streaming fashion.
     Need citation	Specifies whether citations should be included in the response.
     Inference with multi-modal LLM	Specifies whether to display images when you use a multi-modal large language model.

   • Vector retrieval parameters. Three vector retrieval modes are supported:

     • Embedding Only: Retrieves results from the vector database.

     • Keyword Only: Retrieves results based on keywords.

     • Hybrid: Fuses results from multi-channel recall, including both vector database and keyword retrieval.

   • LLM parameters:

     Temperature: Controls the randomness of the generated content. A lower temperature value results in more deterministic outputs, while a higher value leads to more diverse and creative results.

4. Verify model inference.

   On the Chat tab, select the knowledge base index (Index Name) to use, configure the Q&A policy, and use one of the various methods to perform inference and fine-tune your chatbot.

   • Retrieval: Query using the keyword Serverless Computing.

     With Index Name set to default_index and Retrieval Mode set to Embedding Only, the system returns the following results: Doc 1 with a score of 0.82963, which contains the definition and use cases of Hologres Serverless Computing; Doc 2 with a score of 0.80813, which addresses whether Serverless Computing can accelerate a Fixed Plan; and Doc 3, which explains what Hologres is.

   • LLM: Ask the question Can a Fixed Plan use Hologres Serverless Computing for acceleration?.

     The system returns an inference result. After its thought process, the AI responds that a Fixed Plan does not support acceleration with Hologres Serverless Computing and recommends upgrading to the Pro Plan or Enterprise Plan.

   • Chat (Knowledge Base): Ask the question Can a Fixed Plan use Hologres Serverless Computing for acceleration?.

     The system returns an answer explaining that Hologres Serverless Computing cannot directly accelerate online service scenarios for a Fixed Plan and recommends using exclusive computing resources.

Step 4: Verify inference with API calls

After you debug the chatbot's performance on the web UI, you can integrate it into your own business applications by using the API provided by PAI. Follow these steps:

1. Obtain the invocation information for the RAG service.

   1. Click the name of your RAG service to go to the Overview page.

   2. In the Basic Information section, click View Invocation Information.

   3. In the Invocation Information dialog box, obtain the model service's endpoint and token based on your network environment.

2. Make an API call. For more information, see Make API calls for a RAG-based chatbot.