Platform For AI:RAG-based LLM chatbot

Background information

Prerequisites

• A virtual private cloud (VPC), vSwitch, and security group are created. For more information, see Create and manage a VPC and Create a security group.

  Note

  If you use Facebook AI Similarity Search (Faiss) to build a vector database, the preceding prerequisites are not required.

• An Object Storage Service (OSS) bucket or Apsara File Storage NAS (NAS) file system is created to store fine-tuned model files. This prerequisite must be met if you use a fine-tuned model to deploy the chatbot. For more information, see Get started by using the OSS console or Create a file system.

  Note

  If you use Faiss to build a vector database, you must prepare an OSS bucket.

Limits

The vector database and EAS must be deployed in the same region.

Precautions

This practice is subject to the maximum number of tokens of an LLM service and is designed to help you understand the basic retrieval feature of a RAG-based LLM chatbot.

• The chatbot is limited by the server resource size of the LLM service and the default number of tokens. The conversation length supported by the chatbot is also limited.

• If you do not need to perform multiple rounds of conversations, we recommended that you disable the with chat history feature of the chatbot. This effectively reduces the possibility of reaching the limit. For more information, see the How do I disable the with chat history feature of the RAG-based chatbot? section of this topic.

Step 1: Prepare a vector database

You can use one of the following services to build a vector database: Faiss, Elasticsearch, Hologres, OpenSearch, and ApsaraDB RDS for PostgreSQL. When you build a vector database, save the required parameter configurations that are used to connect to the vector database in subsequent operations.

Step 2: Deploy the RAG-based chatbot

1. Go to the EAS page.

   1. Log on to the Platform for AI (PAI) console.

   2. In the left-side navigation pane, click Workspaces. On the Workspaces page, find the workspace to which you want to deploy the model and click its name to go to the Workspace Details page.

   3. In the left-side navigation pane, choose Model Deployment > Elastic Algorithm Service (EAS) to go to the Elastic Algorithm Service (EAS) page.

2. On the Elastic Algorithm Service (EAS) page, click Deploy Service. In the Scenario-based Model Deployment section, click RAG-based Smart Dialogue Deployment.

3. On the RAG-based LLM Chatbot Deployment page, configure the parameters. The following tables describe the key parameters in different sections.

   • Basic Information

     Parameter	Description
     Service Name	The name of the service.
     Model Source	The source of the model. Valid values: Open Source Model and Custom Fine-tuned Model.
     Model Type	The model type. Select a model type based on your business requirements. If you set the Model Source parameter to Custom Fine-tuned Model, you must configure the parameter quantity and precision for the model type.
     Model Settings	If you set the Model Source parameter to Custom Fine-tuned Model, you must configure the path in which the fine-tuned model file is stored. Valid values: Note Make sure that the model file format is compatible with Hugging Face transformers. Mount OSS: the OSS path in which the fine-tuned model file is stored. Mount NAS: the NAS file system in which the fine-tuned model file is stored and the source path of the NAS file system.

   • Resource Configuration

     Parameter	Description
     Resource Configuration	If you set the Model Source parameter to Open Source Model, the system automatically selects an instance type based on the selected model type as the default value. If you set the Model Source parameter to Custom Fine-tuned Model, select an instance type that matches the model. For more information, see the How do I switch to another open source LLM? section of the "Quickly deploy LLMs in EAS" topic.
     Inference Acceleration	Inference acceleration can be enabled for the Qwen, Llama2, ChatGLM, or Baichuan2 model that is deployed on A10 or GU30 instances. The following inference acceleration methods are provided: BladeLLM Inference Acceleration: The BladeLLM inference acceleration engine ensures high concurrency and low latency. You can use BladeLLM to accelerate LLM inference in a cost-effective manner. Open-source vLLM Inference Acceleration

   • Vector Database Settings

     Select a service to build a vector database based on your business requirements.

     FAISS

     Parameter	Description
     Vector Database Type	The service that you want to use to build the vector database. Select FAISS.
     OSS Path	The OSS path of the vector database. Select an OSS path in the current region. You can create an OSS path if no OSS path is available. For more information, see Get started by using the OSS console.

     Elasticsearch

     Parameter	Description
     Vector Database Type	The service that you want to use to build the vector database. Select Elasticsearch.
     Private Endpoint and Port	The private endpoint and port number that you obtained in Step 1. Format: http://Private endpoint:Port number.
     Index Name	The name of the index. You can enter a new index name or an existing index name. If you use an existing index name, the index schema must meet the requirements of the RAG-based chatbot. For example, you can enter the name of the index that is automatically created when you deploy the RAG-based chatbot by using EAS.
     Account	The username that you configured when you created the Elasticsearch cluster in Step 1.
     Password	The password that you configured when you created the Elasticsearch cluster in Step 1.

     Hologres

     Parameter	Description
     Vector Database Type	The service that you want to use to build the vector database. Select Hologres.
     Invocation Information	The Hologres invocation information that you obtained in Step 1.
     Database Name	The name of the database that you created in Step 1.
     Account	The custom account that you created in Step 1.
     Password	The password of the custom account that you created in Step 1.
     Table name	The name of the table. You can enter a new table name or an existing table name. If you use an existing table name, the table schema must meet the requirements of the RAG-based chatbot. For example, you can enter the name of the Hologres table that is automatically created when you deploy the RAG-based chatbot by using EAS.

     OpenSearch

     Parameter	Description
     Endpoint	The public endpoint you obtained in Step 1.
     Instance ID	The ID of the OpenSearch Vector Search Edition instance you created in Step 1.
     Username	The username and password you specified when creating the OpenSearch Vector Search Edition instance in Step 1.
     Password
     Table Name	The name of the index table you configured for the OpenSearch Vector Search Edition instance in Step 1.

     RDS for PostgreSQL

     Parameter	Description
     Host Address	The internal endpoint or public endpoint you obtained in Step 1.
     Port	The actual port number. Default value: 5432.
     Database	The name of the database you created in Step 1.
     Table Name	Specify a table name.
     Account	The priviledged account you created in Step 1.
     Password	The password of the priviledged account you created in Step 1.

   • VPC Configuration

     Parameter	Description
     VPC	If you use Hologres, Elasticsearch, OpenSearch, or RDS PostgreSQL to build a vector database, select the VPC in which the vector database is deployed. Note If you use OpenSearch to build a vector database, you can select a different VPC. However, you need to make sure that the VPC can be accessed over the Internet and the associated EIP is added to the public access whitelist of the OpenSearch instance. For more information, see Use the SNAT feature of an Internet NAT gateway to access the Internet and Configure the public access whitelist. If you use Faiss to build a vector database, you do not need to configure the VPC.
     vSwitch
     Security Group Name

4. Click Deploy.

   If the value in the Service Status column changes to Running, the RAG-based chatbot is deployed.

Step 3: Perform model inference on the web UI

This step describes how to debug the RAG-based chatbot on the web UI. After you test the Q&A performance of the RAG-based chatbot on the web UI, you can call API operations provided by PAI to apply the RAG-based chatbot to your business system. For more information, see the Step 4: Call API operations to perform model inference section of this topic.

1. Configure the RAG-based chatbot

1. After the RAG-based chatbot is deployed, click View Web App in the Service Type column to enter the web UI.

2. Configure the machine learning model.

   • Embedding Model Name: Four models are available. By default, the optimal model is selected.

   • Embedding Dimension: After you configure the Embedding Model Name parameter, the system automatically configures this parameter.

3. Check whether the vector database is connected.

   The system automatically recognizes and applies the vector database settings that are configured when you deploy the chatbot. The settings cannot be modified. If you use Hologres to build the vector database, click Connect Hologres to check whether the vector database in Hologres is connected.

2. Upload business data files

3. Configure model inference parameters

4. Perform model inference

Step 4: Call API operations to perform model inference

1. Obtain the invocation information of the RAG-based chatbot.

   1. Click the name of the RAG-based chatbot to go to the Service Details page.

   2. In the Basic Information section, click View Endpoint Information.

   3. On the Public Endpoint tab, obtain the service endpoint and token.

2. Connect to the vector database on the web UI and upload business data files. For more information, see the 1. Configure the RAG-based chatbot and 2. Upload business data files sections of this topic.

3. Call API operations to perform model inference.

   PAI allows you to call the RAG-based chatbot by using the following API operations in different query modes: service/query/retrieval in retrieval mode, service/query/llm in LLM mode, and service/query in RAG mode. Sample code:

   cURL command

   • Initiate a single-round conversational search request

     • Method 1: Call the service/query/retrieval operation.

       curl -X 'POST'  '<service_url>service/query/retrieval' -H 'Authorization: <service_token>' -H 'accept: application/json' -H 'Content-Type: application/json'  -d '{"question": "What is PAI?"}'
       # Replace <service_url> and <service_token> with the service endpoint and service token that you obtained in Step 1.

     • Method 2: Call the /service/query/llm operation.

       curl -X 'POST'  '<service_url>service/query/llm' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What is PAI?"}'
       # Replace <service_url> and <service_token> with the service endpoint and service token that you obtained in Step 1.

       You can add other adjustable inference parameters such as {"question":"What is PAI?", "temperature": 0.9}.

     • Method 3: Call the service/query operation.

       curl -X 'POST'  '<service_url>service/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What is PAI?"}'
       # Replace <service_url> and <service_token> with the service endpoint and service token that you obtained in Step 1.

       You can add other adjustable inference parameters such as {"question":"What is PAI?", "temperature": 0.9}.

   • Initiate a multi-round conversational search request

     You can initiate a multi-round conversational search request only in RAG and LLM query modes. The following sample code shows an example on how to initiate a multi-round conversational search request in RAG query mode:

     # Send the request.  
     curl -X 'POST'  '<service_url>service/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What is PAI?"}'
     
     # Provide the session ID returned for the request. This ID uniquely identifies a conversation in the conversation history. After the session ID is provided, the corresponding conversation is stored and is automatically included in subsequent requests to call an LLM. 
     curl -X 'POST'  '<service_url>service/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What are the benefits of PAI?","session_id": "ed7a80e2e20442eab****"}'
     
     # Provide the chat_history parameter, which contains the conversation history between you and the chatbot. The parameter value is a list in which each element indicates a single round of conversation in the {"user":"Inputs","bot":"Outputs"} format. Multiple conversations are sorted in chronological order. 
     curl -X 'POST'  '<service_url>service/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question":"What are the features of PAI?", "chat_history": [{"user":"What is PAI", "bot":"PAI is an AI platform provided by Alibaba Cloud..."}]}'
     
     # If you provide both the session_id and chat_history parameters, the conversation history is appended to the conversation that corresponds to the specified session ID.  
     curl -X 'POST'  '<service_url>service/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question":"What are the features of PAI?", "chat_history": [{"user":"What is PAI", "bot":"PAI is an AI platform provided by Alibaba Cloud..."}], "session_id": "1702ffxxad3xxx6fxxx97daf7c"}'

   Python script

   • The following sample code shows an example on how to initiate a single-round conversational search request:

     import requests
     
     EAS_URL = 'http://xxxx.****.cn-beijing.pai-eas.aliyuncs.com'
     headers = {
         'accept': 'application/json',
         'Content-Type': 'application/json',
         'Authorization': 'MDA5NmJkNzkyMGM1Zj****YzM4M2YwMDUzZTdiZmI5YzljYjZmNA==',
     }
     
     
     def test_post_api_query_llm():
         url = EAS_URL + '/service/query/llm'
         data = {
            "question":"What is PAI?"
         }
         response = requests.post(url, headers=headers, json=data)
     
         if response.status_code != 200:
             raise ValueError(f'Error post to {url}, code: {response.status_code}')
         ans = dict(response.json())
         print(f"======= Question =======\n {data['question']}")
         print(f"======= Answer =======\n {ans['answer']} \n\n")
     
     
     def test_post_api_query_retrieval():
         url = EAS_URL + '/service/query/retrieval'
         data = {
            "question":"What is PAI?"
         }
         response = requests.post(url, headers=headers, json=data)
     
         if response.status_code != 200:
             raise ValueError(f'Error post to {url}, code: {response.status_code}')
         ans = dict(response.json())
         print(f"======= Question =======\n {data['question']}")
         print(f"======= Answer =======\n {ans['docs']}\n\n")
     
     
     def test_post_api_query_rag():
         url = EAS_URL + '/service/query'
         data = {
            "question":"What is PAI?"
         }
         response = requests.post(url, headers=headers, json=data)
     
         if response.status_code != 200:
             raise ValueError(f'Error post to {url}, code: {response.status_code}')
         ans = dict(response.json())
         print(f"======= Question =======\n {data['question']}")
         print(f"======= Answer =======\n {ans['answer']}")
         print(f"======= Retrieved Docs =======\n {ans['docs']}\n\n")
     # LLM
     test_post_api_query_llm()
     # Retrieval
     test_post_api_query_retrieval()
     # RAG (Retrieval + LLM)
     test_post_api_query_rag()
     

     Set the EAS_URL parameter to the endpoint of the RAG-based chatbot. Make sure to remove the forward slash (/) at the end of the endpoint. Set the Authorization parameter to the token of the RAG-based chatbot.

   • Initiate a multi-round conversational search request

     You can initiate a multi-round conversational search request only in RAG (Retrieval + LLM) and LLM query modes. Sample code:

     import requests
     
     EAS_URL = 'http://xxxx.****.cn-beijing.pai-eas.aliyuncs.com'
     headers = {
         'accept': 'application/json',
         'Content-Type': 'application/json',
         'Authorization': 'MDA5NmJkN****jNlMDgzYzM4M2YwMDUzZTdiZmI5YzljYjZmNA==',
     }
     
     
     def test_post_api_query_llm_with_chat_history():
         url = EAS_URL + '/service/query/llm'
         # Round 1 query
         data = {
            "question":"What is PAI?"
         }
         response = requests.post(url, headers=headers, json=data)
     
         if response.status_code != 200:
             raise ValueError(f'Error post to {url}, code: {response.status_code}')
         ans = dict(response.json())
         print(f"=======Round 1: Question =======\n {data['question']}")
         print(f"=======Round 1: Answer =======\n {ans['answer']} session_id: {ans['session_id']} \n")
        
         # Round 2 query
         data_2 = {
            "question": "What are the benefits of PAI?",
            "session_id": ans['session_id']
         }
         response_2 = requests.post(url, headers=headers, json=data_2)
     
         if response.status_code != 200:
             raise ValueError(f'Error post to {url}, code: {response.status_code}')
         ans_2 = dict(response_2.json())
         print(f"=======Round 2: Question =======\n {data_2['question']}")
         print(f"=======Round 2: Answer =======\n {ans_2['answer']} session_id: {ans_2['session_id']} \n\n")
     
     
     def test_post_api_query_rag_with_chat_history():
         url = EAS_URL + '/service/query'
        
         # Round 1 query
         data = {
            "question":"What is PAI?"
         }
         response = requests.post(url, headers=headers, json=data)
     
         if response.status_code != 200:
             raise ValueError(f'Error post to {url}, code: {response.status_code}')
         ans = dict(response.json())
     
         print(f"=======Round 1: Question =======\n {data['question']}")
         print(f"=======Round 1: Answer =======\n {ans['answer']} session_id: {ans['session_id']}")
         print(f"=======Round 1: Retrieved Docs =======\n {ans['docs']}\n")
     
         # Round 2 query
         data = {
            "question":"What are the features of PAI?",
            "session_id": ans['session_id']
         }
         response = requests.post(url, headers=headers, json=data)
     
         if response.status_code != 200:
             raise ValueError(f'Error post to {url}, code: {response.status_code}')
         ans = dict(response.json())
     
         print(f"=======Round 2: Question =======\n {data['question']}")
         print(f"=======Round 2: Answer =======\n {ans['answer']} session_id: {ans['session_id']}")
         print(f"=======Round 2: Retrieved Docs =======\n {ans['docs']}")
     # LLM
     test_post_api_query_llm_with_chat_history()
     # RAG (Retrieval + LLM)
     test_post_api_query_rag_with_chat_history()
     

     Set the EAS_URL parameter to the endpoint of the RAG-based chatbot. Make sure to remove the forward slash (/) at the end of the endpoint. Set the Authorization parameter to the token of the RAG-based chatbot.

References

You can also use EAS to deploy the following items:

• You can deploy an LLM application that can be called by using the web UI or API operations. After the LLM application is deployed, use the LangChain framework to integrate enterprise knowledge bases into the LLM application to implement intelligent Q&A and automation features. For more information, see Quickly deploy LLMs in EAS.

• You can deploy an AI video generation model service by using ComfyUI and Stable Video Diffusion models. This helps you complete tasks such as short video generation and animation on social media platforms. For more information, see Use ComfyUI to deploy an AI video generation model service.

FAQ

How do I disable the with chat history feature of the RAG-based chatbot?

On the web UI page of the RAG-based chatbot, do not select Chat history.