:RAG-based LLM chatbot

Step 1: Deploy the RAG service

1. Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).

2. On the Model Online Service (EAS) page, click Deploy Service. In the Scenario-based Model Deployment area, click Deploy Rag-based LLM Chatbot.

   

3. On the Deploy Rag-based LLM Chatbot page, configure the parameters and then click Deploy. When the Service Status changes to Running, the service is successfully deployed. Key parameter descriptions are as follows.

   • Basic Information

     Parameter	Description
     Version	The following two versions are supported for deployment: Integrated LLM Deployment: Deploy the LLM service and RAG service within the same service. Separate LLM Deployment: Only deploy the RAG service. However, within the RAG service, you can freely switch and connect LLM services for greater flexibility.
     Model Type	When you select Integrated LLM Deployment, you need to choose the LLM to deploy. You can select an open-source model based on your specific use case.

   • Resource Deployment

     Parameter	Description
     Deployment Resources	After selecting the model category, the system automatically matches suitable resource specifications. Switching to other resource specifications may cause the model service to fail to start.

   • Vector Database Settings

     RAG supports building vector databases using Faiss (Facebook AI Similarity Search), Elasticsearch, Hologres, OpenSearch, or RDS PostgreSQL. Select a version type based on your scenario to serve as the vector database.

     FAISS

     Faiss enables the creation of a local vector database efficiently, eliminating the necessity to acquire or activate online vector databases.

     Parameter	Description
     Version Type	Select FAISS.
     OSS Address	Select the OSS storage path created in the current region to store the uploaded knowledge base files. If no storage path is available, you can refer to Quick Start for Console to create one. Note If you choose to use a custom fine-tuned model deployment service, ensure that the selected OSS storage path does not overlap with the path of the custom fine-tuned model to avoid conflicts.

     Elasticsearch

     Specify the connection information of an Alibaba Cloud Elasticsearch instance. For information on how to create an Elasticsearch instance and prepare configuration items, see Prepare the vector database Elasticsearch.

     Parameter	Description
     Version Type	Select Elasticsearch.
     Private Network Address/port	Configure the private network address and port of the Elasticsearch instance in the format http://<private network address>:<private network port>. For information on how to obtain the private network address and port number of an Elasticsearch instance, see View the basic information of an instance.
     Index Name	Enter a new index name or an existing index name. For an existing index name, the index schema must meet the requirements of the RAG-based chatbot. For example, you can enter the name of the index that is automatically created when you deploy the RAG-based chatbot by using EAS.
     Account	Specify the logon name that you configured when you created the Elasticsearch instance. The default logon name is elastic.
     Password	Specify the password that you configured when you created the Elasticsearch instance. If you forget the logon password, you can reset the instance access password.

     Hologres

     Specify the connection information of a Hologres instance. If you have not activated a Hologres instance, you can refer to Purchase Hologres for more information.

     Parameter	Description
     Version Type	Select Hologres.
     Invocation Information	Specify the host information of the designated VPC. Go to the instance details page of the Hologres Management Console. In the Network Information area, click Copy after Designated VPC to obtain the host information before the domain name :80.
     Database Name	Specify the database name of the Hologres instance. For more information about how to create a database, see Create a Database.
     Account	Specify the custom account that you created. For more information, see Create a Custom User, where you select Select Member Role and choose Superuser.
     Password	Specify the password of the custom account that you created.
     Table Name	Enter a new table name or an existing table name. For an existing table name, the table schema must meet the requirements of the RAG-based chatbot. For example, you can enter the name of the Hologres table that is automatically created when you deploy the RAG-based chatbot by using EAS.

     OpenSearch

     Specify the connection information of an OpenSearch instance of Vector Search Edition. For information on how to create an OpenSearch instance and prepare configuration items, see Prepare the vector database OpenSearch.

     Parameter	Description
     Version Type	Select OpenSearch.
     Access Address	Specify the public endpoint of the OpenSearch instance of Vector Search Edition. You need to enable the public access feature for the OpenSearch instance of Vector Search Edition. For more information, see Prepare the vector database OpenSearch.
     Instance ID	Obtain the instance ID from the OpenSearch Vector Search Edition instance list.
     Username	Enter the username and password that you specified when you created the OpenSearch instance of Vector Search Edition.
     Password
     Table Name	Enter the name of the index table of the OpenSearch instance of Vector Search Edition that you created. For more information about how to prepare the index table, see Prepare the vector database OpenSearch.

     RDS PostgreSQL

     Specify the connection information of the ApsaraDB RDS for PostgreSQL instance. For information on how to create an ApsaraDB RDS for PostgreSQL instance and prepare configuration items, see Prepare the vector database RDS PostgreSQL.

     Parameter	Description
     Version Type	Select RDS PostgreSQL.
     Host Address	Specify the internal network address of the ApsaraDB RDS for PostgreSQL instance. You can go to the ApsaraDB RDS for PostgreSQL console page and view the Database Connection page of the ApsaraDB RDS for PostgreSQL instance.
     Port	The default value is 5432. Enter the value based on your actual situation.
     Database	Specify the name of the database that you created. For more information about how to create a database and an account, see Create an Account and a Database, where: When you create an account, select Account Type and choose Privileged Account. When you create a database, select Authorized Account and choose the privileged account that you created.
     Table Name	Specify the name of the database table.
     Account	Specify the privileged account and password that you created. For more information about how to create a privileged account, see Create an Account and a Database, where you select Account Type and choose Privileged Account.
     Password

   • Virtual Private Cloud Configuration

     Parameter	Description
     VPC	If you selected Separate LLM Deployment when deploying the RAG service, you must make sure it is connected to the LLM service: Public access: Associate a virtual private cloud (VPC) with the RAG service and configure Internet access. Private access: Associate the same VPC with both the RAG service and the LLM service. If you need to use a model from Model Studio or perform Q&A with online search, you must configure a VPC and enable public access. For more information, see Configure Public Network Connection. Network requirements for the vector database: The Faiss vector database does not require network access. Hologres, Elasticsearch, or RDS PostgreSQL can be accessed by EAS through the public network or private network. Private network access is recommended. Private network access requires that the VPC configured in EAS is consistent with the VPC of the vector database. For more information about how to create a VPC, a switch, and a security group, see Create and Manage a VPC and Create a Security Group. EAS can only access OpenSearch through the public network. For more information about how to configure the access method, see Step 2: Prepare Configuration Items.
     Switch
     Security Group Name

Step 2: Debug on the WebUI page

After the RAG service is successfully deployed, click View Web Application in the Service Method column to launch the WebUI page.

Follow the steps below to upload your knowledge base file on the WebUI page and test the Q&A chatbot.

Step 3: API call

The following content describes how to call the RAG API.

Obtain invocation information

1. Click the RAG service name to go to the Service Details page.

2. In the Basic Information area, click View Invocation Information.

3. On the Invocation Information dialog box, click the Public Address Call tab to obtain the service access address and token.

Upload knowledge base files

You can upload local knowledge base files through the API. You can query the status of the file upload task based on the task_id returned by the upload interface.

In the following example, replace <service_url> with the access address of the RAG service and <service_token> with the token of the RAG service. For more information about how to obtain the access address and token, see Obtain Invocation Information.

• Upload data

  curl -X 'POST' '<service_url>api/v1/upload_data' -H 'Authorization: <service_token>' -H 'Content-Type: multipart/form-data' -F 'files=@<file_path>'
  # Return: {"task_id": "****557733764fdb9fefa063538914da"}

• Query the upload task status

  curl '<service_url>api/v1/get_upload_state?task_id=****557733764fdb9fefa063538914da' -H 'Authorization: <service_token>'
  # Return: {"task_id":"****557733764fdb9fefa063538914da","status":"completed"}

Single-round conversation request

import requests

SERVICE_URL = 'http://xxxx.****.cn-beijing.pai-eas.aliyuncs.com/'
headers = {
    'accept': 'application/json',
    'Content-Type': 'application/json',
    'Authorization': 'MDA5NmJkNzkyMGM1Zj****YzM4M2YwMDUzZTdiZmI5YzljYjZmNA==',
}

def test_post_api_query(url):
    data = {
       "question":"What is PAI?" 
    }
    response = requests.post(url, headers=headers, json=data)

    if response.status_code != 200:
        raise ValueError(f'Error post to {url}, code: {response.status_code}')
    ans = dict(response.json())

    print(f"======= Question =======\n {data['question']}")
    if 'answer' in ans.keys():
        print(f"======= Answer =======\n {ans['answer']}")
    if 'docs' in ans.keys():
        print(f"======= Retrieved Docs =======\n {ans['docs']}\n\n")
 
# LLM 
test_post_api_query(SERVICE_URL + 'api/v1/query/llm')
# Retrieval
test_post_api_query(SERVICE_URL + 'api/v1/query/retrieval')
# Chat (Knowledge Base)
test_post_api_query(SERVICE_URL + 'api/v1/query')

Multi-round conversation request

LLM and Chat (Knowledge Base) support sending multi-round conversation requests. The following code example shows how to do this:

# Send a request. 
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What is PAI?"}'

# Provide the session ID returned for the request. This ID uniquely identifies a conversation in the conversation history. After the session ID is provided, the corresponding conversation is stored and is automatically included in subsequent requests to call an LLM.
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What are the benefits of PAI?","session_id": "ed7a80e2e20442eab****"}'

# Provide the chat_history parameter, which contains the conversation history between you and the chatbot. The parameter value is a list in which each element indicates a single round of conversation in the {"user":"Inputs","bot":"Outputs"} format. Multiple conversations are sorted in chronological order.
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question":"What are its features?", "chat_history": [{"user":"What is PAI?", "bot":"PAI is Alibaba Cloud's platform for AI..."}]}'

# If you provide both the session_id and chat_history parameters, the conversation history is appended to the conversation that corresponds to the specified session ID. 
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question":"What are its features?", "chat_history": [{"user":"What is PAI?", "bot":"PAI is Alibaba Cloud's platform for AI..."}], "session_id": "1702ffxxad3xxx6fxxx97daf7c"}'

import requests

SERVICE_URL = 'http://xxxx.****.cn-beijing.pai-eas.aliyuncs.com'
headers = {
    'accept': 'application/json',
    'Content-Type': 'application/json',
    'Authorization': 'MDA5NmJkN****jNlMDgzYzM4M2YwMDUzZTdiZmI5YzljYjZmNA==',
}

def test_post_api_query_with_chat_history(url):
    # Round 1 query
    data = {
       "question": "What is PAI?"
    }
    response = requests.post(url, headers=headers, json=data)

    if response.status_code != 200:
        raise ValueError(f'Error post to {url}, code: {response.status_code}')
    ans = dict(response.json())
    print(f"=======Round 1: Question =======\n {data['question']}")
    if 'answer' in ans.keys():
        print(f"=======Round 1: Answer =======\n {ans['answer']} session_id: {ans['session_id']}")
    if 'docs' in ans.keys():
        print(f"=======Round 1: Retrieved Docs =======\n {ans['docs']}")
   
    # Round 2 query
    data_2 = {
       "question": "What are the benefits of PAI?",
       "session_id": ans['session_id']
    }
    response_2 = requests.post(url, headers=headers, json=data_2)

    if response.status_code != 200:
        raise ValueError(f'Error post to {url}, code: {response.status_code}')
    ans_2 = dict(response_2.json())
    print(f"=======Round 2: Question =======\n {data_2['question']}")
    if 'answer' in ans.keys():
        print(f"=======Round 2: Answer =======\n {ans_2['answer']} session_id: {ans_2['session_id']}")
    if 'docs' in ans.keys():
        print(f"=======Round 2: Retrieved Docs =======\n {ans['docs']}")
    print("\n")

# LLM
test_post_api_query_with_chat_history(SERVICE_URL + "api/v1/query/llm")
# Chat (Knowledge Base)
test_post_api_query_with_chat_history(SERVICE_URL + "api/v1/query")

Precautions

This practice is subject to the maximum number of tokens of an LLM service and is designed to help you understand the basic retrieval feature of a RAG-based LLM chatbot:

• The chatbot is limited by the server resource size of the LLM service and the default number of tokens. The conversation length supported by the chatbot is also limited.

• If multi-round conversation is not required, it is recommended to disable the with chat history feature to effectively reduce the possibility of reaching the limit.

  WebUI operation method: On the Chat tab of the RAG service WebUI page, deselect the Chat history check box.

  

References

You can also use EAS to deploy the following items:

• Deploy an LLM application that can be called by using the web UI or API operations. After the LLM application is deployed, use the LangChain framework to integrate enterprise knowledge bases into the LLM application and implement intelligent Q&A and automation features. For more information, see Deploy an LLM Application in 5 Minutes by Using EAS.

• Deploy an AI video generation model service by using ComfyUI and Stable Video Diffusion models. This helps you complete tasks such as short video generation and animation on social media platforms. For more information, see AI Video Generation - ComfyUI Deployment.

• Deploy a model service based on Stable Diffusion WebUI by configuring a few parameters. For more information, see AI Painting - SDWebUI Deployment.

• When you deploy a RAG-based LLM chatbot, only the fees related to EAS resources are charged. For more information about billing, see Model Online Service (EAS) Billing Description. If you use other products such as Bailian, vector databases, or Object Storage Service (OSS) during the use of the RAG-based LLM chatbot, they are billed separately in the corresponding products.