Platform For AI:Deploying large language models on PAI-EAS

cURL

curl -X POST /v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: " \
    -d '{
        "model": "",
        "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "hello"
        }
        ],
        "max_tokens":1024,
        "temperature": 0.7,
        "top_p": 0.8,
        "stream":true
    }'

Where:

• Replace <EAS_ENDPOINT> and <EAS_TOKEN> with the endpoint and token of your deployed service.

• Replace <model_name> with the model name. For vLLM/SGLang, you can retrieve the model name from the model list API at /v1/models.

  curl -X GET /v1/models -H "Authorization: "

OpenAI SDK

Install the OpenAI SDK:  pip install openai. Use it to interact with the service.

from openai import OpenAI

# 1. Configure the client
# Replace  with the token of your deployed service
openai_api_key = ""
# Replace  with the endpoint of your deployed service
openai_api_base = "/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 2. Get the model name
# For BladeLLM, set model = "". BladeLLM does not require the model parameter and does not support getting the model name using client.models.list(). Set it to an empty string to meet the OpenAI SDK's mandatory parameter requirement.
models = client.models.list()
model = models.data[0].id
print(model)

# 3. Send a chat request
# Supports streaming (stream=True) and non-streaming (stream=False) output
stream = True
chat_completion = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hello"},
    ],
    model=model,
    top_p=0.8,
    temperature=0.7,
    max_tokens=1024,
    stream=stream,
)

if stream:
    for chunk in chat_completion:
        print(chunk.choices[0].delta.content, end="")
else:
    result = chat_completion.choices[0].message.content
    print(result)

Python requests library

For scenarios without OpenAI SDK dependency, use the requests library.

import json
import requests

# Replace  with the endpoint of your deployed service
EAS_ENDPOINT = ""
# Replace  with the token of your deployed service
EAS_TOKEN = ""
# Replace  with the model name. You can get it from the model list API at /v1/models. For BladeLLM, this API is not supported. You can omit the "model" field or set it to "".
model = ""

url = f"{EAS_ENDPOINT}/v1/chat/completions"
headers = {
    "Content-Type": "application/json",
    "Authorization": EAS_TOKEN,
}

stream = True
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "hello"},
]

req = {
    "messages": messages,
    "stream": stream,
    "temperature": 0.7,
    "top_p": 0.8,
    "max_tokens": 1024,
    "model": model,
}
response = requests.post(
    url,
    json=req,
    headers=headers,
    stream=stream,
)

if stream:
    for chunk in response.iter_lines(chunk_size=8192, decode_unicode=False):
        msg = chunk.decode("utf-8")
        # The following code processes streaming responses in Server-Sent Events (SSE) format
        if msg.startswith("data:"):
            info = msg[6:]
            if info == "[DONE]":
                break
            else:
                resp = json.loads(info)
                if resp["choices"][0]["delta"].get("content") is not None:
                    print(resp["choices"][0]["delta"]["content"], end="", flush=True)
else:
    resp = json.loads(response.text)
    print(resp["choices"][0]["message"]["content"])

Dify

1. Install the OpenAI-API-compatible model provider

   Click your profile picture in the upper-right corner and select Settings. In the navigation pane on the left, select Model Providers. If OpenAI-API-compatible is not in the Model List, find it in the list below and click Install.

   

2. Add a model

   Click Add Model in the lower-right corner of the OpenAI-API-compatible card and configure the parameters as follows:

   • Model Type: Select LLM.

   • Model Name: For vLLM deployments, send a GET request to the /v1/models API to retrieve the name. For example, enter Qwen3-8B.

   • API Key: Enter the EAS service token.

   • API endpoint URL: Enter the Internet endpoint of the EAS service. Note: Append /v1 to the end.

3. Usage

   1. On the Dify main page, click Create Blank App. Select the Chatflow type, enter the application name and other information, and then click Create.

   2. Click the LLM node, select the model you added, and set the context and prompt.

   3. Click Preview in the upper-right corner and enter a question.

      

Chatbox

1. Go to Chatbox. Download and install the version for your device, or click Launch Web App to use the web version. This example uses macOS on an M3 chip.

2. Add a model provider. Click Settings, add a model provider, enter a name, such as pai, and select OpenAI API Compatible for the API Mode.

   

3. Select the pai model provider and configure the following parameters.

   • API Key: Enter the EAS service token.

   • API Host: Enter the Internet endpoint of the EAS service. Note: Append /v1 to the end.

   • API Path: Leave this empty.

   • Model: Click Get to add a model. If the inference engine is BladeLLM, which does not support retrieval through the API, click New to enter the model name.

   

4. Test the conversation. Click New Chat. In the lower-right corner of the text input box, select the model service.

   

Cherry Studio

1. Install the client

   Visit Cherry Studio to download and install the client.

   You can also download it from https://github.com/CherryHQ/cherry-studio/releases.

2. Configure the model service.

   1. Click the Settings button in the lower-left corner. In the Model Service section, click Add. For Provider Name, enter a custom name, such as PAI. Set the Provider Type to OpenAI. Click OK.

   2. For API Key, enter the EAS service token. For API Address, enter the Internet endpoint of the EAS service.

   3. Click Add. For Model ID, enter the model name. For vLLM deployments, send a GET request to the /v1/models API to retrieve the name. For example, enter Qwen3-8B. Note that the name is case-sensitive.

   4. Next to the API Key input box, click Test to confirm connectivity.

3. Quickly test the model

   Return to the dialog box. At the top, select the model and start a conversation.