Platform For AI:Deploying large language models on PAI-EAS

cURL

curl -X POST /v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: " \
    -d '{
        "model": "",
        "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "hello"
        }
        ],
        "max_tokens":1024,
        "temperature": 0.7,
        "top_p": 0.8,
        "stream":true
    }'

Where:

• Replace <EAS_ENDPOINT> and <EAS_TOKEN> with the endpoint and token of your deployed service.

• Replace <model_name> with the model name. For vLLM/SGLang, you can retrieve the model name from the model list API at /v1/models.

  curl -X GET /v1/models -H "Authorization: "

OpenAI SDK

Install the OpenAI SDK: pip install openai. Use it to interact with the service.

from openai import OpenAI

# 1. Configure the client
# Replace  with the token of your deployed service
openai_api_key = ""
# Replace  with the endpoint of your deployed service
openai_api_base = "/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 2. Get the model name
# For BladeLLM, set model = "". BladeLLM does not require the model parameter and does not support client.models.list(). Set it to an empty string to meet the OpenAI SDK's mandatory parameter requirement.
models = client.models.list()
model = models.data[0].id
print(model)

# 3. Send a chat request
# Supports streaming (stream=True) and non-streaming (stream=False) output
stream = True
chat_completion = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hello"},
    ],
    model=model,
    top_p=0.8,
    temperature=0.7,
    max_tokens=1024,
    stream=stream,
)

if stream:
    for chunk in chat_completion:
        print(chunk.choices[0].delta.content, end="")
else:
    result = chat_completion.choices[0].message.content
    print(result)

Python requests library

For scenarios without OpenAI SDK dependency, use the requests library.

import json
import requests

# Replace with your deployed service endpoint
EAS_ENDPOINT = ""
# Replace with your deployed service token
EAS_TOKEN = ""
# Replace with model name from /v1/models API
# For BladeLLM: Omit "model" field or set to ""
model = ""

# Construct API endpoint URL
url = f"{EAS_ENDPOINT}/v1/chat/completions"
headers = {
    "Content-Type": "application/json",
    "Authorization": EAS_TOKEN,
}

# Enable streaming for real-time responses
stream = True
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "hello"},
]

# Build request payload
req = {
    "messages": messages,
    "stream": stream,
    "temperature": 0.7,  # Controls randomness (0.0-2.0)
    "top_p": 0.8,        # Controls diversity (0.0-1.0)
    "max_tokens": 1024,  # Maximum response length
    "model": model,
}

# Send POST request with timeout and error handling
try:
    response = requests.post(
        url,
        json=req,
        headers=headers,
        stream=stream,
        timeout=30  # Prevent hanging on slow model
    )
    response.raise_for_status()  # Raise error for 4xx/5xx

    # Process response based on streaming mode
    if stream:
        # Handle Server-Sent Events (SSE) format
        for chunk in response.iter_lines(chunk_size=8192, decode_unicode=False):
            msg = chunk.decode("utf-8")
            if msg.startswith("data:"):
                info = msg[6:]
                if info == "[DONE]":
                    break
                else:
                    resp = json.loads(info)
                    if resp["choices"][0]["delta"].get("content") is not None:
                        print(resp["choices"][0]["delta"]["content"], end="", flush=True)
    else:
        # Handle non-streaming response
        resp = json.loads(response.text)
        print(resp["choices"][0]["message"]["content"])

except requests.exceptions.Timeout:
    print("Error: Request timeout - model may be loading or overloaded")
except requests.exceptions.HTTPError as e:
    print(f"HTTP Error {response.status_code}: {response.text}")
except Exception as e:
    print(f"Unexpected error: {e}")

Dify

1. Install the OpenAI-API-compatible model provider

   Click your profile picture in the upper-right corner and select Settings. In the navigation pane on the left, select Model Providers. If OpenAI-API-compatible is not in the Model List, find it in the list below and click Install.

   

2. Add a model

   Click Add Model in the lower-right corner of OpenAI-API-compatible card and configure these parameters:

   • Model Type: Select LLM.

   • Model Name: For vLLM deployments, send a GET request to /v1/models API to retrieve name. For example, enter Qwen3-8B.

   • API Key: Enter EAS service token.

   • API endpoint URL: Enter Internet endpoint of EAS service. Note: Append /v1 to end.

3. Usage

   1. On the Dify main page, click Create Blank App. Select the Chatflow type, enter the application name and other information, and then click Create.

   2. Click the LLM node, select the model you added, and set the context and prompt.

   3. Click Preview in the upper-right corner and enter a question.

      

Chatbox

1. Go to Chatbox. Download and install the version for your device, or click Launch Web App to use the web version. This example uses macOS on an M3 chip.

2. Add a model provider. Click Settings, add a model provider, enter a name, such as pai, and select OpenAI API Compatible for the API Mode.

   

3. Select pai model provider and configure these parameters.

   • API Key: Enter EAS service token.

   • API Host: Enter Internet endpoint of EAS service. Note: Append /v1 to end.

   • API Path: Leave empty.

   • Model: Click Get to add a model. If inference engine is BladeLLM, which does not support retrieval through API, click New to enter model name.

   

4. Test the conversation. Click New Chat. In the lower-right corner of the text input box, select the model service.

   

Cherry Studio

1. Install the client

   Visit Cherry Studio to download and install the client.

   You can also download it from https://github.com/CherryHQ/cherry-studio/releases.

2. Configure the model service.

   1. Click Settings button in lower-left corner. In Model Service section, click Add. For Provider Name, enter a custom name, such as PAI. Set Provider Type to OpenAI. Click OK.

   2. For API Key, enter EAS service token. For API Address, enter Internet endpoint of EAS service.

   3. Click Add. For Model ID, enter model name. For vLLM deployments, send a GET request to /v1/models API to retrieve name. For example, enter Qwen3-8B. Note that name is case-sensitive.

   4. Next to the API Key input box, click Test to confirm connectivity.

3. Quickly test model

   Return to dialog box. At top, select model and start a conversation.