Platform For AI:Deploying large language models on PAI-EAS

cURL

curl -X POST /v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: " \
    -d '{
        "model": "",
        "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "hello"
        }
        ],
        "max_tokens":1024,
        "temperature": 0.7,
        "top_p": 0.8,
        "stream":true
    }'

Where:

• Replace <EAS_ENDPOINT> and <EAS_TOKEN> with the endpoint and token of your deployed service.

• Replace <model_name> with the model name. For vLLM/SGLang, you can retrieve the model name from the model list API at /v1/models.

  curl -X GET /v1/models -H "Authorization: "

OpenAI SDK

Install the OpenAI SDK: pip install openai. Use it to interact with the service.

from openai import OpenAI

# 1. Configure the client
# Replace  with the token of your deployed service
openai_api_key = ""
# Replace  with the endpoint of your deployed service
openai_api_base = "/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# 2. Get the model name
# For BladeLLM, set model = "". It doesn't require model or support client.models.list().
models = client.models.list()
model = models.data[0].id
print(model)

# 3. Send a chat request
# Supports streaming (stream=True) and non-streaming (stream=False) output
stream = True
chat_completion = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hello"},
    ],
    model=model,
    top_p=0.8,
    temperature=0.7,
    max_tokens=1024,
    stream=stream,
)

if stream:
    for chunk in chat_completion:
        print(chunk.choices[0].delta.content, end="")
else:
    result = chat_completion.choices[0].message.content
    print(result)

Python requests library

For scenarios without OpenAI SDK dependency, use the requests library.

import json
import requests

# Replace with your deployed service endpoint
EAS_ENDPOINT = ""
# Replace with your deployed service token
EAS_TOKEN = ""
# Replace with model name from /v1/models API
# For BladeLLM: Omit "model" field or set to ""
model = ""

# Construct API endpoint URL
url = f"{EAS_ENDPOINT}/v1/chat/completions"
headers = {
    "Content-Type": "application/json",
    "Authorization": EAS_TOKEN,
}

# Enable streaming for real-time responses
stream = True
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "hello"},
]

# Build request payload
req = {
    "messages": messages,
    "stream": stream,
    "temperature": 0.7,  # Controls randomness (0.0-2.0)
    "top_p": 0.8,        # Controls diversity (0.0-1.0)
    "max_tokens": 1024,  # Maximum response length
    "model": model,
}

# Send POST request with timeout and error handling
try:
    response = requests.post(
        url,
        json=req,
        headers=headers,
        stream=stream,
        timeout=30  # Prevent hanging on slow model
    )
    response.raise_for_status()  # Raise error for 4xx/5xx

    # Process response based on streaming mode
    if stream:
        # Handle Server-Sent Events (SSE) format
        for chunk in response.iter_lines(chunk_size=8192, decode_unicode=False):
            msg = chunk.decode("utf-8")
            if msg.startswith("data:"):
                info = msg[6:]
                if info == "[DONE]":
                    break
                else:
                    resp = json.loads(info)
                    if resp["choices"][0]["delta"].get("content") is not None:
                        print(resp["choices"][0]["delta"]["content"], end="", flush=True)
    else:
        # Handle non-streaming response
        resp = json.loads(response.text)
        print(resp["choices"][0]["message"]["content"])

except requests.exceptions.Timeout:
    print("Error: Request timeout - model may be loading or overloaded")
except requests.exceptions.HTTPError as e:
    print(f"HTTP Error {response.status_code}: {response.text}")
except Exception as e:
    print(f"Unexpected error: {e}")

Dify

1. Install the OpenAI-API-compatible model provider

   Go to your profile picture > Settings > Model Providers. If OpenAI-API-compatible is not in the Model List, find it below and click Install.

   

2. Add a model

   Click Add Model on the OpenAI-API-compatible card and configure:

   • Model Type: Select LLM.

   • Model Name: Retrieve from the /v1/models API (vLLM). Example: Qwen3-8B.

   • API Key: Enter EAS service token.

   • API endpoint URL: EAS Internet endpoint with /v1 appended.

3. Usage

   1. On the Dify main page, click Create Blank App. Select Chatflow, enter the app name, and click Create.

   2. Click the LLM node, select the model you added, and set the context and prompt.

   3. Click Preview in the upper-right corner and enter a question.

      

Chatbox

1. Go to Chatbox and install the client, or click Launch Web App for the web version.

2. Click Settings and add a model provider. Enter a name (such as pai) and select OpenAI API Compatible as the API Mode.

   

3. Select the pai provider and configure:

   • API Key: Enter EAS service token.

   • API Host: Enter Internet endpoint of EAS service. Note: Append /v1 to end.

   • API Path: Leave empty.

   • Model: Click Get to retrieve the model. For BladeLLM (does not support API retrieval), click New and enter the model name manually.

   

4. Click New Chat and select the model in the lower-right corner of the input box.

   

Cherry Studio

1. Install the client

   Visit Cherry Studio to download and install the client.

   You can also download it from https://github.com/CherryHQ/cherry-studio/releases.

2. Configure the model service.

   1. Click Settings in the lower-left corner. In Model Service, click Add. Set Provider Name (such as PAI) and Provider Type to OpenAI. Click OK.

   2. Enter the EAS service token as API Key and the Internet endpoint as API Address.

   3. Click Add and enter the model name as Model ID. Retrieve the name from the /v1/models API (for example, Qwen3-8B). Case-sensitive.

   4. Click Test next to API Key to verify connectivity.

3. Quickly test model

   Return to the dialog box, select the model at the top, and start a conversation.