:Large language models

Core concepts

• Model: A logical encapsulation of a backend large language model service. It defines the model's source (API, deployment, or custom), access credential, and basic configuration.

• Model proxy: A high-performance LLM gateway deployed on Alibaba Cloud Function Compute (FC), built on the open source LiteLLM project. It provides a unified access endpoint for your models and executes advanced policies such as routing, disaster recovery, and concurrency control.

• Proxy mode: A single-target reverse proxy that binds the proxy instance to one backend model and forwards all requests directly to it. Suitable for simple, single-model scenarios.

• Load balancing mode: A multi-target load balancer that distributes requests across multiple backend models based on polling or weighted policies. Suitable for scenarios that combine multiple models to optimize cost and performance.

System architecture and data flow

1. Client/Application: Calls the API endpoint that is exposed by the model proxy over HTTP/HTTPS. The interface is compatible with the OpenAI Chat Completions API.

2. Model proxy: Receives requests and routes them to a backend model based on the configured proxy or load balancing mode and advanced policies, such as fallbacks and retries.

3. Model service:

   • API model: The request is forwarded to the API of an external service provider, such as OpenAI or Anthropic.

   • Custom model: The request is forwarded to a private model service that you specify, which must comply with the OpenAI API specification.

   • Deployed model: The request is forwarded to an open source model that you deployed on Function Compute Serverless GPU through this platform.

4. Observability (Simple Log Service SLS): The model proxy layer collects the call volume, latency, and error information for all requests. It supports trend analysis, Tracing Analysis, alerting, and more.

Connect to an API model

Use this method to call cloud-based LLM services such as OpenAI, Anthropic, and Google Gemini.

1. On the Large language model management page, click Add model, and then select API model.

2. Configure the model information:

   • Name: Set a unique and easily identifiable name for the model.

   • Description: Describe the use case to help with team collaboration.

   • Service provider: Select the model's service provider from the drop-down list. By default, five popular providers are displayed (Tongyi Qianwen, Zhipu AI, Moonshot, DeepSeek, and Custom Service). You can search for more providers (20+ providers, 260+ models). The AI Gateway provider is also available. This option affects the default path and format of the API call.

   • API endpoint: Enter the base URL of the model service. Do not include specific API paths. The platform automatically appends paths such as /chat/completions based on the service provider type. When you select Tongyi Qianwen, the system automatically populates the Dashscope URL: https://dashscope-intl.aliyuncs.com/compatible-mode/v1.

   • Configure specific models: Select the checkboxes or manually enter the specific models that you want to use. You must select at least one specific model.

   • Credential configuration: Select Use existing credential or API key to associate your API key. Make sure the credential matches the selected Service provider. When you select the AI Gateway provider, the credential type is automatically set to none.

   • Model parameter settings (Optional): Configure default parameters such as temperature, max_tokens, top_p, and stop. You can also enable: Supports Tool Calling, Support Thinking Mode, and Vision Support.

   Provider-specific notes:

   • When you select the AI Gateway provider, you must configure additional settings: Network Type (Internet/Private Network), Select Domain Name, Select model API, and Consumer (optional).

   • When you select overseas providers such as OpenAI, Anthropic, or Google Gemini, the console displays a legal compliance notice. Make sure your use case complies with all applicable laws and regulations.

3. Click Create model.

Connect to a custom model

Use this method to connect to any model service deployed in a VPC or on the internet, as long as it is compatible with the OpenAI API specification.

1. On the Large language model management page, click Add model. Set Model type to API model and Service provider to Custom service.

2. Configure the model information:

   • VPC network configuration: If your model is deployed in a VPC, configure the corresponding VPC, Subnet, and Security group here to ensure network connectivity.

   • API endpoint: Enter the full base URL of your model service, for example, https://api.example.com/v1.

   • Configure specific models: Enter the model name that you want to use in your API calls, for example, my-private-llama. You can optionally enable tool calling or vision capabilities.

   • Credential configuration: Select Use existing credential or API key to associate your API key. Make sure the credential matches the selected Service provider. Otherwise, the call will fail.

3. Click Create model.

Network and protocol requirements:

• The security group must allow access from the security group where the model proxy is located.

• The service must strictly follow the OpenAI /v1/chat/completions request and response format.

Deploy an open source model

Use this method to deploy an open source model to Alibaba Cloud Function Compute FunModel with one click, giving you a private and elastically scalable model API service.

1. On the Large language model management page, click Add model, and then select One-click deployment.

2. Select the model to deploy and the required compute resources:

   • Open source model: Select a model from the list of supported models.

   • Deployment configuration: Select a suitable GPU specification (GPU Compute-Optimized L20 Series, supporting single-card 48 GB GPU memory/64 GB memory or dual-card 48 GB×2/128 GB memory). The deployment is billed based on the usage duration of Function Compute (FC) Serverless GPU resources.

3. To access third-party services, add a Credential configuration.

4. Click Start deployment. The deployment process usually takes 5 to 15 minutes. You can view detailed deployment logs in the FunModel console to troubleshoot issues during the deployment process.

Common operations in the model list

The edit form varies depending on the model type:

• API model: You can modify the model name, model description, API endpoint, model selection, access credentials, and workspace. The service provider cannot be changed.

• Governance instance (model proxy): You can modify the instance name, instance description, and configured model selection. Additional settings are available under More Settings: logs, ARMS, network, and LiteLLM version.

• Deployed model: Displays basic model information (read-only). CPU, memory, GPU, and other resource configurations must be managed in the FunModel console.

The model product page also provides call templates for cURL, Python, and Node.js, so you can connect directly to the backend model from your local environment or application.

Create a model proxy

1. On the Large language model management page, click Model proxy.

2. Mode selection:

   • Proxy mode: Proxies a single model. Suitable for simple scenarios.

   • Load balancing mode: Proxies multiple models and distributes traffic based on weights.

3. Configure basic information:

   • Name: For example, my-first-proxy or proxy-SOYTcS.

   • Description: Briefly describe the purpose.

   • Proxy Region: Example: China (Hangzhou)

4. Select models:

   • Proxy mode: In the Select model section, choose a large language model that you have added, and then select a specific model.

   • Load balancing mode: In Model management, add multiple model cards. For each card, select:

     • A configured large language model

     • A specific model name

     • A weight (for example, 50 / 50)

5. Credential configuration (inbound access credential):

   This determines the identity authentication method for clients to access the model proxy. For more information about creating a credential, see Credential management.

6. Select Recommended specifications:

   • Starter Edition (1-core, 2 GB): For personal use or low traffic.

   • Standard Edition (2-core, 4 GB): For small and medium-sized enterprises.

   • Professional Edition (4-core, 8 GB): For high-concurrency scenarios.

7. Click Start deployment. The first deployment usually takes 1 to 2 minutes.

After the deployment is complete, you can see the following on the model proxy's Details page:

• Running mode (Proxy / Load balancing)

• API endpoint

• List of bound models and their weights

• Access credential name

• Advanced configuration

Model proxy advanced configuration

On the model proxy product page, click Edit administration to configure the following policies. Changes take effect dynamically:

How to call a model proxy

1. Configure parameters

   The Use cases tab of the model proxy product page provides generated call examples. When you make a call, configure the parameters as follows:

   • Request method: POST

   • Request URL:
     https://{model-proxy-domain-name}/models/{model-proxy-name}/v1/chat/completions
     
     
     
     
     
     
     
     

   • Request headers (Headers):

     • x-acs-parent-id: The current Alibaba Cloud account ID. This is automatically provided in the console example.

     • X-API-Key: The name of the model proxy access credential, for example, agentRun-api.

     • Content-Type: application/json

   • Request body (Body) (compatible with OpenAI Chat Completions):

     • model: The name of the specific model configured under this model proxy, for example, qwen3-max or deepseek-r1.

     • messages: An array of conversation messages.

2. Call example

   curl -X POST \
     https://1836055070146700.agentrun-data.ap-southeast-1.aliyuncs.com/models/bailian-model/v1/chat/completions \
     -H "x-acs-parent-id: 1836055070146700" \
     -H "X-API-Key: agentRun-api" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "deepseek-r1",
       "messages": [
         { "role": "user", "content": "Write a poem within 20 characters" }
       ]
     }'

   All requests through this proxy endpoint are recorded in SLS, and the corresponding metrics appear on the Observability page in the console.

   If you call the backend model API directly and bypass the model proxy, the platform cannot observe or manage those requests.

Observability features

On the Observability tab of the model proxy product page, you can view:

• Call volume: Today's / Total number of calls

• Response performance: Average latency, maximum latency, and response time distribution

• Error statistics: Number of errors and error rate

• Call trend: View call fluctuations over 1 hour, 1 day, 7 days, or 30 days.

• Tracing Analysis: View the complete trace of a single request to distinguish between gateway latency and model latency.

• Log analysis: Query the raw log in SLS to identify the cause of errors. Check fields such as error_message and response_body.

Troubleshooting process

When a request fails or performance is abnormal, troubleshoot in the following order:

1. Check the access credential
   Confirm that the API key in the request header matches the inbound credential bound to the model proxy and has not expired.
   
   
   
   
   
   
   
   

2. Check the model configuration and backend credential
   In Large language model management, view the corresponding model and confirm the following:
   
   
   
   
   
   
   
   

   • The API endpoint is correct and reachable.

   • The associated service provider's API key is valid and has sufficient quota.

3. Troubleshoot using observability logs
   On the model proxy product page, go to Observability > Log analysis. Filter for failed requests and focus on the following:
   
   
   
   
   
   
   
   

   • status_code

   • error_message

   • response_body

4. Analyze the trace
   Determine whether the performance issue is in the gateway (proxy) or the backend model to perform targeted optimization.
   
   
   
   
   
   
   
   

Configuration specifications

• API model: The service provider, API endpoint, and credential must match. This prevents errors such as 401 or 403 that are caused by a mismatched endpoint or key.

• Deployed model: Select a suitable GPU specification based on the number of model parameters and GPU memory requirements. This avoids resource waste or out-of-memory (OOM) errors.

• Custom model: Strictly follow the OpenAI API specification. In addition, ensure that the VPC network and security group configurations allow the model proxy to access the target service.

Cost description and optimization