Platform For AI:Parameters for custom deployment in the console

Parameter

Description

Elastic Resource Pool

This parameter is available only if you set the Resource Type parameter to EAS Resource Group.

You can turn on Elastic Resource Pool and configure your resources based on the instructions in the Deployment Resources section.

If you enable Elastic Resource Pool and the dedicated resource group that you use to deploy services is fully occupied, the system automatically adds pay-as-you-go instances to the public resource group during scale-outs. The added instances are billed as public resources. The instances in the public resource group are released first during scale-ins. For more information, see Elastic resource pool.

Additional System Disk

This parameter is available if you set the Resource Type parameter to Public Resources or EAS Resource Group and configure an elastic resource pool.

Configure additional system disks for the EAS service. Unit: GB. Valid values: 0 to 2000. You have a free quota of 30 GB on the system disk. If you specify 20 in the field, the available storage space is 50 GB.

Additional system disks are billed based on their capacity and usage duration. For more information, see Billing of EAS.

Distributed Inference

The instances of a service are deployed on multiple machines. This can resolve the issue that a model with ultra-large-scale parameters cannot be deployed on a single machine. For more information, see Multi-machine distributed inference.

Rolling Update

• Number of Instances Exceeding Expectation: the maximum number of additional instances that can be created for the service during a rolling update. You can set this parameter to a positive integer, which specifies the number of additional instances. You can also set this parameter to a percentage, such as 2%, which specifies the ratio of the number of additional instances to the original number of service instances. The default value is 2%. The higher the value, the faster the service is updated. For example, if you set the number of service instances to 100 and set this parameter to 20, 20 additional instances are immediately created when you update the service.

• Maximum Number of Unavailable Instances: the maximum number of service instances that become unavailable during a rolling update. During a rolling update, the system can release existing instances to free up resources for new instances. This prevents update failures caused by insufficient resources. If a dedicated resource group is used, this parameter is set to 1 by default. If the public resource group is used, this parameter is set to 0 by default. For example, if you set this parameter to N, N instances are immediately stopped when a service update starts.

  Note

  If idle resources are sufficient, you can set this parameter to 0. If you set this parameter to a large value, service stability may be affected. This is because a larger value results in a reduced number of available instances during a service update and heavier workloads for each instance. When specifying this parameter, you must consider service stability and the resources you require.

Shared Memory

Configure shared memory for the instance to perform read and write operations on the memory without data copy or transfer. Unit: GB.

High-priority Resource Rescheduling

After enabling high-priority resource rescheduling, EAS periodically creates probe instances on high-priority resources while the service runs. If the probe instance is scheduled, more probe instances are created exponentially until the scheduling fails. At the same time, after the scheduled probe instance completes initialization and enters the ready state, it will replace the instance on the low-priority resource. This feature can resolve the following issues:

• During the rolling update of services, instances that are being terminated still occupy resources. As a result, newly created instances are started on the public resource group. Due to the limit of public resources, subsequent new instances are rescheduled back to the dedicated resource group.

• When a preemptible instance and a common instance are used at the same time, the system periodically checks whether the preemptible instance is available. If available, the common instance is migrated to the preemptible instance.

GPU Driver

Specify the GPU driver version. Example: 550.127.08.

Parameter

Description

LLM Intelligent Router

Turn on LLM Intelligent Router and select an LLM Intelligent Router service that you deployed. If no LLM intelligent router is available, you can click Create LLM Intelligent Router to create an intelligent router. For more information, see Use LLM Intelligent Router to improve inference efficiency.

LLM Intelligent Router is a special EAS service that can be bound with an LLM inference service. When the LLM inference service has multiple instances, the LLM Intelligent Router can dynamically distributes requests based on backend load. This ensures that the computing power and memory resources of each inference instance are evenly allocated, improving the resource efficiency of the cluster.

Service Response Timeout Period

The timeout period of the server for each request. Unit: seconds. Default value: 5.

Graceful Shutdown

• Graceful Shutdown Time: the maximum amount of time allowed for a graceful shutdown. Unit: seconds. Default value: 30. EAS services use the rolling update policy. Before an instance is released, it enters the Terminating state and continues to process the requests that it received during the period of time that you specify, while the system switches the traffic to other instances. The instance is released after the instance finishes processing the requests. Therefore, the duration of the graceful shutdown process must be within the value of this parameter. If the time required to process requests is long, you can increase the value of this parameter to ensure that all requests that are in progress can be processed when the system updates the service.

  Important

  If you set this parameter to a small value, service stability may be affected. If you set this parameter to a large value, the service update may be prolonged. We recommend that you use the default value unless you have special requirements.

• Send SIGTERM: Valid values:

  • false: When a service instance enters the EXIT state, the system does not send the SIGTERM signal. This is the default value.

  • true: When a service instance enters the EXIT state, the system immediately sends the SIGTERM signal to the main process. After the signal is received, the main process in the service performs a custom graceful shutdown in the signal processing function. If the signal is not processed, the main process may exit immediately after receiving the signal, causing the graceful shutdown to fail.

Save Call Records

You can enable this feature to persistently save all service requests and responses to MaxCompute tables or Simple Log Service. Turn on Save Call Records and select a save method:

• MaxCompute

  • MaxCompute Project: Select an existing project from the drop-down list. If no project is available, you can click Create MaxCompute Project to create a project. For more information, see Use the MaxCompute console to create a project.

  • MaxCompute Table: Specify a name for the table. When you deploy the service, the system automatically creates a table in the MaxCompute project.

• Simple Log Service

  • Simple Log Service Project: specifies a project that is used to isolate and manage resources in Simple Log Service. Select an existing project. If no project is available, click Create Simple Log Service Project to create a project. For more information, see Manage a project.

  • Logstore: used to collect, store, and query log data in Simple Log Service. During logstore configuration and service deployment, the system automatically creates a logstore in a Simple Log Service project.

Task Mode

You can enable this feature to deploy an inference service as an elastic job service. For more information, see Overview.

Configure Secure Encryption Environment

You can configure the system trust management service to ensure that data, models, and code information can be securely encrypted during service deployment and invocation, implementing a secure and verifiable inference service.

The secure encryption environment is designed for mounted storage files. You need to mount storage files before you enable this feature.

Tracing Analysis

Alibaba Cloud images with the built-in tracing analysis component support the tracing analysis feature. For images without the built-in component, you must add the following commands to enable the tracing analysis feature.

• Add aliyun-bootstrap -a install && aliyun-instrument python app.py to the Command parameter. This command is used to install the ARMS agent for Python and start the application with the agent. app.py is the main file that provides prediction services for the image.

• Add aliyun-bootstrap to the Third-party Library Settings parameter. This command is used to download the agent installer from the PyPI repository.