A service group manages multiple services that process traffic through a unified endpoint. The system distributes incoming traffic to these services based on a defined allocation policy. This method is ideal for canary releases, elastic scaling, and heterogeneous resource scheduling. This topic describes how to create a service group, view its traffic endpoint, and modify the traffic allocation policy.
Use cases
-
Canary release
Add a production service and a canary service to the same group. Then, allocate a small amount of traffic to the canary service. When you release a new version, update the canary service first and monitor its performance. If issues occur, roll back the canary service or stop it and shift its traffic to the production service. If the new version runs as expected, you can deploy it to the production service. After the update, you can scale the canary service down to zero replicas or keep it to handle a small amount of traffic.
-
Subscription-based and pay-as-you-go elastic scaling
To handle baseline demand, deploy subscription-based services in a dedicated resource group with a fixed replica count. To manage traffic spikes and reduce costs, use pay-as-you-go services in a public resource group.
-
Use heterogeneous hardware resources
In GPU-accelerated scenarios, if a specific GPU model becomes unavailable in a region, it can prevent a service from scaling out. A service group allows you to overcome this by dynamically creating services with different GPU models that are compatible with various CUDA environments. This allows multiple services to use heterogeneous resources to support the same business scenario. Because the service group's traffic endpoint remains unchanged, these changes are transparent to the front end.
Create a service group
You can assign a service to a service group when you create the service.
When you create a service and assign it to a group, the system creates the group if it does not exist. If the group already exists, the service is simply added to it. The group is automatically deleted when its last member service is removed.
The following example shows how to create a service group named pmml and assign two services, pmml_prod and pmml_grey, to it.
Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
-
On the Group Service tab, click Create Group and Service.
-
On the Custom Deployment page, configure the parameters and click Deploy.
Key parameters:
-
Service Name: Configure the service name based on the on-screen prompts. In this example, set it to pmml_prod.
-
Group: Select New Group and set the value to pmml for this example.
For information about other parameters, see Custom deployment.
-
-
Repeat steps 2 and 3 to create the pmml_grey service and assign it to the pmml service group.
After the services are created, go to the Group Service tab and click the group name pmml to go to the group details page. This page shows the list of services that belong to the group.
By default, newly added services do not receive traffic from the group. To enable traffic, see Modify the traffic allocation policy.
View traffic endpoints
A service group has a unified traffic endpoint. Each service in the group also has its own individual endpoint.
-
Group Traffic Entry: Traffic to this endpoint is distributed to different services based on the allocation policy. Services in a service group can be created or deleted, but the endpoint address remains unchanged.
-
Service-specific Traffic Entry: This endpoint is tied to the lifecycle of a specific service. Traffic always flows to the specified service. When the service is deleted, the endpoint is also destroyed. After you switch the group traffic, you still need to use this endpoint address to access the service and perform online debugging.
How to obtain:
-
On the Inference Service tab, click the name of the target service to go to the Overview page.
-
In the Basic Information section, click View Endpoint Information.
-
In the dialog box that appears, view the call address and token information.
Endpoint
Address format
Example
Group Traffic Entry
<endpoint>/api/predict/<group_name>http://182848887922****.vpc.cn-shanghai.pai-eas.aliyuncs.com/api/predict/pmmlService-specific Traffic Entry
<endpoint>/api/predict/<group_name>.<service_name>http://182848887922****.vpc.cn-shanghai.pai-eas.aliyuncs.com/api/predict/pmml
Modify the traffic allocation policy
On the group details page, you can switch the traffic allocation method next to the group name at the top. EAS currently supports two traffic allocation methods:
-
Replica Allocation: Traffic is dynamically allocated based on the replica count of each service. For example, if Service A has 1 replica and Service B has 3 replicas, Service A receives 25% of the traffic and Service B receives 75%. After you disable traffic allocation, the service no longer participates in group traffic distribution but remains accessible individually.
-
Custom Allocation: Traffic is distributed based on the weight of each service. For example, if Service A has a traffic weight of 100 and Service B has a traffic weight of 400, Service A receives 20% of the traffic and Service B receives 80%.
Note:
-
Turn on the Traffic Allocation switch for a service to enable traffic distribution. Turn off the switch to disable traffic distribution for the service.
ImportantRegardless of the traffic allocation method, after you turn off the Traffic Allocation switch, the service no longer participates in group traffic distribution but remains accessible individually.
-
To use Weight for Custom Allocation, click the edit button in the Traffic Weight column to set the Traffic Weight for the service.
-
You can also use API operations to adjust the traffic weight and traffic state of services. For more information, see ReleaseService.