Container Service for Kubernetes:Resource profiling

Limits

Use resource profiling in the ACK console

Install the resource profiling component

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, click the name of a cluster and choose Cost Management Suite > Cost Optimization in the left-side navigation pane.
3. On the Cost Optimization page, click the Resource Profiling tab.
4. On the Resource Profiling tab, click Install or Upgrade. After the installation or update is complete, the Resource Profiling tab appears.
   Note

   • If this is the first time you install the resource profiling component, you must first install ack-koordinator or update ack-koordinator to the latest version. It requires about 2 minutes to install or update ack-koordinator.
   • If the version of ack-koordinator is earlier than 0.7.0, refer to Migrate ack-koordinator from the Marketplace page to the Add-ons page to migrate and update ack-koordinator.

Manage resource profiling policies

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, click the name of a cluster and choose Cost Management Suite > Cost Optimization in the left-side navigation pane.
3. On the Cost Optimization page, click the Resource Profiling tab and click Policy.
4. In the Policy dialog box, set the parameters and click OK.
   

   Parameter	Description	Valid value
   Policy Name	The name of the policy.	The name must be 1 to 63 characters in length, and can contain letters, digits, hyphens (-), underscores (_), and periods (.). The name must start and end with a letter or a digit.
   Namespaces	The namespaces for which you want to enable resource profiling. Resource profiling is enabled for the workload types that you specify in the selected namespaces.	The namespaces in the cluster. You can specify multiple namespaces.
   Workload Type	The workload types for which you want to enable resource profiling. Resource profiling is enabled for the workload types that you specify in the selected namespaces.	The following workload types are supported: Deployment, StatefulSet, and DaemonSet. You can specify multiple workload types.
   Resource Redundancy Rate	The redundancy rate that is specified in the resource profiling policy. For more information, see the following section.	The value must be positive or 0. You can select 30%, 50%, or 70% or specify a custom redundancy rate.

In most cases, the resource utilization of an application remains below 100% due to reasons such as the limit of physical resources (such as hyper-threading) and the need for a resource buffer to handle traffic spikes during peak hours. Therefore, the resource redundancy rate is required when the administrator configures resources for an application based on metrics such as the queries per second (QPS). If the difference between the recommended resource request and the original resource request exceeds the specified redundancy rate, the system recommends that you decrease the resource request. For more information about the resource profiling algorithm, see Overview of application profiles section.

Overview of application profiles

After you configure the resource profiling policy, you can view the details of resource profiles on the Resource Profiles page. The following table describes the columns that are displayed on the page.

View the application profile

On the Resource Profiling tab, click the name of a workload to go to the profile details page. On the details page, you can view basic information about the workload, view the resource curve of each container of the workload, and modify the resource specifications of the workload.

Modify the resource specifications of the workload

In the Change Resource Configuration section in the lower part of the page, you can change the resource specifications of each container based on the recommended resource specifications provided by ACK. The following table describes the parameters.

Parameter	Description
Resource Request	The original resource request of the container.
Resource Limit	The original resource limit of the container.
Recommended Value	The resource request that is recommended by ACK.
Resource Redundancy Rate	The resource redundancy rate that is specified in the resource profiling policy. You can specify the new resource request based on the redundancy rate and the recommended resource request. In the preceding figure, the new CPU request is calculated based on the following formula: 4.28 cores × (1 + 30%) = 5.6 cores.
New Resource Request	The new resource request that you want to use.
New Resource Limit	The new resource limit that you want to use. If topology-aware CPU scheduling is enabled for the workload, the CPU limit must be an integer.

After you set the parameters, click Submit. The system starts to update the resource specifications of the workload. You are redirected to the details page of the workload. After the resource specifications are updated, the controller performs a rolling update on the workload and recreates the pods.

Use resource profiling at the CLI

Procedure

1. Use the following YAML template to enable resource specification recommendation for your workloads.
   You can use the RecommendationProfile CRD to generate resource profiles for your workloads and obtain recommendations on resource specifications for containers in your workloads. You can specify the namespaces and workload types to which a RecommendationProfile CRD is applied.

   apiVersion: autoscaling.alibabacloud.com/v1alpha1
   kind: RecommendationProfile
   metadata:
     # The name of the RecommendationProfile CRD. If you want to create a non-namespaced RecommendationProfile CRD, do not specify a namespace. 
     name: profile-demo
   spec:
     # The workload types for which you want to enable resource profiling. 
     controllerKind:
     - Deployment
     # The namespaces for which you want to enable resource profiling. 
     enabledNamespaces:
     - recommender-demo

   The following table describes the parameters in the YAML template.

   Parameter	Type	Description
   metadata.name	String	The name of the resource object. If you want to create a non-namespaced RecommendationProfile CRD, do not specify a namespace.
   spec.controllerKind	String	The workload types for which you want to enable resource profiling. Valid values: Deployment, StatefulSet, and DaemonSet.
   spec.enabledNamespaces	String	The namespaces for which you want to enable resource profiling.

   Important To generate accurate recommendations on resource specifications, we recommend that you wait at least one day after you enable resource profiling for your workloads. This way, you can obtain a sufficient amount of historical data.
2. Run the kubectl get recommendations -o yaml command to obtain recommendations on resource specifications for your workloads.
   After you enable resource profiling for your workloads, ack-koordinator provides recommendations on resource specifications for each container in your workloads. The recommendations are stored in the Recommendation CRD. The following code block shows the content of a Recommendation CRD that stores the recommendation on resource specifications for a workload named cpu-load-gen:

   apiVersion: autoscaling.alibabacloud.com/v1alpha1
   kind: Recommendation
   metadata:
     labels:
       alpha.alibabacloud.com/recommendation-workload-apiVersion: app-v1
       alpha.alibabacloud.com/recommendation-workload-kind: Deployment
       alpha.alibabacloud.com/recommendation-workload-name: cpu-load-gen
     name: f20ac0b3-dc7f-4f47-b3d9-bd91f906****
     namespace: recommender-demo
   spec:
     workloadRef:
       apiVersion: apps/v1
       kind: Deployment
       name: cpu-load-gen
   status:
     recommendResources:
       containerRecommendations:
       - containerName: cpu-load-gen
         target:
           cpu: 4742m
           memory: 262144k
         originalTarget: # The intermediate results provided by the algorithm that is used to generate the recommendation on resource specifications. We recommend that you do not use the intermediate results. 
          # ...

   To facilitate data retrieval, the Recommendation CRD is generated in the same namespace as the workload. In addition, the Recommendation CRD has specific labels that record the API version, type, and name of the workload. The following table describes the labels.

   Label Key	Description	Example
   alpha.alibabacloud.com/recommendation-workload-apiVersion	The API version of the workload. The value of the label conforms to the Kubernetes specifications. Forward slashes (/) are replaced by hyphens (-).	app-v1 (Original form: app/v1)
   alpha.alibabacloud.com/recommendation-workload-kind	The type of the workload, for example, Deployment or StatefulSet.	Deployment
   alpha.alibabacloud.com/recommendation-workload-name	The name of the workload. The value of the label conforms to the Kubernetes specifications and cannot exceed 63 characters in length.	cpu-load-gen

   The recommendations on resource specifications for each container are displayed in the status.recommendResources.containerRecommendations section. The following table describes the fields.

   Field	Description	Format	Example
   containerName	The name of the container.	string	cpu-load-gen
   target	The recommendations on CPU and memory resources.	map[ResourceName]resource.Quantity	cpu: 4742mmemory: 262144k
   originalTarget	The intermediate results provided by the algorithm that is used to generate the recommendation on resource specifications. We recommend that you do not use the intermediate results. If you want to use the intermediate results, Submit a ticket.	-	-

   Note The recommended minimum CPU resources are 0.025 vCPUs. The recommended minimum memory resources are 250 MB.
3. Optional:Verify the results on the Prometheus Monitoring page in the ACK console.
   ack-koordinator allows you to check resource profiles on the Prometheus Monitoring page in the ACK console.

   • If this is the first time you use Prometheus dashboards, reset the dashboards and install the Resource Profile dashboard. For more information about how to reset Prometheus dashboards, see Reset dashboards.

     To view details about the collected resource profiles on the Prometheus Monitoring page in the ACK console, perform the following steps:

     1. Log on to the ACK console.
     2. In the left-side navigation pane of the ACK console, click Clusters.
     3. On the Clusters page, find the cluster that you want to manage and click its name or click Details in the Actions column.
     4. In the left-side navigation pane of the cluster details page, choose Operations > Prometheus Monitoring.
     5. On the Prometheus Monitoring page, choose Cost Analysis/Resource Optimization > Resource Profiling.

        On the Resource Profiling tab, you can view details about the collected resource profiles. The details include the resource requests, resource usage, and recommended resource specifications of containers. For more information, see Enable Prometheus Service.

        
   • If you use a self-managed Prometheus monitoring system, you can use the following metrics to configure dashboards:

     # Information about the containers for which you want to generate recommendations on CPU resources. 
     slo_manager_recommender_recommendation_workload_target{exported_namespace="$namespace", workload_name="$workload", container_name="$container", resource="cpu"}
     # Information about the containers for which you want to generate recommendations on memory resources. 
     slo_manager_recommender_recommendation_workload_target{exported_namespace="$namespace", workload_name="$workload", container_name="$container", resource="memory"}

Examples

1. Create a file named cpu-load-gen.yaml and copy the following content to the file:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: cpu-load-gen
     labels:
       app: cpu-load-gen
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: cpu-load-gen-selector
     template:
       metadata:
         labels:
           app: cpu-load-gen-selector
       spec:
         containers:
         - name: cpu-load-gen
           image: registry.cn-zhangjiakou.aliyuncs.com/acs/slo-test-cpu-load-gen:v0.1
           command: ["cpu_load_gen.sh"]
           imagePullPolicy: Always
           resources:
             requests:
               cpu: 8 # Request eight vCPUs for the application. 
               memory: "1G"
             limits:
               cpu: 12
               memory: "2G"

2. Run the following command to deploy the cpu-load-gen application:

   kubectl apply -f cpu-load-gen.yaml

3. Create a file named recommender-profile.yaml with the following YAML template:

   apiVersion: autoscaling.alibabacloud.com/v1alpha1
   kind: RecommendationProfile
   metadata:
     name: profile-demo
   spec:
     controllerKind:
     - Deployment
     enabledNamespaces: # Enable recommendations on resource specifications for all Deployments in the default namespace. 
     - default

4. Run the following command to enable resource profiling for the application that you created:

   kubectl apply -f recommender-profile.yaml

5. Run the following command to obtain recommendations on resource specifications for the application that you created.
   Note To generate accurate recommendations on resource specifications, we recommend that you wait at least one day after you enable resource profiling for your workloads. This way, you can obtain a sufficient amount of historical data. For more information about the labels, see Labels.

   kubectl get recommendations -l \
     "alpha.alibabacloud.com/recommendation-workload-apiVersion=apps-v1, \
     alpha.alibabacloud.com/recommendation-workload-kind=Deployment, \
     alpha.alibabacloud.com/recommendation-workload-name=cpu-load-gen" -o yaml

   Expected output:

   apiVersion: autoscaling.alibabacloud.com/v1alpha1
   kind: Recommendation
   metadata:
     creationTimestamp: "2022-02-09T08:56:51Z"
     labels:
       alpha.alibabacloud.com/recommendation-workload-apiVersion: app-v1
       alpha.alibabacloud.com/recommendation-workload-kind: Deployment
       alpha.alibabacloud.com/recommendation-workload-name: cpu-load-gen
     name: f20ac0b3-dc7f-4f47-b3d9-bd91f906****
     namespace: recommender-demo
   spec:
     workloadRef:
       apiVersion: apps/v1
       kind: Deployment
       name: cpu-load-gen
   status:
     conditions:
     - lastTransitionTime: "2022-02-09T08:56:52Z"
       status: "True"
       type: RecommendationProvided
     recommendResources:
       containerRecommendations:
       - containerName: cpu-load-gen
         target:
           cpu: 4742m # The recommended CPU resources are 4.742 vCPUs. 
           memory: 262144k
         originalTarget: # The intermediate results provided by the algorithm that is used to generate the recommendation on resource specifications. We recommend that you do not use the intermediate results. 
           #...

Analyze the results

Compare the resource specifications in Step 1 and Step 5. The requested amount of CPU resources is greater than the amount of CPU resources in the recommendation. You can reduce the CPU request of the application to save resources in the cluster.

FAQ

How does the resource profiling algorithm work?

What types of applications is resource profiling suitable for?

Resource profiling is suitable for online applications.

In most cases, the resource specifications that are recommended by the resource profiling feature can meet the resource requirements of your applications. Offline applications use batch processing and require high throughput. Offline applications allow resource contention so as to improve resource utilization. If you enable resource profiling for offline applications, resource waste may occur. In most cases, key system components are deployed in active/standby mode and have multiple replicas. In most cases, the resources that are allocated to standby replicas are idle. As a result, the results generated by the resource profiling algorithm are not accurate. In the preceding cases, we recommend that you do not directly use the resource specifications recommended by resource profiling. ACK will provide updates on how to specify resource specifications based on the recommendations provided by resource profiling in these cases.

Can I directly use the resource specifications recommended by resource profiling when I specify the resource request and limit of a container?

The recommendations provided by resource profiling are based on the current resource conditions of the workload. You must specify resource specifications based on the recommendations and the characteristics of the workload. If the workload needs a resource buffer to handle traffic spikes or perform failovers in an active zone-redundancy architecture, or the workload is resource-sensitive, we recommend that you specify resource specifications that are higher than the recommended resource specifications.

How do I view resource profiles if I use a self-managed Prometheus monitoring system?

Follow Step 3 in the Use resource profiling at the CLI section to configure Grafana dashboards.

How do I delete resource profiles and resource profiling rules?

Resource profiles are stored in the Recommendation CRD. Resource profiling rules are stored in the RecommendationProfile CRD. You can run the following command to delete all resource profiles and resource profiling rules:

# Delete all resource profiles. 
kubectl delete recommendation -A --all

# Delete all resource profiling rules. 
kubectl delete recommendationprofile -A --all