Container Service for Kubernetes:Observability FAQ

Symptoms

If you followed the instructions in Collect ACK cluster container logs but cannot find the logs on the Preview page of the Simple Log Service console, the log collection may have failed. To troubleshoot this issue, check the configuration, the status of collection components and nodes, and the operational logs of the log collection container.

Before you begin

Note the following points when you collect logs from container files.

• The log collection component collects only incremental logs. If the log file is not updated after the Logtail configuration is applied, the component does not collect logs from the file. For more information, see Read logs.

• You can collect logs only from files that use the default container storage or are mounted to a local path. Other storage methods are not supported.

• After logs are collected, you must create an index before you can query and analyze the logs in the Logstore. For more information, see Create an index.

Step 1: Check the collection configuration

Verify that the Logtail configuration is correct. Make sure that all log collection settings are accurate and that the container continuously generates logs.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  # Set the resource name, which must be unique within the current Kubernetes cluster. This name is also used for the created collection configuration. The configuration does not take effect if the name is a duplicate.
  name: example-k8s-file
spec:
  # Specify the destination project.
  project:
    name: k8s-log-<YOUR_CLUSTER_ID>
  logstores:
    # Create a Logstore named k8s-file.
    - name: k8s-file
  # Define the collection configuration.
  config:
    # Log sample (optional).
    sample: |
      2024-06-19 16:35:00 INFO test log
      line-1
      line-2
      end
    # Define the input plugin.
    inputs:
      # Use the input_file plugin to collect multi-line text logs from the container.
      - Type: input_file
        # The file path in the container.
        FilePaths:
          - /data/logs/app_1/**/test.LOG
        # Enable the container discovery feature.
        EnableContainerDiscovery: true
        # Add container information filter conditions. The relationship between multiple options is "AND".
        CollectingContainersMeta: true
        ContainerFilters:
          # Specify the namespace of the pod where the container to be collected is located. Regular expression matching is supported.
          K8sNamespaceRegex: default
          # Specify the name of the container to be collected. Regular expression matching is supported.
          IncludeK8sLabel:
            app: ^(.*app.*)$
        # Enable multi-line log collection. For single-line log collection, delete this configuration.
        Multiline:
          # Select the custom start pattern mode.
          Mode: custom
          # Configure the start pattern regular expression.
          StartPattern: '\d+-\d+-\d+\s\d+:\d+:\d+'
    # Define the processing plugin.
    processors:
      # Use the regex parsing plugin to parse logs.
      - Type: processor_parse_regex_native
        # Source field name.
        SourceKey: content
        # Regular expression for parsing. Use capturing groups "()" to capture the fields to be extracted.
        Regex: (\d+-\d+-\d+\s\S+)(.*)
        # List of extracted fields.
        Keys: ["time", "detail"]
    # Define the output plugin.
    flushers:
      # Use the flusher_sls plugin to output to the specified Logstore.
      - Type: flusher_sls
        # Make sure this Logstore exists.
        Logstore: k8s-file
        # Make sure the endpoint is correct.
        Endpoint: cn-beijing.log.aliyuncs.com
        Region: cn-beijing
        TelemetryType: logs

Step 2: Check the status of the collection component and machine group

Confirm that a collection component, such as Logtail or LoongCollector, is deployed and running on each worker node, and that the number of OK heartbeats from the collection containers matches the number of worker nodes.

1. Check the status of the collection component pod.

   1. Connect to the cluster.

   2. Run the following command to check whether all related collection pods are in the Running state. If a pod is in an abnormal state, see Troubleshoot pod exceptions.

      kubectl get pods -n kube-system -l 'k8s-app in (loongcollector-ds,logtail)' 

      The output is similar to the following:

      NAME                      READY   STATUS    RESTARTS   AGE
      loongcollector-ds-fn5zn   1/1     Running   0          3d19h
      loongcollector-ds-ks76g   1/1     Running   0          3d19h

2. Check the heartbeat status of the machine group.

   1. Log on to the Simple Log Service console.

   2. In the Projects section, click the one you want.

      

   3. In the navigation pane on the left, choose Resource > Machine Groups.

   4. In the machine group list, click the destination machine group.

   5. On the Machine Group Configuration page, note the number of machines with a heartbeat status of OK. Verify that this number matches the number of worker nodes in the cluster. For more information, see Heartbeat Status Solution.

Step 3: View the LoongCollector (Logtail) operational logs

Check the collection container for collection errors or error messages to further analyze the cause of the problem.

1. Enter the collection component pod.

   kubectl exec -it -n kube-system loongcollector-ds-XXXX  -- bash

2. Logtail logs are stored in the /usr/local/ilogtail/ directory of the Logtail container. The filenames are ilogtail.LOG and logtail_plugin.LOG. You can log on to the Logtail container and run the following commands to view the log files:

   # Open the /usr/local/ilogtail/ directory.
   cd /usr/local/ilogtail
   
   # View the ilogtail.LOG and logtail_plugin.LOG files.
   cat ilogtail.LOG
   cat logtail_plugin.LOG

3. View the alarm metric of the error log and find the corresponding solution in Common error types for data collection in Simple Log Service.

Other O&M operations

• View the status of Simple Log Service components in a Kubernetes cluster

  View the status of Simple Log Service components in a Kubernetes cluster

  Run the following command to view the status and information of the Simple Log Service deployment.

  kubectl get deploy -n kube-system | grep -E 'alibaba-log-controller|loongcollector-operator'

  The following result is returned:

  NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
  alibaba-log-controller   1/1     1            1           11d

  Run the following command to view the status information of the DaemonSet resource.

  kubectl get ds  -n kube-system | grep -E 'logtail-ds|loongcollector-ds'

  The following result is returned:

  NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR  AGE
  logtail-ds   2         2         2       2            2           **ux           11d

• View the version number, IP address, and startup time of Logtail

  The information is stored in the /usr/local/ilogtail/app_info.json file of the Logtail container.

  kubectl exec logtail-ds-****k -n kube-system cat /usr/local/ilogtail/app_info.json

  A result similar to the following is returned.

  {
     "UUID" : "",
     "hostname" : "logtail-****k",
     "instance_id" : "0EB****_172.20.4.2_1517810940",
     "ip" : "172.20.4.2",
     "logtail_version" : "0.16.2",
     "os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
     "update_time" : "2018-02-05 06:09:01"
  }

• Transfer ACK cluster logs to a project in another Alibaba Cloud account

Symptoms

This topic describes how to delete a project and what to do if you cannot delete a project and receive an "insufficient permissions" error message.

Solution

For more information about how to delete a project or Logstore, see Manage projects and Manage a Logstore. If a project fails to be deleted, see What do I do if the "Operation denied, insufficient permissions" error message is returned when I delete a project?

Cause

1. Application monitoring is suspended.

2. The ARMS agent is not loaded as expected at the pod where the application resides.

Solution

1. Check whether application monitoring is paused.

   1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring > Application List.

   2. On the Application List page, select a region in the top navigation bar and click the name of the application.

      If the application is not found, proceed to Step 2: Check whether the ARMS agent is loaded as expected.

   3. If you are using the new Application Real-Time Monitoring Service (ARMS) console, choose Configuration > Custom Configurations in the top navigation bar of the application details page. In the Probe switch settings section, check whether application monitoring is suspended.

      • If Pause application monitoring is turned on, turn off the switch and click Save.

      • If Pause application monitoring is turned off, proceed to Step 2: Check whether the ARMS agent is loaded as expected.

   4. If you are using the old ARMS console, click Application Settings in the left-side navigation pane of the application details page. On the page that appears, click the Custom Configuration tab. In the Agent Switch Settings section, check whether Probe Master Switch is turned on.

      • If Probe Master Switch is turned off, turn on Probe Master Switch and click Save in the lower part of the page.

      • If Probe Master Switch is turned on, proceed to Step 2: Check whether the ARMS agent is loaded as expected.

2. Check whether the probe is loaded correctly.

   1. Log on to the ACK console. In the left-side navigation pane, click Clusters. On the Clusters page, click the name of the cluster to go to the cluster details page.

   2. In the left-side navigation pane, choose Workloads > Pods.

   3. On the Pods page, select the namespace in which your application resides, find the application, and then click Edit YAML in the Actions column.

   4. In the Edit YAML dialog box, check whether the YAML file contains initContainers.

      

      • If the YAML file does not contain initContainers, the pod has not been injected into one-pilot-initcontainer. Perform Step 5.

      • If the YAML file contains initContainers, the pod has been injected into one-pilot-initcontainer. Perform Step 8.

   5. In the left-side navigation pane of the cluster details page, choose Workloads > Pods. On the page that appears, set the Namespace parameter to ack-onepilot. Check if any pod named ack-onepilot-* with completed rolling updates exists in the pod list.

      • If the specified pod exists, perform Step 6.

      • If the specified pod does not exist, install the ack-onepilot component from the application market. For more information, see How do I install ack-onepilot and uninstall arms-pilot?

   6. In the left-side navigation pane of the cluster details page, choose Workloads > Deployments or StatefulSets. On the page that appears, find the application and choose > Edit YAML in the Actions column. In the Edit YAML dialog box, check whether the YAML file contains the following labels in the spec.template.metadata section:

      labels:
        armsPilotAutoEnable: "on"
        armsPilotCreateAppName: "<your-deployment-name>"    # Replace <your-deployment-name> with the actual application name. 
        armsSecAutoEnable: "on"    # If you want to connect the application to Application Security, you must configure this parameter.

      • If the YAML file contains the labels, perform Step 7.

      • If the YAML file does not contain the labels, perform the following operations: In the Edit YAML dialog box, add the labels to the spec > template > metadata section and replace <your-deployment-name> with the actual application name. Then, click Update.

   7. In the left-side navigation pane of the cluster details page, choose Workloads > Pods. On the page that appears, find the pod and choose More > Logs in the Actions column to check whether the pod logs of ack-onepilot report a Security Token Service (STS) error in the "Message":"STS error" format.

      • If the error is reported, authorize the cluster of the application and restart the pod of the application. For more information, see Install the ARMS agent for Java applications deployed in ACK.

      • If the error is not reported, submit a ticket.

   8. In the left-side navigation pane of the cluster details page, choose Workloads > Pods, find the pod and click Edit YAML in the Actions column. In the Edit YAML dialog box, check whether the YAML file contains the following javaagent parameter:

      -javaagent:/home/admin/.opt/ArmsAgent/aliyun-java-agent.jar

      Note

      If you use an ARMS agent earlier than 2.7.3.5, replace aliyun-java-agent.jar in the preceding code with arms-bootstrap-1.7.0-SNAPSHOT.jar. We recommend that you upgrade the agent to the latest version at the earliest opportunity.

      • If the YAML file contains the parameter, find the pod on the Pods page and click Terminal in the Actions column to go to the command line page. Run the following command to check whether the logs file contains a log file with the .log file extension. Then, submit a ticket.

        cd /home/admin/.opt/ArmsAgent/logs

      • If the YAML file does not contain the parameter, submit a ticket.

Symptoms

The ARMS Addon Token does not exist in the target cluster.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters. On the Clusters page, click the name of the cluster to go to the cluster details page.

2. In the left-side navigation pane, choose Configurations > Secrets.

3. In the upper part of the page, select kube-system from the Namespace drop-down list to check whether addon.arms.token is enabled.

   

Solution

Grant Container Service for Kubernetes access permissions to ARMS resources.

Symptoms

• The value displayed in the namespace column on the custom dashboard is not updated after you change the namespace of your application.

  

• After you change the cluster of your application, the data for rate, errors, and duration (RED) metrics is displayed normally but no data is displayed for container monitoring metrics, such as CPU and memory.

Possible cause

Container-related parameters, such as Namespace and ClusterId, are configured when the application is created and the values of these parameters cannot be automatically updated. If you change the cluster or namespace of your application, the container-related data may fail to be queried or displayed.

Solution

• Delete the application, recreate the application, and then report monitoring data again. For more information, see Delete applications.

  This method causes the loss of historical data.

• Submit a ticket.

Background

Typically, the ack-onepilot component specifies the mount path for Application Real-Time Monitoring Service (ARMS) agents for Java by injecting the environment variable JAVA_TOOL_OPTIONS. However, you may need to customize this path for scenarios such as:

• Centralized configuration management

  Manage the mount path through a Kubernetes ConfigMap to ensure environment consistency.

• Persistent storage

  Store agent files in a custom persistent volume claim (PVC) to meet enterprise security or O&M requirements.

Solution

To customize the mount path for ARMS agents for Java, these version requirements must be met:

• ack-onepilot: V4.1.0 or later.

• ARMS agent for Java: V4.2.2 or later. You can control the version of your ARMS agent for Java.

1. Add the disableJavaToolOptionsInjection annotation to the Kubernetes workload, such as a deployment, that requires a custom mount path.

   The ack-onepilot component will not automatically set the mount path or other Java Virtual Machine (JVM) parameters using the environment variable JAVA_TOOL_OPTIONS.

   1. To view the YAML file of the deployment, run the following command:

      kubectl get deployment {Deployment name} -o yaml

      Note

      If you're not sure about the deployment name, run the following command to list all deployments:

      kubectl get deployments --all-namespace

      Then, find the one you want in the results and view its YAML file.

   2. Run the following command to edit the YAML file:

      kubectl edit deployment {Deployment name} -o yaml

   3. In the YAML file, add the following labels to spec.template.metadata:

      labels:
        armsPilotAutoEnable: "on"
        armsPilotCreateAppName: "<your-deployment-name>"    # The name of your deployment.
        disableJavaToolOptionsInjection: "true" # If you want to customize the mount path for the ARMS agent for Java, set this parameter to true.

2. Replace the default mount path /home/admin/.opt/AliyunJavaAgent/aliyun-java-agent.jar in your Java startup script or command with your custom path:

   java -javaagent:/home/admin/.opt/AliyunJavaAgent/aliyun-java-agent.jar ... ... -jar xxx.jar

   Other information such as the reporting region and license key is provided by ack-onepilot through environment variables.

Symptoms

How do I report data from Region A to Region B?

Solution

1. Update the ack-onepilot component to V4.0.0 or later.

2. Add the ARMS_REPORT_REGION environment variable to the ack-onepilot-ack-onepilot application in the ack-onepilot namespace. The value must be the ID of a region where ARMS is available. Example: cn-hangzhou or cn-beijing.

3. Restart the existing application or deploy a new application to report data across regions.

   Note

   After the environment variable is added, all applications deployed in the cluster report data to the region specified in the previous step.

Background

The old Application Monitoring agent arms-pilot is no longer maintained. You can install the new agent ack-onepilot to monitor your applications. ack-onepilot is fully compatible with arms-pilot. You can seamlessly install ack-onepilot without the need to modify application configurations. This topic describes how to uninstall arms-pilot and install ack-onepilot.

Solution

• You must install ack-onepilot in an ACK cluster V1.16 or later. If your cluster is earlier than V1.16, upgrade the cluster first. For more information, see Update the Kubernetes version of an ACK cluster.

• You must uninstall arms-pilot before installing ack-onepilot. If you have both ack-onepilot and arms-pilot installed, the ARMS agent cannot be mounted. If arms-pilot is not completely uninstalled, ack-onepilot does not work because it regards that arms-pilot is still working in the environment.

• When you uninstall arms-pilot and install ack-onepilot, the configurations of arms-pilot cannot be automatically synchronized to ack-onepilot. We recommend that you record the configurations and then manually configure ack-onepilot.

1. Uninstall arms-pilot.

   1. Log on to the ACK console. On the Clusters page, click the name of the cluster.

   2. In the left-side navigation pane, choose Applications > Helm.

   3. On the Helm page, find arms-pilot and click Delete in the Actions column.

   4. In the Delete message, click OK.

2. Check whether arms-pilot has been uninstalled.

   Go to the cluster details page of the ACK console. In the left-side navigation pane, choose Workloads > Deployments. On the Deployments page, select arms-pilot from the Namespace drop-down list, and check whether the pods of the namespace are deleted as expected.

   Note

   If you have modified the namespace to which arms-pilot belongs, select the new namespace.

3. Install ack-onepilot.

   1. Log on to the ACK console. On the Clusters page, click the name of the cluster.

   2. In the left-side navigation pane, click Add-ons. On the Add-ons page, search for ack-onepilot.

   3. Click Install on the ack-onepilot card.

      Note

      By default, the ack-onepilot component supports 1,000 pods. For every additional 1,000 pods in the cluster, you must add 0.5 CPU cores and 512 MB memory for the component.

   4. In the dialog box that appears, configure the parameters and click OK. We recommend that you use the default values.

      Note

      After you install ack-onepilot, you can upgrade, configure, or uninstall it on the Add-ons page.

4. Check whether ack-onepilot has been installed.

   Go to the cluster details page of the ACK console. In the left-side navigation pane, choose Workloads > Deployments. On the Deployments page, select ack-onepilot from the Namespace drop-down list, and check whether the pods of the namespace are running as expected.

   

Symptoms

If you enable Prometheus monitoring and see the message No dashboard is found on the Operations > Prometheus Monitoring page for the target cluster, follow the procedure below to resolve the issue.

Solution

1. Reinstall the Prometheus monitoring component.

   1. Disable Prometheus monitoring.

   2. Reinstall the component:

      1. After confirming that the uninstallation is complete, click Install, and then click OK in the dialog box.

      2. After the installation is complete, return to the Prometheus Monitoring page to check whether the issue is resolved.

         If the issue persists, proceed to the next step.

2. Check the Prometheus instance connection.

   1. In the navigation pane on the left of the ARMS console, click Integration Management.

   2. On the Integrated Environments tab, check the Container Service list for a container environment with the same name as your cluster.

      • If no corresponding container environment exists: See Connect using the ARMS or Prometheus console.

      • If you have a container environment, click Configure Agent in the Actions column for the target environment to open the Configure Agent page.

        Check whether the installed agents run as expected.

Cause

Data may not be displayed in Managed Service for Prometheus because the synchronization task with the Alibaba Cloud Prometheus service failed, which leads to resource registration failure, or because the Prometheus instance was not provisioned correctly. Follow the process below to check and resolve the issue.

Solution

1. Check the status of the provisioning task for Managed Service for Prometheus.

   1. Log on to the Container Service for Kubernetes console. In the left navigation pane, click Clusters.

   2. On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Workloads > Jobs.

   3. On the Jobs page, set Namespace to arms-prom at the top of the page, and then click o11y-init-environment to verify that the job was successful.

      If it is not successful, it may be because synchronizing with the Alibaba Cloud Prometheus service and registering resources failed. You can view its pod logs to find the specific reason for the failure. For more information, see Troubleshoot pod exceptions.

      If the pod no longer exists, proceed with the following steps.

2. Reinstall the Prometheus monitoring component.

   1. Disable Prometheus monitoring.

   2. Reinstall the component:

      1. After confirming that the uninstallation is complete, click Install, and then click OK in the dialog box.

      2. After the installation is complete, return to the Prometheus Monitoring page to check whether the issue is resolved.

         If the issue persists, proceed to the next step.

3. Check the Prometheus instance provisioning.

   1. In the navigation pane on the left of the ARMS console, click Integration Management.

   2. On the Integrated Environments tab, check the Container Service list for a container environment with the same name as your cluster.

      • If no corresponding container environment exists: See Connect using the ARMS or Prometheus console.

      • If you have a container environment, click Configure Agent in the Actions column for the target environment to open the Configure Agent page.

        Check whether the installed agents run as expected.

Symptoms

When you uninstall and reinstall the Prometheus agent, the following error message appears:

rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: ClusterRole, namespace: , name: arms-pilot-prom-k8s

Cause

After you run commands to manually uninstall the Prometheus agent, resources such as roles may fail to be deleted.

Solution

1. Run the following command to find the ClusterRoles of the Prometheus agent:

   kubectl get ClusterRoles --all-namespaces | grep prom

2. Run the following command to delete the ClusterRoles that are queried in the previous step:

    kubectl delete ClusterRole [$Cluster_Roles] -n arms-prom

   Note

   The [$Cluster_Roles] parameter specifies the ClusterRoles that are queried in the previous step.

3. If the issue persists after you delete the ClusterRoles, view the value of kind in the error message to check whether resources other than ClusterRoles exist. Perform the preceding operations to delete them in sequence.

Background

You must check the version of the ack-arms-prometheus component that is deployed in your cluster and whether it needs to be updated.

Solution

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, find the one you want to manage and click its name. In the left navigation pane, click Add-ons.

3. On the Component Management page, click the Log And Monitoring tab and find the ack-arms-prometheus component.

   The current version is displayed below the component name. If a later version is available, click Upgrade next to the version number to upgrade the component.

   Note

   The Upgrade button is available only if the installed component is not the latest version.

Cause

GPU monitoring may fail to deploy if a GPU node has taints. You can perform the following steps to check for taints on a GPU node.

Solution

1. Run the following command to view the taints of the target GPU node.

   If the GPU node has custom taints, you will find entries related to them in the output. This topic uses a taint with a key of test-key, a value of test-value, and an effect of NoSchedule as an example:

   kubectl describe node cn-beijing.47.100.***.***

   Expected output:

   Taints:test-key=test-value:NoSchedule

2. You can handle the taints on the GPU node in one of the following ways.

   • Run the following command to delete the taints of the GPU node.

     kubectl taint node cn-beijing.47.100.***.*** test-key=test-value:NoSchedule-

   • Declare a toleration for the taints of the GPU node to allow pods to be scheduled to the node.

     # 1. Run the following command to edit ack-prometheus-gpu-exporter.
     kubectl edit daemonset -n arms-prom ack-prometheus-gpu-exporter
     
     # 2. Add the following fields to the YAML file to declare the toleration for the taints.
     # Omit other fields.
     # The tolerations field is added above the containers field and at the same level as the containers field.
     tolerations:
     - key: "test-key"
       operator: "Equal"
       value: "test-value"
       effect: "NoSchedule"
     containers:
      # Omit other fields.

Background

If you delete only the namespace of Prometheus Monitoring for Alibaba Cloud, residual configurations may remain and cause reinstallation to fail. To completely and manually delete all ARMS-Prometheus configurations, perform the following operations.

Solution

• Delete the arms-prom namespace.

  kubectl delete namespace arms-prom

• Delete the ClusterRoles.

  kubectl delete ClusterRole arms-kube-state-metrics
  kubectl delete ClusterRole arms-node-exporter
  kubectl delete ClusterRole arms-prom-ack-arms-prometheus-role
  kubectl delete ClusterRole arms-prometheus-oper3
  kubectl delete ClusterRole arms-prometheus-ack-arms-prometheus-role
  kubectl delete ClusterRole arms-pilot-prom-k8s
  kubectl delete ClusterRole gpu-prometheus-exporter
  kubectl delete ClusterRole o11y:addon-controller:role
  kubectl delete ClusterRole arms-aliyunserviceroleforarms-clusterrole

• Delete the ClusterRoleBindings.

  kubectl delete ClusterRoleBinding arms-node-exporter
  kubectl delete ClusterRoleBinding arms-prom-ack-arms-prometheus-role-binding
  kubectl delete ClusterRoleBinding arms-prometheus-oper-bind2
  kubectl delete ClusterRoleBinding arms-kube-state-metrics
  kubectl delete ClusterRoleBinding arms-pilot-prom-k8s
  kubectl delete ClusterRoleBinding arms-prometheus-ack-arms-prometheus-role-binding
  kubectl delete ClusterRoleBinding gpu-prometheus-exporter
  kubectl delete ClusterRoleBinding o11y:addon-controller:rolebinding
  kubectl delete ClusterRoleBinding arms-kube-state-metrics-agent
  kubectl delete ClusterRoleBinding arms-node-exporter-agent
  kubectl delete ClusterRoleBinding arms-aliyunserviceroleforarms-clusterrolebinding

• Delete the Roles and RoleBindings.

  kubectl delete Role arms-pilot-prom-spec-ns-k8s
  kubectl delete Role arms-pilot-prom-spec-ns-k8s -n kube-system
  kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s
  kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s -n kube-system

After you manually delete the ARMS-Prometheus resources, navigate to Operations Management > Component Management in the Container Service for Kubernetes (ACK) console to reinstall the ack-arms-prometheus component.

Cause

When you deploy the ack-arms-prometheus component, an "xxx in use" error is reported. This is likely due to a resource being occupied or a residual resource, which causes the component installation to fail.

Solution

1. On the Clusters page, click the name of the target cluster. In the navigation pane on the left of the cluster details page, choose Applications > Helm.

2. On the Helm page, verify that ack-arms-prometheus exists.

   • Delete ack-arms-prometheus from the Helm page and then reinstall it on the Add-ons page. For more information, see Manage components.

   • Not found:

     1. If ack-arms-prometheus is not found, it indicates that residual resources remain from the deletion of the ack-arms-prometheus Helm Release. You must then manually and completely delete ARMS-Prometheus.

     2. Re-enable Prometheus monitoring.

Symptoms

When you try to install the ack-arms-prometheus component, a "Component Not Installed" message appears first, and the installation still fails on the second attempt.

Solution

• Check whether the ack-arms-prometheus component is already installed.

  1. On the Clusters page, click the name of the target cluster. In the navigation pane on the left of the cluster details page, choose Applications > Helm.

  2. On the Helm page, verify that ack-arms-prometheus exists.

     • Delete ack-arms-prometheus from the Helm page and then reinstall it on the Add-ons page. For more information, see Manage components.

     • Not found:

       1. If ack-arms-prometheus is not found, it indicates that residual resources remain from the deletion of the ack-arms-prometheus Helm Release. You must then manually and completely delete ARMS-Prometheus.

       2. Re-enable Prometheus monitoring.

• Check the logs of ack-arms-prometheus for errors.

  1. In the navigation pane on the left of the cluster details page, choose Workloads > Deployments.

  2. At the top of the Deployments page, set Namespace to arms-prom and click arms-prometheus-ack-arms-prometheus.

  3. Click the Logs tab and check the logs for errors.

• Check whether an error occurred during the agent installation.

  1. Log on to the ARMS console. In the navigation pane on the left, click Integration Management.

  2. On the Integration Management tab, find the target container environment in the Container Service list. In the Actions column, click Configure Agent to open the Configure Agent page.

Symptoms

After you deploy open source Prometheus, you must configure it to send alert notifications using DingTalk.

Solution

1. Obtain the webhook URL of your DingTalk chatbot. For more information, see Scenario 3: Implement Kubernetes monitoring and alerting with DingTalk chatbot.

2. On the Parameters wizard page, find the dingtalk section, set enabled to true, and then specify the webhook URL of your DingTalk chatbot in the token field. For more information, see Configure DingTalk alert notifications in Alert configurations.

Symptoms

Can't install release with errors: rpc error: code = Unknown desc = object is being deleted: customresourcedefinitions.apiextensions.k8s.io "xxxxxxxx.monitoring.coreos.com" already exists

Solution

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com

Symptoms

After you deploy open source Prometheus, the email alerts that you configured do not send alert notifications.

Solution

Make sure that the value of smtp_auth_password is the SMTP authorization code instead of the logon password of the email account. Make sure that the SMTP server endpoint includes a port number.

Symptoms

After you deploy open source Prometheus, when you click Update YAML, the error message "The current cluster is temporarily inaccessible. Please try again later or submit a ticket for feedback" appears.

Solution

If the configuration file of Tiller is overlarge, the cluster cannot be accessed. To solve this issue, you can delete some annotations in the configuration file and mount the file to a pod as a ConfigMap. You can specify the name of the ConfigMap in the configMaps fields of the prometheus and alertmanager sections. For more information, see the second method in Mount a ConfigMap to Prometheus.

Symptoms

After you deploy open source Prometheus, you may need to perform further configuration to enable its features.

Solution

After prometheus-operator is deployed, you can perform the following steps to enable the features of prometheus-operator. Go to the cluster details page and choose Applications > Helm in the left-side navigation pane. On the Helm page, find ack-prometheus-operator and click Update in the Actions column. In Update Release panel, configure the code block to enable the features. Then, click OK.

Symptoms

When choosing a storage solution, how do I choose between TSDB and Alibaba Cloud disks, and how do I configure the data retention policy?

Solution

Symptoms

After you deploy open source Prometheus, the Grafana dashboard is not displayed correctly.

Solution

Go to the cluster details page and choose Applications > Helm in the left-side navigation pane. On the Helm page, find ack-prometheus-operator and click Update in the Actions column. In the Update Release panel, check whether the value of the clusterVersion field is correct. If the Kubernetes version of your cluster is earlier than 1.16, set clusterVersion to 1.14.8-aliyun.1. If the Kubernetes version of your cluster is 1.16 or later, set clusterVersion to 1.16.6-aliyun.1.

Cause

After you delete the ack-prometheus namespace, the related resource configurations may be retained. In this case, you may fail to install ack-prometheus again. You can perform the following operations to delete the related resource configurations:

Solution

1. Delete role-based access control (RBAC)-related resource configurations.

   1. Run the following commands to delete the related ClusterRoles:

      kubectl delete ClusterRole ack-prometheus-operator-grafana-clusterrole
      kubectl delete ClusterRole ack-prometheus-operator-kube-state-metrics
      kubectl delete ClusterRole psp-ack-prometheus-operator-kube-state-metrics
      kubectl delete ClusterRole psp-ack-prometheus-operator-prometheus-node-exporter
      kubectl delete ClusterRole ack-prometheus-operator-operator
      kubectl delete ClusterRole ack-prometheus-operator-operator-psp
      kubectl delete ClusterRole ack-prometheus-operator-prometheus
      kubectl delete ClusterRole ack-prometheus-operator-prometheus-psp

   2. Run the following commands to delete the related ClusterRoleBindings:

      kubectl delete ClusterRoleBinding ack-prometheus-operator-grafana-clusterrolebinding
      kubectl delete ClusterRoleBinding ack-prometheus-operator-kube-state-metrics
      kubectl delete ClusterRoleBinding psp-ack-prometheus-operator-kube-state-metrics
      kubectl delete ClusterRoleBinding psp-ack-prometheus-operator-prometheus-node-exporter
      kubectl delete ClusterRoleBinding ack-prometheus-operator-operator
      kubectl delete ClusterRoleBinding ack-prometheus-operator-operator-psp
      kubectl delete ClusterRoleBinding ack-prometheus-operator-prometheus
      kubectl delete ClusterRoleBinding ack-prometheus-operator-prometheus-psp

2. Run the following command to delete the related CRD objects:

   kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
   kubectl delete crd alertmanagers.monitoring.coreos.com
   kubectl delete crd podmonitors.monitoring.coreos.com
   kubectl delete crd probes.monitoring.coreos.com
   kubectl delete crd prometheuses.monitoring.coreos.com
   kubectl delete crd prometheusrules.monitoring.coreos.com
   kubectl delete crd servicemonitors.monitoring.coreos.com
   kubectl delete crd thanosrulers.monitoring.coreos.com

Symptoms

In the Alert Center, the alert rule synchronization status shows the message The Project does not exist : k8s-log-xxx.

Cause

SLS Event Center resources have not been created.

Solution

1. In the Simple Log Service console, confirm whether the quota limit has been reached. For more information about resources, see Basic resource limits.

   1. If the quota limit has been reached, delete unnecessary projects, or submit a ticket to apply for an increase in the project resource quota limit. For information about how to delete a project, see Manage a project.

   2. If the limit has not been reached, proceed with the following steps.

2. Reinstall the ack-node-problem-detector component.

   Reinstalling the component recreates the default project named k8s-log-xxxxxx.

   1. Uninstall the ack-node-problem-detector component.

      1. On the management page of the target cluster in the Container Service for Kubernetes console, in the left navigation pane, choose O&M > Component Management.

      2. Click the Logs and Monitoring tab. Then, click Uninstall on the ack-node-problem-detector component card and click Confirm in the dialog box.

   2. After the uninstallation is complete, install the ack-node-problem-detector component.

      1. In the left navigation pane, choose O&M > Alerting

      2. On the Alert Rules page, click Start Installation. The console automatically creates a project, and installs and upgrades the component.

3. On the Alert Rules page, disable the corresponding alert rule set. Wait until its Alert Rule Status changes to Rule Disabled, and then re-enable the rule set to retry.

Symptom

An alert rule fails to sync during configuration or deployment, and the system displays an error message such as this rule have no xxx contact groups reference. As a result, notifications for this alert rule are not delivered.

Cause

The alert rule synchronization fails, and an error message similar to this rule have no xxx contact groups reference is reported.

Solution

1. You have created a contact and added it to a contact group.

2. To the right of the target alert rule set, click Edit Notification Policy to configure which contact groups subscribe to the alert rule set.

Why is no data returned for kubectl top pod/node?

Why is partial data missing for kubectl top pod/node?

What do I do if HPA cannot obtain metrics data?

Why does HPA create extra pods during a rolling deployment?

--metric-resolution=15s
--enable-hpa-rolling-update-skipped=true