Container Service for Kubernetes:Enable Tracing Analysis for data plane components

Prerequisites

Before you begin, make sure that you have:

• An ACK managed cluster or ACK dedicated cluster running Kubernetes 1.28 or later. For upgrade instructions, see Manually upgrade ACK clusters.

• Helm 3.9 or later installed. For upgrade instructions, see Update Helm V2 to Helm V3.

• Managed Service for OpenTelemetry activated and authorized. For setup instructions, see Prepare for using Managed Service for OpenTelemetry.

Enabling this feature incurs Managed Service for OpenTelemetry charges in addition to your ACK cluster fees. For free quota and pricing details, see Billing overview.

How it works

Enabling kubelet tracing involves three stages:

1. Retrieve the endpoint and authentication token from Managed Service for OpenTelemetry, then store the token in a cluster Secret so that the OpenTelemetry Collector can read the configuration and authentication information.

2. Deploy the OpenTelemetry Collector in DaemonSet mode — one agent per node — to collect, process, and forward observability data. Because kubelet runs on every node, a DaemonSet deployment ensures each kubelet instance has a local Collector to send traces to.

3. Configure kubelet tracing parameters (endpoint and sampling rate) so that trace data is forwarded to Managed Service for OpenTelemetry through the OpenTelemetry Collector.

Step 1: Get the endpoint and authentication token

1. Log on to the Managed Service for OpenTelemetry console. In the left-side navigation pane, click Integration Center.

2. In the Open Source Frameworks section, click the OpenTelemetry card.

3. Select a data reporting region, set Connection Type to Alibaba Cloud VPC Network and Export Protocol to gRPC. Save the endpoint information for later use.

   Resources are automatically initialized in the region that you access for the first time. Follow the instructions on the page.

   

Step 2: Store the authentication token in a Secret

Store the authentication token in a Secret in your ACK cluster and reference it through environment variables in the OpenTelemetry Collector. The Secret must be in the same namespace as the OpenTelemetry Collector you deploy in Step 3.

By default, the OpenTelemetry Collector is deployed in the otel-collector namespace. Create the namespace manually, or use a different namespace and select it during installation.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the cluster name. In the left-side navigation pane, choose Configurations > Secrets.

3. Click Create and configure the Secret with the following parameters:

   Parameter	Value	Note
   Name	opentelemetry-exporter-config	Important Use this exact value. Custom names are not supported.
   Type	Opaque
   Name (in the Add Secret dialog)	grpc-token	Important Use this exact value. Custom names are not supported.
   Value	The authentication token obtained in Step 1.
   Encode Data Values Using Base64	Enable

Step 3: Install the OpenTelemetry Collector

The OpenTelemetry Collector receives, processes, and exports observability data from multiple open source sources — including Jaeger, Prometheus, and Fluent Bit — so you don't need to run a separate collector for each.

All examples in this step use a DaemonSet deployment so that each node has a dedicated Collector agent next to its kubelet instance.

1. Log on to the ACK console. In the left-side navigation pane, choose Marketplace > Marketplace.

2. Search for opentelemetry-collector and click the card.

3. In the upper-right corner, click Deploy. In the dialog box, select your cluster and the same namespace as the Secret created in Step 2. Follow the on-screen instructions to proceed.

4. Select a chart version (use the default version), then configure the following parameters: `exporters` configuration — replace <internal endpoint> with the endpoint from Step 1:

   Parameter	Value	Description
   mode	"daemonset"	Deploys the Collector as a DaemonSet so that every node gets a local agent.
   enableXtraceTokenFromSecret	true	Reads the authentication token from the Secret via environment variables instead of storing it as plaintext. Make sure you have completed Step 2 before enabling this option.
   exporters	See configuration below	Adds an otlp exporter pointing to the Managed Service for OpenTelemetry endpoint.
   receivers	See configuration below	Adds an otlp/kubelet receiver listening on port 4317 for kubelet trace data.
   service.pipelines.traces	See configuration below	Connects the otlp/kubelet receiver to the otlp exporter through a processing pipeline.
   hostNetwork	true	Lets the Collector use the host's IP address and port for communication.

   exporters:
     otlp:
       endpoint: <internal endpoint>
       tls:
         insecure: true
       headers:
         Authentication: ${env:OPENTELEMETRY_GRPC_TOKEN}

   `receivers` configuration:

   receivers:
     otlp/kubelet:
       protocols:
         grpc:
           endpoint: 127.0.0.1:4317

   `service.pipelines.traces` configuration:

   service:
     pipelines:
       traces:
         exporters:
           - debug
           - otlp
         processors:
           - memory_limiter
           - batch
           - resource
         receivers:
           - otlp
           - jaeger
           - zipkin
           - otlp/kubelet

   `hostNetwork` configuration:

   hostNetwork: true

5. Install the chart. The page redirects to the Helm page after installation completes.

Verify the installation: In the left-side navigation pane, choose Workloads > DaemonSets. Select the component namespace. If a DaemonSet named opentelemetry-collector-agent appears with all pods in Running status, the Collector is installed successfully.

Step 4: Enable tracing for kubelet

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the cluster name. In the left-side navigation pane, choose Nodes > Node Pools.

3. Find the target node pool. In the Actions column, click > Kubelet Configuration.

4. In the Custom Parameters section, click + Custom Parameters to add the following two parameters:

   Depending on your overall configuration, Tracing Analysis may have an impact on cluster network and CPU performance. In case of performance problems, you can reduce the sample rate according to the load of the cluster.

   Name	Sub-parameter	Value	Description
   tracing	endpoint	localhost:4317	Sends kubelet trace data to the otlp/kubelet receiver configured in Step 3.
   tracing	samplingRatePerMillion	See sampling rate table below	Controls what fraction of requests are sampled.

   Sampling rate values:

   Value	Sampling rate
   1000000	100% (all requests)
   100000	10%
   10000	1%

   

5. Select the nodes to update, configure Batch Update Policy (Maximum Number of Nodes to Repair per Batch), and click submit.

Step 5: View tracing data

1. Log on to the Managed Service for OpenTelemetry console. In the left-side navigation pane, click Applications.

2. Select a region at the top of the page, then click the kubelet application name to view trace information. The application page includes the following tabs. For full details, see Application details.

   • Trace Explorer: Shows the trace and topology of kubelet requests. Enter resources.k8s.cluster.name : "ClusterId" in the search box to filter by cluster. Click a trace ID to see details including the topology view, service count, and interface call count. > Note: Get the cluster ID from the cluster list in the ACK console.

   • Provided Services: Shows the request count, error count, and average duration for each interface provided by kubelet.

   • Dependencies: Shows requests to services that kubelet depends on, such as containerd, including request count, error count, and average duration.

What's next

To monitor and record traces for the API server of a cluster, see Enable Tracing Analysis for cluster control plane components.