Alibaba Cloud Service Mesh:Configure ASM to report tracing data

Connect to your ACK cluster with kubectl and create the namespace:

   kubectl create namespace opentelemetry-operator-system

Install the OpenTelemetry Operator with Helm: This skips cert-manager and auto-generates certificates for the admission webhooks.

   helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
   helm install --namespace=opentelemetry-operator-system opentelemetry-operator open-telemetry/opentelemetry-operator \
     --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" \
     --set admissionWebhooks.certManager.enabled=false \
     --set admissionWebhooks.autoGenerateCert.enabled=true

Verify the Operator pod is running: Expected output: Both containers in the pod should show Running status.

   kubectl get pod -n opentelemetry-operator-system

   NAME                                      READY   STATUS    RESTARTS   AGE
   opentelemetry-operator-854fb558b5-pvllj   2/2     Running   0          1m

Create a collector.yaml file with the following content.

Replace the placeholders with your actual values:

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     labels:
       app.kubernetes.io/managed-by: opentelemetry-operator
     name: default
     namespace: opentelemetry-operator-system
     annotations:
       sidecar.istio.io/inject: "false"
   spec:
     config: |
       extensions:
         zpages:
           endpoint: 0.0.0.0:55679
       receivers:
         otlp:
           protocols:
             grpc:
               endpoint: 0.0.0.0:4317
       exporters:
         debug:
           verbosity: detailed
         otlp:
           endpoint: ${ENDPOINT}
           tls:
             insecure: true
           headers:
             Authentication: ${TOKEN}
       service:
         extensions: [zpages]
         pipelines:
           traces:
             receivers: [otlp]
             processors: []
             exporters: [otlp, debug]
     ingress:
       route: {}
     managementState: managed
     mode: deployment
     observability:
       metrics: {}
     podDisruptionBudget:
       maxUnavailable: 1
     replicas: 1
     resources: {}
     targetAllocator:
       prometheusCR:
         scrapeInterval: 30s
       resources: {}
     upgradeStrategy: automatic

Key configuration details:

• The sidecar.istio.io/inject: "false" annotation prevents ASM from injecting a sidecar into the Collector pod, which would cause a routing loop.

• The debug exporter logs trace data to stdout for troubleshooting. Remove it after verifying the setup.

• The trace pipeline routes spans from the OTLP receiver to both the OTLP exporter (Managed Service for OpenTelemetry) and the debug exporter.

Deploy the Collector:

   kubectl apply -f collector.yaml

Verify the Collector pod is running: Expected output:

   kubectl get pod -n opentelemetry-operator-system

   NAME                                      READY   STATUS    RESTARTS   AGE
   opentelemetry-operator-854fb558b5-pvllj   2/2     Running   0          3m
   default-collector-5cbb4497f4-2hjqv        1/1     Running   0          30s

Verify the Collector service exists: Expected output: The default-collector service on port 4317 receives trace data from ASM.

   kubectl get svc -n opentelemetry-operator-system

   opentelemetry-operator           ClusterIP   172.16.138.165   <none>        8443/TCP,8080/TCP   3m
   opentelemetry-operator-webhook   ClusterIP   172.16.127.0     <none>        443/TCP             3m
   default-collector              ClusterIP   172.16.145.93    <none>        4317/TCP   30s
   default-collector-headless     ClusterIP   None             <none>        4317/TCP   30s
   default-collector-monitoring   ClusterIP   172.16.136.5     <none>        8888/TCP   30s

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

On the Mesh Management page, click your ASM instance name. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

In the Tracing Analysis Settings section, set Sampling Percentage to 100 and click Submit.

In the left-side navigation pane, choose Observability Management Center > Tracing Analysis. Configure the following fields:

Click Collect ASM Tracing Data to Managed Service for OpenTelemetry.

Deploy the Bookinfo and sleep test applications. For more information, see Deploy an application in an ACK cluster that is added to an ASM instance.

• bookinfo.yaml

• sleep.yaml

  sleep.yaml

     apiVersion: v1
     kind: ServiceAccount
     metadata:
       name: sleep
     ---
     apiVersion: v1
     kind: Service
     metadata:
       name: sleep
       labels:
         app: sleep
         service: sleep
     spec:
       ports:
       - port: 80
         name: http
       selector:
         app: sleep
     ---
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: sleep
     spec:
       replicas: 1
       selector:
         matchLabels:
           app: sleep
       template:
         metadata:
           labels:
             app: sleep
         spec:
           terminationGracePeriodSeconds: 0
           serviceAccountName: sleep
           containers:
           - name: sleep
             image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
             command: ["/bin/sleep", "infinity"]
             imagePullPolicy: IfNotPresent
             volumeMounts:
             - mountPath: /etc/sleep/tls
               name: secret-volume
           volumes:
           - name: secret-volume
             secret:
               secretName: sleep-secret
               optional: true

Send a test request through the mesh:

   kubectl exec -it deploy/sleep -c sleep -- curl productpage:9080/productpage?u=normal

Check the Collector logs for exported spans: This confirms the Collector received and exported trace data.

   2023-11-20T08:44:27.531Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 3}

View traces in the Application Real-Time Monitoring Service (ARMS) console:

1. Log on to the ARMS console.

2. In the left-side navigation pane, choose Application Monitoring > Trace Explorer. Select your region in the upper-left corner.

3. In the Service Name section, select the sleep application. Tracing data appears on the right. ASM injects a sidecar proxy into the sleep application. When the application sends a request, the sidecar acts as the egress gateway and generates a span for each outbound call.

4. Find a trace ID and click Details in the Actions column to view the full trace with span-level latency.