How to export tracing data from ASM - Alibaba Cloud Service Mesh

Prerequisites

An ACK cluster has been added to the ASM instance. For more information, see Add a cluster to an ASM instance.
You have activated Managed Service for OpenTelemetry on your Alibaba Cloud account. For more information about billing, see Billing.
An ingress gateway has been deployed in the ASM instance. For more information, see Create an ingress gateway.

Export tracing data to Managed Service for OpenTelemetry

Select the tab that corresponds to your ASM instance version. To upgrade your instance, see Upgrade an ASM instance.

Versions earlier than 1.17.2.35

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
On the Base Information page, click Settings. In the Settings Update panel, select Enable Tracing Analysis, set the Sampling Percentage, set Sampling Method to Managed Service for OpenTelemetry, and then click OK.
In the left-side navigation pane, choose Observability Management Center > Tracing Analysis. You are redirected to the Managed Service for OpenTelemetry console. After completing the configuration, you can log on to the Managed Service for OpenTelemetry console and view monitoring metrics such as the number of requests, response times, and number of spans for your applications on the Overview Analysis page.

For more information about tracing, see What is Managed Service for OpenTelemetry?

Note

To disable this feature, deselect Enable Tracing Analysis in the Settings Update panel and then click OK.

Versions 1.17.2.35 to earlier than 1.18.0.124

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Tracing Analysis.
On the Tracing Analysis page, click Collect ASM Tracing Data to Managed Service for OpenTelemetry, and then in the Confirm dialog box, click OK.
Click Open the Managed Service for OpenTelemetry Console to view the tracing data.

For more information about tracing, see What is Managed Service for OpenTelemetry? After the configuration is complete, you can log on to the Managed Service for OpenTelemetry console and view monitoring metrics such as the number of requests, response times, and number of spans for your applications on the Overview Analysis page.

Note

If you no longer need this feature, on the Tracing Analysis page, click Disable Collection, and then in the Confirm dialog box, click OK.

Versions 1.18.0.124 to earlier than 1.22.6.89

For these instance versions, you cannot configure the export to Managed Service for OpenTelemetry directly in the console. Instead, you must manually deploy and configure an OpenTelemetry Collector.

Step 1: Deploy the OpenTelemetry Operator

In the kubeconfig environment for your ACK cluster, run the following command to create the opentelemetry-operator-system namespace:
```
kubectl create namespace opentelemetry-operator-system
```

Run the following command to install the OpenTelemetry Operator in the opentelemetry-operator-system namespace by using Helm:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install --namespace=opentelemetry-operator-system opentelemetry-operator open-telemetry/opentelemetry-operator \
  --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.autoGenerateCert.enabled=true

Run the following command to check if the OpenTelemetry Operator is running properly:

kubectl get pod -n opentelemetry-operator-system

Expected output:

NAME                                      READY   STATUS    RESTARTS   AGE
opentelemetry-operator-854fb558b5-pvllj   2/2     Running   0          1m

A STATUS of Running indicates that the OpenTelemetry Operator is running.

Step 2: Create the OpenTelemetry Collector

Create a file named collector.yaml with the following content.

In the YAML file, replace ${ENDPOINT} with the gRPC VPC access point and ${TOKEN} with the authentication token. To obtain the access point and authentication token for Managed Service for OpenTelemetry, see Access and authentication.

Collector.yaml

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: default
  namespace: opentelemetry-operator-system
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  config: |
    extensions:
      zpages:
        endpoint: 0.0.0.0:55679 
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    exporters:
      debug:
        verbosity: detailed
      otlp:
        endpoint: ${ENDPOINT}
        tls:
          insecure: true
        headers:
          Authentication: ${TOKEN}
    service:
      extensions: [zpages]
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp, debug]
  ingress:
    route: {}
  managementState: managed
  mode: deployment
  observability:
    metrics: {}
  podDisruptionBudget:
    maxUnavailable: 1
  replicas: 1
  resources: {}
  targetAllocator:
    prometheusCR:
      scrapeInterval: 30s
    resources: {}
  upgradeStrategy: automatic

In the kubeconfig environment for your ACK cluster, run the following command to deploy the collector to the cluster:
```
kubectl apply -f collector.yaml
```

Run the following command to check if the collector started properly:

kubectl get pod -n opentelemetry-operator-system

Expected output:

NAME                                      READY   STATUS    RESTARTS   AGE
opentelemetry-operator-854fb558b5-pvllj   2/2     Running   0          3m
default-collector-5cbb4497f4-2hjqv        1/1     Running   0          30s

This output indicates that the collector started successfully.

Run the following command to check if the service was created:

kubectl get svc -n opentelemetry-operator-system

Expected output:

opentelemetry-operator           ClusterIP   172.16.138.165   <none>        8443/TCP,8080/TCP   3m
opentelemetry-operator-webhook   ClusterIP   172.16.127.0     <none>        443/TCP             3m
default-collector              ClusterIP   172.16.145.93    <none>        4317/TCP   30s
default-collector-headless     ClusterIP   None             <none>        4317/TCP   30s
default-collector-monitoring   ClusterIP   172.16.136.5     <none>        8888/TCP   30s

The output indicates that the service was created successfully.

Step 3: Enable tracing in the ASM console

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.
On the Observability Settings page, in the Tracing Analysis Settings section, set Sampling Percentage to 100 and click Submit.
In the left-side navigation pane, choose Observability Management Center > Tracing Analysis. Set OpenTelemetry Service Address/Domain to default-collector.opentelemetry-operator-system.svc.cluster.local and OpenTelemetry Service Port to 4317. Then, click Collect ASM Tracing Data to Custom Managed Service for OpenTelemetry.

Versions 1.22.6.89 and later

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Tracing Analysis.
On the Tracing Analysis page, in the Export to Alibaba Cloud Observable Link section, select an Export Method, such as Zipkin. In the Fill in the configuration section, enter the required information and click Submit.
Click Go to Managed Service for OpenTelemetry console to view the tracing data. For more information about tracing, see What is Managed Service for OpenTelemetry?

Note
If you no longer need this feature, on the Tracing Analysis page, click Export, and then in the Confirm dialog box, click OK.

Export tracing data to a self-managed system

The export method for a self-managed system depends on your ASM instance version. Select the tab that corresponds to your instance version.

Versions earlier than 1.18.0.124

For ASM instances earlier than 1.17.2.28: Log on to the ASM console. On the Base Information page of the target instance, click Settings, select Enable Tracing Analysis, configure the parameters as needed, and then click OK.
For ASM instances of version 1.17.2.28 or later but earlier than 1.18.0.124: For information about how to enable tracing, see Tracing settings.

Versions 1.18.0.124 to earlier than 1.22.6.89

For this version range, configure the export to a self-managed system on the Observability Management Center > Tracing Analysis page of the target instance in the ASM console.

Parameters

Parameter	Description
OpenTelemetry Domain Name (FQDN)	The fully qualified domain name (FQDN) of your self-managed system service. Example: `otel.istio-system.svc.cluster.local`.
OpenTelemetry Service Port	The service port of the self-managed system. Example: `8090`.

Versions 1.22.6.89 and later

On the Observability Management Center > Tracing Analysis page of your instance in the ASM console, select and configure your self-managed tracing system.

Important

When exporting tracing data to a self-managed system, ensure it is a service within the mesh. If the system is outside the service mesh, you can register it with the mesh by creating a ServiceEntry. For more information, see ServiceEntry.

Parameters

Export to OpenTelemetry

Parameter		Description
Service domain name (full FQDN)		The FQDN of the self-managed system service. Example: `otel.istio-system.svc.cluster.local`.
Service Port		The service port of the self-managed system. Example: `8090`.
Export over gRPC	Timeout	Optional. The request timeout in seconds for exporting tracing data. Example: `1`. This feature is disabled by default.
Export over gRPC	Request Header	Optional. Request headers to include when exporting tracing data. Example: `authentication: token-xxx`. This field is empty by default.
Export over HTTP	Request path	The request path for exporting tracing data. Example: `/api/v2/spans`. The default value is `/`.
	Timeout	Optional. The request timeout in seconds for exporting tracing data. Example: `1`. This feature is disabled by default.
	Request Header	Optional. Request headers to include when exporting tracing data. Example: `authentication: token-xxx`. This field is empty by default.

Export to Zipkin

Parameter	Description
Domain name of Zipkin	The FQDN of the self-managed system service. Example: `zipkin.istio-system.svc.cluster.local`.
Request Port	The service port of the self-managed system. Example: `8090`.
Request Path	The request path for exporting tracing data. The default value is `/api/v2/spans`.

Export to SkyWalking

Parameter	Description
Skywalking service domain name	The FQDN of the self-managed system service. Example: `skywalking.istio-system.svc.cluster.local`.
Request Port	The service port of the self-managed system. Example: `8090`.

Verify the data export

Deploy applications

Deploy the sample Bookinfo application.

Create bookinfo.yaml with the following contentbookinfo.yaml。

Expand to viewYAMLYAML content

apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
    service: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-details
  labels:
    account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-details-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: ratings
  labels:
    app: ratings
    service: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-ratings
  labels:
    account: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v1
  labels:
    app: ratings
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v1
  template:
    metadata:
      labels:
        app: ratings
        version: v1
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
      - name: ratings
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-ratings-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  labels:
    app: reviews
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-reviews-v1:1.20.1
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v2
  labels:
    app: reviews
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v2
  template:
    metadata:
      labels:
        app: reviews
        version: v2
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-reviews-v2:1.20.1
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v3
  labels:
    app: reviews
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v3
  template:
    metadata:
      labels:
        app: reviews
        version: v3
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-reviews-v3:1.20.1
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
    service: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-productpage
  labels:
    account: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9080"
        prometheus.io/path: "/metrics"
      labels:
        app: productpage
        version: v1
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-productpage-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir: {}
---

Deploy the Bookinfo application to the data plane cluster.

kubectl --kubeconfig=${DATA_PLANE_KUBECONFIG} apply -f bookinfo.yaml

Deploy the sleep application.

Create a file named sleep.yaml with the following content.

Sleep.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
---

Deploy the sleep application.

kubectl --kubeconfig=${DATA_PLANE_KUBECONFIG} apply -f sleep.yaml

Send test requests

Run the following command to send test requests:

 kubectl exec -it deploy/sleep -- sh -c 'for i in $(seq 1 100); do curl -s productpage:9080/productpage > /dev/null; done'

View the exported data

The following steps show the result of exporting data to Managed Service for OpenTelemetry.

Log on to the Managed Service for OpenTelemetry console.
In the left-side navigation pane, click Application List. You can see the exported data.

The Application List page displays monitoring data for each service, including its Name (such as sleep.default, details.default, productpage.default, reviews.default, and ratings.default), Health, Today's Requests, Today's Errors, Response Time, and the Response Time (Last 30 Min) trend.