Enable distributed tracing in ASM - Alibaba Cloud Service Mesh

Service Mesh (ASM) integrates with Managed Service for OpenTelemetry. Managed Service for OpenTelemetry provides a wide range of tools to help you efficiently identify the performance bottlenecks of distributed applications. For example, you can use these tools to view trace data, display trace topologies, analyze application dependencies, and count the number of requests. This helps you improve the efficiency of developing and troubleshooting distributed applications. This topic describes how to enable distributed tracing in ASM.

Prerequisites

Managed Service for OpenTelemetry is activated for your Alibaba Cloud account. For more information about billing of this service, see Billing rules.
An ASM instance is created and Managed Service for OpenTelemetry is enabled for the instance. For more information, see Collect ASM tracing data to Managed Service for OpenTelemetry.

Background information

Distributed tracing can be used to profile and monitor applications, especially those built using a microservices model. Although Istio proxies can automatically send spans, they need some hints to tie together the entire trace. Applications need to propagate appropriate HTTP headers so that the spans sent by Istio proxies can be correctly correlated into a single trace. To do this, applications need to collect the following headers and propagate them from inbound requests to all outbound requests:

x-request-id
x-b3-traceid
x-b3-spanid
x-b3-parentspanid
x-b3-sampled
x-b3-flags
x-ot-span-context

Deployment example

Deploy applications in an ASM instance, and then check the deployed applications. You can find that all applications use the required HTTP headers. For example, the following sample Python application named productpage extracts the required headers from an HTTP request by using OpenTracing libraries:

def getForwardHeaders(request):
    headers = {}

    # x-b3-*** headers can be populated using the opentracing span
    span = get_current_span()
    carrier = {}
    tracer.inject(
        span_context=span.context,
        format=Format.HTTP_HEADERS,
        carrier=carrier)

    headers.update(carrier)

    # ...

    incoming_headers = ['x-request-id']

    # ...

    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val

    return headers

The following sample Java application named reviews also extracts the HTTP headers:

@GET
@Path("/reviews/{productId}")
public Response bookReviewsById(@PathParam("productId") int productId,
                            @HeaderParam("end-user") String user,
                            @HeaderParam("x-request-id") String xreq,
                            @HeaderParam("x-b3-traceid") String xtraceid,
                            @HeaderParam("x-b3-spanid") String xspanid,
                            @HeaderParam("x-b3-parentspanid") String xparentspanid,
                            @HeaderParam("x-b3-sampled") String xsampled,
                            @HeaderParam("x-b3-flags") String xflags,
                            @HeaderParam("x-ot-span-context") String xotspan) {

  if (ratings_enabled) {
    JsonObject ratingsResponse = getRatings(Integer.toString(productId), user, xreq, xtraceid, xspanid, xparentspanid, xsampled, xflags, xotspan);

Access example

Enter a URL in the format of http://{IP address of the ingress gateway}/productpage in the address bar of your browser, and then press the Enter key. You are navigated to the page of the Bookinfo application.

View applications

On the Applications page of the Managed Service for OpenTelemetry console, you can view the key metrics of all monitored applications, including the number of requests and errors in the current day and the health status. You can also configure labels for applications so that you can filter applications by label.

Log on to the Managed Service for OpenTelemetry console.
In the left-side navigation pane, click Applications. At the top of the Applications page, select a region as needed.

View application details

On the Application Details page of an application in the Managed Service for OpenTelemetry console, you can view the key metrics, call topology, and traces of the application on each server on which it is deployed.

Log on to the Managed Service for OpenTelemetry console.
In the left-side navigation pane, click Applications. At the top of the Applications page, select a region as needed. Then, click the name of the application that you want to check.
In the left-side navigation pane, click Application Details. Then, click All or select a server, represented by an IP address, from the server list on the left.
On the Overview tab, you can view the call topology and key metrics. You can also click the Traces tab to view the traces of the application on the selected server. A maximum of 100 traces are listed in descending order of the amount of time consumed.

View the waterfall chart of a trace

The waterfall chart of a trace shows the information of the trace, including the log generation time, status, IP address or server name, service name, and timeline.

On the Application Details page of the application that you want to check, click the Traces tab. In the trace list, click the ID of the trace that you want to check.
On the page that appears, view the waterfall chart of the trace.
Note
- The IP Address column may display IP addresses or server names. The information displayed depends on the display settings that are configured on the Application Settings page. For more information, see Manage applications and tags.
- You can move the pointer over a service name to view the information of the service, including the duration, start time, tags, and log events. For more information, see Application details.

FAQ

Why am I unable to view the trace information after I collect tracing data to Managed Service for OpenTelemetry in the ASM console?

View the trace push logs.
1. Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
2. Run the following command to view the trace push logs of the pods related to the tracing-on-external-zipkin deployment in the istio-system namespace:
```
kubectl logs "$(kubectl get pods -n istio-system  -l app=tracing -o jsonpath='{.items[0].metadata.name}')" -n istio-system -c  nginx
```
  The status code 406 is recorded in the trace push logs.
View the quota on the number of service requests per day and the actual number of service requests on the previous day.
1. Log on to the Managed Service for OpenTelemetry console.
2. In the left-side navigation pane, click Cluster Configurations to view the quota on the number of service requests per day and the actual number of service requests on the previous day.
You can find that the actual number of service requests on the previous day is greater than the quota on the number of service requests per day that you set in the Managed Service for OpenTelemetry console.
Modify the quota on the number of service requests per day.
If the actual number of service requests in a day is greater than the quota on the number of service requests per day, the reported data is discarded and no trace information is displayed. To resolve the issue, you must increase the quota on the number of service requests per day to make it greater than the actual number of service requests in a day.
1. In the left-side navigation pane of the Managed Service for OpenTelemetry console, click Cluster Configurations.
2. On the Cluster Configurations tab, modify the value of the Configuration quota parameter in the Quota configuration section. Increase the value to make it greater than the actual number of service requests in a day. Then, click Save.
3. In the Cue message that appears, click OK.

Why is the traceid changed after the request is processed by the ingress gateway or sidecar?

The possible cause is that the b3 headers of the request initiated by your application are incomplete, and therefore, Envoy believes that trace information is incomplete and regenerates b3 headers. If you want to use a custom traceid, include at least headers x-b3-trace-id and x-b3-spanid in your request.