How to implement unified tracing for applications - Alibaba Cloud Service Mesh

Alibaba Cloud Service Mesh (ASM) provides out-of-the-box tracing capabilities for applications within the service mesh. However, to achieve end-to-end tracing that includes external applications communicating with in-mesh services, you must use Managed Service for OpenTelemetry to unify the call traces. This guide shows how to use Managed Service for OpenTelemetry to trace a request from an external application to an application inside an ASM instance, creating a single, unified call trace.

Prerequisites

A Service Mesh (ASM) instance is created. The ACK cluster is added to the ASM instance. For more information, see Create an ASM instance and Add a cluster to an ASM instance.
An ingress gateway is deployed in the ASM instance to allow access to in-mesh applications. For more information, see Create an ingress gateway.
An environment with Python 2.7 installed is available for running the out-of-mesh sample application.
Managed Service for OpenTelemetry is enabled. For more information, see Export tracing data from Service Mesh (ASM).

Notes

This guide deploys a Python application named ExternalProxy. This application runs an HTTP server. Accessing the root path of ExternalProxy calls the productpage service of the Bookinfo application running inside the service mesh.
This guide uses Managed Service for OpenTelemetry as the tracing backend. If you use a self-managed tracing system compatible with Zipkin, expose its reporting endpoint to the external application. You can then skip Step 2: Get the endpoint for Managed Service for OpenTelemetry and proceed directly to Step 3: Deploy the out-of-mesh ExternalProxy application, using your system's endpoint as the reporting address.

Step 1: Deploy the in-mesh Bookinfo application

Run the following command to deploy the Bookinfo application to the cluster in your ASM instance.
You can download the YAML file for the Bookinfo application from GitHub.
```
kubectl --kubeconfig=${DATA_PLANE_KUBECONFIG} apply -f bookinfo.yaml
```
Run the following command to deploy the VirtualService for the Bookinfo application.
You can download the YAML file for the VirtualService from GitHub.
```
kubectl --kubeconfig=${ASM_KUBECONFIG} apply -f virtual-service-all-v1.yaml
```
Run the following command to deploy the DestinationRule for the Bookinfo application.
You can download the YAML file for the DestinationRule from Github.
```
kubectl --kubeconfig=${ASM_KUBECONFIG} apply -f destination-rule-all.yaml
```
Run the following command to deploy the Gateway for the Bookinfo application.
You can download the YAML file for the Gateway from GitHub.
```
kubectl --kubeconfig=${ASM_KUBECONFIG} apply -f bookinfo-gateway.yaml
```

Step 2: Get the OpenTelemetry endpoint

Log on to the Managed Service for OpenTelemetry console. In the left-side navigation pane, click Overview.
On the Overview page, click the Access process tab, and then click View access point information.
View the endpoint.
1. At the top of the page, select the target region.
  
  You must select the region where the ACK cluster for the Bookinfo application is deployed. Otherwise, the tracing data cannot be correlated.
2. On the Access point information tab, turn on the Show token switch. Choose a public network or private network endpoint based on where your ExternalProxy application is deployed.
  
  Because ExternalProxy reports data using the Zipkin v1 API, you need a v1 endpoint. Click the Zipkin tab and find the appropriate endpoint in the table. Typically, you should use a v2 endpoint. However, for specific scenarios like Istio or Molten integrations, a v1 endpoint is required. When using the Sleuth component, the base URL must not include /api/v2/spans.

Step 3: Deploy the out-of-mesh ExternalProxy application

Important

The sidecar proxy forwards all parts of an incoming request, including HTTP headers and the request body, to the upstream service. This means that tracing-related headers are also forwarded.
To ensure trace context propagation, your application must forward the required tracing headers in its outbound requests, following standard community practices.

Save the following code as a file named ExternalProxy.py in your external application's runtime environment.

Replace {XTRACE_ZIPKIN_V1_ENDPOINT} with the endpoint from the previous step. If you are reporting to a self-managed tracing system, use its reporting endpoint instead.

Replace {INGRESS_GATE_WAY_IP} with the IP address of your ACK cluster's ingress gateway.

ExternalProxy.py

import requests
from flask import Flask
from py_zipkin.zipkin import zipkin_span,create_http_headers_for_new_span
from py_zipkin.util import generate_random_64bit_string
from py_zipkin.util import ZipkinAttrs
import time
app = Flask(__name__)
def do_stuff(trace_id, span_id, parent_span_id):
    time.sleep(2)
    headers = create_http_headers_for_new_span()
    headers["X-B3-TraceId"] = trace_id
    headers["X-B3-SpanId"] = span_id
    headers["X-B3-ParentSpanId"] = parent_span_id
    print "SEND TO INGRESS HEADERS : {0}".format(headers)
    r = requests.get('http://${INGRESS_GATE_WAY_IP}/productpage', headers=headers)
    return 'OK'
def http_transport(encoded_span):
    # encoding prefix explained in https://github.com/Yelp/py_zipkin#transport
    body=encoded_span
    zipkin_url="${XTRACE_ZIPKIN_V1_ENDPOINT}"
    headers = {"Content-Type": "application/x-thrift"}
    # You'd probably want to wrap this in a try/except in case POSTing fails
    r=requests.post(zipkin_url, data=body, headers=headers)
    print(body)
@app.route('/')
def index():
    with zipkin_span(
        service_name='external-proxy',
        span_name='external-proxy/inbound',
        transport_handler=http_transport,
        port=5000,
        sample_rate=100, 
    ) as inbound_span:
    do_stuff(inbound_span.zipkin_attrs.trace_id, inbound_span.zipkin_attrs.span_id, inbound_span.zipkin_attrs.parent_span_id)
    return 'OK', 200
if __name__=='__main__':
    app.run(host="0.0.0.0",port=5000,debug=True)

Run the following command to start the ExternalProxy application.

python ExternalProxy.py

 * Serving Flask app "main" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 317-792-686

Run the following command to send a request to the ExternalProxy application.
```
curl localhost:5000
```
Expected output:
```
OK
```

Step 4: View the trace information

Log on to the Managed Service for OpenTelemetry console.
In the left-side navigation pane, click Applications. Select the region at the top of the page, and then click ExternalProxy in the application list.
In the left-side navigation pane, click Application details. Click the Traces tab, and then click a Trace ID to view the details of the call trace.

A complete call trace is now formed between the external application, ExternalProxy, and the in-mesh Bookinfo application. The top of the trace details page displays summary information, including the Start time, Duration, Services, Depth, and Total spans. Below the summary, a tree view shows the hierarchical relationship of each span. The columns include Span name, Timeline, Service name, Start time, IP address, and Status. In this example, the call trace starts from external-proxy, passes through productpage, and then calls the details, reviews, and ratings services in sequence. Some spans, such as the ahas_rate_limit_cluster rate-limiting span, may appear with an error status.