Use the ASM fallback mechanism - Alibaba Cloud Service Mesh

A fallback mechanism defines an alternative action when a service call fails. If a microservice fails or becomes unavailable, the mechanism calls a backup service to handle requests, ensuring system stability and availability. For example, if a service endpoint is unavailable, you can use a fallback mechanism to forward requests to a backup service version. This ensures client requests are handled without errors or interruptions. ASM supports this mechanism through the fallback parameter in a VirtualService. This topic describes how to use the fallback mechanism in ASM.

Prerequisites

You have an ASM instance of Enterprise Edition or Ultimate Edition, version 1.17.2.22 or later. For more information, see Create an ASM instance.
Note
If your ASM instance is earlier than version 1.17, update the instance to 1.17.2.22 or later, or submit a ticket for technical support. For more information about how to update an instance, see Upgrade an ASM instance.
An ACK cluster has been added to the ASM instance. For more information, see Add a cluster to an ASM instance.
The Bookinfo sample is deployed. For more information, see Deploy an application in a cluster associated with an ASM instance.
The sidecar version on the data plane is 1.17 or later.
Note
You can view the sidecar version of each business pod on the Instances Status page on the ASM console. For more information, see Upgrade management.

Configuration

This topic uses the reviews service from the Bookinfo sample application as an example. When the productpage service accesses the v1, v2, and v3 versions of the reviews service, if the v3 version is unavailable, the fallback mechanism routes requests to the v2 version. This prevents the service from returning a 503 error.

You can click Configuration file to download the YAML files used in this topic.

Step 1: Access the Bookinfo sample

Create a file named reviews.yaml with the following content to declare the v1, v2, and v3 versions of the reviews service.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3

In your KubeConfig environment, run the following command to deploy the DestinationRule.
```
kubectl apply -f reviews.yaml
```
Use one of the following methods to obtain the IP address of the ingress gateway.
- Method 1: Run the following command.
```
kubectl get svc -n istio-system  istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```
- Method 2: Obtain the IP address of the ingress gateway from the Ingress Gateway page on the ASM console. For more information, see Obtain the endpoint of an ASM gateway.
In a browser, visit http://${YourGatewayIp}/productpage.
${YourGatewayIp} is the gateway IP that you obtained in the previous step. You can identify the version by the value of Reviews served by or the stars. Version v1 has no stars, version v2 has black stars, and version v3 has red stars.
For example, the value reviews-v2 indicates version v2, which displays black stars.
Refresh the page multiple times. Requests are now load-balanced across the v1, v2, and v3 versions of the reviews service.

Step 2: Define a route and fallback rule

Create a file named reviews-route-fallback-sample1.yaml with the following content.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-route
  namespace: default
spec:
  hosts:
    - reviews
  http:
    - route:
        - destination:
            host: reviews
            subset: v3
          fallback:
            target:
              host: reviews
              subset: v2

In your KubeConfig environment for the ASM instance, run the following command to deploy the route and fallback rule for the reviews service.
```
kubectl apply -f reviews-route-fallback-sample1.yaml
```
In a web browser, visit http://${YourGatewayIp}/productpage and keep refreshing the page.
You will see that requests are consistently routed to the v3 version of the reviews service. After refreshing, the page shows that the book review service is provided by reviews-v3, and the review includes red star ratings.
Simulate a failure of the reviews-v3 version by scaling its replicas to 0:
```
kubectl scale deployment reviews-v3 --replicas=0
```

In your browser, visit http://${YourGatewayIp}/productpage and repeatedly refresh the page.

You will see that requests correctly fall back to the v2 version of the reviews service. You can verify that a fallback occurred by adding fallback-related fields to the custom access log format and then checking the logs.

Verify the fallback

Add the following fields to the custom access log format. For more information, see Customize the content of data plane access logs.

Field	Value	Description
fallback_path	%DYNAMIC_METADATA(com.aliyun.fallback:fallback-path)%	The specific path of the fallback. For example, `A:B` indicates that requests fall back from A to B. `A:B:C` indicates that requests fall back from A to B, and then from B to C if B is also unhealthy.
fallback_final_cluster_name	%DYNAMIC_METADATA(com.aliyun.fallback:final-cluster)%	If a fallback occurs, this is the destination cluster. For example, if `service1\|v1` does not exist, requests fall back to `service\|base`.
fallback_result	%DYNAMIC_METADATA(com.aliyun.fallback:fallback-result)%	The result of the fallback. If the fallback fails, the request failure is attributed to the original destination cluster.

View the logs of the istio-proxy for productpage-v1.

The following log shows a fallback from reviews-v3 to reviews-v2:

{
    "authority":"reviews:9080",
    "authority_for":"reviews:9080",
    "bytes_received":"0",
    "bytes_sent":"442",
    "downstream_local_address":"192.168.255.46:9080",
    "downstream_remote_address":"172.16.0.252:57238",
    "duration":"10",
    "fallback_path":"outbound|9080|v3|reviews.default.svc.cluster.local:outbound|9080|v2|reviews.default.svc.cluster.local",
    "fallback_final_cluster_name":"outbound|9080|v2|reviews.default.svc.cluster.local",
    "fallback_result":"fallback successful",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/reviews/0",
    "protocol":"HTTP/1.1",
    "request_id":"15b2dffc-5f3f-4060-b9fa-898eab08****",
    "requested_server_name":"-",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"-",
    "start_time":"2023-05-30T07:02:26.990Z",
    "trace_id":"18b3aed8af41****",
    "upstream_cluster":"outbound|9080|v2|reviews.default.svc.cluster.local",
    "upstream_host":"172.16.0.11:9080",
    "upstream_local_address":"172.16.0.252:44448",
    "upstream_service_time":"9",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"-"
}

You can find the following new fields in the log.

"fallback_path":"outbound|9080|v3|reviews.default.svc.cluster.local:outbound|9080|v2|reviews.default.svc.cluster.local"
"fallback_final_cluster_name":"outbound|9080|v2|reviews.default.svc.cluster.local"
"fallback_result":"fallback successful"

The log shows the original destination was v3. Because the v3 instance was unavailable, the request fell back to v2.

Step 3: Configure fallback with weighted routing

Run the following command to make the reviews-v3 version available again.
```
kubectl scale deployment reviews-v3 --replicas=1
```

Create a file named reviews-route-fallback-sample2.yaml with the following content to modify the reviews-route definition.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-route
  namespace: default
spec:
  hosts:
    - reviews 
  http:
    - route:
        - destination:
            host: reviews 
            subset: v3
          fallback:
            target:
              host: reviews 
              subset: v2
          weight: 50
        - destination:
            host: reviews 
            subset: v2
          fallback:
            target:
              host: reviews 
              subset: v1
          weight: 50
      retries:
        attempts: 0

Run the following command to deploy the new route and fallback rule for the reviews service.
```
kubectl apply -f reviews-route-fallback-sample2.yaml
```
In your browser, visit http://${YourGatewayIp}/productpage and repeatedly refresh the page.
You will see that requests are routed to the v2 and v3 versions of the reviews service at a 50:50 ratio. In this example, retries are disabled to make the result clearer.
Run the following command to scale the replicas of v3 to 0 and verify that its fallback rule works as expected.
```
kubectl scale deployment reviews-v3 --replicas=0
```
Refresh the productpage page multiple times. You will see that requests are consistently routed to the v2 version, which is the expected behavior.
Run the following command to scale the replicas of v2 to 0.
```
kubectl scale deployment reviews-v2 --replicas=0
```
If you repeatedly refresh the productpage page, you will find that it fails to access the reviews service about 50% of the time. The other 50% of the time, requests are sent to the v2 version. Because the v2 version is unhealthy, these requests fall back to the v1 version. After you run the command and access the BookInfo application's product page, the reviews section displays the red error title Error fetching product reviews! and the message Sorry, product reviews are currently unavailable for this book.. This indicates that the product reviews service becomes unavailable after the reviews-v2 replicas are scaled down to 0.

Run the following command to view the logs.

kubectl logs -f  deployment/productpage-v1  -c istio-proxy --tail=10

Expected output:

{
    "authority":"reviews:9080",
    "authority_for":"reviews:9080",
    "bytes_received":"0",
    "bytes_sent":"19",
    "downstream_local_address":"192.168.255.46:9080",
    "downstream_remote_address":"172.16.0.252:47738",
    "duration":"0",
    "fallback_path":"outbound|9080|v3|reviews.default.svc.cluster.local:outbound|9080|v2|reviews.default.svc.cluster.local",
    "fallback_final_cluster_name":"-",
    "fallback_result":"fallback cluster is unhealthy",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/reviews/0",
    "protocol":"HTTP/1.1",
    "request_id":"b207a764-b6d7-4ef8-bc71-59f264c3****",
    "requested_server_name":"-",
    "response_code":"503",
    "response_flags":"UH",
    "route_name":"-",
    "start_time":"2023-05-30T07:32:08.999Z",
    "trace_id":"a40c32a7b2cf****",
    "upstream_cluster":"outbound|9080|v3|reviews.default.svc.cluster.local",
    "upstream_host":"-",
    "upstream_local_address":"-",
    "upstream_service_time":"-",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"-"
}

You can see 503 logs for productpage-v1. Based on the reviews-route weighted routing configuration, 50% of requests from productpage are routed to the v3 version of the reviews service. Because the v3 version is unavailable, the sidecar (istio-proxy) attempts to fall back from v3 to the v2 version based on a fallback rule. Because the v2 version is also unhealthy, the request is sent to the v3 version. You can confirm this by checking the "upstream_cluster":"outbound|9080|v3|reviews.default.svc.cluster.local" field.