This series of articles was written by Kai Yu. Shi Xie from Alibaba Cloud Container Service also contributed to this article.
In recent years, the trend of enterprise infrastructure cloud-native is becoming clearer. From the initial IaaS to the current microservice, the customer's demand for fine granularity and observability is stronger. Container networks have been developing at a high speed to meet customers' requirements for higher performance and higher density. This inevitably brings high barriers and challenges to customers' observability of cloud-native networks. In order to improve the observability of the cloud-native network and facilitate the readability of business links for customers and frontend and backend students, ACK and AES jointly developed the ack net-exporter and cloud-native network data plane observability series to help them understand the cloud-native network architecture system, simplify the observability threshold of the cloud-native network, and optimize the experience of customer operation and maintenance and after-sales students in handling difficult problems. This will improve the stability of the cloud-native network link.
△ ASM Example
△ Istio Data Plane Diagram
The emergence of Kubernetes breaks the boundaries of computing resources (such as underlying servers and underlying networks), bringing unlimited possibilities for flexible business deployment, rapid recovery, elastic scaling, and resource efficiency maximization. However, business scenarios are infinite. With the rapid development of microservices, businesses have more refined requirements for the same service, different versions, and traffic control. It is best to achieve traffic control, observability, etc., in the pod dimension. These are impossible on Kubernetes.
Is there a way to achieve complete business isolation between the above and microservices? The emergence of Istio brings a relatively perfect solution to this, allowing application users and developers to pay more attention to the development iteration of the business itself. Istio leverages the pod concept of Kubernetes. It automatically injects istio-proxy containers and initial containers when each injected pod is deployed based on user configurations. The purpose of the initial container is to modify the iptables rule of the individual network namespace of the pod, so the traffic that needs to be proxied enters the istio-proxy listening port and istio-proxy listens to two ports of inbound and outbound. According to the mesh configuration, the proxy implementation and intervention of the inbound and outbound traffic can be realized. The carriers injected by the same istio are all considered to be within the same service mesh. The calls between them have been separated from the service level and will hit the endpoint configured by the relevant istio cluster. As such, we can implement pod-based traffic management, observability, and security configurations.
This article is the sixth part of [Analysis of Container Network Data Link], which mainly introduces the forwarding links of data plane links in ASM Istio mode. First, it can detect the reasons for the performance of customer access results in different scenarios and help customers optimize the business architecture by understanding the forwarding links of the data plane in different scenarios. On the other hand, through an in-depth understanding of forwarding links, when encountering container network jitter, customer O&M and Alibaba Cloud developers can know which link points to deploy and observe manually to delimit the direction and cause of the problem.
By default, ASM provides a webhook controller to automatically inject sidecar proxies into the pods of applications. Run the following command to see that the cluster injected by ASM has a mutatingwebhookconfiguration of istio-sidecar-injector-1-15-3. View webhook, and you can see that one of them is istio-inject: the pod in the namespace with the enabled label will be automatically injected when it is created.
In addition to the namespace dimension, there are the pod dimension, other annotation methods, and other dimensions to determine whether the Kubernetes cluster is added to the Istio service mesh. ASM allows you to add a Container Service for the Kubernetes cluster to an ASM instance. You must inject a sidecar proxy into the pod of an application deployed in the ACK cluster to make full use of ASM. In addition to manual injection, we recommend enabling automatic injection to simplify deployment. ASM provides visualized injection configurations. Please see Enable Automatic Injection [1] for more information.
By describing the injected pods, you can find that there are two more containers in addition to the configured business container: istio-proxy and init container: istio-init. The images of the two containers are the same, but the commands they run are different. This has the advantage that only one image needs to be pulled, saving time for pulling images.
Init container takes advantage of Kubernetes features, a special container with privileges that runs before the application container in the Pod is started. Init containers can include utilities and installation scripts not present in some application images. Each Pod can contain multiple Containers and multiple Init Containers. It is similar to ordinary containers but has unique points.
Now that we understand the role of the Init container, let's take a look at what istio-init did during the startup process. You can use the following command:
kubectl logs -n istio-inject productpage-v1-797d845774-dndmk -c istio-init
It can be seen that istio-init has generated and configured a series of iptables rules during the startup process (such as outbound forwarding to port 15001, inbound forwarding to port 15006, access to port 15008, and direct return without traffic hijacking). Is there any way to customize the configuration? View the pod information to see the configured startup parameters. These redirect inbound and outbound traffic to the specified port based on the rules.
-p: All outbound traffic is redirected to port 15001 by iptables.
-z: All inbound traffic is redirected to port 15006 by iptables.
-u: Used to exclude user ID 1337, which can be regarded as the envoy application using UID 1337
-m: Traffic redirection mode, REDIRECT or TPROXY
-i: The range of addresses for outbound redirection. * indicates that all outbound traffic is redirected.
-x: Refers to the range of IP addresses that will be excluded from the outbound redirection.
-b: The list of redirected inbound ports
-d: The list of ports excluded from the redirected inbound ports
From the perspective of the pod, we regard the pod as a whole, which contains istio-proxy containers and application containers.
According to the preceding iptables rules, we can summarize the ports forwarded by the inbound proxy, such as 80. In the network namespace of the pod, the netfilter module passes through the following process: Client -> RREROUTING -> ISTIO_INBOUND -> ISTIO_IN_REDIRECT -> INPUT -> Envoy 15006(Inbound)-> OUTPUT -> ISTIO_OUT-> POSTROUTING -> APP. As such, inbound traffic is forwarded to the sidecar container and then to the listening port of the application container. Between steps 5 and 6, the traffic is processed according to the configured istio rules.
According to the preceding iptables rules, we can summarize the ports forwarded by the inbound proxy, such as 80. In the netfilter module of the network namespace of the pod, the process is APP > OUTPUT -> ISTIO_OUTPUT -> ISTIO_REDIRECT -> Envoy 15001(Outbound)-> OUTPUT -> ISTIO_OUTDST-> POSTROT. As such, outbound traffic is forwarded to the sidecar container and then to the destination listening port. Between steps d and e, the traffic is processed based on the configured Istio rules.
For some inbound ports or custom ports, we do not need it to pass through the sidecar container. Iptables rules will be set to prevent the qualified inbound traffic from being forwarded to the 15006 port and directly forward to the application container listening port RREROUTING -> ISTIO_INBOUND -> INPUT -> APP.
For some outbound ports or custom ports, we do not need it to pass through the sidecar container. Iptables rules will be set to prevent the qualified inbound traffic from being forwarded to the 15001 port and directly leaving the Pod network namespace APP -> OUTPUT -> ISTIO_OUTPUT -> POSTROUTING -> DST.
You can see that 15001 and 15006 are listened to by the Envoy application, which is an istio-proxy container program. When the Init container is started, the mode in which inbound and outbound traffic is redirected to Envoy is specified as REDIRECT or TPROXY. In REDIRECT mode, after a pod is injected with a sidecar proxy, all inbound traffic is redirected from Envoy. Envoy sends the traffic to the application bound to the local IP address (127.0.0.1). Therefore, the application cannot see the original IP address. How can we maintain the client source IP during service access in a service mesh environment? You can use the TPROXY mode. ASM supports the TPROXY mode. Please visit this link for more information
In TPROXY mode, the iptables of the network namespace of the pod have the mangle configuration.
We know service mesh will inject two containers within each injected Pod: istio-init and istio-proxy. Once the relevant configuration changes are made on the mesh control surface, they will be sent to each istio-proxy container through the pilot to take effect. Istio uses the xDS service interface to dynamically deliver relevant configurations. xDS includes Listener Discover Service (LDS), Cluster Discover Service (CDS), Endpoint Discovery Service (EDS), and Route Discover Service (RDS). In general, in the process of updating the configuration, you should update Cluster-> the endpoint of Cluster-> Listener corresponding to Cluster and Endpoint-> Route starts to update Listener information of the new configuration-> delete the cluster and endpoint not in use to ensure traffic is lossless during the update process. However, these xDS interfaces are independent of each other, so when the configuration is issued, some DS with certain dependencies causes some traffic to be discarded if the configurations take effect, which is unacceptable in some production environments.
Service mesh uses gRPC streams for ADS aggregation and discovery services to ensure the consistency of data plane configurations. A gRPC stream is used to ensure the call sequence of each xDS interface and avoid data configuration mismatch caused by the independence of each interface. Please refer to this link for more information
You can see that istio-proxy starts the pilot-agent program, and pilot-agent starts the child process /usr/local/bin/envoy as the parent process. /etc/istio/proxy/envoy-rev.json is the configuration file initialized by Envoy.
The information includes the node to which the istio-proxy belongs, the pod, the Istio version, the ID of the ACK cluster, the ASM version, and the required ports.
Istio-proxy related logs, management ports, and other information
ADS-related configuration information (such as the API protocol, version, and timeout period)
It contains five clusters (prometheus_stats, agent, sds-grpc, xds-grpc, and zipkin) and a listener that listens on 15090. xds-grpc cluster corresponds to the ADS configuration in the previous dynamic_resources. prometheus_stats cluster and 15090 are used to provide prometheus collection ports. The zipkin cluster is the calling address of the external zipkin server.
For distributed tracing analysis, collector_cluster here is the zipkin cluster defined in the previous static_resources.
From the section above, we know the traffic will be hijacked and processed by their respective istio-proxy when two injected pods access each other. As long as the istio-proxy logs of the client and server are analyzed and processed, the traffic can be interpreted for observability. We are still using official examples here. Access http://<gatewayIP>/productpage. The productpage application automatically calls the details service and the reviews service. The link between productpage and details is used as an example for analysis.
The productpage-v1-797d845774-dndmk IP is 10.0.1.130, the name of the svc where details are applied is details, the svc address is 192.168.1.125, and the svc port is 9080.
Request sender productpage-v1-797d845774-dndmk istio-proxy logs:
{"upstream_host":"10.0.1.127:9080","downstream_remote_address":"10.0.1.130:49586","downstream_local_address":"192.168.1.125:9080","duration":6,"upstream_cluster":"outbound|9080||details.istio-inject.svc.cluster.local","path":"/details/0","protocol":"HTTP/1.1","upstream_local_address":"10.0.1.130:50026","method":"GET","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","route_name":"default","request_id":"834147c2-435f-94a7-af11-8491df9ab4f8","start_time":"2023-01-31T14:23:20.603Z","upstream_transport_failure_reason":null,"upstream_service_time":"5","response_flags":"-","bytes_received":0,"authority_for":"details:9080","authority":"details:9080","requested_server_name":null,"istio_policy_status":null,"trace_id":"9712c9f3da936a8c927f227bfe536c16","response_code":200,"x_forwarded_for":null,"bytes_sent":178}Request accepter details-v1-6758dd9d8d-dtbdc istio-proxy logs:
{"x_forwarded_for":null,"start_time":"2023-01-31T14:23:20.608Z","method":"GET","response_flags":"-","route_name":"default","istio_policy_status":null,"requested_server_name":"outbound_.9080_._.details.istio-inject.svc.cluster.local","bytes_received":0,"request_id":"834147c2-435f-94a7-af11-8491df9ab4f8","response_code":200,"upstream_host":"10.0.1.127:9080","trace_id":"9712c9f3da936a8c927f227bfe536c16","downstream_remote_address":"10.0.1.130:50026","protocol":"HTTP/1.1","bytes_sent":178,"upstream_transport_failure_reason":null,"downstream_local_address":"10.0.1.127:9080","upstream_local_address":"127.0.0.6:46225","authority":"details:9080","authority_for":"details:9080","upstream_service_time":"0","upstream_cluster":"inbound|9080||","duration":1,"path":"/details/0","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"}Log Content Interpretation:
- "upstream_host":"10.0.1.127:9080", ---- For outbound, this is the address and port of an upstream endpoint.
- downstream_remote_address":"10.0.1.130:49586"," ---- For outbound, this is the pod-ip: random port 1.
- downstream_local_address":"192.168.1.125:9080","---- For outbound, this is the destination svc-ip:svc-port.
- duration":6," ---- The entire request time, in milliseconds.
- upstream_cluster":"outbound | 9080 | | details.istio-inject.svc.cluster.local",---- cluster information
- "path":"/details/0",
- "protocol":"HTTP/1.1",
- "upstream_local_address":"10.0.1.130:50026", ---- For outbound, this is the pod-ip: random port 2.
- "method":"GET",
- "user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
- "route_name":"default",---- Route name
- "request_id":"834147c2-435f-94a7-af11-8491df9ab4f8",
- "start_time":"2023-01-31T14:23:20.603Z",
- "upstream_transport_failure_reason":null,
- "upstream_service_time":"5",---- Upstream return request time, in ms.
- "response_flags":"-",---- Return flags, details about the connection or return.
- "bytes_received":0,
- "authority_for":"details:9080",
- "authority":"details:9080",
- "requested_server_name":null,
- "istio_policy_status":null,
- "trace_id":"9712 c9f3da936a8c927f227bfe536c16",---- This ID is a unique value and can be corresponded to istio-proxy in the upstream.
- "response_code":200,---- Return the status code.
- "x_forwarded_for":null
- "bytes_sent":178Please visit the official link for details about log interpretation.
The upstream host indicates the target of the request made from Envoy.
This value is upstream pod-ip : pod-port for an outbound cluster.This value is this pod-ip : pod-port for an inbound cluster.
The local address of the current envoy in the upstream connection. This value is pod-ip : random port 2 for the outbound cluster. This value is 127.0.0.6: random port 3 for the inbound cluster. Here, 127.0.0.6 corresponds to the iptables in [1.2 Pod Traffic Forwarding-Init Container], which will exempt the traffic from 127.0.0.6 from the istio proxy because this traffic is sent from the sidecar itself.
The local address of the current envoy in the downstream connection.
This value is destination service-ip : service-port for an outbound cluster. This value is current pod-ip : pod-port for an inbound cluster, which should correspond to the downstream upstream_host.
This value is current pod-ip : random port for an outbound cluster. This value is downstream pod-ip : random port 2 for an inbound cluster, which corresponds to the downstream upstream_local_address.
In the official example, use productpage to access the reviews service.
Through Kubernetes cluster resources, we can see that there are three versions of reviews: v1, v2, and v3, with one pod, respectively. The svc reviews are in ClusterIP mode, the svc port is 9080, and the targetport is the pod 9080 port. v1, v2, and v3 are added to the endpointslice of reviews svc. If the ProductPage pod in the cluster is not injected by Istio, the reviews.istio-inject service is evenly forwarded to the v1, v2, and v3 pods by netfilter in round-robin mode. Each pod should bear 1/3 of the traffic.
In a traditional Kubernetes cluster, Kubernetes resources cannot be used to control the traffic distribution of different versions. However, we have this demand in the actual production environment. For example, v1 is an online service version that carries major business traffic. v2 is a pre-online version that has been developed. In essence, it is not expected to affect online traffic. It may be necessary to divert 5% of online traffic to the pre-release version for a period to determine whether there is a problem with the new version. After that, the drainage ratio is expanded to 100% before v1 is offline, realizing smooth migration from the business perspective. For example, v3 is a test version, and we want to observe whether the self-disaster recovery and recovery behavior of the business meets expectations when the traffic exceeds the time limit due to network fluctuations. Previously, this requirement needed to write the fuse code in the business code, and a re-release is needed in different fuse environments. So, this kind of traffic control can be easily implemented in ASM Istio.
The following is a configuration of vs and dr in ASM Istio:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
creationTimestamp: '2023-01-30T06:25:21Z'
generation: 1
name: reviews
namespace: istio-inject
resourceVersion: '651722274'
uid: 63f715c9-b253-4fbb-8351-5313371df14e
spec:
hosts:
- reviews.istio-inject.svc.cluster.local
http:
- name: route
route:
- destination:
host: reviews
subset: v1
weight: 10
- destination:
host: reviews
subset: v2
weight: 40
- destination:
host: reviews
subset: v3
weight: 50The reviews vs define the rules for accessing reviews.istio-inject.svc.cluster.local http protocol in the cluster. It indicates v1 version weight 10%, v2 version weight 40%, and v3 version weight 50%.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
creationTimestamp: '2023-01-30T06:28:46Z'
generation: 2
name: reviews
namespace: istio-inject
resourceVersion: '654863578'
uid: fdbdfcea-1fcd-453e-96fb-ce41c91ded9b
spec:
host: reviews
subsets:
- labels:
version: v1
name: v1
- labels:
version: v2
name: v2
- labels:
version: v3
name: v3
trafficPolicy:
connectionPool:
http:
http2MaxRequests: 1000
maxRequestsPerConnection: 10
tcp:
maxConnections: 100
outlierDetection:
baseEjectionTime: 15m
consecutive5xxErrors: 7
interval: 5mThe reviews dr defines several versions of reviews in the cluster and defines related traffic policies. http2MaxRequests indicates the maximum number of HTTP requests. maxRequestsPerConnection indicates the maximum number of requests per connection. The maximum number of TCP connections is 100. In the circuit breaker configuration, the backend is detected once every 5min for 7 consecutive 5xx times, and the endpoint is removed for 15min.
The pilot sends the service mesh configuration to the istio-proxy in each injected pod through the xDS interface. Is there a way to view the relevant loaded configuration information for the istio-proxy in each pod? istio-proxy exposes the management port through port 15000. We can obtain the relevant configuration information through the command, as shown in the figure.
Among them, the complete configuration information can be obtained through curl 127.0.0.1:15000/config_dump. Since this configuration information exceeds more than 10000 lines, we will not show it all here. Those interested can study it by themselves. The following will make a relevant display and brief explanation for the key information (such as cluster, Listener, endpoint, and route) in this config_dump information. At the same time, it echoes the xDS mentioned above.
kubectl exec -n istio-inject productpage-v1-797d845774-dndmk -c istio-proxy -it -- curl 127.0.0.1:15000/config_dump
The initialization configuration of Envoy is the same as the preceding envoy-rev0.json. drainDuration -- The time (in seconds) that Envoy closes the connection during the hot restart. Default value: 45s.
parentShutdownDuration -- The maximum amount of time to wait before shutting down the parent process during a warm restart. Default value: 60s. This number should be greater than drainDuration.
terminationDrainDuration -- Default value: 5s. The time when the proxy is allowed to close the connection before it is closed. We know pod traffic passes through istio before it is forwarded to the application containers. When the application is released, if the existing connection is closed and the istio-proxy container is guaranteed to exit after the application container completely closes the existing connection, this value needs to be considered during the release update. This value needs to be considered to realize a smooth business update.
The static resources in config_dump are from envoy-rev0.json, which contains configurations (such as prometheus_stats, agent, sds-grpc, xds-grpc, and zipkin).
The dynamic resource is the configuration of ASM Istio that is delivered to each istio-proxy container through the xDS interface. The preceding reviews dr and vs configurations are also delivered through the ASM control side. We next focus on dynamic resource allocation.
The listener is used by Envoy to receive traffic requests to Envoy. Listener can be bound to IP Sock and Unix Domain Socket or not bound to specific ports. Instead, listener can receive traffic forwarded from other listeners. ASM Istio uses the Envoy listener feature to forward requests to different listeners.
For example, productpage accesses reviews, and productpage accesses the 9080 port of reviews. According to the preceding example, we know that when productpage container accesses an external 9080 port, it will be forwarded to the 15001 port first. Therefore, let's take a look at the 15001 port listeners.
Envoy creates listeners on the 15001 port. All requests forwarded by iptables are forwarded to the 15001 port of Envoy. As shown in the configuration, after Envoy receives the traffic, it will not do the relevant business action but according to configuration use_original_dst: true and the destination port of the request to the corresponding listeners for processing.
Then, there must be a question. If the destination port of the request is not configured with relevant listeners settings, how should the traffic be processed? This depends on the configuration of outboundTrafficPolicy. Please visit this link for more information.
| Name | Description |
| REGISTRY_ONLY | outbound traffic will be restricted to services defined in the service registry as well as those defined through ServiceEntries |
| ALLOW_ANY | outbound traffic to unknown destinations will be allowed, in case there are no services or ServiceEntries for the destination port |
The name of the outbound listener is 0.0.0.0_9080, which indicates that 9080 ports sent to any IP address are covered by this listener. bind_to_port: false indicates that the listener is not bound to the tcp port, and the traffic is forwarded with virtualOutbound. First of all, we need to distinguish whether this listener is for inbound or outbound direction. In the inbound direction, the traffic will pass through the listeners of the virtualInbound port 15006 but will not enter the listeners of 0.0.0.0_9080.
The configuration shows that there is no special filtering condition, indicating that any traffic is accepted, where config_source is ads, indicating this is from dynamic discovery.
The image above shows that the services of reviews, ratings, and details are all port 9080, and these applications are injected by the same service mesh. How does Envoy know which service is the 9080 of the destination address accessed by productpage? How can we judge if the address that is not 9080 by the destination port is not in the mesh? How can we deal with it? From the figure above route_config_name: 9080, we can see there is a 9080 routing rule, which specifies different processing results for different request destinations, which we will discuss in the next section.
As we learned earlier, the listeners outbound 0.0.0.0_9080 routing rule is used to route the 9080 port of the productpage application to the reviews 9080. The following is all the information about the 9080 route. The following five virtual_hosts are allow_any, details, productpage, ratings, and reviews. Among them, the last four correspond to four different outbound clusters and allow_any corresponds to PassthroughCluster. When an outbound request finds the corresponding Cluster rule, it will pass directly by default.
Some users may wonder why the productpage service does not directly call the ratings service and why the productpage Envoy configuration contains the ratings information. This is because the control plane of ASM Istio cannot detect how each service on the data plane is called. Therefore, ASM Istio sends all configuration information to the injected envoy to ensure the consistency of the mesh configuration. However, as the number of mesh service configurations increases, each envoy receives more configuration information that is not related to this envoy. This affects the use of envoy resources. If you have good Envoy development capabilities and are familiar with business calls, you can use sidecar rules to modify egress and ingress. Please visit this link for more information.
{
"version_info": "2023-01-30T06:25:21Z/19",
"route_config": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "9080",
"virtual_hosts": [
{
"name": "allow_any",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "PassthroughCluster",
"timeout": "0s",
"max_grpc_timeout": "0s"
},
"name": "allow_any"
}
],
"include_request_attempt_count": true
},
{
"name": "details.istio-inject.svc.cluster.local:9080",
"domains": [
"details.istio-inject.svc.cluster.local",
"details.istio-inject.svc.cluster.local:9080",
"details",
"details:9080",
"details.istio-inject.svc",
"details.istio-inject.svc:9080",
"details.istio-inject",
"details.istio-inject:9080",
"192.168.1.125",
"192.168.1.125:9080"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|9080||details.istio-inject.svc.cluster.local",
"timeout": "0s",
"retry_policy": {
"retry_on": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"num_retries": 2,
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
503
]
},
"max_stream_duration": {
"max_stream_duration": "0s",
"grpc_timeout_header_max": "0s"
}
},
"decorator": {
"operation": "details.istio-inject.svc.cluster.local:9080/*"
},
"name": "default"
}
],
"include_request_attempt_count": true
},
{
"name": "istio-ingressgateway.istio-system.svc.cluster.local:9080",
"domains": [
"istio-ingressgateway.istio-system.svc.cluster.local",
"istio-ingressgateway.istio-system.svc.cluster.local:9080",
"istio-ingressgateway.istio-system",
"istio-ingressgateway.istio-system:9080",
"istio-ingressgateway.istio-system.svc",
"istio-ingressgateway.istio-system.svc:9080",
"192.168.1.250",
"192.168.1.250:9080"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|9080||istio-ingressgateway.istio-system.svc.cluster.local",
"timeout": "0s",
"retry_policy": {
"retry_on": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"num_retries": 2,
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
503
]
},
"max_stream_duration": {
"max_stream_duration": "0s",
"grpc_timeout_header_max": "0s"
}
},
"decorator": {
"operation": "istio-ingressgateway.istio-system.svc.cluster.local:9080/*"
"name": "default"
}
],
"include_request_attempt_count": true
},
},
{
"name": "productpage.istio-inject.svc.cluster.local:9080",
"domains": [
"productpage.istio-inject.svc.cluster.local",
"productpage.istio-inject.svc.cluster.local:9080",
"productpage",
"productpage:9080",
"productpage.istio-inject.svc",
"productpage.istio-inject.svc:9080",
"productpage.istio-inject",
"productpage.istio-inject:9080",
"192.168.6.226",
"192.168.6.226:9080"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|9080||productpage.istio-inject.svc.cluster.local",
"timeout": "0s",
"retry_policy": {
"retry_on": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"num_retries": 2,
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
503
]
},
"max_stream_duration": {
"max_stream_duration": "0s",
"grpc_timeout_header_max": "0s"
}
},
"decorator": {
"operation": "productpage.istio-inject.svc.cluster.local:9080/*"
},
"name": "default"
}
],
"include_request_attempt_count": true
},
{
"name": "ratings.istio-inject.svc.cluster.local:9080",
"domains": [
"ratings.istio-inject.svc.cluster.local",
"ratings.istio-inject.svc.cluster.local:9080",
"ratings",
"ratings:9080",
"ratings.istio-inject.svc",
"ratings.istio-inject.svc:9080",
"ratings.istio-inject",
"ratings.istio-inject:9080",
"192.168.1.172",
"192.168.1.172:9080"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|9080||ratings.istio-inject.svc.cluster.local",
"timeout": "0s",
"retry_policy": {
"retry_on": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"num_retries": 2,
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
503
]
},
"max_stream_duration": {
"max_stream_duration": "0s",
"grpc_timeout_header_max": "0s"
}
},
"decorator": {
"operation": "ratings.istio-inject.svc.cluster.local:9080/*"
},
"name": "default"
}
],
"include_request_attempt_count": true
},
{
"name": "reviews.istio-inject.svc.cluster.local:9080",
"domains": [
"reviews.istio-inject.svc.cluster.local",
"reviews.istio-inject.svc.cluster.local:9080",
"reviews",
"reviews:9080",
"reviews.istio-inject.svc",
"reviews.istio-inject.svc:9080",
"reviews.istio-inject",
"reviews.istio-inject:9080",
"192.168.4.113",
"192.168.4.113:9080"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "outbound|9080|v1|reviews.istio-inject.svc.cluster.local",
"weight": 10
},
{
"name": "outbound|9080|v2|reviews.istio-inject.svc.cluster.local",
"weight": 40
},
{
"name": "outbound|9080|v3|reviews.istio-inject.svc.cluster.local",
"weight": 50
}
],
"total_weight": 100
},
"timeout": "0s",
"retry_policy": {
"retry_on": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"num_retries": 2,
"retry_host_predicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"host_selection_retry_max_attempts": "5",
"retriable_status_codes": [
503
]
},
"max_stream_duration": {
"max_stream_duration": "0s",
"grpc_timeout_header_max": "0s"
}
},
"metadata": {
"filter_metadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/istio-inject/virtual-service/reviews"
}
}
},
"decorator": {
"operation": "reviews:9080/*"
},
"name": "route"
}
],
"include_request_attempt_count": true
}
],
"validate_clusters": false
},
"last_updated": "2023-01-30T06:25:21.804Z"
},For example, let's use productpage to call reviews. Envoy matches the domain in VirtualHost based on the domains in the HTTP header. Therefore, you can see the domains list the long and short domain names of reviews in the cluster and the address of svc. Match "/" will be routed to three clusters "outbound | 9080 | v1 | reviews.istio-inject.svc.cluster.local", "outbound | 9080 | v2 | reviews.istio-inject.svc.cluster.local" and "outbound | 9080 | v3 | reviews.istio-inject.svc.cluster.local" with weights of 10, 40, 50 and the name' route'. Are you familiar with this? Yes, this is the same as the setting in the vs of reviews in the previous [1.3 Envoy configuration-background], so our relevant configuration information in vs will eventually take effect after the configuration of the route is loaded.
We can see that the default timeout is 0s, the default number of retries is 2, and the retry condition is connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes.
According to the preceding section, when productpage accesses reviews, istio-proxy in the productpage matches the 9080 routing-> one of the three clusters ("outbound | 9080 | v1 | reviews.istio-inject.svc.cluster.local", "outbound | 9080 | v2 | reviews.istio-inject.svc.cluster.local", and "outbound | 9080 | v3 | reviews.istio-inject.svc.cluster.local") based on the information about the vs configuration. Here, we take "outbound | 9080 | v1 | reviews.istio-inject.svc.cluster.local" cluster as an example.
As shown in the outbound|9080|v1|reviews.istio-inject.svc.cluster.local cluster configuration, its type is EDS, which indicates that the cluster endpoint is from dynamic discovery, and eds_config in dynamic discovery indicates that it is delivered by ads. Similarly, you can see the configurations familiar with the settings in dr of reviews in [1.3 Envoy Configuration-Background], such as connectionpool, outlierDetection, and other related configuration information. Finally, Envoy will be converted to cluster configuration and take effect after loading.
Next, let's explore a few other special clusters:
Envoy will discard requests that cannot be processed by the backend by default, so there will be a unified blackholecluster, without any specified backend svc. Any traffic that does not match the backend will be forwarded to this cluster.
Contrary to BlackHoleCluter, the request sent to the PassthroughCluster will be sent directly to the original destination in the request connection. Type: ORIGINAL_DST indicates that it is sent to the original destination address: port.
An inbound cluster is used for inbound requests on a pod. For reviews, the corresponding inbound cluster only has one inbound | 9080.
The content of the endpoint file shows that the reviews cluster "outbound | 9080 | v1 | reviews.istio-inject.svc.cluster.local" only has one endpoint, which is reviews-v1-74fb8fdbd8-qwsjq pod ip 10.0.3.29.
At this point, we have roughly sorted out the access between the two services in the service mesh, istio-proxy log interpretation, and configuration correspondence.
The config_dump file in the previous article is too long and laborious to interpret. Service mesh provides the asmctl tool to easily interpret listeners, route, cluster, endpoint, etc. Please visit this link for more information.
[root@shycmain ~]# asmctl --asmconfig asmconfig pc listeners productpage-v1-797d845774-dndmk.istio-inject --port 9080
ADDRESS PORT MATCH DESTINATION
0.0.0.0 9080 Trans: raw_buffer; App: HTTP Route: 9080
0.0.0.0 9080 ALL PassthroughCluster
[root@shycmain ~]# asmctl --asmconfig asmconfig pc routes productpage-v1-797d845774-dndmk.istio-inject --name 9080
NOTE: This output only contains routes loaded via RDS.
NAME DOMAINS MATCH VIRTUAL SERVICE
9080 details /*
9080 istio-ingressgateway.istio-system /*
9080 productpage /*
9080 ratings /*
9080 reviews /* reviews.istio-inject
[root@shycmain ~]# asmctl --asmconfig asmconfig pc cluster productpage-v1-797d845774-dndmk.istio-inject --fqdn reviews.istio-inject.svc.cluster.local
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
reviews.istio-inject.svc.cluster.local 9080 - outbound EDS reviews.istio-inject
reviews.istio-inject.svc.cluster.local 9080 v1 outbound EDS reviews.istio-inject
reviews.istio-inject.svc.cluster.local 9080 v2 outbound EDS reviews.istio-inject
reviews.istio-inject.svc.cluster.local 9080 v3 outbound EDS reviews.istio-inject
[root@shycmain ~]# asmctl --asmconfig asmconfig pc endpoint productpage-v1-797d845774-dndmk.istio-inject --cluster "outbound|9080|v1|reviews.istio-inject.svc.cluster.local"
ENDPOINT STATUS OUTLIER CHECK CLUSTER
10.0.3.29:9080 HEALTHY OK outbound|9080|v1|reviews.istio-inject.svc.cluster.localThis article focuses on the traffic forwarding link of the data plane that is injected into a pod in ASM Istio service mesh mode. Istio flexible injection enables you to customize the configuration and observation of traffic in the pod dimension, bringing more possibilities from the business link perspective. After you configure the gateway, virtual service, and destinationrule in the service mesh, the rules are sent to the Envoy through xDS. Then, the rules are finally reflected in the traffic forwarding rules through the listeners, route, cluster, and endpoint links. Then, when the use of ASM does not meet expectations, these links are the direction to be considered. In addition to traffic management, ASM provides convenient use of security, authentication, and observability. The configuration of these aspects will eventually be reflected in the configuration of related mesh service resources. Interested parties can refer to the official ASM documents[2].
[1] Enable automatic sidecar proxy injection: https://www.alibabacloud.com/help/en/alibaba-cloud-service-mesh/latest/enable-automatic-sidecar-injection-by-using-multiple-methods
[2] ASM Official Document: https://www.alibabacloud.com/help/en/alibaba-cloud-service-mesh
[3] https://istio.io/latest/docs/reference/config/networking/sidecar/
[4] https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage
Use Alibaba Cloud Serverless Kubernetes + AIGC to Build a Personal Code Assistant
Observability | Best Practices for Monitoring NGINX Ingress Gateways with Prometheus
165 posts | 12 followers
FollowXi Ning Wang(王夕宁) - June 10, 2020
Alibaba Cloud Indonesia - April 10, 2023
Xi Ning Wang(王夕宁) - July 1, 2021
Alibaba Developer - September 22, 2020
Alibaba Container Service - September 14, 2022
Alibaba Container Service - September 14, 2022
165 posts | 12 followers
Follow
ACK One
Provides a control plane to allow users to manage Kubernetes clusters that run based on different infrastructure resources
Learn More
Container Service for Kubernetes
Alibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.
Learn More
Cloud-Native Applications Management Solution
Accelerate and secure the development, deployment, and management of containerized applications cost-effectively.
Learn More
Function Compute
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn MoreMore Posts by Alibaba Cloud Native
Kidd Ip August 10, 2023 at 12:27 am
Thank you for the great sharing in open source area!