Alibaba Cloud Service Mesh:Manage Spring Cloud services

Demo introduction

In this example, Spring Cloud with the Nacos registry is used. You can download the demo sample code from the nacos-examples directory on GitHub.

The xxx string in a response indicates the specific parameter in the corresponding echo request. For example, if the /echo/world request is sent to the provider service of V1, the Hello Nacos Discovery From v1world response is returned.

If the consumer and provider services are not deployed in ASM instances, that is, sidecar injection is not enabled for the service pods, you can access the services but you cannot manage the services by using Istio resources.

Step 1: Enable support for Spring Cloud on the ASM control plane

Method 1: Applicable to all Spring Cloud implementations and registries

1. Use kubectl to connect to the ASM instance. For more information, see Use kubectl to connect to an ASM instance.
2. Create an Envoy filter in the Alibaba Cloud Container Service for Kubernetes (ACK) cluster that is added to the ASM instance.
   Use the Envoy filter to enable com.aliyun.reverse_dns filter for the service.
   1. Create an any-spring-cloud-support.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: EnvoyFilter
      metadata:
        labels:
          provider: "asm"
          asm-system: "true"
        name: any-spring-cloud-support
        namespace: istio-system
      spec:
        configPatches:
        - applyTo: HTTP_FILTER
          match:
            proxy:
              proxyVersion: "^1.*"
            context: SIDECAR_OUTBOUND
            listener:
              portNumber: 8070
              filterChain:
                filter:
                  name: "envoy.filters.network.http_connection_manager"
                  subFilter:
                    name: "envoy.filters.http.router"
          patch:
            operation: INSERT_BEFORE
            value: # reverse_dns filter specification
             name: com.aliyun.reverse_dns
             typed_config:
               "@type": "type.googleapis.com/udpa.type.v1.TypedStruct"
               type_url: type.googleapis.com/envoy.config.filter.reverse_dns.v3alpha.CommonConfig
               value:
                 pod_cidrs:
                 - "10.0.128.0/18"

      Modify the parameter settings in the file based on your business requirements. Some parameters are described as follows:

      • portNumber: specifies the port number of the Spring Cloud service. If there are multiple ports, you can delete this parameter. If the ports can be converged, you can create multiple Envoy filters and configure a port number in each Envoy filter.
      • pod_cidrs: specifies the pod CIDR block of the ACK or serverless Kubernetes (ASK) cluster. To obtain the pod CIDR block of an ACK or ASK cluster, perform the following operations: Log on to the Container Service for Kubernetes (ACK) console, go to the Clusters page and click the cluster. On the cluster details page that appears, click the Basic Information tab. If Terway is used as the Container Network Interface (CNI) plug-in, go to the cluster details page and click the Cluster Resources tab to obtain the pod vSwitch CIDR block.

Method 2: Applicable to only Nacos registries

1. Use kubectl to connect to the ASM instance. For more information. see Use kubectl to connect to an ASM instance.
2. Create a service entry.
   1. Create an external-nacos-svc.yaml file that contains the following content:

      kind: ServiceEntry
      metadata:
        name: external-nacos-svc
      spec:
        hosts:
        - "NACOS_SERVER_HOST"  ## Replace this variable with the endpoint of your Nacos server host. Example: mse-xxx-p.nacos-ans.mse.aliyuncs.com. 
        location: MESH_EXTERNAL
        ports:
        - number: 8848
          name: http
        resolution: DNS

      In the preceding YAML file, port 8848 is the default port used by Nacos. If you use a self-managed Nacos server and a custom port number, set the number parameter to the port number that you use.
   2. Run the following command to create a service entry:

      kubectl apply -f external-nacos-svc.yaml

3. Create an Envoy filter.
   1. Create an external-envoyfilter.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: EnvoyFilter
      metadata:
        labels:
          provider: "asm"
          asm-system: "true"
        name: nacos-subscribe-lua
        namespace: istio-system
      spec:
        configPatches:
          # The first patch adds the lua filter to the listener/http connection manager.
        - applyTo: HTTP_FILTER
          match:
            proxy:
              proxyVersion: "^1.*"
            context: SIDECAR_OUTBOUND
            listener:
              portNumber: 8848
              filterChain:
                filter:
                  name: "envoy.filters.network.http_connection_manager"
                  subFilter:
                    name: "envoy.filters.http.router"
          patch:
            operation: INSERT_BEFORE
            value: # lua filter specification
             name: envoy.lua
             typed_config:
                "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
                inlineCode: |
                   -- copyright: ASM (Alibaba Cloud ServiceMesh)
                   function envoy_on_request(request_handle)
                     local request_headers = request_handle:headers()
                     -- /nacos/v1/ns/instance/list?healthyOnly=false&namespaceId=public&clientIP=10.122.63.81&serviceName=DEFAULT_GROUP%40%40service-provider&udpPort=53174&encoding=UTF-8
                     local path = request_headers:get(":path")
                     if string.match(path,"^/nacos/v1/ns/instance/list") then
                       local servicename = string.gsub(path,".*&serviceName.*40([%w.\\_\\-]+)&.*","%1")
                       request_handle:streamInfo():dynamicMetadata():set("context", "request.path", path)
                       request_handle:streamInfo():dynamicMetadata():set("context", "request.servicename", servicename)
                       request_handle:logInfo("subscribe for serviceName: " .. servicename)
                     else
                       request_handle:streamInfo():dynamicMetadata():set("context", "request.path", "")
                     end
                   end
                   function envoy_on_response(response_handle)
                     local request_path = response_handle:streamInfo():dynamicMetadata():get("context")["request.path"]
                     if request_path == "" then
                        return
                     end
                     local servicename = response_handle:streamInfo():dynamicMetadata():get("context")["request.servicename"]
                     response_handle:logInfo("modified response ip to serviceName:" .. servicename)
                     local bodyObject = response_handle:body(true)
                     local body= bodyObject:getBytes(0,bodyObject:length())
                     body = string.gsub(body,"%s+","")
                     body = string.gsub(body,"(ip\":\")(%d+.%d+.%d+.%d+)","%1"..servicename)
                     response_handle:body():setBytes(body)
                   end

   2. Run the following command to create an Envoy filter:

      kubectl apply -f external-envoyfilter.yaml

Step 2: Deploy Spring Cloud services in an ACK cluster

1. Use kubectl to connect to the ACK cluster. For more information, see Connect to ACK clusters by using kubectl.
2. Run the following command to deploy Spring Cloud services:

   export NACOS_ADDRESS=xxxx # Replace xxxx with the endpoint of the MSE Nacos registry or a self-managed Nacos registry. We recommend that you use a virtual private cloud (VPC) endpoint. 
   wget https://alibabacloudservicemesh.oss-cn-beijing.aliyuncs.com/asm-labs/springcloud/demo.yaml -O demo.yaml
   sed -e "s/NACOS_SERVER_CLUSTERIP/$NACOS_ADDRESS/g" demo.yaml |kubectl apply -f -

3. Run the following command to check the Spring Cloud services:

    kubectl get pods

   Expected output:

   consumer-bdd464654-jn8q7       2/2     Running     0          25h
   provider-v1-66bc67fb6d-46pgl   2/2     Running     0          25h
   provider-v2-76568c45f6-85z87   2/2     Running     0          25h

Step 3: Create an Istio gateway and a virtual service

1. Use kubectl to connect to the ASM instance. For more information, see Use kubectl to connect to an ASM instance.
2. Create an Istio gateway.
   1. Create a test-gateway.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: test-gateway
      spec:
        selector:
          istio: ingressgateway # use istio default controller
        servers:
        - port:
            number: 80
            name: http
            protocol: HTTP
          hosts:
          - "*"

   2. Run the following command to create an Istio gateway:

      kubectl apply -f test-gateway.yaml

3. Create a virtual service.
   1. Create a consumer.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: consumer
      spec:
        hosts:
        - "*"
        gateways:
        - test-gateway
        http:
        - match:
          - uri:
              prefix: /
          route:
          - destination:
              host: consumer.default.svc.cluster.local
              port:
                number: 8080

   2. Run the following command to create a virtual service:

      kubectl apply -f consumer.yaml

Step 4: Check whether ASM can manage the Spring Cloud services

1. Query the IP address of the ingress gateway.
   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
   3. On the Ingress Gateway page, view the IP address of the ingress gateway in the Kubernetes Service column.
2. Run the following command to initiate requests from the ingress gateway to the Spring Cloud consumer service:

   curl <IP address of the ingress gateway>/echo/world

   Expected output:

   Hello Nacos Discovery From v1world
   Hello Nacos Discovery From v2world
   Hello Nacos Discovery From v1world
   Hello Nacos Discovery From v2world

   The output shows that traffic is routed to V1 and V2 of the provider service in polling mode by default.
3. Create a destination rule and a virtual service.
   1. Create a service-provider.yaml file that contains the following content:

      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: service-provider
      spec:
        host: service-provider
        subsets:
        - name: v1
          labels:
            label: v1
        - name: v2
          labels:
            label: v2
                                      

   2. Run the following command to create a destination rule:

      kubectl apply -f service-provider.yaml

   3. Create a service-provider1.yaml file that contains the following content.
      The virtual service to be created defines that /echo/hello requests are routed to the provider service of V1 and other requests are routed to the provider service of V2.

      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: service-provider
      spec:
        hosts:
        - service-provider
        http:
        - name: "hello-v1"
          match:
          - uri:
              prefix: "/echo/hello"
          route:
          - destination:
              host: service-provider
              subset: v1
        - name: "default"
          route:
          - destination:
              host: service-provider
              subset: v2

   4. Run the following command to create a virtual service:

      kubectl apply -f service-provider1.yaml

4. Run the following command to initiate requests to the Spring Cloud consumer service:

   curl <IP address of the ingress gateway>/echo/hello

   Expected output:

   Hello Nacos Discovery From v1hello
   Hello Nacos Discovery From v1hello

   The output shows that /echo/hello requests are routed to the provider service of V1 and other requests are routed to the provider service of V2. This indicates that Spring Cloud traffic is taken over by Istio and Custom Resource Definitions (CRDs) provided by Istio can be used to configure routing rules. In this case, ASM can manage the Spring Cloud services.

FAQ

What do I do if the Spring Cloud services that I deployed do not take effect?

1. Check whether traffic blocking is enabled for the port or IP address of Nacos.
   • If you use reverse DNS lookup, you must block the IP address of the pod.
   • If you use Lua scripts, you must block the Nacos server IP address and cluster IP address.
2. Make sure that an Envoy filter is created before you create deployments so as to intercept the registration process. If specific deployments are created before you create the Envoy filter, you must enable rolling update for the deployments.
3. Check whether you have created Kubernetes service resources and provided a cluster IP address for the Spring Cloud services.
4. Make sure that the client SDK of Nacos is of a version earlier than 2.0. The client SDK of V2.0 or later uses gRPC to connect to the server, which is beyond the application scope of the solution provided in this topic.
5. Check the sidecar version of the services. If the sidecar image version is earlier than 1.13.4.32, you may have updated only the control plane of the ASM instance but not updated the data plane. In this case, enable rolling update for the deployments of the services.