Alibaba Cloud Service Mesh:Access services over WebSocket connections in ASM

Deploy a WebSocket client and a WebSocket server

1. Connect to the ACK cluster by using kubectl. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

2. Deploy a WebSocket server.

   In this example, the WebSocket server for Python provided by the WebSocket community is used. For more information about how to modify the configurations for deploying the WebSocket server, see Deploy to Kubernetes.

   1. Create the websockets-server.yaml file that contains the following content:

      apiVersion: v1
      kind: Service
      metadata:
        name: websockets-server
        labels:
          app: websockets-server
      spec:
        type: ClusterIP
        ports:
          - port: 8080
            targetPort: 80
            name: http-websocket
        selector:
          app: websockets-server
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: websockets-server
        labels:
          app: websockets-server
      spec:
        selector:
          matchLabels:
            app: websockets-server
        template:
          metadata:
            labels:
              app: websockets-server
          spec:
            containers:
            - name: websockets-test
              image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/istio-websockets-test:1.0
              ports:
              - containerPort: 80

   2. Deploy the WebSocket server in the default namespace.

      kubectl apply -f websockets-server.yaml -n default

3. Deploy a WebSocket client.

   In this example, a Dockerfile is used to build an image for a WebSocket client. The websockets-client.yaml file contains the configurations for using the image of the WebSocket client to deploy the WebSocket client to a cluster.

   FROM python:3.9-alpine
   RUN pip3 install websockets

   1. Create the websockets-client.yaml file that contains the following content:

      apiVersion: v1
      kind: Service
      metadata:
        name: websockets-client
        labels:
          app: websockets-client
      spec:
        type: ClusterIP
        ports:
          - port: 8080
            targetPort: 80
            name: http-websockets-client
        selector:
          app: websockets-client
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: websockets-client-sleep
        labels:
          app: websockets-client
      spec:
        selector:
          matchLabels:
            app: websockets-client
        template:
          metadata:
            labels:
              app: websockets-client
          spec:
            containers:
            - name: websockets-client
              image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/istio-websockets-client-test:1.0
              command: ["sleep", "14d"]

   2. Deploy the WebSocket client in the default namespace.

      kubectl apply -f websockets-client.yaml -n default

Establish WebSocket connections over HTTP/1.1

Even if the WebSocket client sends multiple requests on one connection to the WebSocket server, the logs of sidecar proxies contain only one access log entry for the WebSocket connection. Envoy treats WebSocket connections as TCP byte streams and can recognize only requests and responses that contain the HTTP Upgrade header. Therefore, Envoy generates an access log entry only after the connection is closed. In addition, the log entry does not contain details of each TCP request.

1. Use one of the following methods to open the CLI shell:

   • Method 1:

     1. Log on to the ACK console and click Clusters in the left-side navigation pane.

     2. On the Clusters page, click the name of the cluster that you want to manage and choose Workloads > Pods in the left-side navigation pane.

     3. On the Pods page, find websockets-client and click Terminal in the Actions column. Then, click websockets-client.

   • Method 2:

     Run the following command to open the CLI shell:

     kubectl exec -it -n <namespace> websockets-client-sleep...  -c websockets-client -- sh

2. Run the following command to access the WebSocket server:

   python3 -m websockets ws://websockets-server.<namespace>.svc.cluster.local:8080

   Expected output:

   Connected to ws://websockets-server.default.svc.cluster.local:8080.

   If you enter "hello" and "world", the two words are returned as expected.

   > hello
   < hello
   > world
   < world
   Connection closed: 1000 (OK).

3. Query the logs of sidecar proxies.

   • Query the logs of the sidecar proxy for the WebSocket client.

     1. On the Pods page, click the name of the container of the WebSocket client.

     2. Click the Logs tab and select istio-proxy from the Container drop-down list.

        You can see that the logs contain an entry about the WebSocket connection over HTTP/1.1.

        {...."upstream_host":"10.208.0.105:80","bytes_sent":23,"protocol":"HTTP/1.1",....}

   • Query the logs of the sidecar proxy for the WebSocket server.

     1. On the Pods page, click the name of the container of the WebSocket server.

     2. Click the Logs tab and select istio-proxy from the Container drop-down list.

        You can see that the logs contain an entry about the WebSocket connection over HTTP/1.1.

        {...."downstream_local_address":"10.208.0.105:80","upstream_local_address":"127.0.**.**:53983","protocol":"HTTP/1.1",....}

Establish WebSocket connections over HTTP/2

If you use Istio of a version earlier than 1.12, WebSocket connections cannot be established to the WebSocket server by upgrading HTTP/1.1 connections to HTTP/2 connections. In this case, the HTTP status code 503 is returned.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  labels:
    provider: asm
  name: websockets-server
spec:
  host: websockets-server
  trafficPolicy:
    connectionPool:
      http:
        h2UpgradePolicy: DO_NOT_UPGRADE

If you use Istio 1.12 or later, you can perform the following steps to establish WebSocket connections by using HTTP/2.

1. Create a destination rule.

   1. Log on to the ASM console.

   2. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.

   4. On the details page of the ASM instance, choose Traffic Management Center > DestinationRule in the left-side navigation pane. On the page that appears, click Create from YAML.

   5. On the Create page, select default from the Namespace drop-down list and copy the following content to the code editor. Then, click Create.

      apiVersion: networking.istio.io/v1beta1
      kind: DestinationRule
      metadata:
        labels:
          provider: asm
        name: websockets-server
      spec:
        host: websockets-server
        trafficPolicy:
          connectionPool:
            http:
              h2UpgradePolicy: UPGRADE

      h2UpgradePolicy: A value of UPGRADE indicates that HTTP/2 is enabled for the WebSocket server.

2. Create an Envoy filter.

   By default, WebSocket does not work with the HTTP/2 protocol. However, Envoy supports tunneling WebSocket over HTTP/2. This way, all communication can be performed over HTTP/2 in the ASM instance. You can set the allow_connect parameter of an Envoy filter to true for the destination workload. After that, HTTP/2 connections are supported by the WebSocket server.

   1. Log on to the ASM console.

   2. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.

   4. On the details page of the ASM instance, choose Plugin Extension Center > Market Place in the left-side navigation pane.

   5. On the Market Place page, click Template that sets the allow_connect parameter to true to allow updated protocol connections.

   6. On the Plugin Detail page, click the Plugin Config tab. In the Plugin Effective scope section, select Workload Scope and click Add workloads to effective scope.

   7. In the Add workloads to effective scope dialog box, set the Namespace parameter to default and Workload Type to Deployment. In the Select workloads section, select websockets-server, click the icon to add websockets-server to the selected section, and then click OK.

   8. In the YAML code editor of the Plugin Config section, enter patch_context: SIDECAR_INBOUND, turn on Plugin Switch, and wait until the plug-in is enabled.

      After the plug-in is enabled, ASM automatically creates an Envoy filter. The following code is an example YAML file of the Envoy filter:

      apiVersion: networking.istio.io/v1alpha3
      kind: EnvoyFilter
      metadata:
        name: h2-upgrade-wss
        labels:
          asm-system: 'true'
          provider: asm
      spec:
        workloadSelector:
          labels:
            app: websockets-server
        configPatches:
        - applyTo: NETWORK_FILTER
          match:
            context: SIDECAR_INBOUND
            proxy:
              proxyVersion: '^1\.*.*'
            listener:
              filterChain:
                filter:
                  name: "envoy.filters.network.http_connection_manager"
          patch:
            operation: MERGE
            value:
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                http2_protocol_options:
                  allow_connect: true

3. Run the following command to access the WebSocket server:

   python3 -m websockets ws://websockets-server.<namespace>.svc.cluster.local:8080

   Expected output:

   Connected to ws://websockets-server.default.svc.cluster.local:8080.

   If you enter "hello" and "world", the two words are returned as expected.

   > hello
   < hello
   > world
   < world
   Connection closed: 1000 (OK).

4. Query the logs of sidecar proxies.

   • Query the logs of the sidecar proxy for the WebSocket client.

     1. On the Pods page, click the name of the container of the WebSocket client.

     2. Click the Logs tab and select istio-proxy from the Container drop-down list.

        You can see that the logs contain an entry about the WebSocket connection over HTTP/1.1. This indicates that the WebSocket client sends requests by using HTTP/1.1.

        {...."authority":"websockets-server.default.svc.cluster.local:8080","upstream_service_time":null,"protocol":"HTTP/1.1",....}

   • Query the logs of the sidecar proxy for the WebSocket server.

     1. On the Pods page, click the name of the container of the WebSocket server.

     2. Click the Logs tab and select istio-proxy from the Container drop-down list.

        You can see that the logs contain an entry about the WebSocket connection over HTTP/2. This indicates that the WebSocket server receives requests whose protocols are upgraded to HTTP/2.

        {...."method":"GET","upstream_local_address":"127.0.**.**:34477","protocol":"HTTP/2",....}