Alibaba Cloud Service Mesh:In-place canary migration from Istio to ASM

Step 1: Add the cluster to ASM

Add your K8s cluster to the ASM instance. You can add an Alibaba Cloud Container Service for Kubernetes (ACK) cluster or a self-built cluster registered with ACK. For instructions, see Add a cluster to an ASM instance. During the process, select the Ignore istio-system namespace check option. If the cluster you are migrating is not yet registered with ACK, use the registered cluster feature of ACK One to register it first.

After the cluster is added to the ASM instance, an ASMMigrateFromIstio resource is automatically created. This resource controls the entire migration process. Run the following command to view its content:

kubectl --kubeconfig=${ASM_KUBECONFIG} get asmmigratefromistio

Expected output:

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMMigrateFromIstio
metadata:
  name: default
spec:
  desiredState: Init
  retryCounter: 0
  advancedOptions:
    stopIstioSystemInjectionDisabling: false
status:
  message: ""
  retryCounter: 0
  state: Init

The following table describes the key fields in this resource.

Change the ASMMigrateFromIstio state

To proceed to the next migration phase, you must manually change the state of the ASMMigrateFromIstio resource. For example, to move from the Init phase to the next phase, you change the state to SetupIstioForMigrate.

The following commands show how to change the state for each phase.

• SetupIstioForMigrate

  kubectl --kubeconfig=${ASM_KUBECONFIG} patch asmmigratefromistio default --type='merge' -p '{"spec":{"desiredState":"SetupIstioForMigrate"}}'

• Migrating

  kubectl --kubeconfig=${ASM_KUBECONFIG} patch asmmigratefromistio default --type='merge' -p '{"spec":{"desiredState":"Migrating"}}'

• Finished

  kubectl --kubeconfig=${ASM_KUBECONFIG} patch asmmigratefromistio default --type='merge' -p '{"spec":{"desiredState":"Finished"}}'

Use the SetupIstioForMigrate state as an example. Run the following command to verify that the state was changed successfully:

kubectl get asmmigratefromistio default -o yaml

Expected output:

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMMigrateFromIstio
metadata:
  name: default
spec:
  desiredState: SetupIstioForMigrate
  retryCounter: 0
status:
  message: ""
  retryCounter: 0
  state: SetupIstioForMigrate

The state change is successful if the value of status.state is SetupIstioForMigrate.

Configuration for east-west gateway clusters

By default, ASM disables automatic sidecar injection in the istio-system namespace. However, if your cluster uses an Istio east-west gateway, the gateway's Pods require the MutatingWebhook to replace their images. This process requires that sidecar injection is not explicitly disabled for the namespace. Therefore, if you use an Istio east-west gateway, run the following command to prevent the ASM control plane from disabling injection in the istio-system namespace.

kubectl --kubeconfig=${ASM_KUBECONFIG} patch asmmigratefromistio default --type='merge' -p '{"spec":{"advancedOptions":{"disableIstioSystemLabelReconciliation": true}}}' 

Then, manually remove the label that ASM has applied to the istio-system namespace:

1. Edit the configuration of the istio-system namespace.

   kubectl --kubeconfig=${ACK_KUBECONFIG_PATH} edit ns istio-system

2. Manually remove the istio-injection: disable label, and then save your changes and exit.

Step 2: Prepare Istio for migration

1. Change the state to SetupIstioForMigrate.

   Important

   This command reconfigures Istiod for migration and triggers a rolling restart.

2. In this phase, ASM adds the following configuration to the Istio control plane. This enables Istio Sidecars and ASM mesh proxies to establish mTLS communication during the migration.

   defaultConfig:
     proxyMetadata:
       PROXY_CONFIG_XDS_AGENT："true"

   You must manually restart all workloads with an injected Istio Sidecar to apply this configuration. Run the following command to verify that the configuration has been applied to a specific Pod:

   kubectl get pod ${POD_NAME} -o yaml|grep PROXY_CONFIG_XDS_AGENT

   If the command returns a non-empty output, the workload is correctly configured. After all workloads with an injected Istio Sidecar are correctly configured, you can proceed to the Migrating state.

Step 3: Start the migration

In this phase, you transition to the Migrating state. You can then gradually replace Istio Sidecars with ASM mesh proxies across your workloads.

1. Change the state to Migrating. After the state is successfully changed, first apply all your Istio API resources to the ASM instance. Then, follow the steps below to migrate Istio Sidecars, ingress gateways, egress gateways, and east-west gateways to their ASM equivalents.

2. (Optional) Deploy an ASM cross-cluster gateway.

   In ASM, workloads with an injected mesh proxy automatically use the ASM cross-cluster gateway for cross-cluster communication, which is the equivalent of an Istio east-west gateway. If you use an east-west gateway in your Istio setup, you must deploy the ASM cross-cluster gateway before you inject the ASM mesh proxy into any workload. This ensures that cross-cluster communication is not interrupted.

   In the ASM console, configure the network name for the cluster to match your current Istio network topology. For clusters that have an Istio east-west gateway deployed, enable the cross-cluster gateway. For more information, see Configure network settings for a cluster and enable the cross-cluster mesh proxy.

   After enabling the cross-cluster gateway, a LoadBalancer Service is automatically created, which incurs fees for the underlying CLB instance. Click OK to confirm the configuration.

   Note

   The network configuration in ASM must be identical to the configuration in Istio. For example, if cluster A is in network-1 and cluster B is in network-2 in your Istio setup, you must use the same network names in ASM.

3. Migrate Istio Sidecars to ASM mesh proxies.

   $ kubectl --kubeconfig=${K8S_KUBECONFIG_PATH} -n ${NAMESPACE} patch ${DEPLOYMENT_NAME} --type='merge' -p '{"spec":{"template":{"metadata":{"labels":{"sidecar.asm.aliyun.com/inject":"true"}}}}}'

   Important

   This command modifies the workload labels, which triggers a rolling restart. Whether traffic is disrupted depends on several factors, including support for graceful shutdown in the application protocol (such as HTTP, HTTP/2, and gRPC), correct configuration of the Istio Sidecar for graceful shutdown, and proper application behavior (for example, if all requests can be completed within the graceful shutdown period). To minimize potential disruption, perform this operation during off-peak hours.

4. Migrate the Istio ingress gateway to the ASM ingress gateway.

   For more information, see Migrate a self-built Istio ingress gateway to an ASM gateway.

5. Migrate the Istio egress gateway to the ASM egress gateway. If an egress gateway is enabled in the cluster, follow these steps to migrate it:

   1. Create an egress gateway in ASM. For more information, see Create an egress gateway. Create other required resources, such as a ServiceEntry to register external services, gateway rules to enable forwarding ports for the egress gateway, and a virtual service to direct traffic to the egress gateway. For more information, see Manage traffic to external services on ASM.

   2. Modify the virtual service for the external service that you want to migrate to point to the ASM egress gateway and set the desired weight.

      The following YAML file is an example. For a complete example, see Egress gateway for HTTP traffic.

      apiVersion: networking.istio.io/v1
      kind: VirtualService
      metadata:
        name: direct-cnn-through-egress-gateway
      spec:
        hosts:
        - edition.cnn.com
        gateways:
        - istio-egressgateway
        - mesh
        http:
        - match:
          - gateways:
            - mesh
            port: 80
          route:
          - destination:
              host: istio-egressgateway.istio-system.svc.cluster.local
              port:
                number: 80
            weight: 99
          - destination: # Add a new destination that points to the ASM egress gateway and adjust its weight.
              host: asm-egressgateway.istio-system.svc.cluster.local # Assumes your ASM egress gateway is named asm-egressgateway.
              port:
                number: 80
            weight: 1

      In this example, traffic to edition.cnn.com is split between the Istio egress gateway and the ASM egress gateway at a 99:1 ratio. You can gradually adjust this ratio until all traffic is routed to the ASM egress gateway. Monitor the access logs of the Istio egress gateway to verify that no more traffic is passing through it.

Step 4: Finalize the migration

Before you proceed, ensure that you have completed all of the following tasks:

• All Istio Sidecars are removed.

• No traffic is passing through the Istio east-west, ingress, or egress gateways.

• All Istio API configurations have been imported into ASM.

• Workloads injected with the ASM mesh proxy are behaving as expected.

After confirming that all prerequisites are met, you can uninstall Istio from the cluster using the istioctl command that corresponds to your Istio version.

$ istioctl --kubeconfig=${K8S_KUBECONFIG_PATH} uninstall --purge

After the uninstallation is complete, change the state to Finished. If the state transition fails, check the status.message field for information about remaining Istio resources that require manual cleanup. After cleanup, increment the spec.retryCounter value by 1 to retry the operation. The migration is complete when the status.state field is Finished.