Use Terraform to create or modify ASM custom resources - Alibaba Cloud Service Mesh

Service Mesh (ASM) supports using Terraform Kubernetes Provider to operate custom resources in ASM or modify mesh features starting from version 1.22.6.109. This topic introduces how to use Terraform to create or modify ASM custom resources through two examples.

Prerequisites

A kubectl client is connected to the ACK cluster. For more information, see Get a cluster kubeconfig and connect to the cluster using kubectl.
(Optional) Terraform is installed and configured locally.
You can also use Cloud Shell to perform all the operations demonstrated in this topic. Before you start, switch the Terraform version to ensure that the Terraform version is greater than 0.14.
(Optional) The tfk8s tool is installed.

Preparations

Before starting the demonstration process, we recommend that you create an empty directory as the Terraform project directory and create a provider.tf file in the directory with the following content.

provider "kubernetes" {
  config_path = "~/.kube/config"
}

This configuration file specifies the kubeconfig used by the Terraform Kubernetes Provider.

The complete directory structure in this topic is as follows:

terraform-Project         # Terraform project directory
├── asmmeshconfig.tf      # Scenario 2 tf file
├── virtualservice.tf     # Scenario 1 tf file
├── provider.tf           # provider file
└── resources             # Used to store resource files, such as yaml, json, etc.
    └── demo.yaml         # Scenario 1 resource file

After the provider.tf file is created, initialize the Terraform project.

terraform init

Scenario 1: Use Terraform to create a VirtualService resource

Create a VirtualService resource file named demo.yaml in the resources directory.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: my-productpage-rule
  namespace: istio-system
spec:
  hosts:
  - productpage.prod.svc.cluster.local # ignores rule namespace
  http:
  - timeout: 5s
    route:
    - destination:
        host: productpage.prod.svc.cluster.local

Create virtualservice.tf.

resource "kubernetes_manifest" "virtualservice_demo" {
  manifest = yamldecode(file("./resources/demo.yaml"))
}

Check the changes to Terraform resources.

terraform plan

Expected output:

Expand to view terraform plan output

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # kubernetes_manifest.virtualservice_demo will be created
  + resource "kubernetes_manifest" "virtualservice_demo" {
      + manifest = {
          + apiVersion = "networking.istio.io/v1"
          + kind       = "VirtualService"
          + metadata   = {
              + name      = "my-productpage-rule"
              + namespace = "istio-system"
            }
          + spec       = {
              + hosts = [
                  + "productpage.prod.svc.cluster.local",
                ]
              + http  = [
                  + {
                      + route   = [
                          + {
                              + destination = {
                                  + host = "productpage.prod.svc.cluster.local"
                                }
                            },
                        ]
                      + timeout = "5s"
                    },
                ]
            }
        }
      + object   = {
          + apiVersion = "networking.istio.io/v1"
          + kind       = "VirtualService"
          + metadata   = {
              + annotations                = (known after apply)
              + creationTimestamp          = (known after apply)
              + deletionGracePeriodSeconds = (known after apply)
              + deletionTimestamp          = (known after apply)
              + finalizers                 = (known after apply)
              + generateName               = (known after apply)
              + generation                 = (known after apply)
              + labels                     = (known after apply)
              + managedFields              = (known after apply)
              + name                       = "my-productpage-rule"
              + namespace                  = "istio-system"
              + ownerReferences            = (known after apply)
              + resourceVersion            = (known after apply)
              + selfLink                   = (known after apply)
              + uid                        = (known after apply)
            }
          + spec       = {
              + exportTo = (known after apply)
              + gateways = (known after apply)
              + hosts    = [
                  + "productpage.prod.svc.cluster.local",
                ]
              + http     = [
                  + {
                      + corsPolicy               = {
                          + allowCredentials    = (known after apply)
                          + allowHeaders        = (known after apply)
                          + allowMethods        = (known after apply)
                          + allowOrigin         = (known after apply)
                          + allowOrigins        = (known after apply)
                          + exposeHeaders       = (known after apply)
                          + maxAge              = (known after apply)
                          + unmatchedPreflights = (known after apply)
                        }
                      + delegate                 = {
                          + name      = (known after apply)
                          + namespace = (known after apply)
                        }
                      + directResponse           = {
                          + body   = {
                              + bytes  = (known after apply)
                              + string = (known after apply)
                            }
                          + status = (known after apply)
                        }
                      + fault                    = {
                          + abort = {
                              + grpcStatus = (known after apply)
                              + http2Error = (known after apply)
                              + httpStatus = (known after apply)
                              + percentage = {
                                  + value = (known after apply)
                                }
                            }
                          + delay = {
                              + exponentialDelay = (known after apply)
                              + fixedDelay       = (known after apply)
                              + percent          = (known after apply)
                              + percentage       = {
                                  + value = (known after apply)
                                }
                            }
                        }
                      + headerToDynamicSubsetKey = (known after apply)
                      + headers                  = {
                          + request  = {
                              + add    = (known after apply)
                              + remove = (known after apply)
                              + set    = (known after apply)
                            }
                          + response = {
                              + add    = (known after apply)
                              + remove = (known after apply)
                              + set    = (known after apply)
                            }
                        }
                      + match                    = (known after apply)
                      + mirror                   = {
                          + host   = (known after apply)
                          + port   = {
                              + number = (known after apply)
                            }
                          + subset = (known after apply)
                        }
                      + mirrorPercent            = (known after apply)
                      + mirrorPercentage         = {
                          + value = (known after apply)
                        }
                      + mirror_percent           = (known after apply)
                      + mirrors                  = (known after apply)
                      + name                     = (known after apply)
                      + redirect                 = {
                          + authority    = (known after apply)
                          + derivePort   = (known after apply)
                          + port         = (known after apply)
                          + redirectCode = (known after apply)
                          + scheme       = (known after apply)
                          + uri          = (known after apply)
                        }
                      + retries                  = {
                          + attempts              = (known after apply)
                          + perTryTimeout         = (known after apply)
                          + retryOn               = (known after apply)
                          + retryRemoteLocalities = (known after apply)
                        }
                      + rewrite                  = {
                          + authority       = (known after apply)
                          + uri             = (known after apply)
                          + uriRegexRewrite = {
                              + match   = (known after apply)
                              + rewrite = (known after apply)
                            }
                        }
                      + route                    = [
                          + {
                              + destination = {
                                  + host   = "productpage.prod.svc.cluster.local"
                                  + port   = {
                                      + number = (known after apply)
                                    }
                                  + subset = (known after apply)
                                }
                              + fallback    = {
                                  + case    = (known after apply)
                                  + target  = {
                                      + host   = (known after apply)
                                      + port   = {
                                          + number = (known after apply)
                                        }
                                      + subset = (known after apply)
                                    }
                                  + targets = (known after apply)
                                }
                              + headers     = {
                                  + request  = {
                                      + add    = (known after apply)
                                      + remove = (known after apply)
                                      + set    = (known after apply)
                                    }
                                  + response = {
                                      + add    = (known after apply)
                                      + remove = (known after apply)
                                      + set    = (known after apply)
                                    }
                                }
                              + weight      = (known after apply)
                            },
                        ]
                      + timeout                  = "5s"
                    },
                ]
              + tcp      = (known after apply)
              + tls      = (known after apply)
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Create and verify the resource.

Create the resource.
```
terraform apply --auto-approve
```

Verify the resource.

kubectl get VirtualService -n istio-system

Expected output:

NAME                  GATEWAYS   HOSTS                                    AGE
my-productpage-rule              ["productpage.prod.svc.cluster.local"]   77s

(Optional) Clean up the resource.

terraform destroy -target=kubernetes_manifest.virtualservice_demo --auto-approve

Scenario 2: Modify the ASMMeshConfig resource to control mesh features

This scenario demonstrates how to modify an existing ASMMeshConfig resource to disable the default service mesh feature that rewrites health checks for Pods.

Import the ASMMeshConfig resource.

Use the tfk8s tool

Generate asmmeshconfig.tf.

kubectl get asmmeshconfig default -o yaml | tfk8s --strip -o asmmeshconfig.tf

The above command will directly generate the asmmeshconfig.tf file with the expected content as follows.

resource "kubernetes_manifest" "asmmeshconfig_default" {
  manifest = {
    "apiVersion" = "istio.alibabacloud.com/v1beta1"
    "kind" = "ASMMeshConfig"
    "metadata" = {
      "name" = "default"
    }
    "spec" = {
      "accessLogConfiguration" = {}
      "ambientConfiguration" = {
        "enabled" = false
        "redirectMode" = ""
        "waypoint" = {}
        "ztunnel" = {}
      }
      "cniConfiguration" = {
        "enabled" = true
        "excludeNamespaces" = "istio-system,kube-system"
        "repair" = {}
      }
      "enableGatewayAPI" = true
      "gatewayAPIInferenceExtension" = {}
      "ingressControllerMode" = "OFF"
      "ingressSelector" = "ingressgateway1"
      "ingressService" = "istio-ingressgateway1"
      "sidecarInjectorWebhookConfiguration" = {}
      "smcEnabled" = false
    }
  }
}

Import the ASMMeshConfig resource into the terraform state.

terraform import kubernetes_manifest.asmmeshconfig_default "apiVersion=istio.alibabacloud.com/v1beta1,name=default,kind=ASMMeshConfig"

Expected output:

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

╷
│ Warning: Apply needed after 'import'
│ 
│ Please run apply after a successful import to realign the resource state to the configuration in Terraform.
╵

Execute the terraform apply command to align the Terraform and Kubernetes resource states.

terraform apply --auto-approve

Partial expected output:

...
# kubernetes_manifest.asmmeshconfig_default will be updated in-place
~ resource "kubernetes_manifest" "asmmeshconfig_default" {
      + manifest = {
          + apiVersion = "istio.alibabacloud.com/v1beta1"
          + kind       = "ASMMeshConfig"
          + metadata   = {
              + annotations                = null
              + creationTimestamp          = null
...

Use the terraform command

Import the ASMMeshConfig resource into the state.

terraform import kubernetes_manifest.asmmeshconfig_default "apiVersion=istio.alibabacloud.com/v1beta1,name=default,kind=ASMMeshConfig"

Expected output:

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

╷
│ Warning: Apply needed after 'import'
│ 
│ Please run apply after a successful import to realign the resource state to the configuration in Terraform.
╵

Generate asmmeshconfig.tf.

terraform show -no-color > asmmeshconfig.tf

Expected file content:

resource "kubernetes_manifest" "asmmeshconfig_default" {
    object = {
        apiVersion = "istio.alibabacloud.com/v1beta1"
        kind       = "ASMMeshConfig"
        metadata   = {
            annotations                = null
            creationTimestamp          = null
            deletionGracePeriodSeconds = null
            deletionTimestamp          = null
            finalizers                 = null
            generateName               = null
            generation                 = null
            labels                     = null
            managedFields              = null
            name                       = "default"
            namespace                  = null
            ownerReferences            = null
            resourceVersion            = null
            selfLink                   = null
            uid                        = null
        }
        spec       = {
            accessLogConfiguration         = {}
            adaptiveSchedulerConfiguration = {}
            ambientConfiguration           = {
                redirectMode = ""
                waypoint     = {}
                ztunnel      = {}
            }
            cniConfiguration               = {
                enabled = true
                repair  = {}
            }
            localityLbSetting              = {
                enabled = true
            }
        }
    }
}

Change object to manifest in the asmmeshconfig.tf file and remove parameters with null values. The following is the adjusted content of asmmeshconfig.tf.

resource "kubernetes_manifest" "asmmeshconfig_default" {
    manifest = {
        apiVersion = "istio.alibabacloud.com/v1beta1"
        kind       = "ASMMeshConfig"
        metadata   = {
            name                       = "default"
        }
        spec       = {
            accessLogConfiguration         = {}
            adaptiveSchedulerConfiguration = {}
            ambientConfiguration           = {
                redirectMode = ""
                waypoint     = {}
                ztunnel      = {}
            }
            cniConfiguration               = {
                enabled = true
                repair  = {}
            }
            localityLbSetting              = {
                enabled = true
            }
        }
    }
}

Execute the terraform apply command to align the Terraform and Kubernetes resource states.

terraform apply --auto-approve

Partial expected output:

...
# kubernetes_manifest.asmmeshconfig_default will be updated in-place
~ resource "kubernetes_manifest" "asmmeshconfig_default" {
      + manifest = {
          + apiVersion = "istio.alibabacloud.com/v1beta1"
          + kind       = "ASMMeshConfig"
          + metadata   = {
              + annotations                = null
              + creationTimestamp          = null
...

The parameters of terraform import in the steps above are explained as follows:

Parameter	Description
kubernetes_manifest	Terraform resource type, corresponding to the resource type in `asmmeshconfig.tf`.
asmmeshconfig_default	Terraform resource name, corresponding to the resource name in `asmmeshconfig.tf`.
apiVersion	The API version registered in the Kubernetes CRD. You can view the apiVersion of the resource by executing `kubectl get ${resource type} ${resource name}`.
kind	The resource type registered in the Kubernetes CRD. You can view the kind of the resource by executing `kubectl get ${resource type} ${resource name}`.
name	The name of the ASMMeshConfig resource to be imported, which is `default` in this case. Since the ASMMeshConfig resource is a cluster-level Kubernetes resource, no namespace is specified here. For namespace-level resources, you can specify the namespace using `namespace=${resource namespace}`.

Edit asmmeshconfig.tf to add the value of spec.sidecarInjectorWebhookConfiguration.rewriteAppHTTPProbe field as false to disable the default service mesh feature that rewrites health checks for Pods.

resource "kubernetes_manifest" "asmmeshconfig_default" {
  manifest = {
    "apiVersion" = "istio.alibabacloud.com/v1beta1"
    "kind" = "ASMMeshConfig"
    "metadata" = {
      "name" = "default"
    }
    "spec" = {
      "accessLogConfiguration" = {}
      "ambientConfiguration" = {
        "enabled" = false 
        "redirectMode" = ""
        "waypoint" = {}
        "ztunnel" = {}
      }
      "cniConfiguration" = {
        "enabled" = true
        "excludeNamespaces" = "istio-system,kube-system"
        "repair" = {}
      }
      "enableGatewayAPI" = true
      "gatewayAPIInferenceExtension" = {}
      "ingressControllerMode" = "OFF"
      "ingressSelector" = "ingressgateway1"
      "ingressService" = "istio-ingressgateway1"
      "sidecarInjectorWebhookConfiguration" = {
        "rewriteAppHTTPProbe" = false
      }
      "smcEnabled" = false
    }
  }
}

Check the changes.

terraform plan

Expected output:

kubernetes_manifest.asmmeshconfig_default: Refreshing state...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # kubernetes_manifest.asmmeshconfig_default will be updated in-place
  ~ resource "kubernetes_manifest" "asmmeshconfig_default" {
      ~ manifest = {
          ~ spec       = {
              ~ sidecarInjectorWebhookConfiguration = {
                  + rewriteAppHTTPProbe = false
                }
                # (9 unchanged attributes hidden)
            }
            # (3 unchanged attributes hidden)
        }
      ~ object   = {
          ~ spec       = {
              ~ sidecarInjectorWebhookConfiguration = {
                  + rewriteAppHTTPProbe = false
                }
                # (9 unchanged attributes hidden)
            }
            # (3 unchanged attributes hidden)
        }
    }
Plan: 0 to add, 1 to change, 0 to destroy.

Note

If the changes in manifest and object are inconsistent, it indicates that the Kubernetes resource has been modified through other means, and the actual state of the Kubernetes resource is different from the state recorded by Terraform. You can execute terraform refresh to update Terraform's state.

Apply the changes.

terraform apply --auto-approve

Expected output:

kubernetes_manifest.asmmeshconfig_default: Modifying...
kubernetes_manifest.asmmeshconfig_default: Modifications complete after 1s
...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.