Implement a phased release in Kubernetes clusters with toolkit-maven-plugin - Enterprise Distributed Application Service

toolkit-maven-plugin automates canary releases for Spring Cloud, Dubbo, and High-speed Service Framework (HSF) applications deployed on Enterprise Distributed Application Service (EDAS). Instead of updating all instances at once, a canary release rolls out changes to a small subset of instances first, then gradually promotes the update across the remaining instances in batches.

How canary release works

A canary release with toolkit-maven-plugin follows a two-phase process:

Canary phase: Deploy the new version to a specified number of instances (gray). These instances receive live traffic, allowing you to validate the new version before the full rollout.
Batch rollout phase: Roll out the new version to the remaining instances in controlled batches (batch), with a configurable interval (batchWaitTime) between each batch. Batches can run automatically or require manual approval.

The EDAS update strategy type for this process is GrayBatchUpdate.

Prerequisites

Before you begin, make sure that you have:

A Spring Cloud, Dubbo, or HSF application deployed to an EDAS Kubernetes cluster
An AccessKey ID and AccessKey secret for a Resource Access Management (RAM) user (recommended over root account credentials)
The application ID (appId), or both the microservices namespace ID (namespaceId) and application name (appName) of the target application

Set up a canary release

To set up a canary release, add the plug-in dependency, create configuration files, and run the deployment command.

Step 1: Add the plug-in dependency

Add the following dependency to your pom.xml:

<build>
    <plugins>
        <plugin>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>toolkit-maven-plugin</artifactId>
            <version>1.1.9</version>
        </plugin>
    </plugins>
</build>

Note

Always use the latest toolkit-maven-plugin version.

Step 2: Create configuration files

Create three YAML files in the root directory of your packaged project. If you deploy a Maven submodule, create the package configuration file in the submodule directory instead.

Account configuration (toolkit_profile.yaml)

regionId:        # Region where the application is deployed, for example, cn-hangzhou
accessKeyId:     # AccessKey ID of a RAM user
accessKeySecret: # AccessKey secret of a RAM user
jarPath:         # (Optional) Path to the deployment package. Skips Maven packaging if specified.

Package configuration (toolkit_package.yaml)

apiVersion: V1
kind: AppPackage
spec:
  packageType:   # War, FatJar, Image, or url
  packageUrl:    # (Optional) URL of a remote deployment package. Used when packageType is url.
  imageUrl:      # (Optional) Image URL. Used when packageType is Image.

Deployment configuration (toolkit_deploy.yaml)

apiVersion: V1
kind: AppDeployment
spec:
  type: kubernetes
  target:
    appId:        # Application ID. If specified, namespaceId and appName are not required.
    namespaceId:  # Microservices namespace ID. Required if appId is not specified.
    appName:      # Application name. Required if appId is not specified.
  updateStrategy:
    type: GrayBatchUpdate
    grayUpdate:
      gray: 2              # Number of instances to update in the canary phase
    batchUpdate:
      batch: 3             # Number of batches for the remaining instances
      releaseType: auto     # auto or manual
      batchWaitTime: 5      # Minutes to wait between batches

The following table describes the canary release parameters:

Parameter	Description	Example
`appId`	EDAS application ID	`6bbc57a2-a2a0-4edb-**-**********`
`namespaceId`	Microservices namespace ID	`cn-hangzhou:my-namespace`
`gray`	Number of canary instances	`2`
`batch`	Number of rollout batches	`3`
`releaseType`	`auto` (automatic) or `manual` (requires approval)	`auto`
`batchWaitTime`	Minutes between batches	`5`

Step 3: Run the deployment command

Navigate to the directory that contains pom.xml (or the submodule pom.xml) and run:

mvn clean package toolkit:deploy \
  -Dtoolkit_profile=toolkit_profile.yaml \
  -Dtoolkit_package=toolkit_package.yaml \
  -Dtoolkit_deploy=toolkit_deploy.yaml

When the deployment succeeds, the output displays BUILD SUCCESS.

Command parameters:

Parameter	Description
`toolkit:deploy`	Deploys the application after Maven packaging completes.
`-Dtoolkit_profile`	Path to the account configuration file.
`-Dtoolkit_package`	Path to the package configuration file.
`-Dtoolkit_deploy`	Path to the deployment configuration file.
`-Ddeploy_version`	(Optional) Deployment version. Overrides the version in `toolkit_deploy.yaml`. Requires toolkit-maven-plugin 1.0.6 or later.

Note

To skip the -Dtoolkit_profile, -Dtoolkit_package, and -Dtoolkit_deploy parameters, place the configuration files in the same directory as pom.xml and prefix each filename with a dot (.toolkit_profile.yaml, .toolkit_package.yaml, .toolkit_deploy.yaml). The plug-in detects them automatically.

Deployment scenarios

Deploy with a local WAR or FatJar package

Build a WAR or FatJar package locally and deploy it to an existing EDAS application.

toolkit_package.yaml:

apiVersion: V1
kind: AppPackage
spec:
  packageType: War    # Or FatJar

toolkit_deploy.yaml:

apiVersion: V1
kind: AppDeployment
spec:
  type: kubernetes
  target:
    appId:        # Application ID
    namespaceId:  # (Optional) Namespace ID, required if appId is not specified
    appName:      # (Optional) Application name, required if appId is not specified

Deploy with an existing image URL

Deploy an application using a container image that already exists in a registry.

toolkit_package.yaml:

apiVersion: V1
kind: AppPackage
spec:
  packageType: Image
  imageUrl: registry.cn-beijing.aliyuncs.com/test/gateway:latest

toolkit_deploy.yaml:

apiVersion: V1
kind: AppDeployment
spec:
  type: kubernetes
  target:
    appId:        # Application ID
    namespaceId:  # (Optional) Namespace ID, required if appId is not specified
    appName:      # (Optional) Application name, required if appId is not specified

Build and push a local Docker image

Build a Docker image locally, push it to an Alibaba Cloud container image repository, and deploy.

toolkit_package.yaml:

apiVersion: V1
kind: AppPackage
spec:
  packageType: Image
  build:
    docker:
      dockerfile: Dockerfile
      imageRepoAddress:   # Image repository address
      imageTag:           # Image tag
      imageRepoUser:      # Repository username
      imageRepoPassword:  # Repository password

toolkit_deploy.yaml:

apiVersion: V1
kind: AppDeployment
spec:
  type: kubernetes
  target:
    appId:        # Application ID
    namespaceId:  # (Optional) Namespace ID, required if appId is not specified
    appName:      # (Optional) Application name, required if appId is not specified

Configuration reference

Package parameters

apiVersion: V1
kind: AppPackage
spec:
  packageType:         # War, FatJar, Image, or url
  imageUrl:            # Image URL. Required when deploying with an image.
  packageUrl:          # Package URL. Used when packageType is url.
  build:
    docker:
      dockerfile:        # Dockerfile path. Required for local image builds.
      imageRepoAddress:  # Alibaba Cloud image repository address
      imageTag:          # Image tag
      imageRepoUser:     # Repository username
      imageRepoPassword: # Repository password
    oss:
      bucket:            # Object Storage Service (OSS) bucket name
      key:               # OSS object path
      accessKeyId:       # AccessKey ID for OSS access
      accessKeySecret:   # AccessKey secret for OSS access

Deployment parameters

Click to expand the parameters that are supported by the deployment configuration file.

The deployment file supports the following parameters. All parameters are optional unless otherwise noted.

apiVersion: V1
kind: AppDeployment
spec:
  type: kubernetes
  target:
    appName:       # Application name
    namespaceId:   # Microservices namespace ID
    appId:         # Application ID. If specified, namespaceId and appName are not required.
    version:       # Deployment version. Default format: day-hour-minute-second.
    jdk:           # JDK version (Open JDK 7 or Open JDK 8). Not applicable to image deployments.
    webContainer:  # Tomcat version (apache-tomcat-7.0.91). Not applicable to image deployments.
    batchWaitTime: # Interval between batches (minutes)
    command:       # Container startup command. Overrides the default image command.
    commandArgs:   # Startup command arguments
    - 1d
  envs:            # Container environment variables
    - name: ENV_NAME
      value: 'value'

Health checks

Configure liveness and readiness probes to monitor container and application health. Specify one of exec, tcpSocket, or httpGet for each probe.

Containers that fail liveness checks are restarted. Containers that fail readiness checks multiple times are stopped and then restarted, and do not receive traffic from Server Load Balancer (SLB) instances.

  liveness:
    exec:
      command:
        - cat
        - /tmp/healthy
    tcpSocket:
      host: "192.168.1.109"  # (Optional) Defaults to the pod IP address
      port: "8080"
    httpGet:
      path: "/health"
      port: "8080"
      host: "192.168.1.109"  # (Optional) Defaults to the pod IP address
      scheme: "HTTP"       # HTTP or HTTPS
      httpHeaders:
        - name: "Custom-Header"
          value: "value"
    initialDelaySeconds: 5   # Seconds before the first check
    timeoutSeconds: 11       # Seconds before a check times out
    periodSeconds: 5         # Seconds between checks
    successThreshold: 1      # Must be 1 for liveness probes
    failureThreshold: 3      # Consecutive failures before action

  readiness:
    exec:
      command:
        - cat
        - /tmp/healthy
    tcpSocket:
      host: "192.168.1.109"  # (Optional) Defaults to the pod IP address
      port: "8080"
    httpGet:
      path: "/health"
      port: "8080"
      host: "192.168.1.109"  # (Optional) Defaults to the pod IP address
      scheme: "HTTP"
      httpHeaders:
        - name: "Custom-Header"
          value: "value"
    initialDelaySeconds: 5
    timeoutSeconds: 11
    periodSeconds: 5
    successThreshold: 2
    failureThreshold: 3

Lifecycle hooks

  preStop:          # Runs before the container is terminated
    exec:
      command:
        - /bin/bash
        - -c
        - "sleep 30"
    httpGet:
      host: "192.168.1.109"  # (Optional) Defaults to the pod IP address
      path: "/shutdown"
      port: "8080"
      scheme: "HTTP"         # HTTP or HTTPS
      httpHeaders:
        - name: "Custom-Header"
          value: "value"

  postStart:        # Runs after the container starts
    exec:
      command:
        - /bin/bash
        - -c
        - "echo started"
    httpGet:
      host: "192.168.1.109"  # (Optional) Defaults to the pod IP address
      path: "/initialize"
      port: "8080"
      scheme: "HTTP"         # HTTP or HTTPS
      httpHeaders:
        - name: "Custom-Header"
          value: "value"

Configuration mounts

Mount ConfigMap or Secret resources into the container.

  configMountDescs:
    - type: "ConfigMap"           # ConfigMap or Secret
      name: "my-config"
      mountPath: "/home/admin"    # Mount directory
      mountItems:                 # Mount as individual files
        - key: "config-key"
          path: "config-file"
      useSubPath: true            # true: keep existing files; false: overwrite

Java startup parameters

  javaStartUpConfig:
    initialHeapSize:             # -Xms (initial heap size, MB)
      original: 1000
      startup: "-Xms1000m"
    maxHeapSize:                 # -Xmx (max heap size, MB)
      original: 1000
      startup: "-Xmx1000m"
    newSize:                     # -XX:NewSize (initial young generation, MB)
      original: 200
      startup: "-XX:NewSize=200m"
    maxNewSize:                  # -XX:MaxNewSize (max young generation, MB)
      original: 200
      startup: "-XX:MaxNewSize=200m"
    survivorRatio:               # -XX:SurvivorRatio (Eden to Survivor ratio)
      original: 2
      startup: "-XX:SurvivorRatio=2"
    newRatio:                    # -XX:NewRatio (old to young generation ratio)
      original: 8
      startup: "-XX:NewRatio=8"
    permSize:                    # -XX:PermSize (permanent generation, MB)
      original: 512
      startup: "-XX:PermSize=512m"
    maxPermSize:                 # -XX:MaxPermSize (max permanent generation, MB)
      original: 512
      startup: "-XX:MaxPermSize=200m"
    maxDirectMemorySize:         # -XX:MaxDirectMemorySize (MB)
      original: 100
      startup: "-XX:MaxDirectMemorySize=100m"
    threadStackSize:             # -XX:ThreadStackSize
      original: 500
      startup: "-XX:ThreadStackSize=500"
    hsfserverPort:               # HSF server port
      original: 12200
      startup: "-Dhsf.server.port=12200"
    hsfserverMinPoolSize:        # HSF minimum thread pool size
      original: 50
      startup: "-Dhsf.server.min.poolsize=50"
    hsfserverMaxPoolSize:        # HSF maximum thread pool size
      original: 720
      startup: "-Dhsf.server.max.poolsize=720"
    youngGarbageCollector:       # Young generation GC policy
      original: "UseSerialGC"   # UseSerialGC, UseG1GC, UseParNewGC, or UseParallelGC
      startup: "-XX:+UseSerialGC"
    oldGarbageCollector:         # Old generation GC policy
      original: "UseConcMarkSweepGC"   # UseConcMarkSweepGC, UseSerialGC, UseG1GC, UseParNewGC, UseParallelOldGC, or UseParallelGC
      startup: "-XX:+UseConcMarkSweepGC"
    concGCThreads:               # Concurrent GC threads
      original: 5
      startup: "-XX:ConcGCThreads=5"
    parallelGCThreads:           # Parallel GC threads
      original: 5
      startup: "-XX:ParallelGCThreads=5"
    g1HeapRegionSize:            # G1 region size (MB)
      original: 50
      startup: "-XX:G1HeapRegionSize=50m"
    gclogFilePath:               # GC log directory
      original: "/tmp/"
      startup: "-Xloggc:/tmp/"
    useGCLogFileRotation:        # Enable GC log rotation
      original: true
      startup: "-XX:+UseGCLogFileRotation"
    numberOfGCLogFiles:          # Number of GC log files
      original: 5
      startup: "-XX:NumberOfGCLogFiles=5"
    gclogFileSize:               # GC log file size (MB)
      original: 100
      startup: "-XX:GCLogFileSize=100m"
    heapDumpOnOutOfMemoryError:  # Enable out-of-memory (OOM) heap dumps
      original: true
      startup: "-XX:+HeapDumpOnOutOfMemoryError"
    heapDumpPath:                # OOM dump file path
      original: "/tmp/dumpfile"
      startup: "-XX:HeapDumpPath=/tmp/dumpfile"
    customParams:                # Custom JVM parameters
      original: "-Dtest=true"
      startup: "-Dtest=true"

Scheduling rules

  deployAcrossZones: "true"    # Deploy pods across availability zones (recommended)
  deployAcrossNodes: "true"    # Deploy pods across nodes (recommended)

  customTolerations:           # Kubernetes scheduling tolerations
    - key: my-key
      operator: Exists         # Exists or Equal
      effect: NoSchedule       # NoSchedule or NoExecute
    - key: another-key
      operator: Equal
      value: "my-value"
      effect: "NoExecute"
      tolerationSeconds: 300

  customAffinity:
    nodeAffinity:              # Schedule pods based on node labels
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "topology.kubernetes.io/zone"
                operator: "In"
                values:
                  - "cn-hangzhou-a"
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: "topology.kubernetes.io/zone"
                operator: "In"
                values:
                  - "cn-hangzhou-b"
    podAffinity:               # Co-locate pods with matching pods
      requiredDuringSchedulingIgnoredDuringExecution:
        - namespaces:
            - "default"
          topologyKey: "topology.kubernetes.io/zone"
          labelSelector:
            matchExpressions:
              - key: "app"
                operator: "In"
                values:
                  - "my-app"
    podAntiAffinity:           # Spread pods away from matching pods
      requiredDuringSchedulingIgnoredDuringExecution:
        - namespaces:
            - "default"
          topologyKey: "kubernetes.io/hostname"
          labelSelector:
            matchExpressions:
              - key: "app"
                operator: "In"
                values:
                  - "my-app"
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 2
          podAffinityTerm:
            namespaces:
              - "default"
            topologyKey: "kubernetes.io/hostname"
            labelSelector:
              matchExpressions:
                - key: "app"
                  operator: "In"
                  values:
                    - "my-app"