Use ApplicationSets to coordinate multi-environment deployment and application dependencies - Container Service for Kubernetes

This topic describes how to build an automated deployment system that supports application dependency management between development and staging environments by combining the Progressive Syncs feature of Argo CD with the multi-environment resource orchestration capabilities of ApplicationSets.

Background information

Web applications consist of three parts: Frontend, Backend, and Database. When deploying, you need to deploy in the following order: Database > Backend > Frontend to ensure proper dependency management. When using Distributed Cloud Container Platform for Kubernetes (ACK One) GitOps to deploy such applications, each component (Database/Backend/Frontend) corresponds to an Argo CD application. You must handle the deployment dependencies between multiple applications.

ApplicationSets simplify multi-cluster application orchestration. It automatically generates one or more applications based on a single application orchestration template. ApplicationSets is commonly used to manage multi-cluster or multi-environment deployments (dev/staging/prod) of your applications.

The requirements for application dependency management and multi-environment deployment are complex. This topic introduces an advanced usage of ApplicationSet aimed at unified management of application dependencies and multi-environment deployments. The following section describes how it works:

Define applications for multi-environment deployment (dev/staging) by using Matrix Generator and add labels to the applications to identify their names.
Define the creation order of applications with dependencies in rollingSync . For example, deploy app1 first, then deploy app2.

After configuration, deployment will proceed in the defined order: dev-app1>dev-app2>staging-app1>staging-app2, as shown in the following figure:

Progressive Syncs

The Progressive Syncs feature of Argo CD enables intelligent deployment orchestration by using ApplicationSet. This feature controls the creation and update order of applications based on the dependencies and health status of applications managed by ApplicationSets. You can define a steps list, and Progressive Syncs will continuously monitor the health status of applications in each phase. The next phase of operation is triggered only when the status of the application is Healthy.

Note

DaemonSet, StatefulSet, and Argo Rollout are all supported by the Progressive Syncs feature because applications enter the Progressing status during pod deployment. In fact, any resource with health checks that can report a Progressing status is supported by the Progressive Syncs feature.

Directory structure of applications

This example contains two applications, app1 and app2, where app2 depends on app1, and both applications need to be deployed to the dev and staging environments. The example directory structure is as follows:

manifests
└── apps
    ├── env
    │   ├── dev
    │   │   └── config.json
    │   └── staging
    │       └── config.json
    ├── app1
    │   ├── base
    │   │   ├── deployment.yaml
    │   │   ├── kustomization.yaml
    │   │   └── service.yaml
    │   └── overlay
    │       ├── dev
    │       │   └── bj
    │       │       ├── deployment.yaml
    │       │       └── kustomization.yaml
    │       └── staging
    │           └── bj
    │               ├── deployment.yaml
    │               └── kustomization.yaml
    └── app2
        ├── base
        │   ├── deployment.yaml
        │   ├── kustomization.yaml
        │   └── service.yaml
        └── overlay
            ├── dev
            │   └── bj
            │       ├── deployment.yaml
            │       └── kustomization.yaml
            └── staging
                └── bj
                    ├── deployment.yaml
                    └── kustomization.yaml

manifests/apps/env: Configure this parameter according to specific environments.
Important
If you use this example for deployment, you need to fork the example to your own repository and modify cluster_address in config.json to the API server endpoint of your associated cluster.
manifests/apps/app1: Application 1 managed by kustomize. Each environment type may contain multiple regions (the preceding example only shows one: bj).
manifests/apps/app2: Application 2 managed by kustomize. Each environment type may contain multiple regions.

Prerequisites

Step 1: Enable the Progressive Syncs feature

ACK One GitOps only supports enabling Progressive Syncs in high availability mode. The steps to enable it are as follows.

Use the Fleet kubeconfig to run the following command to modify the configuration:

kubectl edit cm -nargocd argocd-cmd-params-cm

Add the applicationsetcontroller.enable.progressive.syncs: "true" configuration to argocd-cmd-params-cm.

apiVersion: v1
data:
  applicationsetcontroller.enable.progressive.syncs: "true"
  ...
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
  ...

Run the following command to restart the argocd-application-controller pod:

 kubectl rollout restart deployment argocd-application-controller -n argocd

Step 2: Create an ApplicationSet in the Fleet to manage application dependencies and multi-environment deployments

Log on to the ACK One console. In the left-side navigation pane, choose Fleet > Multi-cluster GitOps.

Click Create Multi-cluster Application > GitOps to enter the Create Multi-cluster Application - GitOps page, click the YAML Create tab, and use the following example to create an application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: dependency-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/AliyunContainerService/gitops-demo.git
              revision: main
              pathParamPrefix: env
              files:
              - path: "manifests/apps/env/*/config.json"
          - git:
              repoURL: https://github.com/AliyunContainerService/gitops-demo.git
              revision: main
              pathParamPrefix: target
              directories:
              - path: "manifests/apps/app1/overlay/{{.env.path.basename}}/*"
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/AliyunContainerService/gitops-demo.git
              revision: HEAD
              pathParamPrefix: env
              files:
              - path: "manifests/apps/env/*/config.json"
          - git:
              repoURL: https://github.com/AliyunContainerService/gitops-demo.git
              revision: HEAD
              pathParamPrefix: target
              directories:
              - path: "manifests/apps/app2/overlay/{{.env.path.basename}}/*"        
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: app.kubernetes.io/instance
              operator: In
              values:
                - app1
            - key: environment
              operator: In
              values:
                - dev
        - matchExpressions:
            - key: app.kubernetes.io/instance
              operator: In
              values:
                - app2
            - key: environment
              operator: In
              values:
                - dev
        - matchExpressions:
            - key: app.kubernetes.io/instance
              operator: In
              values:
                - app1
            - key: environment
              operator: In
              values:
                - staging
        - matchExpressions:
            - key: app.kubernetes.io/instance
              operator: In
              values:
                - app2
            - key: environment
              operator: In
              values:
                - staging
  template:
    metadata:
      name: '{{.env.path.basename}}-{{index .target.path.segments 2}}-{{.target.path.basename}}'
      labels:
        app.kubernetes.io/instance: '{{index .target.path.segments 2}}'
        environment: '{{.env.path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/AliyunContainerService/gitops-demo.git
        targetRevision: HEAD
        path: '{{.target.path.path}}'
      destination:
        #server: https://kubernetes.default.svc
        server: '{{index .cluster_address .target.path.basename}}'
        namespace: demo
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 5
          backoff:
            duration: 5s
            maxDuration: 3m0s
            factor: 2
        syncOptions:
          - CreateNamespace=true

This example uses Matrix Generator to deploy the application to multiple environments and uses RollingSync of the Progressive Syncs feature to implement dependencies between applications. In this example, the ApplicationSet generates four applications.

In the rollingSync.steps parameter of Progressive Syncs, the matching labels are the labels of the ArgoCD application, so you must specify the generated labels in .spec.template.metadata.labels.

Matrix Generator combines two Git generators. The following are the variables provided by Git generators.

Parameter	Description	Example
`{{.path.basename}}`	The base name of the path of the directory containing the configuration file.	dev staging
`{{.target.path.path}}`	The path of the directory in the Git repository that contains the matching configuration file.	`manifests/apps/app2/overlay/dev/bj` `manifests/apps/app2/overlay/staging/bj`
`{{index .path.segments n}}`	The path of the matching configuration file in the Git repository, split into array elements (n - array index).	app1 app2

Note

If you specify pathParamPrefix, when you want to reference variables like {{.path.basename}}, you must add the prefix path to access the variable. For example, the original variable {{.path.basename}} must be modified to {{.target.path.basename}}, where target is the value of pathParamPrefix.

Important

To ensure the security of the production environment, we recommend that you separate the production environment and development/testing/staging environments into two different ApplicationSets. When generating production environment applications, you should choose the Manual Sync method.

Step 3: View application deployment dependencies and release status

Since each application needs to be released to both development and staging environments, app1 and app2 each need to generate two applications, for a total of four application instances. Their deployment order will be dev-app1>dev-app2>staging-app1>staging-app2, with each application waiting for the previous application to synchronize successfully and reach a healthy state before starting.

Use the console

Log on to the ACK One console. In the left-side navigation pane, choose Fleet > Multi-cluster GitOps.
Find the multi-cluster application and view the value in the Application column.
Applications are deployed in the following order, with expected output as follows:
1. Deploy dev-app1.
2. Deploy dev-app2.
3. Deploy staging-app1.
4. Deploy staging-app2.

Use the CLI

Use the Fleet KubeConfig to execute the following command to view the application status:

 kubectl -nargocd get app

Applications are deployed in the following order, with expected output as follows:

Deploy dev-app1.

NAME              SYNC STATUS   HEALTH STATUS
dev-app1-bj       Synced        Progressing
dev-app2-bj       OutOfSync     Missing
staging-app1-bj   OutOfSync     Missing
staging-app2-bj   OutOfSync     Missing

Deploy dev-app2.

NAME              SYNC STATUS   HEALTH STATUS
dev-app1-bj       Synced        Healthy
dev-app2-bj       Synced        Progressing
staging-app1-bj   OutOfSync     Missing
staging-app2-bj   OutOfSync     Missing

Deploy staging-app1.

NAME              SYNC STATUS   HEALTH STATUS
dev-app1-bj       Synced        Healthy
dev-app2-bj       Synced        Healthy
staging-app1-bj   Synced        Progressing
staging-app2-bj   OutOfSync     Missing

Deploy staging-app2.

NAME              SYNC STATUS   HEALTH STATUS
dev-app1-bj       Synced        Healthy
dev-app2-bj       Synced        Healthy
staging-app1-bj   Synced        Healthy
staging-app2-bj   Synced        Progressing