Build a multi-cluster CI/CD pipeline with ACK One GitOps and Container Registry - Container Service for Kubernetes

This tutorial walks you through building a complete CI/CD pipeline across Dev, Staging, and Production clusters using ACK One GitOps and Container Registry (ACR). After a code commit, ACR automatically builds and pushes a container image. ACK One GitOps detects the new image, writes the updated tag back to your Deployment repository, and deploys the change to each cluster—automatically in Dev, and with manual approval and canary release in Staging and Production.

How it works

The pipeline connects two systems:

CI (Container Registry): Monitors your source code repository. When you push a tag matching the build rule (e.g., release-*), ACR builds a container image and pushes it to the image repository.
CD (ACK One GitOps): Runs ArgoCD Image Updater, which polls the image repository. When a new image tag is detected, it commits the updated tag to the Deployment repository and ArgoCD syncs the change to the target clusters.

Why use separate repositories? Keeping the source code repository and the Deployment repository separate prevents image tag commits from re-triggering CI builds—which would cause an infinite build loop. It also gives you a clean audit trail of what was deployed and when, and lets you control who can push to production environments independently of who writes application code.

Environment sync strategy:

Environment	Sync mode	Rationale
Dev	Automatic	Any detected image change is applied immediately for fast feedback
Staging	Manual	Review the diff and trigger a canary release before wider rollout
Production	Manual	Approve canary release before full rollout

Limitations

Applies only to applications created with ACK One GitOps.
Applies only to applications orchestrated with Kustomize or Helm.

Prerequisites

Before you begin, ensure that you have:

Fleet management enabled. See Enable Fleet management.
The kubeconfig file of the Fleet instance downloaded and kubectl connected to the Fleet instance via the ACK One console.
ACK One GitOps enabled on your Fleet instance. See Log on to the GitOps system.
The latest ArgoCD CLI installed from ArgoCD releases.
The Kruise Rollouts component installed in the Staging and Production clusters and the kubectl-kruise component installed.

Example application

This tutorial uses echo-server as the sample application. Fork the following repositories to your own account—ACK One GitOps needs write access to commit image tag changes back to the Deployment repository:

Business code repo 1 (echo-server): Ealenn/Echo-Server
Business code repo 2 (echo-web-server, multiple Deployments): AliyunContainerService/echo-web-server
Deployment repository (gitops-demo): AliyunContainerService/gitops-demo

Each environment uses a separate values file. Modify image.repository and image.tag in values.yaml per environment:

image:
  repository: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/echo-server
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "v1.0"

For the latest updates to this example, see echo-server example.

Step 1: Create a CI pipeline with Container Registry

Create an image repository in Container Registry, bind it to your forked echo-server project, and add a build rule. See Use a Container Registry Enterprise Edition instance to build an image for the full setup.

Add a build rule that triggers when new tags starting with release- are pushed. ACR then automatically builds a container image and pushes it to the repository. Adjust the regular expression to match your branching strategy.

To allow pulling images without a Kubernetes Secret, turn on Allow Anonymous Users to Pull Public Artifacts from the Container Registry Enterprise Edition instance overview page, or configure aliyun-acr-credential-helper.

If your source code is hosted on GitHub and builds time out when pulling code, turn on Build With Servers Deployed Outside Chinese Mainland.

Step 2: Configure credentials for ACK One GitOps

ACK One GitOps needs credentials to poll the image repository and write image tag changes back to the Deployment repository.

Connect to ACK One GitOps. See Log on to the GitOps system.
Add the Deployment repository to ACK One GitOps. See Add Git repositories to ACK One GitOps.
Create a GitOps application. See Use GitOps to manage applications.

Create a Secret named acr in the argocd namespace to allow ACK One GitOps to authenticate with the image repository:

kubectl -n argocd apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: acr
type: Opaque
stringData:
  acr: <your_username>:<your_password>  # Replace with your image repository credentials.
EOF

The Git repository was added with a username and password or a private key certificate (including repos added with an SSH private key)

Configure write-back credentials so ACK One GitOps can commit image tag changes to the Deployment repository. If you already provided credentials (username/password or SSH private key) when adding the Git repository, add the following annotation to the Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/write-back-method: git

The Git repository was added without a username and password or a private key certificate

If you did not provide credentials when adding the Git repository, create a Secret with the Git credentials and reference it in the annotation:

kubectl -n argocd create secret generic git-creds \
--from-literal=username=<your_username> \
--from-literal=password=<your_password>

Then add the following annotation to the Application (git:secret:argocd/git-creds refers to the Secret named git-creds in the argocd namespace):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/git-creds

Step 3: Configure automated image updates and deploy to multiple clusters

Add annotations to each ArgoCD Application to enable automated image updates. The following example configures updates for two images (echoserver and webserver):

metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: echoserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-server,webserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-web-server
    argocd-image-updater.argoproj.io/echoserver.helm.image-name: image.echoServer.repository
    argocd-image-updater.argoproj.io/echoserver.helm.image-tag: image.echoServer.tag
    argocd-image-updater.argoproj.io/echoserver.update-strategy: latest
    argocd-image-updater.argoproj.io/webserver.helm.image-name: image.echoWebServer.repository
    argocd-image-updater.argoproj.io/webserver.helm.image-tag: image.echoWebServer.tag
    argocd-image-updater.argoproj.io/webserver.update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/git-creds

echoserver in the image-list value is an alias for the image repository address. Separate multiple image addresses with commas (,). Replace with your actual image repository addresses.

The annotations above apply to Helm-managed applications. For Kustomize-managed applications, see Annotations for Helm and Kustomize applications.

Deploy to the Dev cluster

Create app-helm-dev.yaml. Replace ${url} with the Dev cluster server URL (see Use GitOps to manage ACK clusters) and repoURL with your Deployment repository address.

syncPolicy is set to automated—ACK One GitOps applies image updates to Dev immediately without manual intervention.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-helm-dev
  annotations:
    argocd-image-updater.argoproj.io/image-list: echoserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-server,webserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-web-server
    argocd-image-updater.argoproj.io/echoserver.helm.image-name: image.echoServer.repository
    argocd-image-updater.argoproj.io/echoserver.helm.image-tag: image.echoServer.tag
    argocd-image-updater.argoproj.io/echoserver.update-strategy: latest
    argocd-image-updater.argoproj.io/webserver.helm.image-name: image.echoWebServer.repository
    argocd-image-updater.argoproj.io/webserver.helm.image-tag: image.echoWebServer.tag
    argocd-image-updater.argoproj.io/webserver.update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git
spec:
  destination:
    namespace: app-helm-dev
    # https://XX.XX.XX.XX:6443
    server: ${url}
  source:
    path: manifests/helm/echo-server
    repoURL: 'git@github.com:ivan-cai/gitops-demo.git'
    targetRevision: stable-example
    helm:
      valueFiles:
        - values-dev.yaml
  project: default
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

Connect to the ACK One Fleet instance and deploy:

argocd app create -f app-helm-dev.yaml

Deploy to the Staging and Production clusters

Create app-helm-staging.yaml and app-helm-production.yaml. Unlike Dev, these applications do not set syncPolicy: automated—image changes are written back to the Deployment repository, but you trigger the sync manually after reviewing the diff. This gives you a gate before changes reach production-grade environments.

Staging cluster — app-helm-staging.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-helm-staging
  annotations:
    argocd-image-updater.argoproj.io/image-list: echoserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-server,webserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-web-server
    argocd-image-updater.argoproj.io/echoserver.helm.image-name: image.echoServer.repository
    argocd-image-updater.argoproj.io/echoserver.helm.image-tag: image.echoServer.tag
    argocd-image-updater.argoproj.io/echoserver.update-strategy: latest
    argocd-image-updater.argoproj.io/webserver.helm.image-name: image.echoWebServer.repository
    argocd-image-updater.argoproj.io/webserver.helm.image-tag: image.echoWebServer.tag
    argocd-image-updater.argoproj.io/webserver.update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git
spec:
  destination:
    namespace: app-staging
    # https://XX.XX.XX.XX:6443
    server: ${url}
  source:
    path: manifests/helm/echo-server
    repoURL: 'git@github.com:ivan-cai/gitops-demo.git'
    targetRevision: stable-example
    helm:
      valueFiles:
        - values-staging.yaml
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Production cluster — app-helm-production.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-helm-production
  annotations:
    argocd-image-updater.argoproj.io/image-list: echoserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-server,webserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-web-server
    argocd-image-updater.argoproj.io/echoserver.helm.image-name: image.echoServer.repository
    argocd-image-updater.argoproj.io/echoserver.helm.image-tag: image.echoServer.tag
    argocd-image-updater.argoproj.io/echoserver.update-strategy: latest
    argocd-image-updater.argoproj.io/webserver.helm.image-name: image.echoWebServer.repository
    argocd-image-updater.argoproj.io/webserver.helm.image-tag: image.echoWebServer.tag
    argocd-image-updater.argoproj.io/webserver.update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git
spec:
  destination:
    name: ''
    namespace: app-production
    server: 'https://39.98.XX.XX:6443'
  source:
    path: manifests/helm/echo-server
    repoURL: 'git@github.com:ivan-cai/gitops-demo.git'
    targetRevision: test2
    helm:
      valueFiles:
        - values-production.yaml
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

When image changes are detected, verify the Deployment repository is updated, then go to the ArgoCD UI, click REFRESH, check APP DIFF, and click SYNC to apply the changes.

Alternatively, sync from the CLI:

argocd app sync argocd/app-helm-staging
argocd app sync argocd/app-helm-production

To trigger a canary release, see Use Kruise Rollout to implement canary releases based on ACK One GitOps. Approve the canary with:

kubectl-kruise rollout approve rollout/rollouts-demo --kubeconfig <path to kubeconfig file>

This tutorial uses Kruise Rollouts for canary releases. You can also use Argo Rollouts. See Use ACK One GitOps and Argo Rollouts to perform canary releases.

Step 4: Roll back the application

To roll back to a previous version, revert the image tag in the Deployment repository and sync the application in ArgoCD:

In the Deployment repository, edit manifests/helm/echo-server/.argocd-source-${appname}.yaml and set the image tag back to the previous version.
Commit and push the change to the Deployment repository.
In the ArgoCD UI, go to the application page, click REFRESH, then click SYNC to apply the reverted tag. Alternatively, sync from the CLI:
```
argocd app sync argocd/app-helm-staging
argocd app sync argocd/app-helm-production
```

The application returns to the previous image version once the sync completes.

Step 5: Test the CI/CD pipeline

Run through the full pipeline end-to-end to verify each stage produces the expected output.

1. Push a tag to trigger the CI pipeline.

Push a branch that matches the Container Registry build rule:

git clone https://github.com/{xxx}/echo-web-server.git
cd echo-web-server
git push origin HEAD:release-v3

Set the tag name based on your naming convention. The tag must match the release- prefix in the build rule.

2. Verify the container image is built.

Log on to the Container Registry console.
In the top navigation bar, select a region.
In the left-side navigation pane, click Instances.
On the Instances page, click the Enterprise Edition instance.
Choose Repository > Repositories and click the target image repository.
In the left-side navigation pane, click Build. In the Build Log section, confirm a new image is listed.

Expected result: A new image entry appears in the Build Log with a status of Successful.

3. Verify the Image Updater detects the new image.

After the build completes, check the argocd-image-updater logs:

kubectl -nargocd logs argocd-server-<xxxxx> -c argocd-image-updater -f

Expected result: A successful update produces output similar to:

time="2023-07-19T07:35:41Z" level=info msg="Successfully updated image 'demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-web-server:v1.0' to 'demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-web-server:v3-5a7147', but pending spec update (dry run=false)" alias=echowebserver application=echo-web-server
time="2023-07-19T07:35:41Z" level=info msg="Committing 1 parameter update(s) for application echo-web-server" application=echo-web-server

4. Verify the write-back commit appears in the Deployment repository.

Check that manifests/helm/echo-server/.argocd-source-${appname}.yaml is created in the Deployment repository on GitHub. One file is created per cluster (Dev, Staging, and Production). The following figure shows the Dev cluster file:

Expected result: The file exists and contains the updated image tag (e.g., v3-5a7147).

5. Sync Staging and Production.

At this point, the Dev cluster Deployment is updated to v3-5a7147 automatically. Manually sync Staging and Production, then approve the canary release to update those clusters to v3-5a7147.

Expected result: After syncing and approving the canary release, all three clusters run the new image version.

Automated image update settings

Specify images to watch

Add the image-list annotation to specify one or more images for automated updates:

argocd-image-updater.argoproj.io/image-list: <image_spec_list>

image_spec_list is a comma-separated list of image specs in the following format:

[<alias_name>=]<image_path>[:<version_constraint>]

alias_name is a string alias used only within the annotation. Example—set the alias echoserver for the image:

argocd-image-updater.argoproj.io/image-list: echoserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-server:v1.0

Filter by image tag

Trigger updates only for tags that match a regular expression. Use the allow-tags annotation with the regexp: prefix:

argocd-image-updater.argoproj.io/<image_name>.allow-tags: regexp:^v[1-9].*

Example for echoserver:

argocd-image-updater.argoproj.io/echoserver.allow-tags: regexp:^v[1-9].*

Configure an update strategy

Four update strategies are available. The default is semver.

Update strategy	Description
`semver`	Update to the latest version in a semantically sorted list
`latest`	Update to the latest version by creation time. Note that creation time differs from the push time.
`name`	Update to the latest version in an alphabetically sorted list
`digest`	Update to the latest version pushed with a mutable tag

Set the strategy with:

argocd-image-updater.argoproj.io/<image_name>.update-strategy: <strategy>

Example:

argocd-image-updater.argoproj.io/echoserver.update-strategy: semver

Annotations for Helm and Kustomize applications

Helm applications

When values.yaml contains separate fields for the image repository and tag, specify both with helm.image-name and helm.image-tag:

annotations:
  argocd-image-updater.argoproj.io/image-list: echoserver=demo-test-registry.cn-hangzhou.cr.aliyuncs.com/cidemo/echo-server:v1.0
  argocd-image-updater.argoproj.io/echoserver.helm.image-name: image.echoServer.repository
  argocd-image-updater.argoproj.io/echoserver.helm.image-tag: image.echoServer.tag
  argocd-image-updater.argoproj.io/echoserver.update-strategy: latest
  argocd-image-updater.argoproj.io/write-back-method: git

Kustomize applications

For Kustomize, include the tag in the alias (image-list) and specify the base image address (without tag) using kustomize.image-name:

annotations:
  argocd-image-updater.argoproj.io/image-list: <image_alias>=<image_name>:<image_tag>
  argocd-image-updater.argoproj.io/<image_alias>.kustomize.image-name: <original_image_name>

Container Service for Kubernetes:Build a CI/CD pipeline using ACK One GitOps and ACR