Implement phased releases and blue-green deployments using Nginx Ingress - Container Service for Kubernetes

Background

Both phased releases (also called canary releases) and blue-green deployments run a new version alongside the old and route partial traffic to it. Once stable, all traffic switches over and the old version is decommissioned.

The ACK Nginx Ingress Controller supports two annotation-based traffic-splitting methods:

Method	Status	Notes
`canary-*` annotations	Active	The official community method. Use this for all new configurations.
`service-*` annotations	Deprecated	No longer available in Nginx Ingress Controller v1.12 and later. Do not use it.

Use cases

Traffic splitting based on client requests: Route traffic to the new version only when a request contains a specific header or cookie — for example, foo=bar. Other requests continue to the old version. Once stable, switch all traffic.
Traffic splitting based on service weight: Route a fixed percentage of traffic — for example, 20% — to the new version. Gradually increase the percentage until all traffic goes to the new version.

Supported traffic-splitting dimensions:

Request header — suitable for phased releases and A/B testing
Cookie — suitable for phased releases and A/B testing
Query parameter — suitable for phased releases and A/B testing
Service weight — suitable for blue-green deployments

The canary-* annotation method

Annotation reference

All canary configurations require the nginx.ingress.kubernetes.io/canary: "true" annotation. Other annotations define the routing logic.

Annotation	Description	Min. version
`nginx.ingress.kubernetes.io/canary`	Enables the canary feature. Must be set to `true` for any other canary annotation to take effect. Valid values: `true`, `false`.	>=v0.22.0
`nginx.ingress.kubernetes.io/canary-by-header`	Routes requests to the canary service based on a request header. Special values: `always` (always routes to canary), `never` (never routes to canary). If no value is specified, traffic is forwarded whenever the header is present.	>=v0.22.0
`nginx.ingress.kubernetes.io/canary-by-header-value`	Routes requests to the canary service when the header specified by `canary-by-header` matches an exact value. Must be used with `canary-by-header`.	>=v0.30.0
`nginx.ingress.kubernetes.io/canary-by-header-pattern`	Routes requests to the canary service when the header specified by `canary-by-header` matches a regular expression. Must be used with `canary-by-header`.	>=v0.44.0
`nginx.ingress.kubernetes.io/canary-by-cookie`	Routes requests to the canary service based on a cookie, e.g., `nginx.ingress.kubernetes.io/canary-by-cookie: foo`. Cookie values: `always` (`foo=always` routes to canary), `never` (`foo=never` does not route). Traffic is forwarded only when the cookie exists and its value is `always`.	>=v0.22.0
`nginx.ingress.kubernetes.io/canary-weight`	Routes a percentage of requests to the canary service based on weight. Range: `0` to `canary-weight-total` (default 100).	>=v0.22.0
`nginx.ingress.kubernetes.io/canary-weight-total`	Sets the total weight denominator. Default: `100`.	>=v1.1.2

Annotation priority (descending):

canary-by-header > canary-by-cookie > canary-weight

Each Ingress rule supports only one canary Ingress. Additional canary Ingresses are ignored.

Step 1: Deploy the service

Deploy an Nginx service and expose it for Layer 7 domain access with the Nginx Ingress Controller.

Create a Deployment and a Service. Save the following to nginx.yaml.

View YAML file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: old-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      run: old-nginx
  template:
    metadata:
      labels:
        run: old-nginx
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/acs-sample/old-nginx
        imagePullPolicy: Always
        name: old-nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: old-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: old-nginx
  sessionAffinity: None
  type: NodePort

Apply the manifest:

kubectl apply -f nginx.yaml

Create the Ingress. Save the following to ingress.yaml.

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      # Old version of the service.
      - path: /
        backend:
          service:
            name: old-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      # Old version of the service.
      - path: /
        backend:
          serviceName: old-nginx
          servicePort: 80

Apply the manifest:

kubectl apply -f ingress.yaml

Verify the deployment. Get the external IP address:
```
kubectl get ingress
```
Check routing access:
```
curl -H "Host: www.example.com" http://<EXTERNAL_IP>
```
Expected output:
```
old
```

Step 2: Release the new service version

Deploy the new Nginx version and configure canary routing rules.

Create the new Deployment and Service. Save the following to nginx1.yaml.

View YAML file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: new-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: new-nginx
  template:
    metadata:
      labels:
        run: new-nginx
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/acs-sample/new-nginx
        imagePullPolicy: Always
        name: new-nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: new-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: new-nginx
  sessionAffinity: None
  type: NodePort

Apply the manifest:

kubectl apply -f nginx1.yaml

Create a canary Ingress to route traffic to the new version. Three strategies are available. Strategy A: Route by request header Route requests to the new version only when the foo header equals bar. Other requests go to the old version. Save the following to ingress1.yaml. For clusters v1.19 and later:

With `foo: bar` header: 100% of traffic goes to new-nginx (controlled by canary-by-header and canary-by-header-value).
Without `foo: bar` header: 50% of traffic goes to new-nginx (controlled by canary-weight).

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release-canary
  annotations:
    # Enable canary routing.
    nginx.ingress.kubernetes.io/canary: "true"
    # Route to new-nginx only when the foo header equals bar.
    nginx.ingress.kubernetes.io/canary-by-header: "foo"
    nginx.ingress.kubernetes.io/canary-by-header-value: "bar"
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      # New version of the service.
      - path: /
        backend:
          service:
            name: new-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "foo"
    nginx.ingress.kubernetes.io/canary-by-header-value: "bar"
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: new-nginx
          servicePort: 80

Apply the manifest and verify:

kubectl apply -f ingress1.yaml
kubectl get ingress

Test without the header (routed to old version):

curl -H "Host: www.example.com" http://<EXTERNAL_IP>

Expected output: old Test with foo: bar header (routed to new version):

curl -H "Host: www.example.com" -H "foo: bar" http://<EXTERNAL_IP>

Expected output: new

Strategy B: Route by header and split remaining traffic by weight

Route all requests with foo=bar to the new version, and 50% of other requests to the new version. Update ingress1.yaml with the following content.

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    # Route to new-nginx when foo header equals bar.
    nginx.ingress.kubernetes.io/canary-by-header: "foo"
    nginx.ingress.kubernetes.io/canary-by-header-value: "bar"
    # For requests that don't match the header rule, route 50% to new-nginx.
    nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: new-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "foo"
    nginx.ingress.kubernetes.io/canary-by-header-value: "bar"
    nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: new-nginx
          servicePort: 80

Apply the manifest:

kubectl apply -f ingress1.yaml

Strategy C: Route by weight only

Route 50% of traffic to the new version, regardless of headers or cookies. Update ingress1.yaml with the following content.

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    # Route 50% of all traffic to new-nginx. Default total weight is 100.
    nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: new-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: new-nginx
          servicePort: 80

Apply the manifest:

kubectl apply -f ingress1.yaml

Verify: run this command several times. About 50% of responses should return new.

curl -H "Host: www.example.com" http://<EXTERNAL_IP>

Step 3: Complete the traffic cutover

Once the new version is stable, decommission the old version.

Update nginx.yaml to redirect the old Service to the new Deployment.

View YAML file

apiVersion: v1
kind: Service
metadata:
  name: old-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    # Point to the new service version.
    run: new-nginx
  sessionAffinity: None
  type: NodePort

Apply the update:

kubectl apply -f nginx.yaml

Verify all traffic goes to the new version:

kubectl get ingress
curl -H "Host: www.example.com" http://<EXTERNAL_IP>

Expected output: new

Delete the canary Ingress:

kubectl delete ingress gray-release-canary

Delete the old Deployment and new Service:

kubectl delete deploy old-nginx
kubectl delete svc new-nginx

The service-* annotation method

Important

The service-* annotation is no longer available in Nginx Ingress Controller v1.12 and later. Do not use it.

Annotation reference

Annotation Description

nginx.ingress.kubernetes.io/service-match Defines routing rules that map request attributes to a service. Syntax: nginx.ingress.kubernetes.io/service-match: | <service-name>: <match-rule>. Supported match types: header, cookie, query. Match formats: regex /regular expression/, exact "exact value". Examples: new-nginx: header("foo", /^bar$/); new-nginx: header("foo", "bar"); new-nginx: cookie("foo", /^sticky-.+$/); new-nginx: query("foo", "bar").

nginx.ingress.kubernetes.io/service-weight Sets traffic weights between the old and new service versions. Syntax: nginx.ingress.kubernetes.io/service-weight: | <new-svc-name>:<new-svc-weight>, <old-svc-name>:<old-svc-weight>. Example: new-nginx: 20, old-nginx: 60.

Annotation	Description
`nginx.ingress.kubernetes.io/service-match`	Defines routing rules that map request attributes to a service. Syntax: `nginx.ingress.kubernetes.io/service-match: \| <service-name>: <match-rule>`. Supported match types: `header`, `cookie`, `query`. Match formats: regex `/regular expression/`, exact `"exact value"`. Examples: `new-nginx: header("foo", /^bar$/)`; `new-nginx: header("foo", "bar")`; `new-nginx: cookie("foo", /^sticky-.+$/)`; `new-nginx: query("foo", "bar")`.
`nginx.ingress.kubernetes.io/service-weight`	Sets traffic weights between the old and new service versions. Syntax: `nginx.ingress.kubernetes.io/service-weight: \| <new-svc-name>:<new-svc-weight>, <old-svc-name>:<old-svc-weight>`. Example: `new-nginx: 20, old-nginx: 60`.

Step 1: Deploy the service

Deploy an Nginx service and expose it for Layer 7 domain access with the Nginx Ingress Controller.

Create a Deployment and a Service. Save the following to nginx.yaml.

View YAML file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: old-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      run: old-nginx
  template:
    metadata:
      labels:
        run: old-nginx
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/acs-sample/old-nginx
        imagePullPolicy: Always
        name: old-nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: old-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: old-nginx
  sessionAffinity: None
  type: NodePort

Apply the manifest:

kubectl apply -f nginx.yaml

Create the Ingress. Save the following to ingress.yaml.

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      # Old version of the service.
      - path: /
        backend:
          service:
            name: old-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: old-nginx
          servicePort: 80

Apply the manifest:

kubectl apply -f ingress.yaml

Verify the deployment. Get the external IP address:
```
kubectl get ingress
```
Check routing access:
```
curl -H "Host: www.example.com" http://<EXTERNAL_IP>
```
Expected output:
```
old
```

Step 2: Release the new service version

Deploy the new Nginx version and configure routing rules.

Create the new Deployment and Service. Save the following to nginx1.yaml.

View YAML file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: new-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: new-nginx
  template:
    metadata:
      labels:
        run: new-nginx
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/acs-sample/new-nginx
        imagePullPolicy: Always
        name: new-nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: new-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: new-nginx
  sessionAffinity: None
  type: NodePort

Apply the manifest:

kubectl apply -f nginx1.yaml

Modify the gray-release Ingress. Three strategies are available. Strategy A: Route by request header Route requests to the new version only when the foo header matches bar. Update ingress.yaml with the following content. For clusters v1.19 and later:

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release
  annotations:
    # Route to new-nginx when the foo header matches ^bar$.
    nginx.ingress.kubernetes.io/service-match: |
      new-nginx: header("foo", /^bar$/)
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      # Old version of the service.
      - path: /
        backend:
          service:
            name: old-nginx
            port:
              number: 80
        pathType: ImplementationSpecific
      # New version of the service.
      - path: /
        backend:
          service:
            name: new-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release
  annotations:
    nginx.ingress.kubernetes.io/service-match: |
      new-nginx: header("foo", /^bar$/)
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: old-nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: new-nginx
          servicePort: 80

Apply the manifest and verify:

kubectl apply -f ingress.yaml
kubectl get ingress

Test without the header (routed to old version):

curl -H "Host: www.example.com" http://<EXTERNAL_IP>

Expected output: old Test with foo: bar header (routed to new version):

curl -H "Host: www.example.com" -H "foo: bar" http://<EXTERNAL_IP>

Expected output: new

Strategy B: Route by header with weight-based split

Route requests matching foo=bar to both versions, with 50% going to each. Update ingress.yaml with the following content.

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release
  annotations:
    nginx.ingress.kubernetes.io/service-match: |
        new-nginx: header("foo", /^bar$/)
    # Of the matching requests, route 50% to new-nginx and 50% to old-nginx.
    nginx.ingress.kubernetes.io/service-weight: |
        new-nginx: 50, old-nginx: 50
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: old-nginx
            port:
              number: 80
        pathType: ImplementationSpecific
      - path: /
        backend:
          service:
            name: new-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release
  annotations:
    nginx.ingress.kubernetes.io/service-match: |
        new-nginx: header("foo", /^bar$/)
    nginx.ingress.kubernetes.io/service-weight: |
        new-nginx: 50, old-nginx: 50
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: old-nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: new-nginx
          servicePort: 80

Apply the manifest:

kubectl apply -f ingress.yaml

Run the test command several times. About 50% of foo: bar requests should return new.

Strategy C: Route by weight only

Route 50% of traffic to the new version. Update ingress.yaml with the following content.

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release
  annotations:
    # Route 50% of all traffic to new-nginx.
    nginx.ingress.kubernetes.io/service-weight: |
        new-nginx: 50, old-nginx: 50
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: old-nginx
            port:
              number: 80
        pathType: ImplementationSpecific
      - path: /
        backend:
          service:
            name: new-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release
  annotations:
    nginx.ingress.kubernetes.io/service-weight: |
        new-nginx: 50, old-nginx: 50
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: old-nginx
          servicePort: 80
      - path: /
        backend:
          serviceName: new-nginx
          servicePort: 80

Apply the manifest:

kubectl apply -f ingress.yaml

Verify: run this command several times. About 50% of responses should return new.

curl -H "Host: www.example.com" http://<EXTERNAL_IP>

Step 3: Complete the traffic cutover

After the new version is stable, decommission the old version.

For clusters of v1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gray-release
spec:
  ingressClassName: nginx
  rules:
  - host: www.example.com
    http:
      paths:
      # New version of the service.
      - path: /
        backend:
          service:
            name: new-nginx
            port:
              number: 80
        pathType: ImplementationSpecific

For clusters earlier than v1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: gray-release
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: new-nginx
          servicePort: 80

Apply the update:

kubectl apply -f ingress.yaml

Verify all traffic goes to the new version:

kubectl get ingress
curl -H "Host: www.example.com" http://<EXTERNAL_IP>

Expected output: new

Delete the old Deployment and Service:

kubectl delete deploy <Deployment_name>
kubectl delete svc <Service_name>