Migrate from self-managed NGINX to ALB Ingress - Container Service for Kubernetes

Migration process

By resolving a domain name to both the NGINX Ingress and the ALB Ingress and gradually adjusting their weights, you can achieve a seamless client migration. The following diagram outlines the general process for a zero-downtime migration using DNS resolution. Each step is detailed in the Migration example below.

Using DNS resolution for traffic shifting is one method for a zero-downtime migration. This process is provided for reference only.

Migration example

The following example demonstrates the migration process, including specific operations and how they work.

A company has an NGINX Ingress controller installed in its cluster. The controller's Service, an Internet-facing LoadBalancer type, automatically provisions an Internet-facing CLB instance. The company has also configured DNS resolution so that when clients access www.example.net, their requests are resolved to the CLB and then forwarded to the backend pods.

Step 1: Configure the ALB Ingress

As business traffic grows, the NGINX Ingress can no longer meet performance requirements and incurs high O&M costs. The company decides to migrate from NGINX Ingress to ALB Ingress. During the migration, the company uses a migration tool to convert the NGINX Ingress configuration to an ALB Ingress configuration. They also add the new ALB Ingress to the DNS resolution with a lower weight.

Important

The DNS protocol does not allow a domain name to be resolved to both an A record and a CNAME record simultaneously. CLB instances receive requests via IP addresses, while ALB instances expose a default domain name (for more details, see What is Application Load Balancer?). Therefore, you must configure a temporary domain name for the CLB instance to enable weight-based traffic distribution between the NGINX Ingress and the ALB Ingress.

Step 2: Gradually shift traffic to ALB Ingress

After verifying that the ALB Ingress works as expected, they then increase the DNS resolution weight for the ALB Ingress to route more traffic through it.

Step 3: Delete NGINX Ingress resources

After all traffic is fully migrated to the ALB Ingress, the company deletes the NGINX Ingress-related DNS entries, removes the NGINX Ingress resources, and uninstalls the NGINX Ingress controller add-on. This completes the zero-downtime migration from NGINX Ingress to ALB Ingress.

Prerequisites

A kubectl client is connected to the ACK cluster. For more information, see Connect to an ACK cluster using kubectl.

Step 1: Configure the ALB Ingress

The migration tool automatically converts AlbConfig, IngressClass, and Ingress resources. Therefore, when you install the add-on, select None for ALB Instance.

Create a file named ingress2albconfig.yaml and copy the following content into it.

apiVersion: batch/v1
kind: Job
metadata:
  name: ingress2albconfig
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: ingress2albconfig
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/ingress2albconfig:0.0.2
        command: ["/bin/ingress2albconfig", "print", "-A"]
      restartPolicy: Never
      serviceAccount: ingress2albconfig
      serviceAccountName: ingress2albconfig
  backoffLimit: 4
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:ingress2albconfig
rules:
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  - ingressclasses
  verbs:
  - get
  - list
  - watch
  - update
  - create
  - patch
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress2albconfig
  namespace: default
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:ingress2albconfig
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:ingress2albconfig
subjects:
  - kind: ServiceAccount
    name: ingress2albconfig
    namespace: default

Run the following command to start the Job. This Job automatically converts the existing NGINX Ingress controller configurations into AlbConfig, IngressClass, Ingress, and other resources, and then outputs the results. You can view the output in the logs.
```
kubectl apply -f ingress2albconfig.yaml
```
Expected output:
```
job.batch/ingress2albconfig created
clusterrole.rbac.authorization.k8s.io/system:ingress2albconfig created
serviceaccount/ingress2albconfig created
clusterrolebinding.rbac.authorization.k8s.io/system:ingress2albconfig created
```

View the pod that belongs to the Job.

kubectl get pod -l job-name=ingress2albconfig

Expected output:

NAME                READY   STATUS      RESTARTS   AGE
ingress2albconfig-vw***   0/1     Completed   0          16m

View the pod logs.

kubectl logs ingress2albconfig-vw*** # Replace this with the pod name from the previous step.

The following output is expected. After you deploy these resources to the cluster, the ALB Ingress is configured.

apiVersion: alibabacloud.com/v1
kind: AlbConfig
metadata:
  name: from-nginx
spec:
  config:
    accessLogConfig: {}
    edition: Standard
    name: from-nginx
    tags:
    - key: converted/ingress2albconfig
      value: "true"
    zoneMappings:
    - vSwitchId: vsw-xxx # Replace this with the ID of a vSwitch in your VPC.
    - vSwitchId: vsw-xxx # Replace this with the ID of a vSwitch in your VPC.
  listeners:
  - port: 80
    protocol: HTTP
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: from-nginx
spec:
  controller: ingress.k8s.alibabacloud/alb
  parameters:
    apiGroup: alibabacloud.com
    kind: AlbConfig
    name: from-nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]'
  name: from-ingress1
  namespace: default
spec:
  ingressClassName: from-nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: nginx-svc-g1msr
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer: {}

Automatically converted annotations

The migration tool automatically converts the NGINX Ingress annotations in the following table to ALB Ingress annotations.

Important

NGINX Ingress annotations that are not in the following table are ignored during the conversion. After the conversion, verify the ALB Ingress configuration to ensure it functions as expected.

Annotation list

NGINX Ingress annotation	ALB Ingress annotation
`nginx.ingress.kubernetes.io/canary`	`alb.ingress.kubernetes.io/canary`
`nginx.ingress.kubernetes.io/canary-by-header`	`alb.ingress.kubernetes.io/canary-by-header`
`nginx.ingress.kubernetes.io/canary-by-header-value`	`alb.ingress.kubernetes.io/canary-by-header-value`
`nginx.ingress.kubernetes.io/canary-by-cookie`	`alb.ingress.kubernetes.io/canary-by-cookie`
`nginx.ingress.kubernetes.io/canary-weight`	`alb.ingress.kubernetes.io/canary-weight`
`nginx.ingress.kubernetes.io/use-regex`	`alb.ingress.kubernetes.io/use-regex`
`nginx.ingress.kubernetes.io/rewrite-target`	`alb.ingress.kubernetes.io/rewrite-target`
`nginx.ingress.kubernetes.io/ssl-redirect`	`alb.ingress.kubernetes.io/ssl-redirect`
`nginx.ingress.kubernetes.io/enable-cors`	`alb.ingress.kubernetes.io/enable-cors`
`nginx.ingress.kubernetes.io/cors-allow-origin`	`alb.ingress.kubernetes.io/cors-allow-origin`
`nginx.ingress.kubernetes.io/cors-allow-methods`	`alb.ingress.kubernetes.io/cors-allow-methods`
`nginx.ingress.kubernetes.io/cors-allow-headers`	`alb.ingress.kubernetes.io/cors-allow-headers`
`nginx.ingress.kubernetes.io/cors-expose-headers`	`alb.ingress.kubernetes.io/cors-expose-headers`
`nginx.ingress.kubernetes.io/cors-allow-credentials`	`alb.ingress.kubernetes.io/cors-allow-credentials`
`nginx.ingress.kubernetes.io/backend-protocol`	`alb.ingress.kubernetes.io/backend-protocol`
`nginx.ingress.kubernetes.io/load-balance`	`alb.ingress.kubernetes.io/backend-scheduler`
`nginx.ingress.kubernetes.io/upstream-hash-by`	`alb.ingress.kubernetes.io/backend-scheduler-uch-value`

Step 2: Gradually shift traffic to ALB Ingress

Warning

Before switching traffic, compare the forwarding rules of your NGINX Ingress and ALB Ingress, and test them to ensure identical functionality. This prevents unexpected business impacts during the switch.
Switch traffic during off-peak hours.
After you configure the ALB Ingress, perform a canary release by directing a small portion of traffic to it. Verify that the forwarding rules and configurations are correct before you gradually increase the weight. This prevents traffic loss from a direct cutover with mismatched configurations.

Configure a temporary domain for the CLB instance

Because the DNS protocol does not support resolving a domain name to both an A record and a CNAME record simultaneously, and ALB instances use a default domain name to provide services, you must configure a temporary domain name for the CLB instance.

Log on to the Alibaba Cloud DNS console.
On the Public Zone page, find and click the DNS domain name www.example.net that points to the CLB instance to be migrated.
On the Settings page, find the A record that points from www.example.net to the IP address of the CLB instance. In the Actions column, click Edit.
In the Edit Record panel, change the Hostname, and then click OK. In this example, the Hostname is changed to web0. Keep the other parameters unchanged.

On the Settings page, click Add Record. In the Add Record panel, configure the following parameters and click OK.

Parameter	Description
Record Type	Select CNAME from the drop-down list.
Hostname	The prefix of the domain name. In this example, enter `www`.
Query Source	Select Default.
Record Value	Enter the temporary domain name. In this example, enter `web0.example.net`.
TTL	Time To Live (TTL) specifies how long a DNS record is cached on a DNS server. Keep the default value.

Note

After you complete the configuration, the CNAME record resolves www.example.net to web0.example.net, and the A record resolves web0.example.net to the IP address of the CLB instance.

Add a CNAME record for the ALB instance

Run the following command to find the domain name of the ALB instance.

kubectl get albconfig

The following output is expected, in which alb-a8mmh2tqbmrm11****.cn-hangzhou.alb.aliyuncs.com is the domain name of the ALB instance.

NAME         ALBID                    DNSNAME                                               PORT&PROTOCOL   CERTID   AGE
from_nginx   alb-a8mmh2tqbmrm11****   alb-a8mmh2tqbmrm11****.cn-hangzhou.alb.aliyuncs.com                            20m

On the Settings page, click Add Record. In the Add Record panel, configure the following parameters and click OK.

Parameter	Description
Record Type	Select CNAME from the drop-down list.
Hostname	The prefix of the domain name. In this example, enter `www`.
Query Source	Select Default.
Record Value	Enter the domain name of the ALB instance.
TTL	Time To Live (TTL) specifies how long a DNS record is cached on a DNS server. Keep the default value.

Set the traffic weight

On the Settings page, find the CNAME record that you added in the previous step. In the Actions column, click the arrow next to Edit and select Edit Record Set.
In the Edit Record panel, set the weights for the DNS records of the CLB and ALB instances. Set the weight of the record corresponding to the CLB instance to 100, and set the weight of the record corresponding to the ALB instance to 0. Then, click OK.
After confirming no business impact, gradually decrease the weight of the DNS record for the CLB instance while gradually increasing the weight for the ALB instance.

Run the dig command multiple times to verify the traffic shifting effect.

[root@xxx 2uvl6raykm3Z ~]# dig www.xxx.net

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.10 <<>> www.xxx.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31592
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.xxx.net.                IN      A

;; ANSWER SECTION:
www.xxx.net.        5       IN      CNAME   web0.xxx.net.
web0.xxx.net.       5       IN      A       47.xxx.144

;; Query time: 63 msec
;; SERVER: 100.xxx.136#53(100.xxx.136)
;; WHEN: Thu Jan 19 15:46:40 CST 2023
;; MSG SIZE  rcvd: 66

[root@xxx ~]# dig www.xxx.net

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.10 <<>> www.xxx.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14224
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.xxx.net.                IN      A

;; ANSWER SECTION:
www.xxx.net.        5       IN      CNAME   alb-a8mmh2xxxm74.cn-hangzhou.alb.aliyuncs.com.
alb-a8mxxxm74.cn-hangzhou.alb.aliyuncs.com. 60 IN A 116.xxx.54
alb-a8mxxxm74.cn-hangzhou.alb.aliyuncs.com. 60 IN A 118.xxx.39

;; Query time: 4 msec
;; SERVER: 100.xxx.136#53(100.xxx.136)
;; WHEN: Thu Jan 19 15:47:52 CST 2023
;; MSG SIZE  rcvd: 128

After verification, gradually set the weight of the DNS record for the CLB instance to 0 and the weight for the ALB instance record to 100.

If your DNS service provider does not support weight configuration for CNAME records, click here to see an alternative traffic shifting solution.

Step 3: Delete NGINX Ingress resources

After all persistent connections to the NGINX Ingress are closed and no new traffic is being sent to it, you can monitor the system for a period appropriate to your business needs before releasing the redundant resources.

Delete the NGINX Ingress-related DNS entries from the Alibaba Cloud DNS console.

Container Service for Kubernetes:Migrate from self-managed NGINX to ALB Ingress

Migration process

Migration example

Step 1: Configure the ALB Ingress

Step 2: Gradually shift traffic to ALB Ingress

Step 3: Delete NGINX Ingress resources

Prerequisites

Step 1: Configure the ALB Ingress

Automatically converted annotations

Step 2: Gradually shift traffic to ALB Ingress

Configure a temporary domain for the CLB instance

Add a CNAME record for the ALB instance

Set the traffic weight

Step 3: Delete NGINX Ingress resources

Related documents